Release Notes - Version 6

6.1.4.0

(16.09.2022)

Improvements:
  • CICD: Introducing support for mixed_api_auth for users on installs using AD Auth, so that a technical user can be defined and used for interacting with CICD APIs.

  • Business Rules / Businessobjects: Performance improvements in creation and edit.

  • Staging: Added retry for load stability in case driver did not return columnnar metadata at first try.

  • Dependencies State: Performance improvements.

Bugfix:
  • Dependencies calculation: Fixed some changes on objects not being properly reflected in the dependencies, potentially leading to errors of missing dependent objects even they did not exist anymore.

  • Staging - gRPC: Fixed false potentially false positive load state when dataPollTimeout was reached.

  • Staging: Fixed not working add table button.

Breaking Changes:
  • Staging - DataBricks: Added latest driver into distribution. Data Bricks JDBC Driver does not need to be manually loaded anymore. However, this changes the connection string for existing DataBricks Connections. After the update, open up the source system and revert the connection string to apply this change!

Other Changes:
  • Gui: Updated used moment.js component.

Hints for the Installation: Manual Update Scripts / Automatic Model Update
  • MS SQL: Yes / No

  • Oracle: Yes / No

  • Postgres: Yes / No

  • Exasol: Yes / No

  • Snowflake: Yes / No

  • Synapse: Yes / No

6.1.3.0

(29.07.2022)

Improvements:
  • General: Improved logging of staging loads to include the query sent to the source (for custom and regular queries) the job parameters and in case of custom queries the ftl template used.

  • General: Introduced a unique identifier for each job run. It’s logged to the column “unique_job_run_id” in the tables job_load_log, staging_load_log and datavault_load_log. The views for convenient queries are also extended.

  • Staging: Added legacy MS SQL driver based on jTDS to connect to old MS SQL versions (6.5 to 2008)

  • Staging: Added driver for Trino servers

  • Staging: Introduced a new variable ${insert_statement} for custom query ftl scripts e.g. to be able to use CTEs if not allowed in a subquery.

  • Exasol: Enable lock free metadata queries.

Bugfix:
  • Exasol: Fixed links potentially not being loaded if underlying staging table was modified.

  • MS SQL: Fixed removal of properties (e.g. where clause) to be persisted.

  • Staging - Postgres: Postgres sources having columns with a data type “numeric” without scale/precision: Columns are now mapped to a correct type in the staging table (possible thanks to the latest Postgres driver)

  • Staging: Technical fields on Excel and Theobald sources were causing problems on deployment.

  • Business Rules - MS SQL: Fixed issue with stringified quick inserts.

  • Staging: Pasting into the password field of the Add Table dialog not working on first try

  • Staging: Filtering on columns not working in the Add Table dialog.

  • Synapse: Fixed loading pits.

  • Oracle: Missing update script could cause problems on startup.

  • Datavault: Staging column renaming and comments were not correctly applied to satellite columns.

Behavioral Changes:
  • Staging: Default name for new staging tables does remove dots and removes schema for business vault loads.

Other Changes:
  • Snowflake: Update after recent behavior change on Snowflake.

  • Exasol / Staging: Updated Exasol driver to 7.1.11.

  • Postgres / Staging: Updated Postgres driver to 42.4.0.

Hints for the Installation: Manual Update Scripts / Automatic Model Update
  • MS SQL: Yes / No

  • Oracle: Yes / No

  • Postgres: Yes / No

  • Exasol: Yes / No

  • Snowflake: Yes / No

  • Synapse: Yes / No

6.1.2.0

(22.06.2022)

Important bugfix for delta load subset definition! If you are using this, upgrade as quickly as possible!

Improvements:
  • Snowflake: Improved object dependency process. This now allows to also resolve dependencies for views containing UDFs and improves dependency maintenance. However, due to the snowflake side implementation such dependencies information can have a delay from up to 3 hours.

Bugfix:
  • Staging: Fixed delta load subset clause to not be stored properly when editing on installs targeting Oracle/Postgres/Exasol/Snowflake. This may lead to it either being interpreted as full load clause or error on DVB restart. Introduced with Datavault Builder 6.1.0.0. Please check if your clauses are in the correct field after the update: For this it can be compared to an existing state export in the deployment module.

  • Staging - MS SQL: Fixed potential issue with saving source_table_type.

  • Staging - Snowflake: Fixed issue with custom query containing multi-line statements not being stored properly.

  • Staging: Fixed issue with switching source of a staging table when the currently defined source does not exist anymore.

  • Data Lineage: Fixed broken circular relations filter.

Hints for the Installation: Manual Update Scripts / Automatic Model Update
  • MS SQL: No / No

  • Oracle: No / No

  • Postgres: No / No

  • Exasol: No / No

  • Snowflake: Yes / No

  • Synapse: No / No

6.1.1.0

(15.06.2022)

Important bugfix for custom query based loads in connection with intra db loader. If you are using this, upgrade as quickly as possible!

Bugfix:
  • Staging - Custom Query: Fixed custom query intra db column sorting on load, which could lead to data being loaded into wrong columns.

  • Staging - Oracle: Fixed oracle intra db loader with custom query.

  • General: Fixed failed loads on restart. Will take effect on the second start of the Datavault Builder 6.1.1.0!

  • Loading: Fixed potentially failed loads for objects containing special characters in their ids.

  • MS SQL - Business Rules: Fixed business ruleset names to be trimmed and invalid quick inserts.

  • Deployment: Fixed certain business objects not being deployed.

Hints for the Installation: Manual Update Scripts / Automatic Model Update
  • MS SQL: Yes / No

  • Oracle: Yes / No

  • Postgres: Yes / No

  • Exasol: Yes / No

  • Snowflake: Yes / No

  • Synapse: No / No

6.1.0.0

(30.05.2022)

Introducing extended sourcing capabilities!

  • Create a “custom query” for the source data extraction. Check the staging documentation for details. Samples can be found in the knowledge base: https://kb.datavault-builder.com/display/DBKB/Staging+-+Custom+Source+Query)

  • Change the source for a staging table

  • Influence the schema naming part of a staging table

  • Add / modify column name & comment of a staging table column

  • Override staged column data type on creation

  • Add custom column to a staging table

Officially introducing support for Azure Synapse (dedicated pool)! This means, from now on installs targeting Azure Synapse will also be covered with regular update scripts.

Further Improvements:
  • General: Performance improvements for loading and editing.

  • Datavault: Column name and comment is prefilled as defined in staging for satellite and transaction link creation.

Bugfix:
  • General: Fixed maintenance process for caching sync which could lead to load errors.

  • General: Fixed startup sync process to handle pending loads while performing a restart.

  • General: Fixed usage of usernames containing a single quote.

  • Businessobjects: Fixed reverting business key column output name to original column id in scd type 2 business object.

  • Deployment: Fixed business object not being deployed when an object from the same grain already exists.

  • Snowflake: Fixed retry hub load creation when being in inconsistent state after the initial creation failed.

  • Exasol/Snowflake: Fixed delta parameter length limitation.

  • CICD: Fixed exportModel api to include result.

  • Staging: Fixed removal of missing columns from a staging table.

  • Staging: Fixed missing source object type for object on edit.

Behavioral Changes:
  • Deployment: Export state now includes the newly added “source_query_customized” for a staging table. This will show a diff for staging tables of existing model states. Export current state of the model to align the format.

Other Changes:
  • Staging: Catch naming conflicts for column ids using reserved patterns.

Hints for the Installation: Manual Update Scripts / Automatic Model Update
  • MS SQL: Yes / No

  • Oracle: No / No

  • Postgres: No / No

  • Exasol: Yes / No

  • Snowflake: Yes / No

6.0.15.0

(22.04.2022)

Improvements:
Bugfix:
  • Staging - Snowflake: Fixed type mapping for snowflake related sources (also intra database loads) to handle different timestamp types.

  • MS SQL: Fixed startup failure when having cloned structures with same naming in other schema on database.

  • Documentation: Fixed generation timestamp.

Other Changes:
  • Core-Container: Changed logger to no longer create files and only log to stdout.

Hints for the Installation: Manual Update Scripts / Automatic Model Update
  • MS SQL: Yes / No

  • Oracle: No / No

  • Postgres: No / No

  • Exasol: No / No

  • Snowflake: No / No

6.0.14.0

(06.04.2022)

Bugfix:
  • Scheduler: Updated scheduler component to prevent random crashes due to “assertion error”.

  • Operations: Fixed loading issue of PIT which can fail the load or leave it pending infinetly.

  • Operations: Fixed certain error types not properly being logged on post job query execution, leaving job loading infinetly.

  • Businessrules: Fixed issue potentially leading to accesslayer being combined from unrelated business rulesets.

  • Snowflake/Oracle: Fixed issue in rollback of failed business rules saving.

  • Businessrules: Fixed issue leading to business rule not being removed when removing related scd 2 businessobject.

  • Businessrules: Adjusted column sorting of newly created business rule to align with order in businessobject.

  • Exasol: Fixed cache refresh issue when having objects with same name as tracking objects in custom schema.

  • Datavault: Fixed potential issue leaving tracking objects when simultaneously deleting a load and the objects data.

  • Deployment: Fixed business rules dependency not respected for deletion.

  • Deployment: Fixed wrong dependencies calculation for satellites.

  • Deployment: Added missing property “staging table comment” in deployment.

Other Changes:
  • Exasol / Staging: Updated Exasol driver to 7.1.7.

Hints for the Installation: Manual Update Scripts / Automatic Model Update
  • MS SQL: No / No

  • Oracle: No / No

  • Postgres: No / No

  • Exasol: Yes / No

  • Snowflake: No / No

6.0.13.0

(21.03.2022)

Bugfix:
  • Staging - MS SQL: Fixed issue leading to unrecoverable failed staging loads.

  • Satellite: Fixed issue which could lead to duplicate error when loading batch wise.

  • Logging: Fixed issue which could lead to crash of core / connection pool due to out of memory in error case.

  • Data Lineage: Fixed access error mart lineage & included business ruleset alias view into lineage.

  • CICD / Deployment: Fixed issue leading to difference of result between Webgui and CICD APIs.

Other Changes:
  • Transaction Link: Changed default, so that for transaction links foreign keys not created anymore. (Can be overriden in dvb_config.config, config_key “transaction_link_fk_constraints” & “transaction_link_parent_hub_constraints”)

  • Datavault Loading: Changed default, so that batchwise loading and committing of batches is disabled. (Can be overriden in dvb_config.config, config_key “datavault_load_commit_batches” & “datavault_load_sat_in_batches”)

  • Dimensional Model: Aligned sorting of technical columns with sorting of business object definition when saving a business object.

  • Deployment: Changed format of staging tables export to be sorted by column_nq_id instead of column_order. This will require to check in current model state into versioning again.

Hints for the Installation: Manual Update Scripts / Automatic Model Update
  • MS SQL: No / No

  • Oracle: No / No

  • Postgres: No / No

  • Exasol: No / No

  • Snowflake: No / No

6.0.12.0

(04.03.2022)

Improvements:
  • Hashdiff-Satellite: Extended hashdiff satellite load pattern to also support batchbased and delta based loading. For this, the primary key of existing hashdiff satellites will be altered during the startup.

  • MS SQL - Staging: New load pattern for faster performance of staging loads. For this, the _dvb_row_id will be altered during the startup on MS SQL to not be an identity column anymore.

Bugfix:
  • Satellite Conversion: Fixed issue with rounding of conversions with precision / scale leading to unnecessary change detection in records.

  • Satellite: Fixed batch based loading.

  • Snowflake - Business Rules: Fixed database name added into business rules code on cloned database.

  • Deployment/CICD: Fixed false positive change detection for staging table if only column order changed.

  • Exasol - Satellite: Fixed edit of metadata for certain satellites.

  • Operations: Fixed toggling / loading of pit table to only offer connection between main hub and pit.

Hints for the Installation: Manual Update Scripts / Automatic Model Update
  • MS SQL: Yes / Yes

  • Oracle: No / Yes

  • Postgres: No / Yes

  • Exasol: No / Yes

  • Snowflake: Yes / Yes

6.0.11.0

(16.02.2022)

Improvements:
  • Operations: Optimized load control to run more efficiently, also preventing degraded performance over time.

Bugfix:
  • Operations: Fixed issue with long running post job queries to block other jobs.

  • Deployment: Fixed deployment of SCD2 Businessobjects.

  • Satellite: Fixed dvb_row_id not loadable as attribute in satellite.

  • Businessrules: Fixed removal of alias view not working on Exasol.

  • Exasol: Extended reserved keyword list to catch issues with object ids.

  • Hashdiff-Satellite: Fixed hashdiff satellite loading for Snowflake and Exasol - this only affects satellites which have a hashdiff satellite created using the API! Still, the restriction applies, that hashdiff loading is currently only supported for full loads.

  • Businessrules: Fixed validation for Businessrule on Businessrule.

  • Businessrules: Fixed too restrictive ID validation.

  • Snowflake: Introducing additional loading safety to prevent duplicate keys in parallel hub loads.

Hints for the Installation: Manual Update Scripts / Automatic Model Update
  • MS SQL: No / No

  • Oracle: No / No

  • Postgres: No / No

  • Exasol: No / No

  • Snowflake: No / No

6.0.10.0

(26.01.2022)

Bugfix:
  • CICD: Fixed issue for deployments potentially returning “process.stdout.getWindowSize is not a function”.

  • Snowflake: Fixed business key decomposer.

  • Deployment: Fixed issue of Business Objects and Business Rules with incorrect dependencies to be deployed.

  • Snowflake: Fixed lost Business Objects structure on failed rollback.

  • Snowflake: Fixed view dependencies calculation.

  • Datavault: Fixed uniqueness check for satellite creation.

  • Data Lineage: Fixed accesslayer filtering.

  • Operations: Fixed job without load to end immediately when started over a schedule.

  • General: Fixed cause of potential “template session conflicts”.

Other changes:
  • Log4j: Updated to 2.17.1.

  • Neo4j: Removed driver from DVB due to log4j vulnerability. Please check the documentation section “Staging” for manually including the driver if needed.

  • Webgui: Introduction of CICD_PROXY_PASS variable for customized webtraffic redirect.

  • Datavault: Introduction of “No Subject Area” group for objects without a defined subject area.

Hints for the Installation: Manual Update Scripts / Automatic Model Update
  • MS SQL: No / No

  • Oracle: No / No

  • Postgres: No / No

  • Exasol: No / No

  • Snowflake: Yes / No

6.0.9.0

(05.01.2022)

Improvements:
  • MSSQL: Improved transaction handling to prevent conflicts due to schema locks.

  • General: Improved parallel API execution.

Bugfix:
  • Operations - Jobs: Fixed issue with saving job triggers when having multiple dependencies (affects Exasol, Postgres, MS SQL).

  • Postgres/Snowflake: Fixed potential issue with business key decomposer for columns containing special characters.

  • Exasol - Transaction link: Fixed edit of metadata.

  • Exasol - Runtime data: Fixed potential issue with runtime data persistance onto database.

Other changes:
  • Datavault - Tracking Satellites: Last full load time is from now on stored as “end of load” instead of “begin of load”.

  • General: IDs can now start with a digit as first character.

Hints for the Installation: Manual Update Scripts / Automatic Model Update
  • MS SQL: No / No

  • Oracle: No / No

  • Postgres: No / No

  • Exasol: No / No

  • Snowflake: No / No

6.0.8.0

(21.12.2021)

Bugfix:
  • Exasol: Fixed potential cause of “global transaction rollback” in loads.

  • Staging: Fixed saving source systems when running out of system colors.

  • Exasol: Fixed caching issue with auto-created satellites.

  • Operations: Fixed last load duration, now showing last load instead of last successful load.

  • Businessrules: Fixed filter reset when adding a business rule or saving code.

  • Businessobjects: Fixed rollback on error for snowflake and oracle.

  • CICD: Fixed token renewal in deployments.

  • Deployment: Fixed update business ruleset in deployment by removing unneeded putBusinessRulesetBase call.

  • Snowflake: Fixed potential cause for empty view relations.

Other changes:
  • Log4j: Updated to 2.17.0 to fix CVE-2021-45105 and CVE-2021-45046.

  • Logging: Core logs are no longer written into csvlog by default (which will make those logs unaccessible in data viewer) and are rotated by default once a day.

Hints for the Installation: Manual Update Scripts / Automatic Model Update
  • MS SQL: No / No

  • Oracle: No / No

  • Postgres: No / No

  • Exasol: No / No

  • Snowflake: No / No

6.0.7.0

(13.12.2021)

Security Release to fix CVE-2021-44228. The update is recommended for all clients to prevent such attacks on the system.

Datavault Builder is affected by CVE-2021-44228 to a limited extent:

  • As long as the user is not authenticated, no such attacks are possible.

  • However, as soon as a user is logged in, the vulnerability can be exploited by that user.

This update increases the used version for log4j to 2.15.0 to resolve the CVE.

Workaround

If you can not update to this version immediately, you can apply changes in your docker-compose.yml to prevent the CVE, too, by setting log4j option formatMsgNoLookups.

Please have a look at the corresponding knowledge base article here: https://kb.datavault-builder.com/display/DBKB/Installation

Hints for the Installation: Manual Update Scripts / Automatic Model Update
  • MS SQL: No / No

  • Oracle: No / No

  • Postgres: No / No

  • Exasol: No / No

  • Snowflake: No / No

6.0.6.0

(06.12.2021)

Improvements:
Bugfix:
  • Operations: Fixed job dependencies which could not be removed anymore.

  • Staging - gRPC: Fixed an issue with failed loads using occupying core transactions.

  • Logging - Oracle: Fixed broken log synchronization onto Oracle.

  • MS SQL: Fixed blocked edit when adding columns to a staging table for business vault load.

Hints for the Installation: Manual Update Scripts / Automatic Model Update / Additional steps
  • MS SQL: No / No

  • Oracle: No / No

  • Postgres: No / No

  • Exasol: No / No

  • Snowflake: No / No

6.0.5.0

(22.11.2021)

Improvements:
  • General: Introducing batch based logging of runtime data for Snowflake and Exasol.

Bugfix:
  • Businessobjects: Fixed an issue with SCD2 object creation when going multiple levels deep over links.

  • Business rules: Fixed an issue with data preview when sorting / filtering.

  • Deployment/CICD: Fixed an issue when deploying job dependencies.

  • Operations/Lineage: Fixed an issue potentially leading to “maximum stack size exceeded”

  • Logging: Fixed an issue with log orders.

Hints for the Installation: Manual Update Scripts / Automatic Model Update / Additional steps
  • MS SQL: No / No

  • Oracle: No / No

  • Postgres: No / No

  • Exasol: No / No

  • Snowflake: No / No

6.0.3.0 - 6.0.4.0

Internal releases

6.0.2.0

(15.10.2021)

Improvements:
  • General: Improved transaction handling in connection pool to reduce amount of queries to set autocommit state.

  • Staging: Bundling of runtime data writing to reduce number of queries.

  • General: Extended display strings of objects to also show technical ID.

  • Staging - JSON connector: Performance improvements.

Bugfix:
  • Deployment: Fixed an issue which could prevent accessing certain jobs.

  • Businessrules: Fixed failure to create custom Business Rules on SCD2 Type Businessobjects.

  • Deployment - Snowflake: Fixed export format for business keys containing timestamp field(s) on snowflake.

  • Businessobjects: Fixed saving when having on top an unaltered BR (with default config to include in accesslayer) not included in accesslayer.

  • Businessobjects: Fixed an issue preventing to build business objects on prototype satellites.

  • Deployment: Fixed businessobject comment & name not to being deployed.

  • Update Script MS SQL: Fixed indices update pattern to be compatible with older DVB Versions.

  • Connection Pool - Python: Corrected file based loading and error logging.

  • Deployment: Fixed a displaying issue preventing the grid to not be displayed on tab switching.

  • Businessrules: Fixed filter to reset when cleaning search bar.

  • CICD: Fixed a certificate issue when using SSL encrypted webtraffic.

  • Businessrules - Postgres: Fixed handling of custom alias views.

  • Transaction Link: Fixed logging error on creation.

  • Deployment: Fixed deployment order issue of Business Rules for uppercase dbs (Snowflake, Exasol, Oracle).

Other Changes:
  • Exasol: Updated clientdb image to use Exasol 7.1.1

Hints for the Installation: Manual Update Scripts / Automatic Model Update / Additional steps
  • MS SQL: No / No

  • Oracle: No / No

  • Postgres: No / No

  • Exasol: No / No

  • Snowflake: No / No

6.0.1.0

(22.09.2021)

Introducing extended capabilities to run CI CD pipelines. With the additional APIs offered, you can now:

  • automatically export the model from an environment (to check it into git)

  • validate an externally managed model

  • deploy an externally managed model (as part of your pipeline process)

Hint

Please have a look at the CI CD Chapter in the knowledge base (kb.datavault-builder.com) to learn more about the possibilities.

To make use of the APIs, add the cicd service to your docker-compose.yml:

# make sure to use correct indentation
  cicd:
    image: ${DVB_REGISTRY}${DVB_PROJECT}/cicd:${DVB_TAG}
    networks:
      - dvbnet
    restart: always
Further Improvements:
  • Deployment: Better performance when comparing two states.

  • Businessobjects: Introducing scd type 2 output for Business objects! You can now create a historized output with just a couple of clicks. The Datavault Builder will implement the corresponding PIT Tables in the background automatically.

  • Businessobjects: Introducing new mapping feature. Match columns from an object to the existing Business object columns and thereby speed up refactoring of output layers with changed underlying data vault objects.

  • Operations: Extended loads filter drop down to filter for only loads contained in the current job (or also loads not yet in the job).

  • Business Rules: New parameters allow to define custom ID for an alias view so you give the business rule a custom id and mirror the content of it into a specific schema.

  • MS SQL: Changed patterns for loading hubs, satellites and tracking satellites to improve performance.

  • Exasol: New configuration parameters to automatically convert varchar fields to a specific length in the accesslayer. (In dvb_config.config, parameter keys: accesslayer_varchar_precision_before_cast, accesslayer_varchar_precision_after_cast)

  • Data Lineage: Better look, performance and zooming capabilities.

  • Businessobjects: Improved performance of saving.

  • Business Rules: Improved performance of saving.

Bugfix:
  • Deployment: Fixed wrongly calculated dependencies for hub load when selecting for deletion.

  • Operations: Fixed load state filter.

  • Staging: Fixed column order to respect custom order in creation.

Hints for the Installation: Manual Update Scripts / Automatic Model Update / Additional steps
  • MS SQL: Yes / Yes / To keep the upgrade process short, parts of the upgrade can be executed manually. After performing the upgrade steps, existing indices can be changed from non-clustered to clustered with the following script. For details, please check Section Major Upgrade.

  • Oracle: Yes / No

  • Postgres: Yes / No

  • Exasol: Yes / No

  • Snowflake: Yes / No

6.0.0.3

(27.08.2021)

Second beta release

6.0.0.2

(14.07.2021)

First beta release

6.0.0.0 - 6.0.0.1

Internal test releases