Attention

This document is no longer being updated. For the most recent documentation, including the latest release notes for version 6, please refer to Documentation Version 7

Release Notes - Version 6

6.10.1.1

~~6.10.1.0~~

(16.11.2023)

Security Update:

We advise all users to apply the recent update to their installations. This update addresses a security vulnerability that has the potential to allow an attacker to escalate their privileges.

To date, we are not aware of any instances where this vulnerability has been actively exploited in the wild. However, it is always best to remain proactive in maintaining the security of your systems.

It’s worth noting some limitations regarding the severity of this vulnerability:

An attacker must already be logged into the system to exploit this vulnerability.
Only members of the ‘dvb_user’ group have the capability to leverage this issue, meaning its only affecting users with already extensive permissions to do changes within the environment.

Given these constraints, while the risk is mitigated to a certain extent, it remains essential to update your systems to safeguard against potential threats.

Following part is only relevant for customers targeting snowflake: With the update script, a new user and role will be introduced which will be able to handle security sensitive tasks - and at the same time this permissions will be revoked from the current datavaultbuilder role. This role will maintain its own connection pool. Please have a look at the configuration options here: https://kb.datavault-builder.com/display/DBKB/Security

Improvements:

Stability: Self healing connection pool in case of unexpected transaction crashes.

Bugfix:

Stability: Fixed an issue which could lead to a stackoverflow and starvation of connection pool service.
PIT: Fixed load potentially being initiated prior to all underlying loads being finished.
Staging: Fixed potentially duplicated columns for Big Query source.
Business Rules: Fixed filter not being applied when creating new business ruleset.

Hints for the Installation: Manual Update Scripts / Automatic Model Update

MS SQL: No / No
Oracle: No / No
Postgres: No / No
Exasol: No / No
Snowflake: Yes / No
Synapse: No / No

6.10.0.1

(02.10.2023)

Bugfix:

Business Rules: Fixed issue in selecting current business rule and updating publication status in list.

Hints for the Installation: Manual Update Scripts / Automatic Model Update

MS SQL: No / No
Oracle: No / No
Postgres: No / No
Exasol: No / No
Snowflake: No / No
Synapse: No / No

6.10.0.0

(28.09.2023)

New Features:

Dimensional Model: Added functionality to directly open up related business rule.

Improvements:

Oracle: Compatibility with databases having max_string_size not set to extended.
Datavault: Improved loading patterns for cdc based satellite loads.
General: Stability improvement preventing potential error “could not find block containing chunk”.
Operations: Improved updating lineage while having filters active.

Bugfix:

Staging: Fixed business vault loads having schema in table id in case it is not manually set to empty string.
Staging: Fixed column comment not removable.
Deployment: Fixed potential casing issue with id for post job sql query.

Behavior Change:

Staging: When overriding the name of a table, the derived id now follows default capitalization of targeted database.

Hints for the Installation: Manual Update Scripts / Automatic Model Update

MS SQL: No / No
Oracle: No / No
Postgres: No / No
Exasol: No / No
Snowflake: No / No
Synapse: No / No

6.9.1.0

(25.08.2023)

Improvements:

Staging: When editing column names or data types, leaving the field with the mouse saves the changes now (same as pressing enter).
Staging: Removed filter for some unusual source schemas in the drop down menu to be able to use them as a source.
Datavault: Added hiding of single satellites
Dimensional Model: Separated used column display between SCD typ 1 and 2 columns

Bugfix:

Datavault: Fixed wrong BK uniqueness check result in gui in case of NULL values in composite keys.
Datavault: Fixed reordering columns of a composite BK could result in duplicated columns.
Staging: Fixed an issue that could prevent loading when table or column names contain special characters.
Data Viewer - Oracle Fixed broken Data Viewer on Oracle.

Other Changes:

Snowflake: Upgraded driver to 3.14.0.

Hints for the Installation: Manual Update Scripts / Automatic Model Update

MS SQL: No / No
Oracle: No / No
Postgres: No / No
Exasol: No / No
Snowflake: No / No
Synapse: No / No

6.9.0.0

(27.07.2023)

Improvements:

Security: Removal of weak SSL ciphers in webgui / new configuration paramters. Possibility to restrict to TLS 1.3 connections only.
Datavault: Improved loading pattern for CDC based loads.
Staging: Improved aborting of staging loads. Should be faster and more robust now.
General: Improved metadata queries to check if table is empty.
General: Performance improvements, especially for business object creation and editing.

Bugfix:

Datavault - Snowflake: Fixed shifted columns on changing satellite metadata in specific situations.
Datavault - Snowflake: Fixed escaping issue on alias hub metadata.
Staging: gRPC Fixed failing loads when empty datasets are received.
Dimensional Model: Fixed SCD2 BO with alias hubs / same-as-links.
Postgres: On containerized version, fixed vanishing datavault_staging views after restart.
Monitoring: If Grafana is in use, newly introduced security headers will no longer prevent it from loading in browser.

Hints for the Installation: Manual Update Scripts / Automatic Model Update

MS SQL: No / No
Oracle: No / No
Postgres: No / No
Exasol: No / No
Snowflake: No / No
Synapse: No / No

6.8.1.0

(03.07.2023)

Improvements:

Data Viewer: Better performance.

Bugfix:

General: Fixed maintenance check frequently querying the database (also keeping snowflake warehouse alive).

Hints for the Installation: Manual Update Scripts / Automatic Model Update

MS SQL: No / No
Oracle: No / No
Postgres: No / No
Exasol: No / No
Snowflake: No / No
Synapse: No / No

6.8.0.0

(28.06.2023)

Improvements:

Staging - Cross DB Load: Added variable which can be used in custom query for database identifier.
Logging: Improved stability.
Security: Stronger CSS attack prevention.
Operations: Added unique_job_run_id into API response for initiateJob, getJobsState, checkJob.

Bugfix:

General: Fixed broken quoting of “Umlauts”.
Deployment: Fixed potentially missing staging_table_id in export satellites.
Staging - MS SQL: Fixed handling of sqlite integer values in connection with bulk loader.
Dimensional Model: Fixed SCD2 BO creation if no field of grain sat was in output.
Dimensional Model: Fixed column filtering.
Staging: Fixed column filter wronlgy being applied onto column selection.
Staging: Fixed last list entry potentially not being displayed.
Deployment - MS SQL: Fixed handling of datetime datatype.
Authentication: Fixed AzureAD Authentication.
Snowflake: Fixed handling of timestamps as part of business keys.

Behavior Changes:

API - initiateJob: Initiating an already running job will now return a warning in result.technical_message.

Hints for the Installation: Manual Update Scripts / Automatic Model Update

MS SQL: Yes / No
Oracle: No / No
Postgres: No / No
Exasol: No / No
Snowflake: No / No
Synapse: No / No

6.7.0.0

(24.05.2023)

Improvements:

Datavault: Improved loading pattern for CDC based loads. This will compress the data loaded into satellites, i.e. only write entries with actual changes. The pattern change does not affect already loaded data but only newly processed data.
Security: Introducing rate limit for non-authorized API requests.
Security: Changed base image for core to be based on alpine to fix CVEs which were nonfixable on ubuntu.
Dimensional Model: Added switch to use Column ID instead of Name as default for output.
Staging: Improved search to also allow filtering for system comment.

Bugfix:

Security: Fixed an issue leading to passwords being logged in cleartext.
Security: Fixed an issue allowing users of group dvb_user to elevate privileges.
Snowflake: Downgraded driver to 3.13.29. due to discovered issue in batch process handling of latest version. This fixes staging load issues, but will also re-introduce wrong connection pool log messages coming from the driver.
Snowflake: Fixed escaping issue in column comments.
Deployment: Fixed non working deployment grid actions.

Behavior Changes:

Operations: Scheduled jobs are now always executed as “dvb_user”. If you have post job queries defined, make sure the dvb_user has the privileges to execute them.

Other Changes:

Snowflake: Downgraded driver to 3.13.29.
Core: Based on alpine 3.17.3 instead of ubuntu 22.04.

Hints for the Installation: Manual Update Scripts / Automatic Model Update

MS SQL: No / No
Oracle: No / No
Postgres: No / No
Exasol: No / No
Snowflake: No / No
Synapse: No / No

6.6.0.1

(12.05.2023) Internal Release

6.6.0.0

(08.05.2023)

Improvements:

Security: Updated brute force prevention.
Security: Reduced permissions for dvb_admin users.

Bugfix:

Dimensional Model: Fixed potentially wrong join leading to wrong grain for scd2 business objects using the hub as grain.
Dimensional Model: Fixed certain technical attributes (e.g. _dvb_%) not being available as output columns.
Security: Fixed an issue allowing users of group dvb_user to elevate privileges.
Connection Pool: Fixed wrong log messages coming from snowflake driver.

Behavior Changes:

Security: User management functionalities (create_user, update_user_password, delete_user) now require the users current password to make any changes.
Security: New config parameter which can be set in dvb_config.config: ‘login_lock_duration_in_seconds’. The time in seconds that a user is locked after ‘’login_lock_max_unsuccessful’’ failed logins. -1 for permanent locking until unlocked by an admin user (Default 3600).
Security: New config parameter which can be set in dvb_config.config: ‘login_lock_max_unsuccessful’. Number of failed logins before the user account gets locked (Default 15).

Other Changes:

Snowflake: Updated driver to 3.13.30.

Hints for the Installation: Manual Update Scripts / Automatic Model Update

MS SQL: No / Yes
Oracle: No / Yes
Postgres: No / Yes
Exasol: No / Yes
Snowflake: No / Yes
Synapse: No / Yes

6.5.1.0

(17.04.2023)

Improvements:

Snowflake: Added timestamp/time/date conversions on the way into the vault.
Staging: Added support for column aliases used in custom queries for certain source types (e.g. firebird).
Staging: Added new source type: ExactOnline.

Bugfix:

MS SQL: Fixed runtime data not being persisted properly, leading to errors in the logs.
Staging - Snowflake: Fixed default type mapping for data types Object and Array (“Intra / Cross Database Staging”).
PIT: Fixed PIT potentially not being able to be loaded if a same as link is involved.
Stability: Fixed unclosed connections potentially leading to resource starvation.
Snowflake: Fixed TIMESTAMP_LTZ not able to be part of a business key.

Other Changes:

Postgres: Updated reserved keywords list for identifier quoting.
Staging - Elasticsearch: Updated driver to 7.17.9.
Snowflake: Updated driver to 3.13.29.

Hints for the Installation: Manual Update Scripts / Automatic Model Update

MS SQL: No / No
Oracle: No / No
Postgres: No / No
Exasol: No / No
Snowflake: No / No
Synapse: No / No

6.5.0.0

(23.03.2023)

Improvements:

Configuration: Added a docker-compose parameter to import own CA certificates in the connection pool (used for JDBC & API connections)
Staging: Changed layout to directly access actions. Added icon for tables with defined custom source query or cdc column.
Businessrules: Faster loading and searching of list. Added icon for alias view.
Error Handling: Replaced error handler in GUI to show more details.

Bugfix:

Authentication: Fixed broken logout functionality.
Staging - Oracle: Fixed an issue with Oracle sources when a schema contains many objects.
Snowflake - Datavault: Fixed an issue resetting metadata changes for cdc based satellites.
Dataviewer: Fixed styling.
Dimensional Model: Fixed new columns to be added at the end of the existing ones.

Behavior Changes:

Networking: Since our switch to the Alpine based Connection Pool container in 6.4.0.0, all external host names in staging sources and the docker-compose config (CLIENT_DB_CONNECTIONSTRING) must now be fully qualified domain names (FQDN).
Oracle: Oracle OCI connections are deprecated and can cause errors in specific cases. We strongly advise you to use the THIN mode instead. In Datavault Builder, please edit your staging systems containing Oracle sources and either change the “Driver Type” drop-down menu to “Thin” (new default) or if the URL was edited, change “jdbc:oracle:oci:” to “jdbc:oracle:thin:” to prevent failing loads.

Hints for the Installation: Manual Update Scripts / Automatic Model Update

MS SQL: No / No
Oracle: No / No
Postgres: No / No
Exasol: No / No
Snowflake: No / No
Synapse: No / No

6.4.1.0

(17.02.2023)

Improvements:

Restart: New config parameter which can be set in dvb_config.config: ‘scheduled_jobs_reschedule_on_startup_if_older_than_seconds’
Snowflake: Safety for lately discovered behavior change of Snowflake in dvb_core.systems view which potentially prevents the sync process.
Staging: Introducing new source types for Theobald and MS Dynamics which can be used in connection with grpc.
Stability: Improvement to handle situations when numerous schedules are triggered at exactly the same time.

Bugfix:

Operations: Fixed situation where wrong parameters could be passed into an executed schedule.
Staging - Excel: Fixed broken processing of defined named ranges in excel files.
Exasol: Fixed creation of CDC Satellites for certain fields.
Exasol/Snowflake: Fixed CDC Tracking Satellite SCD1 field for business object output.
Prototype Satellites: Fixed metadata update.
Business Rules: Fixed removal of alias views in “non-Datavault Builder” schemas.

Behavior Changes:

Restart: New config parameter which can be set in dvb_config.config: ‘scheduled_jobs_reschedule_on_startup_if_older_than_seconds’. On core startup, if the next run time of a schedule is more than the provided value (in seconds) in the past (e.g. because Datavault Builder was down), it’s not executed immediately and just rescheduled to the next regular execution point. Set it to -1 to disable rescheduling (old behavior), default is 0 (always reschedule)

Hints for the Installation: Manual Update Scripts / Automatic Model Update

MS SQL: No / No
Oracle: No / No
Postgres: No / No
Exasol: No / Yes
Snowflake: Yes / Yes
Synapse: No / No

6.4.0.0

(03.02.2023)

Improvements:

Exasol: Business rules are saved faster.
Businessrules: Added business rules and a “view or table id” to quick inserts.
Documentation: Allow documentation generation also when parts of the model are inconsistent.

Bugfix:

Businessobjects: Fixed potentially duplicated rows in output for SCD2 Business objects.
Businessrules: Fixed scrolling of quick inserts.
Deployment: Fixed regression of wrongly escaping $ for variables in deployment script.
MS SQL: Fixed recreating previously existing loads for CDC objects.

Behavior Changes:

Exasol: Business rules are saved faster, because only direct dependencies are checked for validation. This means, that all further dependent objects will remain uncompiled until being accessed. This speeds up editing but can slow down the first query performed against these views.
General: As a result of the switch to Alpine containers, all external host names must now be fully qualified (FQDN). This applies to all staging sources and the CLIENT_DB_CONNECTIONSTRING in docker-compose.yml

Other Changes:

Connection Pool: Based on alpine 3.17.1 instead of ubuntu 22.04.
GUI: Updated used libraries for jquery, angular and moments to prevent potential security threats.
Postgres / Staging: Updated Postgres driver to 42.5.1.
Staging - Athena: Updated driver to 2.0.35.
Staging - Databricks: Updated driver to 2.6.32.

Hints for the Installation: Manual Update Scripts / Automatic Model Update

MS SQL: No / Yes
Oracle: No / Yes
Postgres: No / Yes
Exasol: No / Yes
Snowflake: No / Yes
Synapse: No / Yes

6.3.2.0

(20.01.2023)

Improvements:

Deployment: Handle objects with comments containing special characters in export script.
Staging: Improved stability of custom query when many staging tables are executing in parallel.
Business Rules: Improved handling of datatype of missing columns for accesslayer generation.
Stability: Improved error handling and automatic recovery capabilities when unexpected error occurs.
Deployment: Improved logic for dependent business ruleset deployments.

Bugfix:

Deployment: Fixed issue for certain model constellations to accidentally trigger deployment.
Dimensional Model: Fixed issue where in some situations SCD2 Objects could not be created anymore.
Deployment: Fixed deployment of transaction links on environments using Snowflake as database.
Synapse: Fixed duplicated entries in job table.

Behavior Changes:

Synapse: Aligned type mappings for datatypes to not default to varbinary/varchar/nvarchar(max) and instead to varbinary/varchar/nvarchar(8000/8000/4000).

Hints for the Installation: Manual Update Scripts / Automatic Model Update

MS SQL: No / No
Oracle: No / No
Postgres: No / No
Exasol: No / No
Snowflake: No / No
Synapse: No / No

6.3.1.0

(22.12.2022)

New Feature: Added support for Firebird SQL as source type in staging.

Improvements:

Staging: Changed retrieval of columnar metadata during load to better work with drivers doing internal filtering of metadata when table name contains regex like patterns (e.g. Snowflake driver).
MS SQL: Changed install script to distinguish between Azure SQL DB and Azure SQL Managed instance. Mainly relevant for authentication process as Azure SQL Managed instance supports impersonate on logins to make use of cross database loads.
CICD: Allow bigger data models to be deployed over CICD.

Bugfix:

Datavault: Fixed technical fields (e.g. _dvb_filename for a source API JSON/Files) not properly being loaded into Datavault objects.
Deployment: Fixed deploying jobs with job schedules (issue was introduced with 6.3.0.1).
Datavault: Fixed wronlgy aborted loads (rare circumstances when manually started and preload parent is active).

Behavior Changes:

PIT: Changed pit load to only fail in jobs if all dependent loads failed. For manual load, also load if all dependent loads are failed.

Hints for the Installation: Manual Update Scripts / Automatic Model Update

MS SQL: No / No
Oracle: No / No
Postgres: No / No
Exasol: No / No
Snowflake: No / No
Synapse: No / No

6.3.0.1

(23.11.2022)

New Feature: Added support to load CDC Data. Please have a look at the article on our knowledge base about the usage: https://kb.datavault-builder.com/x/NoGZAg

Bugfix:

Staging: Fixed infinetly growing pddb files.
Snowflake: Fixed issue with bookmark storage on snowflake when objects comments contained special characters.
Dimensional Model: Fixed issues with one to one and many to many links under some situations leading to wrong output.
Accesslayer: Fixed potentially not being updated / removed when editing underlying objects.
Alias View: Fixed potentially not being updated / removed when editing underlying objects.
Stability: Fixed potential issue with load cancel.
Datavault: Fixed potential issue when doing parallel deletion of loads and data for the same object.
Security: Fixed file permissions within the containers.
Synapse: Fixed broken intra db / bvl loader.

Behavior Changes:

Dimensional Model: One to one links are now driving from the side they are traversed from. Depending on the data, this can change the output of the business object.
Dimensional Model: Many to many links which are filled from multiple sources delivering the same relation do not duplicate data anymore. Depending on the data, this can change the output of the business object.
Datavault - Loading: If a load is triggered manually and the dependening on load is in state “failed” (e.g. a satellite is triggered, but the related hub load is failed), it will fail until the depending on load is fixed / reloaded successfully first.

Other Changes:

Snowflake: Updated driver to 3.13.24.

Hints for the Installation: Manual Update Scripts / Automatic Model Update

MS SQL: Yes / Yes
Oracle: Yes / Yes
Postgres: Yes / Yes
Exasol: Yes / Yes
Snowflake: Yes / Yes
Synapse: Yes / Yes

6.2.0.0

(04.11.2022)

New Feature: Transaction Links can now be used in the Dimensional Model! This means, you can traverse a transaction link from the direction of its main hub.
Security: Changed base images for webgui & scheduler to be based on alpine 3.16.2 to fix CVEs which were nonfixable on ubuntu 20.04.
Security: Changed base images for connection pool, core & api container to be based on ubuntu 22.04 to fix CVEs which were nonfixable on ubuntu 20.04.

We will continue to migrate remaining containers to alpine based images to fix remaining vulnerabilities which are currently not fixable on ubuntu.

Hints for hardening:

Databricks and/or Athena Drivers are affected by CVE-2022-42003 / CVE-2022-42004. If you are not using these sources, you may set the following environment parameter for the connection pool to remove the corresponding drivers on startup:
```
- REMOVE_JDBC_DRIVERS=Athena Databricks
```
We will ship updated drivers as soon as they are available.

Bugfix:

Snowflake: Fixed custom character length for text like fields not properly returned.
MS SQL: Fixed custom query not being removed.
GUI: Fixed potential error in ui reading’css’.

Other Changes:

Gui: Based on alpine 3.16.2 instead of ubuntu 20.04.
Scheduler: Based on alpine 3.16.2 instead of ubuntu 20.04.
Connection Pool: Based on ubuntu 22.04 instead of ubuntu 20.04.
Core: Based on ubuntu 22.04 instead of ubuntu 20.04.
Api: Based on ubuntu 22.04 instead of ubuntu 20.04.
Staging - Oracle: Safety in loading process to prevent changes to be done to staged data during data vault load.
Staging - Athena: Updated driver to 2.0.33.
Staging - Databricks: Updated driver to 2.6.29.
Staging - MySQL: Updated driver to 8.0.30.
Staging - Redshift: Updated driver to 2.1.0.9.
Postgres: Updated driver to 42.5.0.

Hints for the Installation: Manual Update Scripts / Automatic Model Update

MS SQL: No / No
Oracle: No / No
Postgres: No / No
Exasol: No / No
Snowflake: Yes / No
Synapse: No / No

6.1.5.0

(07.10.2022)

Bugfix:

Datavault: Fixed satellites potentially being wronlgy identified as being prototype.
Logging: Fixed logging in case an error happened prior to selecting data from the source.
Bookmarks: Fixed bookmarks management in Data Viewer and Datavault.

Other Changes:

Gui: Fixed faulty update in 6.1.4.0, leading to usage of 6.1.3.0 gui.

Hints for the Installation: Manual Update Scripts / Automatic Model Update

MS SQL: No / No
Oracle: No / No
Postgres: No / No
Exasol: No / No
Snowflake: No / No
Synapse: No / No

6.1.4.0

(16.09.2022)

Improvements:

CICD: Introducing support for mixed_api_auth for users on installs using AD Auth, so that a technical user can be defined and used for interacting with CICD APIs.
Business Rules / Businessobjects: Performance improvements in creation and edit.
Staging: Added retry for load stability in case driver did not return columnnar metadata at first try.
Dependencies State: Performance improvements.

Bugfix:

Dependencies calculation: Fixed some changes on objects not being properly reflected in the dependencies, potentially leading to errors of missing dependent objects even they did not exist anymore.
Staging - gRPC: Fixed false potentially false positive load state when dataPollTimeout was reached.
Staging: Fixed not working add table button.

Breaking Changes:

Staging - DataBricks: Added latest driver into distribution. Data Bricks JDBC Driver does not need to be manually loaded anymore. However, this changes the connection string for existing DataBricks Connections. After the update, open up the source system and revert the connection string to apply this change!

Other Changes:

Gui: Updated used moment.js component.

Hints for the Installation: Manual Update Scripts / Automatic Model Update

MS SQL: Yes / No
Oracle: Yes / No
Postgres: Yes / No
Exasol: Yes / No
Snowflake: Yes / No
Synapse: Yes / No

6.1.3.0

(29.07.2022)

Improvements:

General: Improved logging of staging loads to include the query sent to the source (for custom and regular queries) the job parameters and in case of custom queries the ftl template used.
General: Introduced a unique identifier for each job run. It’s logged to the column “unique_job_run_id” in the tables job_load_log, staging_load_log and datavault_load_log. The views for convenient queries are also extended.
Staging: Added legacy MS SQL driver based on jTDS to connect to old MS SQL versions (6.5 to 2008)
Staging: Added driver for Trino servers
Staging: Introduced a new variable ${insert_statement} for custom query ftl scripts e.g. to be able to use CTEs if not allowed in a subquery.
Exasol: Enable lock free metadata queries.

Bugfix:

Exasol: Fixed links potentially not being loaded if underlying staging table was modified.
MS SQL: Fixed removal of properties (e.g. where clause) to be persisted.
Staging - Postgres: Postgres sources having columns with a data type “numeric” without scale/precision: Columns are now mapped to a correct type in the staging table (possible thanks to the latest Postgres driver)
Staging: Technical fields on Excel and Theobald sources were causing problems on deployment.
Business Rules - MS SQL: Fixed issue with stringified quick inserts.
Staging: Pasting into the password field of the Add Table dialog not working on first try
Staging: Filtering on columns not working in the Add Table dialog.
Synapse: Fixed loading pits.
Oracle: Missing update script could cause problems on startup.
Datavault: Staging column renaming and comments were not correctly applied to satellite columns.

Behavioral Changes:

Staging: Default name for new staging tables does remove dots and removes schema for business vault loads.

Other Changes:

Snowflake: Update after recent behavior change on Snowflake.
Exasol / Staging: Updated Exasol driver to 7.1.11.
Postgres / Staging: Updated Postgres driver to 42.4.0.

Hints for the Installation: Manual Update Scripts / Automatic Model Update

MS SQL: Yes / No
Oracle: Yes / No
Postgres: Yes / No
Exasol: Yes / No
Snowflake: Yes / No
Synapse: Yes / No

6.1.2.0

(22.06.2022)

Important bugfix for delta load subset definition! If you are using this, upgrade as quickly as possible!

Improvements:

Snowflake: Improved object dependency process. This now allows to also resolve dependencies for views containing UDFs and improves dependency maintenance. However, due to the snowflake side implementation such dependencies information can have a delay from up to 3 hours.

Bugfix:

Staging: Fixed delta load subset clause to not be stored properly when editing on installs targeting Oracle/Postgres/Exasol/Snowflake. This may lead to it either being interpreted as full load clause or error on DVB restart. Introduced with Datavault Builder 6.1.0.0. Please check if your clauses are in the correct field after the update: For this it can be compared to an existing state export in the deployment module.
Staging - MS SQL: Fixed potential issue with saving source_table_type.
Staging - Snowflake: Fixed issue with custom query containing multi-line statements not being stored properly.
Staging: Fixed issue with switching source of a staging table when the currently defined source does not exist anymore.
Data Lineage: Fixed broken circular relations filter.

Hints for the Installation: Manual Update Scripts / Automatic Model Update

MS SQL: No / No
Oracle: No / No
Postgres: No / No
Exasol: No / No
Snowflake: Yes / No
Synapse: No / No

6.1.1.0

(15.06.2022)

Important bugfix for custom query based loads in connection with intra db loader. If you are using this, upgrade as quickly as possible!

Bugfix:

Staging - Custom Query: Fixed custom query intra db column sorting on load, which could lead to data being loaded into wrong columns.
Staging - Oracle: Fixed oracle intra db loader with custom query.
General: Fixed failed loads on restart. Will take effect on the second start of the Datavault Builder 6.1.1.0!
Loading: Fixed potentially failed loads for objects containing special characters in their ids.
MS SQL - Business Rules: Fixed business ruleset names to be trimmed and invalid quick inserts.
Deployment: Fixed certain business objects not being deployed.

Hints for the Installation: Manual Update Scripts / Automatic Model Update

MS SQL: Yes / No
Oracle: Yes / No
Postgres: Yes / No
Exasol: Yes / No
Snowflake: Yes / No
Synapse: No / No

6.1.0.0

(30.05.2022)

Introducing extended sourcing capabilities!

Create a “custom query” for the source data extraction. Check the staging documentation for details. Samples can be found in the knowledge base: https://kb.datavault-builder.com/display/DBKB/Staging+-+Custom+Source+Query)

Change the source for a staging table

Influence the schema naming part of a staging table

Add / modify column name & comment of a staging table column

Override staged column data type on creation

Add custom column to a staging table

Officially introducing support for Azure Synapse (dedicated pool)! This means, from now on installs targeting Azure Synapse will also be covered with regular update scripts.

Further Improvements:

General: Performance improvements for loading and editing.
Datavault: Column name and comment is prefilled as defined in staging for satellite and transaction link creation.

Bugfix:

General: Fixed maintenance process for caching sync which could lead to load errors.
General: Fixed startup sync process to handle pending loads while performing a restart.
General: Fixed usage of usernames containing a single quote.
Businessobjects: Fixed reverting business key column output name to original column id in scd type 2 business object.
Deployment: Fixed business object not being deployed when an object from the same grain already exists.
Snowflake: Fixed retry hub load creation when being in inconsistent state after the initial creation failed.
Exasol/Snowflake: Fixed delta parameter length limitation.
CICD: Fixed exportModel api to include result.
Staging: Fixed removal of missing columns from a staging table.
Staging: Fixed missing source object type for object on edit.

Behavioral Changes:

Deployment: Export state now includes the newly added “source_query_customized” for a staging table. This will show a diff for staging tables of existing model states. Export current state of the model to align the format.

Other Changes:

Staging: Catch naming conflicts for column ids using reserved patterns.

Hints for the Installation: Manual Update Scripts / Automatic Model Update

MS SQL: Yes / No
Oracle: No / No
Postgres: No / No
Exasol: Yes / No
Snowflake: Yes / No

6.0.15.0

(22.04.2022)

Improvements:

Staging: New Source Type available: Added Google Big Query. Please check the documentation on the connector usage (https://datavault-builder.com/wp-content/uploads/documentation-portal/latest_6/_modules/staging.html#sources-with-only-shipped-configuration-missing-driver)

Bugfix:

Staging - Snowflake: Fixed type mapping for snowflake related sources (also intra database loads) to handle different timestamp types.
MS SQL: Fixed startup failure when having cloned structures with same naming in other schema on database.
Documentation: Fixed generation timestamp.

Other Changes:

Core-Container: Changed logger to no longer create files and only log to stdout.

Hints for the Installation: Manual Update Scripts / Automatic Model Update

MS SQL: Yes / No
Oracle: No / No
Postgres: No / No
Exasol: No / No
Snowflake: No / No

6.0.14.0

(06.04.2022)

Bugfix:

Scheduler: Updated scheduler component to prevent random crashes due to “assertion error”.
Operations: Fixed loading issue of PIT which can fail the load or leave it pending infinetly.
Operations: Fixed certain error types not properly being logged on post job query execution, leaving job loading infinetly.
Businessrules: Fixed issue potentially leading to accesslayer being combined from unrelated business rulesets.
Snowflake/Oracle: Fixed issue in rollback of failed business rules saving.
Businessrules: Fixed issue leading to business rule not being removed when removing related scd 2 businessobject.
Businessrules: Adjusted column sorting of newly created business rule to align with order in businessobject.
Exasol: Fixed cache refresh issue when having objects with same name as tracking objects in custom schema.
Datavault: Fixed potential issue leaving tracking objects when simultaneously deleting a load and the objects data.
Deployment: Fixed business rules dependency not respected for deletion.
Deployment: Fixed wrong dependencies calculation for satellites.
Deployment: Added missing property “staging table comment” in deployment.

Other Changes:

Exasol / Staging: Updated Exasol driver to 7.1.7.

Hints for the Installation: Manual Update Scripts / Automatic Model Update

MS SQL: No / No
Oracle: No / No
Postgres: No / No
Exasol: Yes / No
Snowflake: No / No

6.0.13.0

(21.03.2022)

Bugfix:

Staging - MS SQL: Fixed issue leading to unrecoverable failed staging loads.
Satellite: Fixed issue which could lead to duplicate error when loading batch wise.
Logging: Fixed issue which could lead to crash of core / connection pool due to out of memory in error case.
Data Lineage: Fixed access error mart lineage & included business ruleset alias view into lineage.
CICD / Deployment: Fixed issue leading to difference of result between Webgui and CICD APIs.

Other Changes:

Transaction Link: Changed default, so that for transaction links foreign keys not created anymore. (Can be overriden in dvb_config.config, config_key “transaction_link_fk_constraints” & “transaction_link_parent_hub_constraints”)
Datavault Loading: Changed default, so that batchwise loading and committing of batches is disabled. (Can be overriden in dvb_config.config, config_key “datavault_load_commit_batches” & “datavault_load_sat_in_batches”)
Dimensional Model: Aligned sorting of technical columns with sorting of business object definition when saving a business object.
Deployment: Changed format of staging tables export to be sorted by column_nq_id instead of column_order. This will require to check in current model state into versioning again.

Hints for the Installation: Manual Update Scripts / Automatic Model Update

MS SQL: No / No
Oracle: No / No
Postgres: No / No
Exasol: No / No
Snowflake: No / No

6.0.12.0

(04.03.2022)

Improvements:

Hashdiff-Satellite: Extended hashdiff satellite load pattern to also support batchbased and delta based loading. For this, the primary key of existing hashdiff satellites will be altered during the startup.
MS SQL - Staging: New load pattern for faster performance of staging loads. For this, the _dvb_row_id will be altered during the startup on MS SQL to not be an identity column anymore.

Bugfix:

Satellite Conversion: Fixed issue with rounding of conversions with precision / scale leading to unnecessary change detection in records.
Satellite: Fixed batch based loading.
Snowflake - Business Rules: Fixed database name added into business rules code on cloned database.
Deployment/CICD: Fixed false positive change detection for staging table if only column order changed.
Exasol - Satellite: Fixed edit of metadata for certain satellites.
Operations: Fixed toggling / loading of pit table to only offer connection between main hub and pit.

Hints for the Installation: Manual Update Scripts / Automatic Model Update

MS SQL: Yes / Yes
Oracle: No / Yes
Postgres: No / Yes
Exasol: No / Yes
Snowflake: Yes / Yes

6.0.11.0

(16.02.2022)

Improvements:

Operations: Optimized load control to run more efficiently, also preventing degraded performance over time.

Bugfix:

Operations: Fixed issue with long running post job queries to block other jobs.
Deployment: Fixed deployment of SCD2 Businessobjects.
Satellite: Fixed dvb_row_id not loadable as attribute in satellite.
Businessrules: Fixed removal of alias view not working on Exasol.
Exasol: Extended reserved keyword list to catch issues with object ids.
Hashdiff-Satellite: Fixed hashdiff satellite loading for Snowflake and Exasol - this only affects satellites which have a hashdiff satellite created using the API! Still, the restriction applies, that hashdiff loading is currently only supported for full loads.
Businessrules: Fixed validation for Businessrule on Businessrule.
Businessrules: Fixed too restrictive ID validation.
Snowflake: Introducing additional loading safety to prevent duplicate keys in parallel hub loads.

Hints for the Installation: Manual Update Scripts / Automatic Model Update

MS SQL: No / No
Oracle: No / No
Postgres: No / No
Exasol: No / No
Snowflake: No / No

6.0.10.0

(26.01.2022)

Bugfix:

CICD: Fixed issue for deployments potentially returning “process.stdout.getWindowSize is not a function”.
Snowflake: Fixed business key decomposer.
Deployment: Fixed issue of Business Objects and Business Rules with incorrect dependencies to be deployed.
Snowflake: Fixed lost Business Objects structure on failed rollback.
Snowflake: Fixed view dependencies calculation.
Datavault: Fixed uniqueness check for satellite creation.
Data Lineage: Fixed accesslayer filtering.
Operations: Fixed job without load to end immediately when started over a schedule.
General: Fixed cause of potential “template session conflicts”.

Other changes:

Log4j: Updated to 2.17.1.
Neo4j: Removed driver from DVB due to log4j vulnerability. Please check the documentation section “Staging” for manually including the driver if needed.
Webgui: Introduction of CICD_PROXY_PASS variable for customized webtraffic redirect.
Datavault: Introduction of “No Subject Area” group for objects without a defined subject area.

Hints for the Installation: Manual Update Scripts / Automatic Model Update

MS SQL: No / No
Oracle: No / No
Postgres: No / No
Exasol: No / No
Snowflake: Yes / No

6.0.9.0

(05.01.2022)

Improvements:

MSSQL: Improved transaction handling to prevent conflicts due to schema locks.
General: Improved parallel API execution.

Bugfix:

Operations - Jobs: Fixed issue with saving job triggers when having multiple dependencies (affects Exasol, Postgres, MS SQL).
Postgres/Snowflake: Fixed potential issue with business key decomposer for columns containing special characters.
Exasol - Transaction link: Fixed edit of metadata.
Exasol - Runtime data: Fixed potential issue with runtime data persistance onto database.

Other changes:

Datavault - Tracking Satellites: Last full load time is from now on stored as “end of load” instead of “begin of load”.
General: IDs can now start with a digit as first character.

Hints for the Installation: Manual Update Scripts / Automatic Model Update

MS SQL: No / No
Oracle: No / No
Postgres: No / No
Exasol: No / No
Snowflake: No / No

6.0.8.0

(21.12.2021)

Bugfix:

Exasol: Fixed potential cause of “global transaction rollback” in loads.
Staging: Fixed saving source systems when running out of system colors.
Exasol: Fixed caching issue with auto-created satellites.
Operations: Fixed last load duration, now showing last load instead of last successful load.
Businessrules: Fixed filter reset when adding a business rule or saving code.
Businessobjects: Fixed rollback on error for snowflake and oracle.
CICD: Fixed token renewal in deployments.
Deployment: Fixed update business ruleset in deployment by removing unneeded putBusinessRulesetBase call.
Snowflake: Fixed potential cause for empty view relations.

Other changes:

Log4j: Updated to 2.17.0 to fix CVE-2021-45105 and CVE-2021-45046.
Logging: Core logs are no longer written into csvlog by default (which will make those logs unaccessible in data viewer) and are rotated by default once a day.

Hints for the Installation: Manual Update Scripts / Automatic Model Update

MS SQL: No / No
Oracle: No / No
Postgres: No / No
Exasol: No / No
Snowflake: No / No

6.0.7.0

(13.12.2021)

Security Release to fix CVE-2021-44228. The update is recommended for all clients to prevent such attacks on the system.

Datavault Builder is affected by CVE-2021-44228 to a limited extent:

As long as the user is not authenticated, no such attacks are possible.

However, as soon as a user is logged in, the vulnerability can be exploited by that user.

This update increases the used version for log4j to 2.15.0 to resolve the CVE.

Workaround

If you can not update to this version immediately, you can apply changes in your docker-compose.yml to prevent the CVE, too, by setting log4j option formatMsgNoLookups.

Please have a look at the corresponding knowledge base article here: https://kb.datavault-builder.com/display/DBKB/Installation

Hints for the Installation: Manual Update Scripts / Automatic Model Update

MS SQL: No / No
Oracle: No / No
Postgres: No / No
Exasol: No / No
Snowflake: No / No

6.0.6.0

(06.12.2021)

Improvements:

Security: Added support for Azure AD/Managed System Identities/Key Vault. Please check the knowledge base for setup instructions. (https://kb.datavault-builder.com/pages/viewpage.action?pageId=18841630)

Bugfix:

Operations: Fixed job dependencies which could not be removed anymore.
Staging - gRPC: Fixed an issue with failed loads using occupying core transactions.
Logging - Oracle: Fixed broken log synchronization onto Oracle.
MS SQL: Fixed blocked edit when adding columns to a staging table for business vault load.

Hints for the Installation: Manual Update Scripts / Automatic Model Update / Additional steps

MS SQL: No / No
Oracle: No / No
Postgres: No / No
Exasol: No / No
Snowflake: No / No

6.0.5.0

(22.11.2021)

Improvements:

General: Introducing batch based logging of runtime data for Snowflake and Exasol.

Bugfix:

Businessobjects: Fixed an issue with SCD2 object creation when going multiple levels deep over links.
Business rules: Fixed an issue with data preview when sorting / filtering.
Deployment/CICD: Fixed an issue when deploying job dependencies.
Operations/Lineage: Fixed an issue potentially leading to “maximum stack size exceeded”
Logging: Fixed an issue with log orders.

Hints for the Installation: Manual Update Scripts / Automatic Model Update / Additional steps

MS SQL: No / No
Oracle: No / No
Postgres: No / No
Exasol: No / No
Snowflake: No / No

6.0.3.0 - 6.0.4.0

Internal releases

6.0.2.0

(15.10.2021)

Improvements:

General: Improved transaction handling in connection pool to reduce amount of queries to set autocommit state.
Staging: Bundling of runtime data writing to reduce number of queries.
General: Extended display strings of objects to also show technical ID.
Staging - JSON connector: Performance improvements.

Bugfix:

Deployment: Fixed an issue which could prevent accessing certain jobs.
Businessrules: Fixed failure to create custom Business Rules on SCD2 Type Businessobjects.
Deployment - Snowflake: Fixed export format for business keys containing timestamp field(s) on snowflake.
Businessobjects: Fixed saving when having on top an unaltered BR (with default config to include in accesslayer) not included in accesslayer.
Businessobjects: Fixed an issue preventing to build business objects on prototype satellites.
Deployment: Fixed businessobject comment & name not to being deployed.
Update Script MS SQL: Fixed indices update pattern to be compatible with older DVB Versions.
Connection Pool - Python: Corrected file based loading and error logging.
Deployment: Fixed a displaying issue preventing the grid to not be displayed on tab switching.
Businessrules: Fixed filter to reset when cleaning search bar.
CICD: Fixed a certificate issue when using SSL encrypted webtraffic.
Businessrules - Postgres: Fixed handling of custom alias views.
Transaction Link: Fixed logging error on creation.
Deployment: Fixed deployment order issue of Business Rules for uppercase dbs (Snowflake, Exasol, Oracle).

Other Changes:

Exasol: Updated clientdb image to use Exasol 7.1.1

Hints for the Installation: Manual Update Scripts / Automatic Model Update / Additional steps

MS SQL: No / No
Oracle: No / No
Postgres: No / No
Exasol: No / No
Snowflake: No / No

6.0.1.0

(22.09.2021)

Introducing extended capabilities to run CI CD pipelines. With the additional APIs offered, you can now:

automatically export the model from an environment (to check it into git)

validate an externally managed model

deploy an externally managed model (as part of your pipeline process)

Hint

Please have a look at the CI CD Chapter in the knowledge base (kb.datavault-builder.com) to learn more about the possibilities.

To make use of the APIs, add the cicd service to your docker-compose.yml:

# make sure to use correct indentation
  cicd:
    image: ${DVB_REGISTRY}${DVB_PROJECT}/cicd:${DVB_TAG}
    networks:
      - dvbnet
    restart: always

Further Improvements:

Deployment: Better performance when comparing two states.
Businessobjects: Introducing scd type 2 output for Business objects! You can now create a historized output with just a couple of clicks. The Datavault Builder will implement the corresponding PIT Tables in the background automatically.
Businessobjects: Introducing new mapping feature. Match columns from an object to the existing Business object columns and thereby speed up refactoring of output layers with changed underlying data vault objects.
Operations: Extended loads filter drop down to filter for only loads contained in the current job (or also loads not yet in the job).
Business Rules: New parameters allow to define custom ID for an alias view so you give the business rule a custom id and mirror the content of it into a specific schema.
MS SQL: Changed patterns for loading hubs, satellites and tracking satellites to improve performance.
Exasol: New configuration parameters to automatically convert varchar fields to a specific length in the accesslayer. (In dvb_config.config, parameter keys: accesslayer_varchar_precision_before_cast, accesslayer_varchar_precision_after_cast)
Data Lineage: Better look, performance and zooming capabilities.
Businessobjects: Improved performance of saving.
Business Rules: Improved performance of saving.

Bugfix:

Deployment: Fixed wrongly calculated dependencies for hub load when selecting for deletion.
Operations: Fixed load state filter.
Staging: Fixed column order to respect custom order in creation.

Hints for the Installation: Manual Update Scripts / Automatic Model Update / Additional steps

MS SQL: Yes / Yes / To keep the upgrade process short, parts of the upgrade can be executed manually. After performing the upgrade steps, existing indices can be changed from non-clustered to clustered with the following script. For details, please check Section Major Upgrade.
Oracle: Yes / No
Postgres: Yes / No
Exasol: Yes / No
Snowflake: Yes / No

6.0.0.3

(27.08.2021)

Second beta release

6.0.0.2

(14.07.2021)

First beta release

6.0.0.0 - 6.0.0.1

Internal test releases