Release Notes - Version 5

5.3.2.0

(27.10.2020)

This update includes a major model upgrade. Make sure to follow the update instructions and do perform the update on a backed up environment first! Depending on the size of your model and data volume, this update will take several hours to complete. After following the update steps and starting up the environment again, do check the core logs “docker-compose logs -f core” for any error message thrown during the upgrade steps.

Besides numerous performance improvments in the UI, we are introducing a further optimized technical implementation for all supported target databases, based on a SCD Type 4 approach.

Up to date, the implementation of object historization was done with a SCD Type 2 (ending to _H) and a SCD Type 1 (_C) object, whereof the content contained within the SCD1 version was also part of the SCD2 object. This meant, that the latest version of the data was basically stored twice.

With our newly released SCD4 approach, this pattern will be discontinued. Instead, an additional object (SCD4) is introduced:
  • The SCD4 Object (_A) will only hold the history, not the current information.

  • The SCD1 Object (_C) will only hold the latest state, no historical information. (It provides the same data as before)

  • The SCD2 Object (_H) will combine both parts to provide the whole history including the latest state. (It provides the same data as before)

The advantage of this is: Faster loading performance (less I/O), less disk space used and simplified management of historical data.

Please check the list below for further improvements and also have a look at our training video section: We added more videos, e.g. about the deployment and code versioning process.

Improvements:
  • General: Better responsiveness and performance

  • General: Better loading performance and storage need due to SCD4 approach

  • Datavault: Display conversion type settings and complete data type in Satellite / Transaction Link Metadata Slider.

  • Snowflake: Updated JDBC Driver (3.12.13), drastically increasing staging speed for time

  • Snowflake: Improved metadata getters to allow the Snowflake optimizer to return metadata quicker.

  • Postgres/MS SQL/Exasol/Snowflake: Increased data type size of row_id to handle larger batches.

  • Deployment: Switch deployment direction does not reload models anymore to allow faster comparison.

  • Staging: Allow cancel test source connection on first try.

  • Staging: Replaced complete staging module in the UI with faster engine.

  • MS SQL: Replaced computed columns with “Instead of” triggers for hash key computation, avoiding schema locks during modelling.

  • MS SQL: The amount of loaded data in the staging tables will no longer slow down the modelling performance when adding new loads.

  • Staging: General introducation of an “Intra / Cross Database Loader”. This allows direct loads within the target instance, which is much faster than using regular source connectors when performing loads within the same target environment. (Cross DB loading is only supported on Snowflake and MS SQL for technical reasons e.g. loading from one MS SQL Database into the Datavault Builder MS SQL Database - Intra DB loading from inside the Datavault Builder Database is possible on all Databases)

  • Exasol: Make use of built-in JSON Functions. This increases the required minimum version for DVB to Exasol 6.2.7

  • MS SQL: Updated JDBC Driver to 8.4.1.

  • MS SQL: Enabled the bulk copy feature again for better staging performance

Bugfix:
  • MSSQL: Rehashing the hub and links hashs, preventing a potential issue with spaces, varchar/nvarchar bks and link hash composition. As the comparison between staged an already loaded business keys is now executed on a binary level even differences that MSSQL treats incorrectly the same are now recognized correctly as different. Example “123” and “123 ” were treated as the same value in the past but will now both be loaded separately. This can lead to new rows being added into your vault.

  • Deployment: Unaltered Default BRs could show difference. Switched the export format to include a column list instead of the code to make the compare stable.

  • Deployment: Job load deletions are now correctly rolled out.

  • Business Objects: Related BO now shows the correct information.

  • Business Objects: Fields of objects were sometimes not available anymore when they were already used in a related BO.

  • SQL Editor: Enabled “Home” & “End”

  • MS SQL: Cleanup of potentially duplicated values in Link History (only affects installs coming from Version 4.0.5.0)

  • Staging: Manually altered connection string was stored, but on edit not retrieved/displayed correctly.

  • SQL Editor: Fixed issue where sometimes only 2 or 3 lines were visible.

  • Business Objects: Fixed an issue where not all fields could be refactored in the GUI.

  • Satellites/Transaction Links: Fixed an issue where the cache and the actual metadata of the objects could get out of sync.

  • Staging: Fixed an issue, where the source system creation could fail due to missing colors in the system colors pool.

  • Snowflake: Fixed reading column conversion from Database

Hints for the Installation: Manual Update Scripts / Automatic Model Update
  • MS SQL: Yes / Yes - 1. To properly enable the “Intra / Cross Database Loader”, the database users will now be assigned to a (disabled!) login on master database level. This is part of the update scripts. One of the update Scripts (mssql_update_masterdb) will create the logins on the master db level. This script has to be executed first. Then, the second script (mssql_update_clientdb) will drop the existing users and recreate them assigned to the login just created. HINT: In case you are using a custom user instead of the authenticator, make sure to replace “authenticator” in the scripts with “authenticator, YOURUSERNAME”.

  • Oracle: Yes / Yes

  • Postgres: Yes / Yes

  • Exasol: Yes / Yes 1. New mininum Exasol Verion is 6.2.7 - 2. In case you are upgrading DVB and also going to Exasol 7, make sure to include EXASOL_USE_CHAR_FOR_HASH=true as environment variable for the core service in case you would like to continue using CHAR instead of HASHTYPE for the hash fields. Otherwise, the update will automatically alter the field type. 3. The default port within the exasol docker container has changed to 8563. If you are upgrading an existing environment with exasol running within a docker container, you will need to add ENVIRONMENT_DATABASE_DB_PORT=8888 to the service environment variables to overwrite the used port.

  • Snowflake: Yes / Yes 1. To properly enable the “Intra / Cross Database Loader”, replace in the CLIENT_DB_CONNECTIONSTRING variable of the core service in your docker-compose.yml the part CLIENT_METADATA_REQUEST_USE_CONNECTION_CTX=true with CLIENT_METADATA_REQUEST_USE_CONNECTION_CTX=false&CLIENT_METADATA_USE_SESSION_DATABASE=true

5.0.0.0_b2.8

(28.08.2020)

Snowflake: Handle changed behavior of RIGHT Function

Improvements:
  • Staging: Added MariaDB as a source type.

  • Staging: Added SQLite as a source type.

  • General: Updated Snowflake JDBC Driver to 3.12.11

  • MS SQL: Performance improvement as the database is not called anymore for quotename

Bugfix:
  • Staging: Fixed potential escaping problem of staging table columns during staging load

  • Snowflake: Handle changed behavior of RIGHT Function

Hints for the Installation: Manual Update Scripts / Automatic Model Update
  • MS SQL: No / No

  • Oracle: No / No

  • Postgres: No / No

  • Exasol: No / No

  • Snowflake: Yes / No

5.0.0.0_b2.7

(06.08.2020)

Improvements:
  • Staging: New Source Type available: Added Amazon Athena.

  • General: Handle display zoom if custom scaling is defined.

  • Deployment: Handle broken dependencies by notifying user, e.g. if a business rule references a non existing business object.

Bugfix:
  • Data Vault: Fixed delete satellite dependency check to be less sensitive.

  • Staging: Initiating a system load while a staging load was running caused “invalid input syntax” error

  • Data Lineage: Fixed clause leading to lineage being stuck in “not up to date” unless manually refreshed

  • Dimensional Model: Do not reset manual ordering of Businessobject when using “double-click” or “+”

  • Documentation: Store initiating username instead of servername.

Hints for the Installation: Manual Update Scripts / Automatic Model Update
  • MS SQL: No / No

  • Oracle: No / No

  • Postgres: No / No

  • Exasol: No / No

  • Snowflake: No / No

5.0.0.0_b2.6

(16.07.2020)

Improvements:
  • Snowflake: Updated JDBC Driver, drastically increasing staging speed (known limitation: based on our tests, the driver does not yet allow this for all data types , e.g. time)

  • Oracle: Extended data type for BK column in hash to NVARCHAR(2000)

Bugfix:
  • Deplyoment: Fix export of columns with pattern “_dvb_”

  • Dimensional Model: Fix renaming and commenting

  • Exasol: Fixed deleting Businessobject with dependent unaltered business rule

  • Data Lineage: Fixed missing Business Vault Systems in Job Lineage

  • Dimensional Model: Fixed replacing of columns

  • Deployment: Fixed hub load dependencies selection

  • Deployment: Do not call deleteHubData for alias hubs

Hints for the Installation: Manual Update Scripts / Automatic Model Update
  • MS SQL: No / No

  • Oracle: No / No

  • Postgres: No / No

  • Exasol: No / No

  • Snowflake: Yes / No

5.0.0.0_b2.5

(02.07.2020)

Improvements:
  • Oracle: Performance of base views

  • Connection Pool: Stability improvements (impacting also loading stability)

  • Documentation: Moved documentation storage into core for compatibility. This means, the documentation has to be regenerated after removing the core container.

  • Business Rules: Open underlying Business object directly from the business rulesets list

  • Deployment: If retry of a failed deployment is successful, overwrite failed state

Bugfix:
  • Operations: Fixed editing jobs, so it directly updates the information without a need for a cache refresh

  • Operations: Fixed command line content to not be overwritten / reset when switching from module

  • General: Re-Enabled edge-chromium support

  • Documentation: Fix “Cannot read property ‘regenerate’ of undefined

  • Oracle: Filter recycle bin objects so that startup/caching is not affected

Hints for the Installation: Manual Update Scripts / Automatic Model Update
  • MS SQL: Yes / No

  • Oracle: Yes / No

  • Postgres: Yes / No

  • Exasol: Yes / No

  • Snowflake: Yes / No

5.0.0.0_b2.4

(26.06.2020)

Improvements:
  • General: Allow copying of result/error messages. Prevent Overflow of messages.

Bugfix:
  • Connection Pool: Prevent insufficient privileges error

  • Datavault: Return error if updating metadta fails

  • Operations: Fixed escaping of single quote ‘ in job schedule parameters

  • Business Rulesets: Fix access errormart related problems, potentially not allowing to edit a view and having outdated access errormarts

  • Datavault: Fix bookmark restoring

Hints for the Installation: Manual Update Scripts / Automatic Model Update
  • MS SQL: No / No

  • Oracle: No / No

  • Postgres: No / No

  • Exasol: No / No

  • Snowflake: No / No

5.0.0.0_b2.3

(22.06.2020) Internal Release

5.0.0.0_b2.2

(28.05.2020)

Improvements:
  • Business Rules: Handle business rules missing include_in_accesslayer in metadata

Bugfix:
  • Staging: Fix loading time type. Hint: On exasol milliseconds are not yet loaded

  • Datavault: Fixed escaping of mssql satellite and transaction link creation (datavault staging view)

  • General: Fixed potential race condition issue in connection pool

  • Exasol/Snowflake: Fixed caching issue in hub and link load deletion, causing other loads from the same staging table to fail

Hints for the Installation: Manual Update Scripts / Automatic Model Update
  • MS SQL: No / No

  • Oracle: No / No

  • Postgres: No / No

  • Exasol: No / No

  • Snowflake: No / No

5.0.0.0_b2.1

(20.05.2020)

Bugfix:
  • Exasol: Corrected type mapping for integer values in staging

Hints for the Installation: Manual Update Scripts / Automatic Model Update
  • MS SQL: No / No

  • Oracle: No / No

  • Postgres: No / No

  • Exasol: No / No

  • Snowflake: No / No

5.0.0.0_b2.0

(15.05.2020)

Improvements:
  • Deployment: Compare Environment Configurations to show if the environments behave differently

  • Jobs: Extend sql post job queries length on oracle, exasol and snowflake, also extend error messages in log tables

  • Datavault: fix Do not open search bar if object present

  • Command Line: Command line CTRL + ENTER to do the same as CTRL + S

  • Deployment: Allow direction changes before connecting

  • Command Line: Handle long error messages and let user copy it

Bugfix:
  • Staging: Handle Citext Type on postgres and default to string otherwise

  • Deployment: Handle dependencies when selecting top categories (Data Vault, Staging, …)

Hints for the Installation: Manual Update Scripts / Automatic Model Update
  • MS SQL: No / No

  • Oracle: Yes / No

  • Postgres: No / No

  • Exasol: Yes / No

  • Snowflake: Yes / No

5.0.0.0_b1.27

(08.05.2020)

Improvements:

Bugfix:
  • General: Escape JSON Strings in comments

  • General: Row counts in same transaction as the load, otherwise the count may be wrong in the log

  • General: Ensure transaction are commited or rollbacked before triggering dependent loads

Hints for the Installation: Manual Update Scripts / Automatic Model Update
  • MS SQL: No / No

  • Oracle: No / No

  • Postgres: No / No

  • Exasol: No / No

  • Snowflake: No / No

5.0.0.0_b1.26

(30.04.2020)

Improvements:

Bugfix:
  • Exasol: Fixed exasol bug for special characters in comments (or other metadata)

  • Staging: Fixed issue when äöü (or any other unicode escape sequences) occurs in source filename or sheet name (e.g. excel)

  • Businessobjects: Handle different names for related Businessobjects

Hints for the Installation: Manual Update Scripts / Automatic Model Update
  • MS SQL: No / No

  • Oracle: No / No

  • Postgres: No / No

  • Exasol: No / No

  • Snowflake: No / No

5.0.0.0_b1.25

(23.04.2020)

Improvements:

Bugfix:
  • Staging: Compatability Fix for JDBC Drivers wrongly using column names instead of column labels

Hints for the Installation: Manual Update Scripts / Automatic Model Update
  • MS SQL: No / No

  • Oracle: No / No

  • Postgres: No / No

  • Exasol: No / No

  • Snowflake: No / No

5.0.0.0_b1.24

(20.04.2020)

Improvements:

Bugfix:
  • Staging: Let ‘-‘ sign be a valid character in source_staging_table_id (coming from the source)

Hints for the Installation: Manual Update Scripts / Automatic Model Update
  • MS SQL: No / No

  • Oracle: No / No

  • Postgres: No / No

  • Exasol: No / No

  • Snowflake: No / No

5.0.0.0_b1.23

(16.04.2020)

Improvements:

Bugfix:
  • Staging: Fixed issue when opening staging table properties after loading a staging table was not possible

Hints for the Installation: Manual Update Scripts / Automatic Model Update
  • MS SQL: No / No

  • Oracle: No / No

  • Postgres: No / No

  • Exasol: No / No

  • Snowflake: No / No

5.0.0.0_b1.22

(09.04.2020)

Improvements:

Bugfix:
  • Snowflake: Changed business key generation by adding coalesce

Hints for the Installation: Manual Update Scripts / Automatic Model Update
  • MS SQL: No / No

  • Oracle: No / No

  • Postgres: No / No

  • Exasol: No / No

  • Snowflake: No / No

5.0.0.0_b1.21

(08.04.2020)

Improvements:
  • Staging: Fixed Excel Connect

  • Businessrules: Remove not existent objects from quick inserts

Bugfix:
  • Oracle/Snowflake: Deploy Columns with conversions

  • Exasol: Cached Metadata of columns (conversion, name, comment)

  • General: Prevent “no valid connection available” error during loads

  • Snowflake: Allow load of sat > 2 Billion records (variable overflow)

  • General: On startup log in core log if a failure happens (refresh config)

Hints for the Installation: Manual Update Scripts / Automatic Model Update
  • MS SQL: No / No

  • Oracle: Yes / No

  • Postgres: No / No

  • Exasol: Yes / No

  • Snowflake: Yes / No

5.0.0.0_b1.2 - 5.0.0.0_b1.20

Internal test releases

5.0.0.0_b1.1

(04.11.2019)

First Beta Release with Connection Pool and Cache.