Release Notes - Version 6

6.0.15.0

(22.04.2022)

Improvements:
Bugfix:
  • Staging - Snowflake: Fixed type mapping for snowflake related sources (also intra database loads) to handle different timestamp types.

  • MS SQL: Fixed startup failure when having cloned structures with same naming in other schema on database.

  • Documentation: Fixed generation timestamp.

Other Changes:
  • Core-Container: Changed logger to no longer create files and only log to stdout.

Hints for the Installation: Manual Update Scripts / Automatic Model Update
  • MS SQL: Yes / No

  • Oracle: No / No

  • Postgres: No / No

  • Exasol: No / No

  • Snowflake: No / No

6.0.14.0

(06.04.2022)

Bugfix:
  • Scheduler: Updated scheduler component to prevent random crashes due to “assertion error”.

  • Operations: Fixed loading issue of PIT which can fail the load or leave it pending infinetly.

  • Operations: Fixed certain error types not properly being logged on post job query execution, leaving job loading infinetly.

  • Businessrules: Fixed issue potentially leading to accesslayer being combined from unrelated business rulesets.

  • Snowflake/Oracle: Fixed issue in rollback of failed business rules saving.

  • Businessrules: Fixed issue leading to business rule not being removed when removing related scd 2 businessobject.

  • Businessrules: Adjusted column sorting of newly created business rule to align with order in businessobject.

  • Exasol: Fixed cache refresh issue when having objects with same name as tracking objects in custom schema.

  • Datavault: Fixed potential issue leaving tracking objects when simultaneously deleting a load and the objects data.

  • Deployment: Fixed business rules dependency not respected for deletion.

  • Deployment: Fixed wrong dependencies calculation for satellites.

  • Deployment: Added missing property “staging table comment” in deployment.

Other Changes:
  • Exasol / Staging: Updated Exasol driver to 7.1.7.

Hints for the Installation: Manual Update Scripts / Automatic Model Update
  • MS SQL: No / No

  • Oracle: No / No

  • Postgres: No / No

  • Exasol: Yes / No

  • Snowflake: No / No

6.0.13.0

(21.03.2022)

Bugfix:
  • Staging - MS SQL: Fixed issue leading to unrecoverable failed staging loads.

  • Satellite: Fixed issue which could lead to duplicate error when loading batch wise.

  • Logging: Fixed issue which could lead to crash of core / connection pool due to out of memory in error case.

  • Data Lineage: Fixed access error mart lineage & included business ruleset alias view into lineage.

  • CICD / Deployment: Fixed issue leading to difference of result between Webgui and CICD APIs.

Other Changes:
  • Transaction Link: Changed default, so that for transaction links foreign keys not created anymore. (Can be overriden in dvb_config.config, config_key “transaction_link_fk_constraints” & “transaction_link_parent_hub_constraints”)

  • Datavault Loading: Changed default, so that batchwise loading and committing of batches is disabled. (Can be overriden in dvb_config.config, config_key “datavault_load_commit_batches” & “datavault_load_sat_in_batches”)

  • Dimensional Model: Aligned sorting of technical columns with sorting of business object definition when saving a business object.

  • Deployment: Changed format of staging tables export to be sorted by column_nq_id instead of column_order. This will require to check in current model state into versioning again.

Hints for the Installation: Manual Update Scripts / Automatic Model Update
  • MS SQL: No / No

  • Oracle: No / No

  • Postgres: No / No

  • Exasol: No / No

  • Snowflake: No / No

6.0.12.0

(04.03.2022)

Improvements:
  • Hashdiff-Satellite: Extended hashdiff satellite load pattern to also support batchbased and delta based loading. For this, the primary key of existing hashdiff satellites will be altered during the startup.

  • MS SQL - Staging: New load pattern for faster performance of staging loads. For this, the _dvb_row_id will be altered during the startup on MS SQL to not be an identity column anymore.

Bugfix:
  • Satellite Conversion: Fixed issue with rounding of conversions with precision / scale leading to unnecessary change detection in records.

  • Satellite: Fixed batch based loading.

  • Snowflake - Business Rules: Fixed database name added into business rules code on cloned database.

  • Deployment/CICD: Fixed false positive change detection for staging table if only column order changed.

  • Exasol - Satellite: Fixed edit of metadata for certain satellites.

  • Operations: Fixed toggling / loading of pit table to only offer connection between main hub and pit.

Hints for the Installation: Manual Update Scripts / Automatic Model Update
  • MS SQL: Yes / Yes

  • Oracle: No / Yes

  • Postgres: No / Yes

  • Exasol: No / Yes

  • Snowflake: Yes / Yes

6.0.11.0

(16.02.2022)

Improvements:
  • Operations: Optimized load control to run more efficiently, also preventing degraded performance over time.

Bugfix:
  • Operations: Fixed issue with long running post job queries to block other jobs.

  • Deployment: Fixed deployment of SCD2 Businessobjects.

  • Satellite: Fixed dvb_row_id not loadable as attribute in satellite.

  • Businessrules: Fixed removal of alias view not working on Exasol.

  • Exasol: Extended reserved keyword list to catch issues with object ids.

  • Hashdiff-Satellite: Fixed hashdiff satellite loading for Snowflake and Exasol - this only affects satellites which have a hashdiff satellite created using the API! Still, the restriction applies, that hashdiff loading is currently only supported for full loads.

  • Businessrules: Fixed validation for Businessrule on Businessrule.

  • Businessrules: Fixed too restrictive ID validation.

  • Snowflake: Introducing additional loading safety to prevent duplicate keys in parallel hub loads.

Hints for the Installation: Manual Update Scripts / Automatic Model Update
  • MS SQL: No / No

  • Oracle: No / No

  • Postgres: No / No

  • Exasol: No / No

  • Snowflake: No / No

6.0.10.0

(26.01.2022)

Bugfix:
  • CICD: Fixed issue for deployments potentially returning “process.stdout.getWindowSize is not a function”.

  • Snowflake: Fixed business key decomposer.

  • Deployment: Fixed issue of Business Objects and Business Rules with incorrect dependencies to be deployed.

  • Snowflake: Fixed lost Business Objects structure on failed rollback.

  • Snowflake: Fixed view dependencies calculation.

  • Datavault: Fixed uniqueness check for satellite creation.

  • Data Lineage: Fixed accesslayer filtering.

  • Operations: Fixed job without load to end immediately when started over a schedule.

  • General: Fixed cause of potential “template session conflicts”.

Other changes:
  • Log4j: Updated to 2.17.1.

  • Neo4j: Removed driver from DVB due to log4j vulnerability. Please check the documentation section “Staging” for manually including the driver if needed.

  • Webgui: Introduction of CICD_PROXY_PASS variable for customized webtraffic redirect.

  • Datavault: Introduction of “No Subject Area” group for objects without a defined subject area.

Hints for the Installation: Manual Update Scripts / Automatic Model Update
  • MS SQL: No / No

  • Oracle: No / No

  • Postgres: No / No

  • Exasol: No / No

  • Snowflake: Yes / No

6.0.9.0

(05.01.2022)

Improvements:
  • MSSQL: Improved transaction handling to prevent conflicts due to schema locks.

  • General: Improved parallel API execution.

Bugfix:
  • Operations - Jobs: Fixed issue with saving job triggers when having multiple dependencies (affects Exasol, Postgres, MS SQL).

  • Postgres/Snowflake: Fixed potential issue with business key decomposer for columns containing special characters.

  • Exasol - Transaction link: Fixed edit of metadata.

  • Exasol - Runtime data: Fixed potential issue with runtime data persistance onto database.

Other changes:
  • Datavault - Tracking Satellites: Last full load time is from now on stored as “end of load” instead of “begin of load”.

  • General: IDs can now start with a digit as first character.

Hints for the Installation: Manual Update Scripts / Automatic Model Update
  • MS SQL: No / No

  • Oracle: No / No

  • Postgres: No / No

  • Exasol: No / No

  • Snowflake: No / No

6.0.8.0

(21.12.2021)

Bugfix:
  • Exasol: Fixed potential cause of “global transaction rollback” in loads.

  • Staging: Fixed saving source systems when running out of system colors.

  • Exasol: Fixed caching issue with auto-created satellites.

  • Operations: Fixed last load duration, now showing last load instead of last successful load.

  • Businessrules: Fixed filter reset when adding a business rule or saving code.

  • Businessobjects: Fixed rollback on error for snowflake and oracle.

  • CICD: Fixed token renewal in deployments.

  • Deployment: Fixed update business ruleset in deployment by removing unneeded putBusinessRulesetBase call.

  • Snowflake: Fixed potential cause for empty view relations.

Other changes:
  • Log4j: Updated to 2.17.0 to fix CVE-2021-45105 and CVE-2021-45046.

  • Logging: Core logs are no longer written into csvlog by default (which will make those logs unaccessible in data viewer) and are rotated by default once a day.

Hints for the Installation: Manual Update Scripts / Automatic Model Update
  • MS SQL: No / No

  • Oracle: No / No

  • Postgres: No / No

  • Exasol: No / No

  • Snowflake: No / No

6.0.7.0

(13.12.2021)

Security Release to fix CVE-2021-44228. The update is recommended for all clients to prevent such attacks on the system.

Datavault Builder is affected by CVE-2021-44228 to a limited extent:

  • As long as the user is not authenticated, no such attacks are possible.

  • However, as soon as a user is logged in, the vulnerability can be exploited by that user.

This update increases the used version for log4j to 2.15.0 to resolve the CVE.

Workaround

If you can not update to this version immediately, you can apply changes in your docker-compose.yml to prevent the CVE, too, by setting log4j option formatMsgNoLookups.

Please have a look at the corresponding knowledge base article here: https://kb.datavault-builder.com/display/DBKB/Installation

Hints for the Installation: Manual Update Scripts / Automatic Model Update
  • MS SQL: No / No

  • Oracle: No / No

  • Postgres: No / No

  • Exasol: No / No

  • Snowflake: No / No

6.0.6.0

(06.12.2021)

Improvements:
Bugfix:
  • Operations: Fixed job dependencies which could not be removed anymore.

  • Staging - gRPC: Fixed an issue with failed loads using occupying core transactions.

  • Logging - Oracle: Fixed broken log synchronization onto Oracle.

  • MS SQL: Fixed blocked edit when adding columns to a staging table for business vault load.

Hints for the Installation: Manual Update Scripts / Automatic Model Update / Additional steps
  • MS SQL: No / No

  • Oracle: No / No

  • Postgres: No / No

  • Exasol: No / No

  • Snowflake: No / No

6.0.5.0

(22.11.2021)

Improvements:
  • General: Introducing batch based logging of runtime data for Snowflake and Exasol.

Bugfix:
  • Businessobjects: Fixed an issue with SCD2 object creation when going multiple levels deep over links.

  • Business rules: Fixed an issue with data preview when sorting / filtering.

  • Deployment/CICD: Fixed an issue when deploying job dependencies.

  • Operations/Lineage: Fixed an issue potentially leading to “maximum stack size exceeded”

  • Logging: Fixed an issue with log orders.

Hints for the Installation: Manual Update Scripts / Automatic Model Update / Additional steps
  • MS SQL: No / No

  • Oracle: No / No

  • Postgres: No / No

  • Exasol: No / No

  • Snowflake: No / No

6.0.3.0 - 6.0.4.0

Internal releases

6.0.2.0

(15.10.2021)

Improvements:
  • General: Improved transaction handling in connection pool to reduce amount of queries to set autocommit state.

  • Staging: Bundling of runtime data writing to reduce number of queries.

  • General: Extended display strings of objects to also show technical ID.

  • Staging - JSON connector: Performance improvements.

Bugfix:
  • Deployment: Fixed an issue which could prevent accessing certain jobs.

  • Businessrules: Fixed failure to create custom Business Rules on SCD2 Type Businessobjects.

  • Deployment - Snowflake: Fixed export format for business keys containing timestamp field(s) on snowflake.

  • Businessobjects: Fixed saving when having on top an unaltered BR (with default config to include in accesslayer) not included in accesslayer.

  • Businessobjects: Fixed an issue preventing to build business objects on prototype satellites.

  • Deployment: Fixed businessobject comment & name not to being deployed.

  • Update Script MS SQL: Fixed indices update pattern to be compatible with older DVB Versions.

  • Connection Pool - Python: Corrected file based loading and error logging.

  • Deployment: Fixed a displaying issue preventing the grid to not be displayed on tab switching.

  • Businessrules: Fixed filter to reset when cleaning search bar.

  • CICD: Fixed a certificate issue when using SSL encrypted webtraffic.

  • Businessrules - Postgres: Fixed handling of custom alias views.

  • Transaction Link: Fixed logging error on creation.

  • Deployment: Fixed deployment order issue of Business Rules for uppercase dbs (Snowflake, Exasol, Oracle).

Other Changes:
  • Exasol: Updated clientdb image to use Exasol 7.1.1

Hints for the Installation: Manual Update Scripts / Automatic Model Update / Additional steps
  • MS SQL: No / No

  • Oracle: No / No

  • Postgres: No / No

  • Exasol: No / No

  • Snowflake: No / No

6.0.1.0

(22.09.2021)

Introducing extended capabilities to run CI CD pipelines. With the additional APIs offered, you can now:

  • automatically export the model from an environment (to check it into git)

  • validate an externally managed model

  • deploy an externally managed model (as part of your pipeline process)

Hint

Please have a look at the CI CD Chapter in the knowledge base (kb.datavault-builder.com) to learn more about the possibilities.

To make use of the APIs, add the cicd service to your docker-compose.yml:

# make sure to use correct indentation
  cicd:
    image: ${DVB_REGISTRY}${DVB_PROJECT}/cicd:${DVB_TAG}
    networks:
      - dvbnet
    restart: always
Further Improvements:
  • Deployment: Better performance when comparing two states.

  • Businessobjects: Introducing scd type 2 output for Business objects! You can now create a historized output with just a couple of clicks. The Datavault Builder will implement the corresponding PIT Tables in the background automatically.

  • Businessobjects: Introducing new mapping feature. Match columns from an object to the existing Business object columns and thereby speed up refactoring of output layers with changed underlying data vault objects.

  • Operations: Extended loads filter drop down to filter for only loads contained in the current job (or also loads not yet in the job).

  • Business Rules: New parameters allow to define custom ID for an alias view so you give the business rule a custom id and mirror the content of it into a specific schema.

  • MS SQL: Changed patterns for loading hubs, satellites and tracking satellites to improve performance.

  • Exasol: New configuration parameters to automatically convert varchar fields to a specific length in the accesslayer. (In dvb_config.config, parameter keys: accesslayer_varchar_precision_before_cast, accesslayer_varchar_precision_after_cast)

  • Data Lineage: Better look, performance and zooming capabilities.

  • Businessobjects: Improved performance of saving.

  • Business Rules: Improved performance of saving.

Bugfix:
  • Deployment: Fixed wrongly calculated dependencies for hub load when selecting for deletion.

  • Operations: Fixed load state filter.

  • Staging: Fixed column order to respect custom order in creation.

Hints for the Installation: Manual Update Scripts / Automatic Model Update / Additional steps
  • MS SQL: Yes / Yes / To keep the upgrade process short, parts of the upgrade can be executed manually. After performing the upgrade steps, existing indices can be changed from non-clustered to clustered with the following script. For details, please check Section Major Upgrade.

  • Oracle: Yes / No

  • Postgres: Yes / No

  • Exasol: Yes / No

  • Snowflake: Yes / No

6.0.0.3

(27.08.2021)

Second beta release

6.0.0.2

(14.07.2021)

First beta release

6.0.0.0 - 6.0.0.1

Internal test releases