Version 7

7.4.3.0

(30.08.2024)

Hints for the Installation:
  • Manual Update Scripts / Automatic Model Update:

    • MS SQL: No / No

    • Oracle: No / No

    • Postgres: No / No

    • Exasol: No / No

    • Snowflake: No / No

    • Synapse: No / No

Improvement:

  • Install: Authenticator passwords now support additional special characters.

  • Staging: Introducing a global parameter for the rounding of numeric values during the staging process.

  • Business Rules: Introducing safety measure to prevent multiple business rules being assigned the same alias view id.

Bugfix:

  • Business Rules: Fixed potential issue when removing materialized Business Ruleset <XYZ> leading to error “Can’t delete the business ruleset because there are still dependent business views referencing it: Custom View: <XYZ>”.

  • Business Rules: Fixed temporarily deleted materialized Business Ruleset to not be removed from job loads.

7.4.2.0

(05.08.2024)

Hints for the Installation:
  • Manual Update Scripts / Automatic Model Update:

    • MS SQL: No / No

    • Oracle: Yes / No

    • Postgres: No / No

    • Exasol: No / No

    • Snowflake: No / No

    • Synapse: No / No

Improvement:

  • Datavault: When starting a link load manually, the dependent hub loads now run in parallel.

  • Operations: Allow usage of a business rule materialization load in jobs when a hub of the underlying business object involves an alias hub.

  • Staging - Oracle: Introducing a parameter to influence the Oracle JDBC driver into an alternative behavior mode (will allow reading NUMBER data types without parameters to better derive default type mapping).

Bugfix:

  • Snowflake: Fixed columnar metadata edit of cdc based satellites.

  • PIT: Fixed potentially broken loads when cdc and non-cdc data is present in links.

  • Deployment: Fixed advanced_properties not being able to be removed.

7.4.1.0

7.4.0.0

(05.07.2024)

Hints for the Installation:
  • Manual Update Scripts / Automatic Model Update:

    • MS SQL: No / No

    • Oracle: No / No

    • Postgres: No / No

    • Exasol: No / No

    • Snowflake: No / No

    • Synapse: No / No

Preview:
Improvement:

  • Business Rules: Data preview settings are now retained when refreshing the data.

  • Snowflake: With configured warehouse selection for jobs, uniqueness checks for hub and link loads now use the same warehouse as the job. (Enterprise Edition - Enterprise Version Server License only)

Bugfix:

  • Dimensional Model: In certain situations, the last column in business objects was not shown in gui for selection.

  • Deployment: When deploying, the column orders of staging tables and satellites were sorted alphabetically in versions 7.3.0.0 & 7.3.0.1.

  • Deployment: Fixed issue with operation delete and empty target definition.

  • Business Rules: On turning on materialization: Fixed wrong replacements if the id partially matches another business rule id.

  • Operations: Aborting a job was not aborting materialized business rule loads.

  • Operations: Jobs with manual load selection may be shown as incomplete when containing business rules loads after switching materialization off.

7.3.0.1

(28.05.2024)

Hints for the Installation:
  • Manual Update Scripts / Automatic Model Update:

    • MS SQL: No / No

    • Oracle: No / No

    • Postgres: No / No

    • Exasol: No / No

    • Snowflake: No / No

    • Synapse: No / No

Bugfix:

  • Deployment: Fixed potential diff after deployment for job in connection with newly introduced job properties.

7.3.0.0

(27.05.2024)

Hints for the Installation:
  • Manual Update Scripts / Automatic Model Update:

    • MS SQL: Yes / No

    • Oracle: Yes / No

    • Postgres: Yes / No

    • Exasol: Yes / No

    • Snowflake: Yes / No

    • Synapse: Yes / No

Features:

  • Snowflake: Introducing option to configure individual Warehouse and Query Tags for a job. Available for selected existing clients with Datavault Builder Enterprise Edition - Enterprise Version Server License.

  • APIs: Introducing new APIs for user management.

Improvement:

  • Deployment: Faster first comparison after startup.

  • Deployment: Performance improvement for comparison, export and package generation.

Bugfix:

  • Deployment: Fixed potential false positive deployment result for failed job deployment.

  • Deployment: Fixed json overflow on larger models.

  • Container Logs: Fixed maintenance process wrongly logging cancellation of non-existing business rules loads.

  • Staging - Custom Query: Fixed wrong load_type in freemarker for metadata queries.

  • Dimensional Model: Fixed filter not working for newly added columns.

  • Synapse: Fixed issue with potential duplication of object metadata.

Other Changes:

  • Staging - Redshift: Updated driver to 2.1.0.28.

7.2.1.0

(03.05.2024)

Hints for the Installation:
  • Manual Update Scripts / Automatic Model Update:

    • MS SQL: No / No

    • Oracle: No / No

    • Postgres: No / No

    • Exasol: No / No

    • Snowflake: No / No

    • Synapse: No / No

  • Manual Update Actions:

    • optional: Update CDC I pointers:

      The new version is introducing a fix for CDC I pointer computation of transaction links.

      Only if you are having CDC Loads for transaction links, it is recommended to run this statement once. Since this can take a long time, we recommend to execute the update statement on the docker host (instead of the usual Operations > Command Line in the webgui, since this would most likely result in a timeout).

      1. Make sure the upgrade to version 7.2.1.0 was successful, login on the webgui or API is possible.

      2. To run the CDC I pointer update, execute the following statement on the docker host:

        docker compose exec core /opt/datavaultbuilder/pgsql/bin/psql -U dbadmin -p 5434 -d datavaultbuilder_core -c "SELECT dvb_core.f_fix_2024_04_22_transaction_link_cdc_pointer(force_execution => true);"

      Warning

      Depending on data volume and number of affected objects this may take a considerable amount of time to execute as it will completely recompute the I pointer tables for any CDC transaction link.

Bugfix:

  • Datavault: Fixed SCD1 output may not be up to date for CDC based transaction link loads.

  • Dataviewer: Fixed aggregators not working for certain column ids.

  • Deployment: Fixed git checkin for orphaned tracking satellites (hub loads without a load).

7.2.0.0

(11.04.2024)

Hints for the Installation:
  • Manual Update Scripts / Automatic Model Update:

    • MS SQL: Yes / No

    • Oracle: Yes / No

    • Postgres: Yes / No

    • Exasol: Yes / No

    • Snowflake: Yes / No

    • Synapse: Yes / No

Features:
Improvement:

  • Webgui: Enforcing more restrictive content security policies to prevent attacks.

  • Staging: Handle databases not supporting database name in generated custom query.

  • Staging: Improved error handling for csv source.

  • Deployment: Improved error message for tracking satellites removal.

  • Business Rules: Handle situation where business ruleset name is null.

  • Staging: Raise error when a used column for a load is trying to be removed from the staging table.

  • Snowflake: Handle business rulesets with non-qualified object references.

Bugfix:

  • Deployment: Fixed counter not updated when checking/unchecking objects for deployment.

  • Deployment: Fixed issue for deployment of prototype satellites without columns.

  • Operations: Fixed satellites without load to be loaded as part of jobs.

  • Deployment: Fixed satellite & transaction link metadata not updated on all technical artefacts.

  • Dimensional Model / Datavault: Fixed suffix id uniqueness checker for scd type 2 business objects and links.

Behavior Change:

  • Permission: Allow deployment related actions (view deployments, export environment) for readonly users (still no actions which would alter the system are allowed).

  • Health-Checks: Use extended version of wget for health-checks prevent false positive unhealthy states.

Other Change:

  • Core: Updated alpine to 3.19.1.

  • API: Updated alpine to 3.19.1.

  • Webgui: Updated alpine to 3.19.1.

  • Connection Pool: Updated alpine to 3.19.1.

7.1.2.0

(07.03.2024)

Hints for the Installation:
  • Manual Update Scripts / Automatic Model Update:

    • MS SQL: No / No

    • Oracle: No / No

    • Postgres: No / No

    • Exasol: No / No

    • Snowflake: No / No

    • Synapse: No / No

  • Manual Update Actions:

    • Exasol: Because of our latest deployment speed improvements, please add/adjust the following parameter to your docker-compose.yml file (The actual value is depending on your model size and how fast your Exasol server can create new connections. 300000 ms (5 Minutes) would be a good starting point):

      connection_pool:
  ..
  environment:
    - CLIENTDB_POOL_CONNECTION_TIMEOUT_MS=300000
  ..
Improvement:

  • Deployment: Performance improvements and stability fixes.

  • Logging: Logs for objects being marked as “Failed because of DVB restart” are now persisted into the corresponding dvb_log table.

  • Snowflake: Improved synchronization of view dependencies between database and backend.

  • Synapse: Improved performance and stability for metadata storing on database.

  • General: Improved connection pool stability in case database unexpectedly closes connections.

  • Snowflake: Improved view dependency getter. Added fallback for objects for which the database does not return dependency information immediately (may this be due to use of UDF in the view or an underlying view) - in this case the dependency is parsed directly from the view definition.

Bugfix:

  • Business Rules: Fixed UI being non-responsive when creating or deleting business rules on certain model constellations.

  • Deployment: Fixed potentially wrong branch or none being selected when comparing against a filtered list of git branches.

  • General: Fixed edge case of renamings being displayed for columns without a defined renaming (rename shows same name as initial name).

  • Deployment: Fixed false positive “missing_in_accesslayer”. (Fix based on improvement for view dependency getter mentioned above).

Behavior Change:

  • Logging: Logs for objects being marked as “Failed because of DVB restart” are now persisted into the corresponding dvb_log table.

  • Deployment: While a deployment is running, deployment actions related to the state of the environment will return a failure to prevent inconsistent exports or compares.

Other Change:

  • Snowflake: Updated driver to 3.15.0.

  • sqlite: Updated driver to 3.45.1.0.

  • Oracle: Udpated driver to 21.13.0.0.

  • Postgres: Updated driver to 42.7.2.

  • Exasol: Updated driver to 7.1.20.

7.1.1.1

(14.02.2024)

Contains all changes from 7.1.1.0 and replaces 7.1.1.0. If you have 7.1.1.0 already installed, switch to 7.1.1.1 as soon as possible.

Bugfix:

  • General: Fixed non-deterministic generation of ids introduced in 7.1.1.0. Can lead to wrongly generated objects, failed object creation and failed loads.

7.1.1.0

(09.02.2024)

Hints for the Installation:
  • Manual Update Scripts / Automatic Model Update:

    • MS SQL: Yes / No

    • Oracle: Yes / No

    • Postgres: Yes / No

    • Exasol: Yes / No

    • Snowflake: Yes / No

    • Synapse: Yes / No

New Features:
Improvement:

  • Startup: Performance improvements for cache refresh speeding up startup (depending on model, about 20-70% faster).

  • Operations: Extended log for jobs to contain triggered_by_job_id and triggered_by_unique_job_run_id. Also included in checkJob API. Allows analysis of job chains.

  • GIT: Extended support for SSH key crypto algorithms (e.g. ed25519).

  • Connection Pool: Reduced logs produced by scheduler.

  • Data Lineage: Introducing handler to filter lineage prior to rendering for larger models.

  • Deployment: Changes in CREATE vs. CREATE OR REPLACE for business rules are now deployable.

  • General: New config parameters to modify the time between persisting log batches to the DWH. See Application Configuration

Bugfix:

  • GIT: Fixed handling of rejected commits, wrong name or comment (Error was not shown in UI).

  • GIT: Fixed unselected updates for object name / comment to not be committed.

  • Webgui: Fixed health check throwing errors in log when using SSL and the CA was not recognized inside webgui container.

  • Data Preview: Fixed preview of top occurrence for hashes.

  • Business Rules: Fixed renaming when view contains certain comments.

Behavior Change:

  • Dimensional Model: Restricting related business objects (same start hub and functional suffix id) to be only created for same SCD Type (1 or 2).

7.1.0.0

(04.01.2024)

Hints for the Installation:
  • Manual Update Scripts / Automatic Model Update:

    • MS SQL: No / No

    • Oracle: No / No

    • Postgres: No / No

    • Exasol: No / No

    • Snowflake: No / No

    • Synapse: No / No

  • Manual Update Actions:

    • Warning

      Port Change for webgui service:

      In the docker-compose.yml file, the previous ports (to the right of the colon):

      webgui:
  ..
  ports:
    - '${DVB_WEBGUI_PORT}:80'
    - '${DVB_WEBGUI_SSL_PORT}:443'
  ..

      have to be changed to new internal ports (8080/8443):

      webgui:
  ..
  ports:
    - '${DVB_WEBGUI_PORT}:8080'
    - '${DVB_WEBGUI_SSL_PORT}:8443'
  ..

      The external ports (on the left side of the colon) should stay exactly the same as they are right now in your file (i.e. a variable or number), so the access to Datavault Builder will work as before.

    • optional: Update CDC L pointers:

      The new version is introducing a behavior change for CDC L pointer computation.

      Only if you are having CDC Loads which have overwritten / corrected historical loads, it is recommended to update the cdc satellites. Since this can take a long time, we recommend to execute the update statement on the docker host (instead of the usual Operations > Command Line in the webgui, since this would most likely result in a timeout).

      1. Make sure the upgrade to version 7.1.0.0 was successful, login on the webgui or API is possible.

      2. To run the CDC L pointer update, execute the following statement on the docker host:

        docker compose exec core /opt/datavaultbuilder/pgsql/bin/psql -U dbadmin -p 5434 -d datavaultbuilder_core -c "SELECT dvb_core.f_fix_2023_11_08_sat_cdc_pointer(force_execution => true);"

      Warning

      Depending on data volume and number of affected objects this may take a considerable amount of time to execute as it will completely recompute the L pointer tables for any CDC satellite.

New Feature:

  • Deployment: Compare states directly from & to GIT.

  • Deployment: Deploy directly from GIT.

  • Deployment: Selectively commit directly to GIT repository.

Documentation on configuration can be found here: https://support.portal.datavault-builder.com/servicedesk/customer/portal/5/article/11993599

Improvement:

  • Deployment: Performance improvements for state export, import and comparison.

  • Webgui: Security hardening. Process now runs as unprivileged user.

  • Synapse: Better transaction handling.

  • Deployment: Faster putting of job with numerous loads in in-/exclude list.

  • Business Rules: Performance improvement for viewing / editing.

  • PIT: Improved loading patterns; Compress PIT and prefer compute vs. IO.

  • General: Improved view code parser.

  • Businessobjects: Introducing hiding options for canvas on right-click onto objects.

  • Datavault: Extended hiding options.

  • Staging: Extended support to write boolean null values from GRPC based sources.

  • Webgui: Optimized background pollers.

Bugfix:

  • General: Fixed edge case issue leading to failure in metadata edit of objects.

  • Synapse: Fixed non-deterministic PIT load when distributing load across multiple nodes.

  • Operations: Fixed post job query containing multiple statements.

  • Operations: Fixed post job query exception hint to not trigger dependent job.

  • UI: Fixed various overflowing boxes.

  • Oracle: Fixed timestamp data preview.

  • Business Rules: Fixed pointer jumping from searchbox into editor.

  • Deployment: Fixed changes in export file (key column_name_detail, spaces).

Behavior Change:

  • Webgui: Changed internal ports from 80 to 8080 and 443 to 8443. Needs manual adjustment in docker-compose.yml.

  • Oracle: Reduced required grants and changed created users as not Oracle maintained for new installs (targeting Oracle).

  • CDC: Changed patterns for pointer tables (_l&_i) to cover backdated overwritting of compressed historical data.

Other Change:

  • API: Updated postgREST to 12.0.1.

7.0.1.1

7.0.1.0

(16.11.2023)

Security Update:

We advise all users to apply the recent update to their installations. This update addresses a security vulnerability that has the potential to allow an attacker to escalate their privileges.

To date, we are not aware of any instances where this vulnerability has been actively exploited in the wild. However, it is always best to remain proactive in maintaining the security of your systems.

It’s worth noting some limitations regarding the severity of this vulnerability:

  • An attacker must already be logged into the system to exploit this vulnerability.

  • Only members of the ‘dvb_user’ group have the capability to leverage this issue, meaning its only affecting users with already extensive permissions to do changes within the environment.

Given these constraints, while the risk is mitigated to a certain extent, it remains essential to update your systems to safeguard against potential threats.

Following part is only relevant for customers targeting snowflake: With the update script, a new user and role will be introduced which will be able to handle security sensitive tasks - and at the same time this permissions will be revoked from the current datavaultbuilder role. This role will maintain its own connection pool. Please have a look at the configuration options here: https://support.portal.datavault-builder.com/servicedesk/customer/portal/5/article/11993900

Improvements:

  • Dimensional Model: Granularity satellite of an existing business object can now be changed in the edit dialogue.

  • Stability: Self healing connection pool in case of unexpected transaction crashes.

  • Stability: Self healing api in case of changed token.

  • Docker: Introducing health state check for services core, api, connection pool and webgui.

  • Deployment: Extended compatibilty with old version 5 exports.

Bugfix:

  • Stability: Fixed an issue which could lead to a stackoverflow and starvation of connection pool service.

  • PIT: Fixed load potentially being initiated prior to all underlying loads being finished.

  • Staging: Fixed potentially duplicated columns for Big Query source.

  • Business Rules: Fixed filter not being applied when creating new business ruleset.

Hints for the Installation: Manual Update Scripts / Automatic Model Update

  • MS SQL: No / No

  • Oracle: No / No

  • Postgres: No / No

  • Exasol: No / No

  • Snowflake: Yes / No

  • Synapse: No / No

7.0.0.6

(12.10.2023)

Internal Release

7.0.0.5

(05.10.2023)

Bugfix:

  • Deployment: Disabled sync of deployment packages to and from database to prevent potential stackoverflow for certain deployments. Deployment logs are still written, but won’t be available anymore in the deployment in case of a service recreation.

Hints for the Installation: Manual Update Scripts / Automatic Model Update

  • MS SQL: No / No

  • Oracle: No / No

  • Postgres: No / No

  • Exasol: No / No

  • Snowflake: No / No

  • Synapse: No / No

7.0.0.4

(29.09.2023)

Bugfix:

  • Business Rules: Fixed issue in selecting current business rule and updating publication status in list.

Hints for the Installation: Manual Update Scripts / Automatic Model Update

  • MS SQL: No / No

  • Oracle: No / No

  • Postgres: No / No

  • Exasol: No / No

  • Snowflake: No / No

  • Synapse: No / No

7.0.0.3

(28.09.2023)

New Features:

  • Deployment:

    • Completely refactored.

    • Faster comparison of two states.

    • Faster deployment due to parallel processing of steps.

    • New APIs for generating packages and monitoring process.

    • New Log tables available for deployments on the database.

    • Object comment now visible in comparison view.

    • Added line-based diff for business ruleset code, customized source queries and post job sql queries.

Improvements:

  • General: More specific categorization of columns to corresponding column type (e.g. hash, bk, attribute, ..).

  • General: Improved column type categorization to reduce risks of naming collisions (e.g. columns ending to _h or _bk).

  • General: Reduced number of main containers to 4 (core, connection_pool, api, webgui) which allows deploying more easily and now supports ACI on Azure.

  • Deployment: New optimized export format.

    • Restructured folder / file paths.

    • Removed “Group”-Files - only one file per object.

    • Added multiline markup for business rules code, custom source query and post job query.

  • Deployment: Handle tracking satellites for which no load exists anymore to be still importable.

Bugfix:

  • Deployment: Fixed issues with dependency selection and dependent step deployments.

  • Deployment: Fixed various deployment order issues.

  • Deployment: Fixed false positive change detections.

  • Deployment: Fixed object filters for CICD APIs.

Other Changes:

  • Architecture: Removed separate scheduler container.

  • Architecture: Removed separate cicd container.

Behavior/Breaking Changes:

  • Deployment: Deployment is now processed on the backend and not from within the UI anymore, meaning the deployment will continue even though the browser is closed.

  • Deployment: New optimized export format to check into git. Old format can still be read, but starting with Version 7 the export can not be retrieved in the old format anymore.

  • Deployment - APIs: Since the CICD container is removed, existing calls to previous CICD APIs are no longer possible. However, to easen the migration, corresponding APIs have been implemented in the core engine. For the migration:

    • Instead of calling url /cicd/ change the calls to /rpc/ (e.g. instead of /cicd/deployModel use /rpc/deployModel)

    • These APIs require a token based authentication and do not accept username + password in the payloads.

      • Therefore: In the script, add a step to call /rpc/login to retrieve a token

      • Set this token into the header of the rpc call for authentication

    • Even though the the compatibility APIs can then be used in the same way, they are based on new backend and will therefore produce a different result then previous CICD APIs. Especially:

      • exportModel: Will only return the model in the new export format (2.0) - in case you need the deprecated format, the export has to be done over the UI

      • APIs won’t return a junit response anymore

Hints for the Installation: Manual Update Scripts / Automatic Model Update

  • MS SQL: Yes / No

  • Oracle: Yes/ No

  • Postgres: Yes / No

  • Exasol: Yes / No

  • Snowflake: Yes / No

  • Synapse: Yes / No

Hint

Make sure to check the section for major upgrade steps for 6.X.X.X to 7.X.X.X and apply required changes!

7.0.0.2

(17.08.2023)

Second release candidate

7.0.0.1

(21.07.2023)

First release candidate

7.0.0.0

Beta releases