Attention

This document is no longer being updated. For the most recent documentation, including the latest release notes for version 6, please refer to Documentation Version 7

Initial Install

The Datavault Builder is shipped as docker images, containing the different parts of the application. This makes not only the setup very easy, but also allows updating one part without touching the others. For the setup, you can use a docker-compose file, such as the one below. It will automatically pull the necessary images from our repository and start the containers.

Steps for initial setup

1. To access the Datavault Builder repository, please provide us your Email address so your account can be initiated.

2. Install docker and latest docker-compose (docker for windows 10 has latest, ubuntu 16.04 needs updates)

3. We will send you a docker-compose.yml file, looking similar to the code shown in Docker-Compose Configuration.

4. Adjust the settings in the docker-compose.yml, such as

   • initial user settings

   • passwords

   • usernames

   • ports (optional)docker

   • and in case of an external database: jdbc connection string

and save the file. For details on the available options, see the subchapters of Docker-Compose Configuration.

Note

In case you install multiple stacks on the same host, make sure to alter the ports settings so you don’t have conflicts. Also, make sure, that not two stacks point at the same database.

1. Generate your custom System Encryption Keys and save them in files in the secrets folder.

2. Optional: Enable SSL-Encryption for the Webtraffic.

3. Open the cmd-line within the folder, containing the docker-compose.yml and login to docker cloud:

   docker login docker.datavault-builder.com
   

4. Download the docker images:

   docker-compose pull
   

5. Start the docker containers:

   docker-compose up -d
   

6. The Datavault Builder Interface can now be reached with a Chrome Browser on the exposed port from the webgui on the machine (default: 80).

Using a non-containered Database

In case you are using a database outside of a docker container, you will need to perform some additional steps in order to install the necessary structures on the database. For this, please see Install DVB Structures on Database.

After the installation of the DVB Structures on your database, adjust the Setting of the CLIENT_DB_CONNECTIONSTRING Parameter for the core Container to point at your database.

The string should then look like:

'CLIENT_DB_CONNECTIONSTRING=jdbc:postgresql://YOURHOST:YOURPORT/datavaultbuilder?stringtype=unspecified&logUnclosedConnections=true'

Note

Do not include the username & password into the connection string. The Datavault Builder will automatically use the technical authenticator for the connection.

Database Requirements

When using the Datavault Builder, by default for the client-database as well a containerized version is started up (if available). Nevertheless, if you would like you can as well connect to a separate instance of your database. In this case, please modify the connection string in the docker-compose/docker-stack file accordingly.

The necessary meta-data structures are not directly deployed through the core to the client-database. If you would like to use a separate client-database, please contact us for the install-script.

Minimal Requirements

When planning on bringing in your own database, please respect the following tested minimal versions and configurations:

General:

• Collation: Case-Sensitive

• Empty database, preferably called “datavaultbuilder”

• Make sure to set the timezone to the same as the Datavault Builder

Database-Specific:

Oracle:

• Min. Version: 12.2

• User with admin rights

• Grants to initially create SYS-Triggers

• 3*2 GB Redo-Log

• 2 GB Undo-Log

• max_string_size = extended (recommended)

MS SQL:

• Min. Version: 2017

• Collation: SQL_Latin1_General_CP1_CS_AS

• Login databaseowner user

• Login authenticator (does not need specific grants)

• Compatibility level 140 or higher:

EXEC dbo.sp_dbcmptlevel @DB, 140

• Set Recovery mode to simple unless taking care of backing up transaction logs:

ALTER DATABASE [DB_NAME] SET RECOVERY SIMPLE;

MS Azure SQL database:

• Login databaseowner user

• Collation: SQL_Latin1_General_CP1_CS_AS

• Login authenticator (does not need specific grants)

Synapse:

• Login databaseowner user

• Collation: SQL_Latin1_General_CP1_CS_AS

• Min. Warehouse: DW200c

EXASOL:

• Min Version: 6.2.7

Postgres:

• Min. Version: 9.6

• ENCODING = ‘UTF8’

• Must be compiled with NAMEDATALEN >= 128 (contact us for more information)

• Must be compiled with extionsions (since we need the pgcrypto extension)

  make world
  make install-world
  

• Needs the 3rd party extension ‘set_user’: https://github.com/pgaudit/set_user/releases/tag/REL2_0_0

  • Compile instructions are in the git repo descriptions https://github.com/pgaudit/set_user#compile-and-install

  • Change this config parameter in postgresql.conf

    shared_preload_libraries = 'plpgsql,set_user'
    

• User dbadmin must be database owner of database (datavaultbuilder by default, but can be different):

  CREATE ROLE dbadmin SUPERUSER CREATEROLE REPLICATION LOGIN;
  ALTER DATABASE datavaultbuilder OWNER TO dbadmin;

Snowflake:

• For the install: User with permissions to create database, database objects, users and user roles. The user must have the following privileges:

  USE ROLE ACCOUNTADMIN;
  USE ROLE SYSADMIN;

Note

Please feel free to contact us for the support of other versions.

Install DVB Structures on Database

To complete the Setup of the Datavault Builder, some Objects (Mainly Tables for Config and Logging and Metadata Views) need to be installed on the target Database. When using a separate, non-containered Version of the Database, then this can be achieved by starting up the containerized version of the clientdabase once, connecting into the container and executing the install scripts.

Using the Helper Container

1. Run a helper container and connect to the shell using bash or windows powershell

docker run -it --rm --entrypoint=/bin/bash --mount type=bind,ro=true,src=$(pwd)/secrets/authenticator_password.txt,dst=/run/secrets/authenticator_password docker.datavault-builder.com/datavaultbuilder/clientdb_<target_db>:6.10.1.1

Hint

1. where <target_db> is mssql, oracle, postgres or exasol

2. the secret file for the authenticator user under secrets/authenticator_password.txt must exists

2. change into folder with source code:

   cd /dvb_sql/
   

3. execute shell script which deploys source code, depending on the database, the script execution may vary.

   Oracle:

   ./create_db.sh '<sys-user>/"<password>"@\"<host>:<port>/<service>.<domain>" as sysdba'
   

   • the sys user must have have the sysdba role for this script to work.

   MS SQL:

   ./create_db.sh <host>[/<instance_name>|,<port>] <sys-user> <password> <database-name>
   
   Sample:
   ./create_db.sh internal.domain.com,1433 sysusername password123 dvb_warehouse
   

   Hint

   1. If the named database does not exists, it will be created

   2. Make sure to either specify an instance_name or a port, but not both!

   Exasol:

   ./create_db.sh <host>:<port> <sys-user> <password>
   

   Postgres:

   ./create_db.sh <host> <port> <database-name> <admin-user> <admin-password>
   

Without the Helper Container

Snowflake:

1. Install the Snowflake python module:

   python -m pip install --upgrade pip
   pip install --upgrade snowflake-connector-python
   

2. Execute python script to install DVB structures:

   python .\snowflake_by_schema_install.py -d <db_name> -w <warehouse_name> -a <snowflake_account> -u <username> -p <password>
   

Hint

The python script will be available in the install package zip file, which you will receive from us.