Docker-Compose Configuration

Files and Structure

The directory name containing the docker-compose.yml file is the default project name for the stack. The created docker container are prefixed with this name. E.g. datavaultbuilder. The files and directories we need are by default structured like this:

datavaultbuilder
|
-- secrets
|   |
|   | -- <secret_file_1>.txt
|   | -- <secret_file_2>.txt
|   | -- ...
|
| -- .env
| -- datavault_builder_license.lic
| -- docker-compose.yml

The secrets directory holds the password and key files needed by the containers. See the description of the different containers for more information The file called .env (be aware of the dot prefix.This means it is hidden in linux - to show it, type ls -al) holds variables needed for all containers:

# defines the docker tag used to choose which version of datavaultbuilder to install
DVB_TAG=4.0.8.3
# timezone for all containers and databases - see TZ under https://en.wikipedia.org/wiki/List_of_tz_database_time_zones
TIMEZONE=Europe/Zurich

The file datavaultbuilder-license.lic holds the license we provided to you

The file docker-compose.yml has the configuration of the docker stack. Please be aware that the whitespaces in a YAML are important! The general stucture looks like this:

version: '3.5'
services:
core:
    <Core container config>

webgui:
    <Webgui container config>

scheduler:
    <Scheduler container config>

api:
    <API container config>

clientdb_<db_type>:
    <in case you use a containerized database>

secrets:
<secret name>:
    file: secrets/<secret file>
#... (see the chapters of the different containers for specification)

volumes:
<volume name>:
#... (see the chapters of the different containers for specification)

Here’s how you configure each container in the code above:

<Core container config>

<Webgui container config>

<Scheduler container config>

<API container config>

<in case you use a containerized database>

See an example of a full directory in our partner portal

Core

   core:
   env_file: .env
   environment:
     - 'CLIENT_DB_CONNECTIONSTRING=jdbc:postgresql://clientdb_postgres:5432/datavaultbuilder?stringtype=unspecified&logUnclosedConnections=true'
     - CLIENT_DB_TYPE=postgres_client_db
     - ENABLE_BETA_FEATURES=false
     - 'USE_UNSECURE_DEFAULT_ENCRYPTION_KEYS=false'
     - 'PLJAVA_VMOPTIONS=-Djava.security.egd=file:///dev/urandom -Xms128M -Xss2M'
     - DOWNLOAD_DEMO_DATA=true
     - 'GUI_USER_NAME=yourName'
     - 'GUI_USER_PASSWORD=yourPassword'
     - 'GUI_USER_GROUP=dvb_admin'
     - 'GUI_USER_EMAIL=your@mail.com'
   image: 'datavaultbuilder/core:{DVB_TAG}'
   volumes:
     - files:/files
   secrets:
     - systems_password_public_key
     - systems_password_private_key
     - systems_password_private_key_password
     - core_dbadmin_password
     - authenticator_password
     - scheduler_password
     - datavault_builder_license
....
   secrets:
       systems_password_public_key:
       file: secrets/systems_password_public_key.txt
       systems_password_private_key:
       file: secrets/systems_password_private_key.txt
       systems_password_private_key_password:
       file: secrets/systems_password_private_key_password.txt
       core_dbadmin_password:
       file: secrets/core_dbadmin_password.txt
       authenticator_password:
       file: secrets/authenticator_password.txt
       scheduler_password:
       file: secrets/scheduler_password.txt
       datavault_builder_license:
       file: datavault_builder_license.lic
   volumes:
       files:

Environment-Variable

Sample-Setting

Purpose

Possible Values

Since DVB Version

CLIENT_DB_
CONNECTIONSTRING
jdbc:postgresql://clientdb_postgres:
5432/datavaultbuilder

jdbc:sqlserver://clientdb_mssql:1433;
databaseName=datavaultbuilder;
integratedSecurity=false;

jdbc:oracle:oci:@clientdb_oracle:1521/
DVBPDB.localdomain

jdbc:exa:clientdbexasol:8888

Specifies onto which processing database the core engine will connect. In case you are using a non-containerd database, specify the JDBC connection string to reach the database.

<4.0.0.0

CLIENT_DB_TYPE

postgres_client_db

Type of the used processing database.

  • postgres_client_db

  • mssql_fdb

  • oracle_client_db

  • exasol_client_db

<4.0.0.0

CLIENT_DB_AUTHENTICATOR_
USERNAME

authenticator

Sets the username of the technical user that impersonates into the needed roles on the processing database

4.0.8.2

CLIENT_DB_CONNECTIONSTRING_
USER_AUTHENTICATION
jdbc:sqlserver://clientdb_mssql:1433;
databaseName=datavaultbuilder;
integratedSecurity=true;
authenticationScheme=JavaKerberos;

Specifies the connection string used to authenticate users on the database. If not set, CLIENT_DB_CONNECTIONSTRING is used instead.

4.0.8.2

ENABLE_BETA_FEATURES

true

Activate beta features in core and gui.

<4.0.0.0

USE_UNSECURE_DEFAULT_
ENCRYPTION_KEYS

true

Make use of predefined encryption keys for system password encryption. Don’t activate this setting in a productive environment!

<4.0.0.0

PLJAVA_VMOPTIONS

Djava.security.egd=file:///dev/urandom -Xms128M -Xss2M

Additional settings used for the JVMs which are started

<4.0.0.0

DOWNLOAD_DEMO_DATA

true

Get some test-csv-files downloaded into the files folder

<4.0.0.0

GUI_USER_NAME

yourName

Initially created admin_user name

<4.0.0.0

GUI_USER_PASSWORD

yourPassword

Password for initially created admin_user

<4.0.0.0

GUI_USER_GROUP

dvb_admin

Group for initial admin user, should always be dvb_admin, will be depracted in the next version

<4.0.0.0

GUI_USER_EMAIL

your@mail.com

Email address of the initially created admin_user name

<4.0.0.0


Secret

Purpose

Since DVB Version

systems_password_
public_key

Public key for the encryption of the stored system passwords on the database.

<4.0.0.0

systems_password_
private_key

Private key for the encryption of the stored system passwords on the database.

<4.0.0.0

systems_password_
private_key_password

Private key password for the encryption of the stored system passwords on the database.

<4.0.0.0

core_dbadmin_password

Password of the user dbadmin which can connect onto the core engine.

<4.0.0.0

authenticator_password

Password of the technical user making user/login independent calls onto the clientdb.

<4.0.0.0

scheduler_password

Password of the scheduler user to connect onto the core and execute scheduled events.

<4.0.0.0

datavault_builder_
license

Mandatory License file necessary for the install. Without a valid license file, the login into the environment will not be possible. Please contact us, in case your license is expired.

4.0.2.0

Generate System Encryption Keys

The encryption-keys are used to encrypt the system passwords configured in the datavault builder. As those will be stored on the processing database, this prevents, that everyone having direct access onto the database can read the passwords in cleartext. Here are some possibilities for the creation of the encryption keys.

On Linux
On a server, you probably need an entropy pool generator, on a client you can just move the mouse. To install an entropy pool generator:

RHEL / CentOS:

sudo yum install rng-tools
sudo rngd -r /dev/urandom

Ubuntu / Debian:

sudo apt-get install rng-tools
sudo rngd -r /dev/urandom

Now we generate the keys.

RHEL 7 / CentOS 7 / Ubuntu 16.04:

gpg --gen-key  (select RSA/RSA, 4096 bits)

Ubuntu 18.04:

gpg --full-generate-key  (select RSA/RSA, 4096 bits)

And export them to files:

gpg -a --export YOUR_EMAIL secrets/systems_password_public_key.txt
gpg -a --export-secret-keys YOUR_EMAIL > secrets/systems_password_private_key.txt
echo YOUR_PASSWORD > secrets/systems_password_private_key_password.txt

Webgui

   webgui:
   env_file: .env
   environment:
     - 'DAV_USER=yourName'
     - 'DAV_PASSWORD=yourPassword'
     - DISABLE_IPV6=false
   image: 'datavaultbuilder/webgui:{DVB_TAG}'
   ports:
     - '80:80'
     - '443:443'
   secrets:
     - ssl_fullchain
     - ssl_private_key
   volumes:
     - files:/files
....
 secrets:
   ssl_fullchain:
     file: secrets/ssl-fullchain.pem
   ssl_private_key:
     file: secrets/domain-com-key.pem
 volumes:
   files:

Environment-Variable

Sample-Setting

Purpose

Since DVB Version

DISABLE_IPV6

false

Turn ipv6 on or off

<4.0.0.0

DAV_USER

yourName

Name of the user being
able to connect to the
Webdav Service. Optional.

<4.0.0.0

DAV_PASSWORD

yourPassword

Password of the user being
able to connect to the
Webdav Service. Optional.

<4.0.0.0


Secret

Purpose

Since DVB Version

ssl_fullchain

SSL certificate as full chain (containing the root certificate, the intermediate certificate if applicable, and the actual certificate of the domain, all in one file, copy them together if needed) in pem-format.

4.0.6.0

ssl_private_key

SSL private key in pem-format.

4.0.6.0

Setup Webdav Access

The Datavault Builder comes with a preconfigured WEBDAV access. This you can use to connect onto the folder /files on the server, upload files and then create a source system for CSV sources to read data from that folder.

To configure the Webdav, simply give the WEBDAV user a name and a password in the webguis environment variables.

Enable SSL-Encryption

If you have an SSL certificate for the domain that points to the Linux hosts, it’s highly recommended to configure that as well, so passwords are not sent in clear text over your LAN.

First we also bind port 443 to the host IP as shown above. Port 80 will now do only a redirection to https, if you don’t want that, you can remove the line - 80:80.

Then you need to put the certificates in your secrets directory and adjust the filenames here in the secrets section (the file: parts).

  • The private key should be in a file that is referenced here as secrets/domain-com.key (please adjust to whatever your actual file is called)

  • The certificate must be available as full chain (containing the root certificate, the intermediate certificate if applicable, and the actual certificate of the domain, all in one file, copy them together if needed). Put this file in the secrets folder as well and adjust secrets/ssl-fullchain.crt to your actual file name.

Scheduler

   scheduler:
   env_file: .env
   environment:
     - MAX_SERVICE_CONNECTION_AGE=3600
     - 'PGAGENT_OPTIONS=-l 2'
   image: 'datavaultbuilder/scheduler:{DVB_TAG}'
   secrets:
     - scheduler_password
....
 secrets:
   scheduler_password:
     file: secrets/scheduler_password.txt

Environment-Variable

Sample-Setting

Purpose

Since DVB Version

MAX_SERVICE_
CONNECTION_AGE

3600

Maximum age for connections to core in seconds. (0=infinite), Default: 0

<4.0.0.0

PGAGENT_OPTIONS

-l 2

additional pgAgent options. e.g. -l 2 for more logs. Default: none

<4.0.0.0


Secret

Purpose

Since DVB Version

scheduler_password

Password of the scheduler user to connect onto the core and execute scheduled events.

<4.0.0.0

API

   api:
   env_file: .env
   environment:
     - CONNECTION_POOL=50
   image: 'datavaultbuilder/api:{DVB_TAG}'
   secrets:
     - authenticator_password
     - core_dbadmin_password
....
 secrets:
   authenticator_password:
     file: secrets/authenticator_password.txt
   core_dbadmin_password:
     file: secrets/core_dbadmin_password.txt

Environment-Variable

Sample-Setting

Purpose

Since DVB Version

CONNECTION_POOL

50

Number of connections between api and core container. Default: 100

<4.0.0.0


Secret

Purpose

Since DVB Version

core_dbadmin_password

Password of the user “dbadmin” which can connect onto the core engine.

<4.0.0.0

authenticator_password

Password of the technical user making user/login independent calls onto the clientdb.

<4.0.0.0