Docker-Compose Configuration

Files and Structure

The directory name containing the docker-compose.yml file is the default project name for the stack. The created docker container are prefixed with this name. E.g. datavaultbuilder. The files and directories we need are by default structured like this:

datavaultbuilder
|
-- secrets
|   |
|   | -- <secret_file_1>.txt
|   | -- <secret_file_2>.txt
|   | -- ...
|
| -- .env
| -- datavault_builder_license.lic
| -- docker-compose.yml

The secrets directory holds the password and key files needed by the containers. See the description of the different containers for more information The file called .env (be aware of the dot prefix.This means it is hidden in linux - to show it, type ls -al) holds variables needed for all containers:

# defines the docker tag used to choose which version of datavaultbuilder to install
DVB_TAG=rel_5_b1.22
# timezone for all containers and databases - see TZ under https://en.wikipedia.org/wiki/List_of_tz_database_time_zones
TIMEZONE=Europe/Zurich

The file datavaultbuilder-license.lic holds the license we provided to you

The file docker-compose.yml has the configuration of the docker stack. Please be aware that the whitespaces in a YAML are important! The general stucture looks like this:

version: '3.5'
services:
core:
    <Core container config>

connection_pool:
    <Connection Pool container config>

webgui:
    <Webgui container config>

prometheus:
    <Metrics provider container config>

 grafana:
    <Metrics provider container config>

scheduler:
    <Scheduler container config>

api:
    <API container config>

clientdb_<db_type>:
    <in case you use a containerized database>

secrets:
<secret name>:
    file: secrets/<secret file>
#... (see the chapters of the different containers for specification)

volumes:
<volume name>:
#... (see the chapters of the different containers for specification)

Here’s how you configure each container in the code above:

<Core container config>

<Connection Pool container config>

<Webgui container config>

<Metrics provider container config>

<Scheduler container config>

<API container config>

<in case you use a containerized database>

See an example of a full directory in our partner portal

Core

 core:
   env_file: .env
   environment:
     - 'CLIENT_DB_CONNECTIONSTRING=jdbc:postgresql://clientdb_postgres:5432/datavaultbuilder?stringtype=unspecified&logUnclosedConnections=true'
     - CLIENT_DB_TYPE=postgres_client_db
     - ENABLE_BETA_FEATURES=false
     - 'USE_UNSECURE_DEFAULT_ENCRYPTION_KEYS=false'
     - 'PLJAVA_VMOPTIONS=-Djava.security.egd=file:///dev/urandom -Xms128M -Xss2M'
     - DOWNLOAD_DEMO_DATA=true
     - 'GUI_USER_NAME=yourName'
     - 'GUI_USER_PASSWORD=yourPassword'
     - 'GUI_USER_GROUP=dvb_admin'
     - 'GUI_USER_EMAIL=your@mail.com'
   image: 'datavaultbuilder/core:{DVB_TAG}'
   volumes:
     - files:/files
   secrets:
     - systems_password_public_key
     - systems_password_private_key
     - systems_password_private_key_password
     - core_dbadmin_password
     - authenticator_password
     - scheduler_password
     - datavault_builder_license
   networks:
     dvbnet:
   restart: always

....
 secrets:
   systems_password_public_key:
     file: secrets/systems_password_public_key.txt
     systems_password_private_key:
     file: secrets/systems_password_private_key.txt
     systems_password_private_key_password:
     file: secrets/systems_password_private_key_password.txt
     core_dbadmin_password:
     file: secrets/core_dbadmin_password.txt
     authenticator_password:
     file: secrets/authenticator_password.txt
     scheduler_password:
     file: secrets/scheduler_password.txt
     datavault_builder_license:
     file: datavault_builder_license.lic
 volumes:
     files:

networks:
 dvbnet:
   driver: bridge
   ipam:
     driver: default

Environment-Variable

Sample-Setting

Purpose

Possible Values

Since DVB Version

CLIENT_DB_
CONNECTIONSTRING
jdbc:postgresql://clientdb_postgres:
5432/datavaultbuilder

jdbc:sqlserver://clientdb_mssql:1433;
databaseName=datavaultbuilder;
integratedSecurity=false;

jdbc:oracle:oci:@clientdb_oracle:1521/
DVBPDB.localdomain

jdbc:exa:clientdbexasol:8888

Specifies onto which processing database the core engine will connect. In case you are using a non-containerd database, specify the JDBC connection string to reach the database.

<4.0.0.0

CLIENT_DB_TYPE

postgres_client_db

Type of the used processing database.

  • postgres_client_db

  • mssql_fdb

  • oracle_client_db

  • exasol_client_db

<4.0.0.0

CLIENT_DB_AUTHENTICATOR_
USERNAME

authenticator

Sets the username of the technical user that impersonates into the needed roles on the processing database

4.0.8.2

CLIENT_DB_CONNECTIONSTRING_
USER_AUTHENTICATION
jdbc:sqlserver://clientdb_mssql:1433;
databaseName=datavaultbuilder;
integratedSecurity=true;
authenticationScheme=JavaKerberos;

Specifies the connection string used to authenticate users on the database. If not set, CLIENT_DB_CONNECTIONSTRING is used instead.

4.0.8.2

ENABLE_BETA_FEATURES

true

Activate beta features in core and gui.

<4.0.0.0

USE_UNSECURE_DEFAULT_
ENCRYPTION_KEYS

true

Make use of predefined encryption keys for system password encryption. Don’t activate this setting in a productive environment!

<4.0.0.0

PLJAVA_VMOPTIONS

Djava.security.egd=file:///dev/urandom -Xms128M -Xss2M

Additional settings used for the JVMs which are started

<4.0.0.0

DOWNLOAD_DEMO_DATA

true

Get some test-csv-files downloaded into the files folder

<4.0.0.0

GUI_USER_NAME

yourName

Initially created admin_user name

<4.0.0.0

GUI_USER_PASSWORD

yourPassword

Password for initially created admin_user

<4.0.0.0

GUI_USER_GROUP

dvb_admin

Group for initial admin user, should always be dvb_admin, will be depracted in the next version

<4.0.0.0

GUI_USER_EMAIL

your@mail.com

Email address of the initially created admin_user name

<4.0.0.0

EXASOL_USE_CHAR_FOR_HASH

false

Make environment use CHAR for hash fields instead of HASHTYPE (compatibility mode for EXASOL 6.*)

5.3.2.0


Secret

Purpose

Since DVB Version

systems_password_
public_key

Public key for the encryption of the stored system passwords on the database.

<4.0.0.0

systems_password_
private_key

Private key for the encryption of the stored system passwords on the database.

<4.0.0.0

systems_password_
private_key_password

Private key password for the encryption of the stored system passwords on the database. (Should be on the first line)

<4.0.0.0

core_dbadmin_password

Password of the user dbadmin which can connect onto the core engine. (Should be on the first line, prevent non alphanumeric charaters except ‘_’ - just choose a longer password instead)

<4.0.0.0

authenticator_password

Password of the technical user making user/login independent calls onto the clientdb. (Should be on the first line, prevent non alphanumeric charaters except ‘_’ - just choose a longer password instead. The password must also fulfill all restrictions of the clientDB since it is used on multiple places.)

<4.0.0.0

scheduler_password

Password of the scheduler user to connect onto the core and execute scheduled events. (Should be on the first line, prevent non alphanumeric charaters except ‘_’ - just choose a longer password instead)

<4.0.0.0

datavault_builder_
license

Mandatory License file necessary for the install. Without a valid license file, the login into the environment will not be possible. Please contact us, in case your license is expired.

4.0.2.0

Connection Pool

 connection_pool:
   env_file: .env
   environment:
     - CORE_POOL_MAX_CONNECTIONS=25
     - CLIENTDB_POOL_MAX_CONNECTIONS=30
     - METRICS_PROVIDER=micrometer-prometheus
   image: 'datavaultbuilder/connection_pool:${DVB_TAG}'
   volumes:
     - files:/files
     - pddb:/opt/datavaultbuilder/var/lib/pddb
   secrets:
     - systems_password_public_key
     - systems_password_private_key
     - systems_password_private_key_password
     - core_dbadmin_password
     - authenticator_password
   networks:
     dvbnet:
       aliases:
         - connectionpool
   restart: always
....

volumes:
 files:
 pddb:

networks:
 dvbnet:
   driver: bridge
   ipam:
     driver: default

Environment-Variable

Sample-Setting

Purpose

Possible Values

Since DVB Version

CORE_
POOL_MAX_CONNECTIONS
30

Specifies the number of jdbc pool connection to the core container.

<5.0.0.0

CLIENTDB_
POOL_MAX_CONNECTIONS
40

Specifies the number of jdbc pool connection to the client db container.

<5.0.0.0

CORE_
POOL_CONNECTION_TIMEOUT_MS
30000

Specifies the time in milliseconds to get a free connection out of.the core pool before raising an exception

<5.0.0.0

CLIENTDB_
POOL_CONNECTION_TIMEOUT_MS
30000

Specifies the time in milliseconds to get a free connection out of.the client db pool before raising an exception

<5.0.0.0

JAVA_OPTS

Optional java parameters

<5.0.0.0

METRICS_PROVIDER
micrometer-prometheus

specificies if connection metrics should be logged and can be shown in the webgui as grafana dashboards

<5.0.0.0


Secret

Purpose

Since DVB Version

systems_password_
public_key

Public key for the encryption of the stored system passwords on the database.

<4.0.0.0

systems_password_
private_key

Private key for the encryption of the stored system passwords on the database.

<4.0.0.0

systems_password_
private_key_password

Private key password for the encryption of the stored system passwords on the database. (Should be on the first line)

<4.0.0.0

core_dbadmin_password

Password of the user dbadmin which can connect onto the core engine. (Should be on the first line, prevent non alphanumeric charaters except ‘_’ - just choose a longer password instead)

<4.0.0.0

authenticator_password

Password of the technical user making user/login independent calls onto the clientdb. (Should be on the first line, prevent non alphanumeric charaters except ‘_’ - just choose a longer password instead. The password must also fulfill all restrictions of the clientDB since it is used on multiple places.)

<4.0.0.0

Webgui

 webgui:
   env_file: .env
   environment:
     - 'DAV_USER=yourName'
     - 'DAV_PASSWORD=yourPassword'
     - DISABLE_IPV6=false
   image: 'datavaultbuilder/webgui:{DVB_TAG}'
   ports:
     - '80:80'
     - '443:443'
   secrets:
     - ssl_fullchain
     - ssl_private_key
   volumes:
     - files:/files
   networks:
     dvbnet:
   restart: always

....
secrets:
   ssl_fullchain:
     file: secrets/ssl-fullchain.pem
   ssl_private_key:
     file: secrets/domain-com-key.pem

volumes:
   files:

networks:
 dvbnet:
   driver: bridge
   ipam:
     driver: default

Environment-Variable

Sample-Setting

Purpose

Since DVB Version

DISABLE_IPV6

false

Turn ipv6 on or off

<4.0.0.0

DAV_USER

yourName

Name of the user being
able to connect to the
Webdav Service. Optional.

<4.0.0.0

DAV_PASSWORD

yourPassword

Password of the user being
able to connect to the
Webdav Service. Optional.

<4.0.0.0


Secret

Purpose

Since DVB Version

ssl_fullchain

SSL certificate as full chain (containing the root certificate, the intermediate certificate if applicable, and the actual certificate of the domain, all in one file, copy them together if needed) in pem-format.

4.0.6.0

ssl_private_key

SSL private key in pem-format.

4.0.6.0

Setup Webdav Access

The Datavault Builder comes with a preconfigured WEBDAV access. This you can use to connect onto the folder /files on the server, upload files and then create a source system for CSV sources to read data from that folder.

To configure the Webdav, simply give the WEBDAV user a name and a password in the webguis environment variables.

Enable SSL-Encryption

If you have an SSL certificate for the domain that points to the Linux hosts, it’s highly recommended to configure that as well, so passwords are not sent in clear text over your LAN.

First we also bind port 443 to the host IP as shown above. Port 80 will now do only a redirection to https, if you don’t want that, you can remove the line - 80:80.

Then you need to put the certificates in your secrets directory and adjust the filenames here in the secrets section (the file: parts).

  • The private key should be in a file that is referenced here as secrets/domain-com.key (please adjust to whatever your actual file is called)

  • The certificate must be available as full chain (containing the root certificate, the intermediate certificate if applicable, and the actual certificate of the domain, all in one file, copy them together if needed). Put this file in the secrets folder as well and adjust secrets/ssl-fullchain.crt to your actual file name.

Metrics dashboard

(for connections and future extensions)

Accessible by adding /grafana to your Datavault Builder URL

 prometheus:
   env_file: .env
   image: 'datavaultbuilder/prometheus:${DVB_TAG}'
   volumes:
     - prometheus:/var/lib/prometheus
   networks:
     - dvbnet
   restart: always

 grafana:
   env_file: .env
   environment:
     - 'GF_SERVER_ROOT_URL=http://your-hostname-or-ip.com/grafana'
     #- 'GF_SECURITY_ADMIN_PASSWORD=someSecretPassword!'
     - GF_AUTH_ANONYMOUS_ENABLED=true
   image: 'datavaultbuilder/grafana:${DVB_TAG}'
   volumes:
     - grafana:/var/lib/grafana
   networks:
     - dvbnet
   restart: always

....

 volumes:
   prometheus:
   grafana:

networks:
 dvbnet:
   driver: bridge
   ipam:
     driver: default

Environment-Variable

Sample-Setting

Purpose

Since DVB Version

GF_SERVER_ROOT_URL

http://your-hostname-or-ip.com/grafana

The URL you use to access datavault builder in the browser, followed by /grafana

5.0.0.0

GF_AUTH_ANONYMOUS_ENABLED

true

Alter this according to the Grafana documentation if you want to restrict access to the metrics dashboard

5.0.0.0

Scheduler

 scheduler:
   env_file: .env
   environment:
     - MAX_SERVICE_CONNECTION_AGE=3600
     - 'PGAGENT_OPTIONS=-l 2'
   image: 'datavaultbuilder/scheduler:{DVB_TAG}'
   secrets:
     - scheduler_password
   networks:
     dvbnet:
   restart: always

....
secrets:
 scheduler_password:
   file: secrets/scheduler_password.txt

networks:
 dvbnet:
   driver: bridge
   ipam:
     driver: default

Environment-Variable

Sample-Setting

Purpose

Since DVB Version

MAX_SERVICE_
CONNECTION_AGE

3600

Maximum age for connections to core in seconds. (0=infinite), Default: 0

<4.0.0.0

PGAGENT_OPTIONS

-l 2

additional pgAgent options. e.g. -l 2 for more logs. Default: none

<4.0.0.0


Secret

Purpose

Since DVB Version

scheduler_password

Password of the scheduler user to connect onto the core and execute scheduled events. (Should be on the first line, prevent non alphanumeric charaters except ‘_’ - just choose a longer password instead)

<4.0.0.0

API

 api:
   env_file: .env
   environment:
     - CONNECTION_POOL=30
   image: 'datavaultbuilder/api:{DVB_TAG}'
   secrets:
     - authenticator_password
     - core_dbadmin_password
   networks:
     dvbnet:
   restart: always

....
 secrets:
   authenticator_password:
     file: secrets/authenticator_password.txt
   core_dbadmin_password:
     file: secrets/core_dbadmin_password.txt
networks:
 dvbnet:
   driver: bridge
   ipam:
     driver: default

Environment-Variable

Sample-Setting

Purpose

Since DVB Version

CONNECTION_POOL

30

Number of connections between api and core container. Default: 30

<4.0.0.0


Secret

Purpose

Since DVB Version

core_dbadmin_password

Password of the user “dbadmin” which can connect onto the core engine. (Should be on the first line, prevent non alphanumeric charaters except ‘_’ - just choose a longer password instead)

<4.0.0.0

authenticator_password

Password of the technical user making user/login independent calls onto the clientdb. (Should be on the first line, prevent non alphanumeric charaters except ‘_’ - just choose a longer password instead. The password must also fulfill all restrictions of the clientDB since it is used on multiple places.)

<4.0.0.0

Client-Database: Postgres

 clientdb_postgres:
   env_file: .env
   environment:
     - EFFECTIVE_CACHE_SIZE=2GB
     - MAINTENANCE_WORK_MEM=128MB
     - MAX_PARALLEL_WORKERS=2
     - SHARED_BUFFERS=2GB
     - WORK_MEM=512MB
   image: 'datavaultbuilder/clientdb_postgres:{DVB_TAG}'
   volumes:
     - postgresdata:/data
   ports:
     - '5433:5432'
   secrets:
     - authenticator_password
     - clientdb_dbadbmin_password
   networks:
     - dvbnet
   restart: always

....
 secrets:
   authenticator_password:
     file: secrets/authenticator_password.txt
   clientdb_dbadbmin_password:
     file: secrets/clientdb_dbadbmin_password.txt

volumes:
 postgresdata:

networks:
 dvbnet:
   driver: bridge
   ipam:
     driver: default

Environment-Variable

Sample-Setting

Purpose

Since DVB Version

EFFECTIVE_CACHE_SIZE

2GB

See postgres documentation

<4.0.0.0

MAINTENANCE_WORK_MEM

128MB

See postgres documentation

<4.0.0.0

SHARED_BUFFERS

2GB

See postgres documentation

<4.0.0.0

WORK_MEM

512MB

See postgres documentation

<4.0.0.0


Secret

Purpose

Since DVB Version

clientdb_dbadmin_password

Password of the user “dbadmin” which can connect onto the database. (Should be on the first line, prevent non alphanumeric charaters except ‘_’ - just choose a longer password instead)

<4.0.0.0

authenticator_password

Password of the technical user making user/login independent calls onto the clientdb.

<4.0.0.0

Client-Database: MS SQL

 clientdb_mssql:
   env_file: .env
   environment:
     - 'SA_PASSWORD=yourPassword'
     - 'ACCEPT_EULA=Y'
     - MSSQL_PID=Developer
     - MSSQL_COLLATION=SQL_Latin1_General_CP1_CS_AS
     - MSSQL_MEMORY_LIMIT_MB=4000
   image: 'datavaultbuilder/clientdb_mssql:{DVB_TAG}'
   ports:
     - '1433:1433'
   secrets:
     - authenticator_password
   volumes:
     - mssqldata:/var/opt/mssql
   networks:
     dvbnet:
   restart: always

....
secrets:
 authenticator_password:
   file: secrets/authenticator_password.txt

volumes:
 mssqldata:

networks:
 dvbnet:
   driver: bridge
   ipam:
     driver: default

Environment-Variable

Sample-Setting

Purpose

Since DVB Version

SA_PASSWORD

yourPWD

Password for sa user on the database

<4.0.0.0

ACCEPT_EULA

Y

Accept MS SQL License agreement

<4.0.0.0

MSSQL_PID

Developer

See microsoft documentation

<4.0.0.0

MSSQL_COLLATION

SQL_Latin1_General_CP1_CS_AS

Set the collation of the database to start

<4.0.0.0

MSSQL_MEMORY_LIMIT_MB

4000

Limits the memory which is allocated to the database. Important to set a useful value here! See microsoft documentation

<4.0.0.0


Secret

Purpose

Since DVB Version

authenticator_password

Password of the technical user making user/login independent calls onto the clientdb. (Should be on the first line, prevent non alphanumeric charaters except ‘_’ - just choose a longer password instead. The password must also fulfill all restrictions of the MsSql Server too, since it is used on multiple places.)

<4.0.0.0

Client-Database: Oracle

 clientdb_oracle:
   env_file: .env
   environment:
     - 'DB_PASSWD=yourPassword'
     - MAX_STRING_SIZE=EXTENDED
     - DB_MEMORY=3GB #GB!
     - REDO_LOG_SIZE=1G
     - UNDO_LOG_SIZE=1G
     - UNDO_LOG_AUTOEXTEND_SIZE=512M
   image: 'datavaultbuilder/clientdb_oracle:{DVB_TAG}'
   ports:
     - '1521:1521'
   secrets:
     - authenticator_password
   volumes:
     - oracledata:/ORCL
   networks:
     - dvbnet
   restart: always

....

secrets:
 authenticator_password:
   file: secrets/authenticator_password.txt

volumes:
 oracledata:

networks:
 dvbnet:
   driver: bridge
   ipam:
     driver: default

Environment-Variable

Sample-Setting

Purpose

Since DVB Version

MAX_STRING_SIZE

extended

Start the Oracle database with extended string size setting

<4.0.0.0

DB_PASSWD

yourPWD

Password for sys user. If you would like to login: SYS AS SYSDBA. Must fulfill all complexity restrictions of the Oracle Server and may not contain special characters such as all sort of quotes or the ‘@’ character.

<4.0.0.0

DB_MEMORY

3GB #GB!

Allocated Memory to the database

4.0.6.0

REDO_LOG_SIZE

1G

Size of the databases redo log for transactions

4.0.6.0

UNDO_LOG_SIZE

1G

Size of the databases undo log for transactions

4.0.6.0

REDO_LOG_AUTOEXTEND_SIZE

512M

Stepsize for autoextending the databases undo log for transactions

4.0.6.0


Secret

Purpose

Since DVB Version

authenticator_password

Password of the technical user making user/login independent calls onto the clientdb. (Should be on the first line, prevent non alphanumeric charaters except ‘_’ - just choose a longer password instead. The password must also fulfill all restrictions of the Oracle Server too, since it is used on multiple places.)

<4.0.0.0

Client-Database: Exasol

 clientdb_exasol:
   env_file: .env
   environment:
     - 'DO_NOT_UPDATE=comment out to auto update on start'
     - 'UPDATE_EXASOL_DB=true'
   image: 'datavaultbuilder/clientdb_exasol:{DVB_TAG}'
   ports:
     - '8888:8888'
   secrets:
     - authenticator_password
   volumes:
     - exasoldata:/exa
   networks:
     dvbnet:
       ipv4_address: 172.16.239.100    # a fix ip out of the subnet defined below - only needed for exasol container
       aliases:
         - clientdbexasol
....
secrets:
 authenticator_password:
   file: secrets/authenticator_password.txt

volumes:
 exasoldata:

networks:
 dvbnet:
   driver: bridge
   ipam:
     driver: default
     config:
     - subnet: 172.16.239.0/24   # a fix ip subnet for the stack - only needed for exasol container, other external exasol databases!

Environment-Variable

Sample-Setting

Purpose

Since DVB Version

DO_NOT_UPDATE

comment out to auto update on start

Will overwrite the dvb structures on the database on startup if commented out

<4.0.0.0

UPDATE_EXASOL_DB

false

Will update the data on the database before startup

<4.1.8.0


Secret

Purpose

Since DVB Version

authenticator_password

Password of the technical user making user/login independent calls onto the clientdb. (Should be on the first line, prevent non alphanumeric charaters except ‘_’ - just choose a longer password instead. The password must also fulfill all restrictions of the Exasol Server too, since it is used on multiple places.)

<4.0.0.0