Docker-Compose Configuration¶
Files and Structure¶
The directory name containing the docker-compose.yml
file is the default project name for the stack. The created docker container are prefixed with this name. E.g. datavaultbuilder. The files and directories we need are by default structured like this:
datavaultbuilder
|
-- secrets
| |
| | -- <secret_file_1>.txt
| | -- <secret_file_2>.txt
| | -- ...
|
| -- .env
| -- datavault_builder_license.lic
| -- docker-compose.yml
The secrets
directory holds the password and key files needed by the containers. See the description of the different containers for more information
The file called .env
(be aware of the dot prefix.This means it is hidden in linux - to show it, type ls -al
) holds variables needed for all containers:
# defines the docker tag used to choose which version of datavaultbuilder to install
DVB_TAG=5.4.0.0
DVB_REGISTRY=docker.datavault-builder.com/
# timezone for all containers and databases - see TZ under https://en.wikipedia.org/wiki/List_of_tz_database_time_zones
TIMEZONE=Europe/Zurich
The file datavaultbuilder-license.lic
holds the license we provided to you
The file docker-compose.yml
has the configuration of the docker stack. Please be aware that the whitespaces in a YAML are important! The general stucture looks like this:
version: '3.5'
services:
core:
<Core container config>
connection_pool:
<Connection Pool container config>
webgui:
<Webgui container config>
prometheus:
<Metrics provider container config>
grafana:
<Metrics provider container config>
scheduler:
<Scheduler container config>
api:
<API container config>
cicd:
<CICD container config>
clientdb_<db_type>:
<in case you use a containerized database>
secrets:
<secret name>:
file: secrets/<secret file>
#... (see the chapters of the different containers for specification)
volumes:
<volume name>:
#... (see the chapters of the different containers for specification)
Here’s how you configure each container in the code above:
<Connection Pool container config>
<Metrics provider container config>
<in case you use a containerized database>
See an example of a full directory in our partner portal
Core¶
core:
env_file: .env
environment:
- 'CLIENT_DB_CONNECTIONSTRING=jdbc:postgresql://clientdb_postgres:5432/datavaultbuilder?stringtype=unspecified&logUnclosedConnections=true'
- CLIENT_DB_TYPE=postgres_client_db
- ENABLE_BETA_FEATURES=false
- 'USE_UNSECURE_DEFAULT_ENCRYPTION_KEYS=false'
- 'PLJAVA_VMOPTIONS=-Djava.security.egd=file:///dev/urandom -Xms128M -Xss2M'
- DOWNLOAD_DEMO_DATA=true
- 'GUI_USER_NAME=yourName'
- 'GUI_USER_PASSWORD=yourPassword'
- 'GUI_USER_GROUP=dvb_admin'
- 'GUI_USER_EMAIL=your@mail.com'
image: '{DVB_REGISTRY}datavaultbuilder/core:{DVB_TAG}'
volumes:
- files:/files
secrets:
- systems_password_public_key
- systems_password_private_key
- systems_password_private_key_password
- core_dbadmin_password
- authenticator_password
- scheduler_password
- datavault_builder_license
networks:
dvbnet:
restart: always
....
secrets:
systems_password_public_key:
file: secrets/systems_password_public_key.txt
systems_password_private_key:
file: secrets/systems_password_private_key.txt
systems_password_private_key_password:
file: secrets/systems_password_private_key_password.txt
core_dbadmin_password:
file: secrets/core_dbadmin_password.txt
authenticator_password:
file: secrets/authenticator_password.txt
scheduler_password:
file: secrets/scheduler_password.txt
datavault_builder_license:
file: datavault_builder_license.lic
volumes:
files:
networks:
dvbnet:
driver: bridge
ipam:
driver: default
Environment-Variable |
Sample-Setting |
Purpose |
Possible Values |
Since DVB Version |
---|---|---|---|---|
CLIENT_DB_ CONNECTIONSTRING |
jdbc:postgresql://clientdb_postgres:
5432/datavaultbuilder
jdbc:sqlserver://clientdb_mssql:1433;
databaseName=datavaultbuilder;
integratedSecurity=false;
jdbc:oracle:oci:@clientdb_oracle:1521/
DVBPDB.localdomain
jdbc:exa:clientdbexasol:8888
|
Specifies onto which processing database the core engine will connect. In case you are using a non-containerd database, specify the JDBC connection string to reach the database. |
<4.0.0.0 |
|
|
postgres_client_db |
Type of the used processing database. |
|
<4.0.0.0 |
CLIENT_DB_AUTHENTICATOR_ USERNAME |
authenticator |
Sets the username of the technical user that impersonates into the needed roles on the processing database |
4.0.8.2 |
|
CLIENT_DB_CONNECTIONSTRING_ USER_AUTHENTICATION |
jdbc:sqlserver://clientdb_mssql:1433;
databaseName=datavaultbuilder;
integratedSecurity=true;
authenticationScheme=JavaKerberos;
|
Specifies the connection string used to authenticate users on the database. If not set, |
4.0.8.2 |
|
|
true |
Activate beta features in core and gui. |
<4.0.0.0 |
|
USE_UNSECURE_DEFAULT_ ENCRYPTION_KEYS |
true |
Make use of predefined encryption keys for system password encryption. Don’t activate this setting in a productive environment! |
<4.0.0.0 |
|
|
Djava.security.egd=file:///dev/urandom -Xms128M -Xss2M |
Additional settings used for the JVMs which are started |
<4.0.0.0 |
|
|
true |
Get some test-csv-files downloaded into the files folder |
<4.0.0.0 |
|
|
yourName |
Initially created |
<4.0.0.0 |
|
|
yourPassword |
Password for initially created |
<4.0.0.0 |
|
|
dvb_admin |
Group for initial admin user, should always be |
<4.0.0.0 |
|
|
your@mail.com |
Email address of the initially created |
<4.0.0.0 |
|
|
false |
Make environment use CHAR for hash fields instead of HASHTYPE (compatibility mode for EXASOL 6.*) |
5.3.2.0 |
|
|
false |
Do not switch into other roles. Instead, use the “authenticator” to execute all actions. |
5.3.2.0 |
|
|
|
Use a custom impersonation template to switch into dvb user roles. E.g. can be used to prefix roles or impersonate into user instead of login on SQL Server. |
5.4.7.0 |
|
|
|
Use a custom revert impersonation template to switch connection back from dvb user roles. |
5.4.7.0 |
|
|
DEBUG |
Levels used for identifying the severity of an event. |
OFF, FATAL, ERROR, WARN, INFO, DEBUG, TRACE, ALL |
5.4.0.0 |
|
VAULT |
Timestamp used for the load time of into the vault layer. Can be VAULT (insert time) or STAGING (extract time) |
VAULT, STAGING |
6.0.3.0 |
Secret |
Purpose |
Since DVB Version |
---|---|---|
systems_password_
public_key
|
Public key for the encryption of the stored system passwords on the database. |
<4.0.0.0 |
systems_password_
private_key
|
Private key for the encryption of the stored system passwords on the database. |
<4.0.0.0 |
systems_password_
private_key_password
|
Private key password for the encryption of the stored system passwords on the database. (Should be on the first line) |
<4.0.0.0 |
core_dbadmin_password |
Password of the user |
<4.0.0.0 |
authenticator_password |
Password of the technical user making user/login independent calls onto the clientdb. (Should be on the first line, prevent non alphanumeric charaters except ‘_’ - just choose a longer password instead. The password must also fulfill all restrictions of the clientDB since it is used on multiple places.) |
<4.0.0.0 |
scheduler_password |
Password of the scheduler user to connect onto the core and execute scheduled events. (Should be on the first line, prevent non alphanumeric charaters except ‘_’ - just choose a longer password instead) |
<4.0.0.0 |
datavault_builder_
license
|
Mandatory License file necessary for the install. Without a valid license file, the login into the environment will not be possible. Please contact us, in case your license is expired. |
4.0.2.0 |
Connection Pool¶
connection_pool:
env_file: .env
environment:
- CORE_POOL_MAX_CONNECTIONS=25
- CLIENTDB_POOL_MAX_CONNECTIONS=30
- METRICS_PROVIDER=micrometer-prometheus
- ADD_CA_CERTIFICATE_PEM_SECRETS=my_ca_cert_1,my_ca_cert_2
image: '{DVB_REGISTRY}datavaultbuilder/connection_pool:${DVB_TAG}'
volumes:
- files:/files
- pddb:/opt/datavaultbuilder/var/lib/pddb
secrets:
- systems_password_public_key
- systems_password_private_key
- systems_password_private_key_password
- core_dbadmin_password
- authenticator_password
- my_ca_cert_1
- my_ca_cert_2
networks:
dvbnet:
aliases:
- connectionpool
restart: always
....
volumes:
files:
pddb:
secrets:
....
my_ca_cert_1:
file: ./secrets/my_ca_cert_1.pem
my_ca_cert_2:
file: ./secrets/my_ca_cert_2.pem
networks:
dvbnet:
driver: bridge
ipam:
driver: default
Environment-Variable |
Sample-Setting |
Purpose |
Possible Values |
Since DVB Version |
---|---|---|---|---|
CORE_ POOL_MAX_CONNECTIONS |
30
|
Specifies the number of jdbc pool connection to the core container. |
<5.0.0.0 |
|
CLIENTDB_ POOL_MAX_CONNECTIONS |
40
|
Specifies the number of jdbc pool connection to the client db container. |
<5.0.0.0 |
|
CORE_ POOL_CONNECTION_TIMEOUT_MS |
30000
|
Specifies the time in milliseconds to get a free connection out of.the core pool before raising an exception |
<5.0.0.0 |
|
CLIENTDB_ POOL_CONNECTION_TIMEOUT_MS |
30000
|
Specifies the time in milliseconds to get a free connection out of.the client db pool before raising an exception |
<5.0.0.0 |
|
JAVA_OPTS |
Optional java parameters |
<5.0.0.0 |
||
METRICS_PROVIDER |
micrometer-prometheus
|
specificies if connection metrics should be logged and can be shown in the webgui as grafana dashboards |
<5.0.0.0 |
|
|
DEBUG |
Levels used for identifying the severity of an event. |
OFF, FATAL, ERROR, WARN, INFO, DEBUG, TRACE, ALL |
5.4.0.0 |
|
Athena |
Space separated list of jdbc drivers which are not used and shall be removed (mainly for environment hardening). E.g. matching Driver name as “Athena*”. |
6.1.6.0 |
|
|
my_ca_cert_1,my_ca_cert_2 |
Comma separated list of additional CA certificates (in PEM format) to be installed. The names listed here must be names of secrets mapped into this container. Needed e.g. for connections over SSL proxies or with company internal CAs. |
6.5.0.0 |
Secret |
Purpose |
Since DVB Version |
---|---|---|
systems_password_
public_key
|
Public key for the encryption of the stored system passwords on the database. |
<4.0.0.0 |
systems_password_
private_key
|
Private key for the encryption of the stored system passwords on the database. |
<4.0.0.0 |
systems_password_
private_key_password
|
Private key password for the encryption of the stored system passwords on the database. (Should be on the first line) |
<4.0.0.0 |
core_dbadmin_password |
Password of the user |
<4.0.0.0 |
authenticator_password |
Password of the technical user making user/login independent calls onto the clientdb. (Should be on the first line, prevent non alphanumeric charaters except ‘_’ - just choose a longer password instead. The password must also fulfill all restrictions of the clientDB since it is used on multiple places.) |
<4.0.0.0 |
Webgui¶
webgui:
env_file: .env
environment:
- 'DAV_USER=yourName'
- 'DAV_PASSWORD=yourPassword'
- DISABLE_IPV6=false
image: '{DVB_REGISTRY}datavaultbuilder/webgui:{DVB_TAG}'
ports:
- '80:80'
- '443:443'
secrets:
- ssl_fullchain
- ssl_private_key
volumes:
- files:/files
networks:
dvbnet:
restart: always
....
secrets:
ssl_fullchain:
file: secrets/ssl-fullchain.pem
ssl_private_key:
file: secrets/domain-com-key.pem
volumes:
files:
networks:
dvbnet:
driver: bridge
ipam:
driver: default
Environment-Variable |
Sample-Setting |
Purpose |
Since DVB Version |
---|---|---|---|
|
false |
Turn ipv6 on or off |
<4.0.0.0 |
|
yourName |
Name of the user being
able to connect to the
Webdav Service. Optional.
|
<4.0.0.0 |
|
yourPassword |
Password of the user being
able to connect to the
Webdav Service. Optional.
|
<4.0.0.0 |
|
600 |
Nginx proxy_connect_timeout in seconds. Optional.
|
5.4.0.0 |
|
600 |
Nginx send_timeout in seconds. Optional.
|
5.4.0.0 |
|
600 |
Nginx proxy_read_timeout in seconds. Optional.
|
5.4.0.0 |
|
600 |
Nginx proxy_send_timeout in seconds. Optional.
|
5.4.0.0 |
|
Nginx proxy_pass for location /cicd/. Optional.
|
6.0.10.0 |
Secret |
Purpose |
Since DVB Version |
---|---|---|
ssl_fullchain |
SSL certificate as full chain (containing the root certificate, the intermediate certificate if applicable, and the actual certificate of the domain, all in one file, copy them together if needed) in pem-format. |
4.0.6.0 |
ssl_private_key |
SSL private key in pem-format. |
4.0.6.0 |
Setup Webdav Access¶
The Datavault Builder comes with a preconfigured WEBDAV access. This you can use to connect onto the folder /files
on the server, upload files and then create a source system for CSV sources to read data from that folder.
To configure the Webdav, simply give the WEBDAV user a name and a password in the webguis environment variables.
Enable SSL-Encryption¶
If you have an SSL certificate for the domain that points to the Linux hosts, it’s highly recommended to configure that as well, so passwords are not sent in clear text over your LAN.
First we also bind port 443
to the host IP as shown above. Port 80
will now do only a redirection to https, if you don’t want that, you can remove the line - 80:80
.
Then you need to put the certificates in your secrets directory and adjust the filenames here in the secrets section (the file: parts).
The private key should be in a file that is referenced here as secrets/domain-com.key (please adjust to whatever your actual file is called)
The certificate must be available as full chain (containing the root certificate, the intermediate certificate if applicable, and the actual certificate of the domain, all in one file, copy them together if needed). Put this file in the secrets folder as well and adjust
secrets/ssl-fullchain.crt
to your actual file name.
Metrics dashboard¶
(for connections and future extensions)
Accessible by adding /grafana
to your Datavault Builder URL
prometheus:
env_file: .env
image: '{DVB_REGISTRY}datavaultbuilder/prometheus:${DVB_TAG}'
volumes:
- prometheus:/var/lib/prometheus
networks:
- dvbnet
restart: always
grafana:
env_file: .env
environment:
- 'GF_SERVER_ROOT_URL=http://your-hostname-or-ip.com/grafana'
#- 'GF_SECURITY_ADMIN_PASSWORD=someSecretPassword!'
- GF_AUTH_ANONYMOUS_ENABLED=true
image: '{DVB_REGISTRY}datavaultbuilder/grafana:${DVB_TAG}'
volumes:
- grafana:/var/lib/grafana
networks:
- dvbnet
restart: always
....
volumes:
prometheus:
grafana:
networks:
dvbnet:
driver: bridge
ipam:
driver: default
Environment-Variable |
Sample-Setting |
Purpose |
Since DVB Version |
---|---|---|---|
GF_SERVER_ROOT_URL |
|
The URL you use to access datavault builder in the browser, followed by /grafana |
5.0.0.0 |
GF_AUTH_ANONYMOUS_ENABLED |
|
Alter this according to the Grafana documentation if you want to restrict access to the metrics dashboard |
5.0.0.0 |
Scheduler¶
scheduler:
env_file: .env
environment:
- MAX_SERVICE_CONNECTION_AGE=3600
- 'PGAGENT_OPTIONS=-l 2'
image: '{DVB_REGISTRY}datavaultbuilder/scheduler:{DVB_TAG}'
secrets:
- scheduler_password
networks:
dvbnet:
restart: always
....
secrets:
scheduler_password:
file: secrets/scheduler_password.txt
networks:
dvbnet:
driver: bridge
ipam:
driver: default
Environment-Variable |
Sample-Setting |
Purpose |
Since DVB Version |
---|---|---|---|
|
|
additional pgAgent options. e.g. |
<4.0.0.0 |
Secret |
Purpose |
Since DVB Version |
---|---|---|
|
Password of the scheduler user to connect onto the core and execute scheduled events. (Should be on the first line, prevent non alphanumeric charaters except ‘_’ - just choose a longer password instead) |
<4.0.0.0 |
API¶
api:
env_file: .env
environment:
- CONNECTION_POOL=30
image: '{DVB_REGISTRY}datavaultbuilder/api:{DVB_TAG}'
secrets:
- authenticator_password
- core_dbadmin_password
networks:
dvbnet:
restart: always
....
secrets:
authenticator_password:
file: secrets/authenticator_password.txt
core_dbadmin_password:
file: secrets/core_dbadmin_password.txt
networks:
dvbnet:
driver: bridge
ipam:
driver: default
Environment-Variable |
Sample-Setting |
Purpose |
Since DVB Version |
---|---|---|---|
|
|
Number of connections between api and core container. Default: 30 |
<4.0.0.0 |
Secret |
Purpose |
Since DVB Version |
---|---|---|
|
Password of the user “dbadmin” which can connect onto the core engine. (Should be on the first line, prevent non alphanumeric charaters except ‘_’ - just choose a longer password instead) |
<4.0.0.0 |
|
Password of the technical user making user/login independent calls onto the clientdb. (Should be on the first line, prevent non alphanumeric charaters except ‘_’ - just choose a longer password instead. The password must also fulfill all restrictions of the clientDB since it is used on multiple places.) |
<4.0.0.0 |
CICD¶
cicd:
image: '${DVB_REGISTRY}datavaultbuilder/cicd:${DVB_TAG}'
networks:
- dvbnet
restart: always
Client-Database: Postgres¶
clientdb_postgres:
env_file: .env
environment:
- EFFECTIVE_CACHE_SIZE=2GB
- MAINTENANCE_WORK_MEM=128MB
- MAX_PARALLEL_WORKERS=2
- SHARED_BUFFERS=2GB
- WORK_MEM=512MB
image: '{DVB_REGISTRY}datavaultbuilder/clientdb_postgres:{DVB_TAG}'
volumes:
- postgresdata:/data
ports:
- '5433:5432'
secrets:
- authenticator_password
- clientdb_dbadbmin_password
networks:
- dvbnet
restart: always
....
secrets:
authenticator_password:
file: secrets/authenticator_password.txt
clientdb_dbadbmin_password:
file: secrets/clientdb_dbadbmin_password.txt
volumes:
postgresdata:
networks:
dvbnet:
driver: bridge
ipam:
driver: default
Environment-Variable |
Sample-Setting |
Purpose |
Since DVB Version |
---|---|---|---|
|
|
<4.0.0.0 |
|
|
|
<4.0.0.0 |
|
|
|
<4.0.0.0 |
|
|
|
<4.0.0.0 |
Secret |
Purpose |
Since DVB Version |
---|---|---|
|
Password of the user “dbadmin” which can connect onto the database. (Should be on the first line, prevent non alphanumeric charaters except ‘_’ - just choose a longer password instead) |
<4.0.0.0 |
|
Password of the technical user making user/login independent calls onto the clientdb. |
<4.0.0.0 |
Client-Database: MS SQL¶
clientdb_mssql:
env_file: .env
environment:
- 'SA_PASSWORD=yourPassword'
- 'ACCEPT_EULA=Y'
- MSSQL_PID=Developer
- MSSQL_COLLATION=SQL_Latin1_General_CP1_CS_AS
- MSSQL_MEMORY_LIMIT_MB=4000
image: '{DVB_REGISTRY}datavaultbuilder/clientdb_mssql:{DVB_TAG}'
ports:
- '1433:1433'
secrets:
- authenticator_password
volumes:
- mssqldata:/var/opt/mssql
networks:
dvbnet:
restart: always
....
secrets:
authenticator_password:
file: secrets/authenticator_password.txt
volumes:
mssqldata:
networks:
dvbnet:
driver: bridge
ipam:
driver: default
Environment-Variable |
Sample-Setting |
Purpose |
Since DVB Version |
---|---|---|---|
|
yourPWD |
Password for sa user on the database |
<4.0.0.0 |
|
|
Accept MS SQL License agreement |
<4.0.0.0 |
|
Developer |
<4.0.0.0 |
|
|
SQL_Latin1_General_CP1_CS_AS |
Set the collation of the database to start |
<4.0.0.0 |
|
|
Limits the memory which is allocated to the database. Important to set a useful value here! See microsoft documentation |
<4.0.0.0 |
Secret |
Purpose |
Since DVB Version |
---|---|---|
|
Password of the technical user making user/login independent calls onto the clientdb. (Should be on the first line, prevent non alphanumeric charaters except ‘_’ - just choose a longer password instead. The password must also fulfill all restrictions of the MsSql Server too, since it is used on multiple places.) |
<4.0.0.0 |
Client-Database: Oracle¶
clientdb_oracle:
env_file: .env
environment:
- 'DB_PASSWD=yourPassword'
- MAX_STRING_SIZE=EXTENDED
- DB_MEMORY=3GB #GB!
- REDO_LOG_SIZE=1G
- UNDO_LOG_SIZE=1G
- UNDO_LOG_AUTOEXTEND_SIZE=512M
image: '{DVB_REGISTRY}datavaultbuilder/clientdb_oracle:{DVB_TAG}'
ports:
- '1521:1521'
secrets:
- authenticator_password
volumes:
- oracledata:/ORCL
networks:
- dvbnet
restart: always
....
secrets:
authenticator_password:
file: secrets/authenticator_password.txt
volumes:
oracledata:
networks:
dvbnet:
driver: bridge
ipam:
driver: default
Environment-Variable |
Sample-Setting |
Purpose |
Since DVB Version |
---|---|---|---|
|
extended |
Start the Oracle database with extended string size setting |
<4.0.0.0 |
|
yourPWD |
Password for sys user. If you would like to login: SYS AS SYSDBA. Must fulfill all complexity restrictions of the Oracle Server and may not contain special characters such as all sort of quotes or the ‘@’ character. |
<4.0.0.0 |
|
|
Allocated Memory to the database |
4.0.6.0 |
|
|
Size of the databases redo log for transactions |
4.0.6.0 |
|
|
Size of the databases undo log for transactions |
4.0.6.0 |
|
|
Stepsize for autoextending the databases undo log for transactions |
4.0.6.0 |
Secret |
Purpose |
Since DVB Version |
---|---|---|
|
Password of the technical user making user/login independent calls onto the clientdb. (Should be on the first line, prevent non alphanumeric charaters except ‘_’ - just choose a longer password instead. The password must also fulfill all restrictions of the Oracle Server too, since it is used on multiple places.) |
<4.0.0.0 |
Client-Database: Exasol¶
clientdb_exasol:
env_file: .env
environment:
- 'DO_NOT_UPDATE=comment out to auto update on start'
- 'UPDATE_EXASOL_DB=true'
image: '{DVB_REGISTRY}datavaultbuilder/clientdb_exasol:{DVB_TAG}'
ports:
- '8888:8888'
secrets:
- authenticator_password
volumes:
- exasoldata:/exa
networks:
dvbnet:
ipv4_address: 172.16.239.100 # a fix ip out of the subnet defined below - only needed for exasol container
aliases:
- clientdbexasol
....
secrets:
authenticator_password:
file: secrets/authenticator_password.txt
volumes:
exasoldata:
networks:
dvbnet:
driver: bridge
ipam:
driver: default
config:
- subnet: 172.16.239.0/24 # a fix ip subnet for the stack - only needed for exasol container, other external exasol databases!
Environment-Variable |
Sample-Setting |
Purpose |
Since DVB Version |
---|---|---|---|
|
comment out to auto update on start |
Will overwrite the dvb structures on the database on startup if commented out |
<4.0.0.0 |
|
false |
Will update the data on the database before startup |
<4.1.8.0 |
Secret |
Purpose |
Since DVB Version |
---|---|---|
|
Password of the technical user making user/login independent calls onto the clientdb. (Should be on the first line, prevent non alphanumeric charaters except ‘_’ - just choose a longer password instead. The password must also fulfill all restrictions of the Exasol Server too, since it is used on multiple places.) |
<4.0.0.0 |