Configuration

If Luden Server is deployed in a standalone mode, it could be configured via single YAML file (read more about deployment options here). The most popular options can be defined as environment variables too. We're aiming to make Luden 100% configurable with env variables, but we're not there yet. Besides, you can host Jitsu config file on external storage - S3, MinIO, HTTP, etc.

We follow convention over configuration so the majority of parameters are optional. To get a base file, clone it from GitHub.

Redis

Luden Server requires Redis. Luden can run without it, however it will work in single-node mode without synchronization, queue persistence, and other high-availability features. It's good enough for development, but not recommended for production.

Redis version 5 and higher are supported. Here's a full list of features that will not work if Redis is not available

• Last events caching (see Events Cache)

• Sources synchronization (see Sources Configuration)

• Retroactive Users Recognition

• Coordination.

Redis credentials should be put into to meta.storage.redis section (see below)

Configuration file structure

Luden config file consists of the following sections:

• server — General configuration parameters such as port, application logs configuration, singer bridge configuration, etc.

• geo — Geo-resolution data (extracting city/state information from the IP address). We currently only support MaxMind as a data provider. see Geo Data resolution

• log — Jitsu writes all events locally and sends them to their destinations (in batch mode). This is where you configure your local temporary path and push frequency.

• sql_debug_log — All SQL statements such as DDL and DML expressions can be stored in separate log files or in stdout.

• api_keys — A set of API Keys objects that identify incoming events JSONs and mapping between destinations is done based on them. see Authorization page

• destinations — A set of targets where the final version of events will be stored. see Destinations Configuration

• sources — A set of data sources to synchronize from. see Sources Configuration

• users_recognition — Luden can update past events with user identifiers on user's identification event!

• coordination — coordination service configuration. It is used in cluster Jitsu deployments. see Scaling Luden

• notifications — notifier configuration. Server starts, system errors, synchronization statuses, and panic information will be sent to it. Currently, only Slack notifications are supported.

• meta.storage - meta storage is the main application storage and it is required for some features. At present, Luden supports only Redis version 5 and higher.

• ui.base_url – base Configurator UI URL for generating links in notifications

• node – node.js process pool size and max heap space in megabytes per process (node is used to execute JavaScript transformations and plugins).

Example:

server:
  name: instance1.domain.com
  port: 8081
  public_url: https://instances.domain.com
  log:
    path: /home/eventnative/data/logs/
  metrics.prometheus.enabled: true

geo.maxmind_path: /home/eventnative/data/config

log:
  path: /home/eventnative/data/logs/events
  rotation_min: 5

sql_debug_log:
  ddl:
  queries:

api_keys:
  - id: my_website_key
    client_secret: '193b6281-f211-47a9-b384-102cf4cd2d55'
    server_secret: 'a6880b16-73d0-476e-840a-804b74f7684d'

destinations:
  redshift:
  bigquery:

sources:
  facebook:
  google_analytics:

users_recognition:
  enabled: true

coordination:
  type: redis

notifications:
  slack:
    url: https://slack_web_hook_url

ui:
  base_url: https://your_configurator_ui_host:5555

meta:
  storage:
    redis:
      host: redis_host
      port: 6379
      password: secret_password
      database: 0

node:
  pool_size: 1 # default
  max_space: 100 # default

You can set ${env.OS_ENV_VAR_NAME} to any configuration parameter in YAML file. Jitsu will get the value from OS ENV (with name OS_ENV_VAR_NAME from the example).

Server

All fields from the server section are optional:

Field	Type	Description	Default value
name	string	Unique instance name. It is used in cluster deployments.	unnamed-server
port	int	TCP port for the server to listen on.	8001
public_url	string	Service public URL. It is used on the welcome HTML page.	Will be got from Host request header
log.path	string	Path to application logs. If not set, app logs will be in stdout.	-
log.rotation_min	int	Log files rotation minutes. If log.path is configured.	-
api_keys_reload_sec	int	If an URL is set in api_keys section, authorization will be reloaded every api_keys_reload_sec seconds. see Authorization page.	1
destinations_reload_sec	int	If an URL is set in destinations section, destinations will be reloaded every destinations_reload_sec seconds. see Destinations.	1
sources_reload_sec	int	If an URL is set in sources section, sources will be reloaded every sources_reload_sec seconds.	1
admin_token	string	see Admin Endpoints page.	-
metrics.prometheus.enabled	boolean	see Application Metrics page.	false
telemetry.disabled.usage	boolean	Flag for disabling telemetry. Luden collects usage metrics about how you use it and how it is working. We don't collect any customer data.	false
metrics.relay.disabled	boolean	Disables extended telemetry metrics collection.	false
metrics.relay.deployment_id	string	Allows to provide deployment ID for extended telemetry collection.	Cluster ID
disable_version_reminder	boolean	Flag for disabling log reminder banner about new Luden versions availability.	false
sync_tasks.store_logs.last_runs	int	Logs for how many task runs must be kept in meta storage. Controlled on Source's collection level. When number of task runs for Source collection exceed provided value – old records get removed from meta storage.	-1 unlimited number of logs
event_enrichment.http_context	boolean	Whether the server should enrich incoming HTTP events with HTTP context (headers, etc.). Please note that when upgrading from Luden 1.41.6 you can switch this setting to true only separately from the upgrade itself, otherwise event data may get corrupted.	false

Log

Luden Server supports destinations in streaming and batch modes. In the case of batch mode, all events are stored in JSON log files locally to path directory, and every rotation_min minutes they are processed and pushed to destinations. All fields from log section are optional:

Field	Type	Description	Default value
path	string	Events log files path.	/home/eventnative/data/logs/events
rotation_min	int	Log files rotation minutes.	5
show_in_server	boolean	Flag for debugging. If true - all events JSON data is written in app logs.	false