Luden Architecture

Luden is a standalone application that doesn't have any requirements such as an inner database or others. Simultaneously some features need Redis, but these features are optional.

Definitions

• Event: An incoming message as a JSON object. Message can be sent to Luden via API or JavaScript library.

• Destination: Usually a database, but can be a different type of service as well. The destination is a service that contains structured records and supports an INSERT-like statement.

• Table: A table in DB or any other structure within the destination.

• BatchHeader: Describes a structure of batch: fields and types along with table name.

Destination Modes

Luden's destination can be operated in two modes:

• Stream: Data (events) is being sent to a destination as soon as possible.

• Batch: Data is being written to local disk and sent to a destination in batches, once in N (configurable) minutes.

Transformation and Mapping Step

In this step, all configured mappings are applied to a JSON object and as a result, BatchHeader is generated. Read about JavaScript Transformation and Schema and Mapping configuration. After all mapping are applied, JSON is flattened, and all special characters and spaces in field names are replaced with underscores. Also values types recognition happens here, read about Typecast.