Index Model

🧑🏼‍🚀 LeonaLog Index Model

Leona is transparently managing one or more sets of Elasticsearch indices to optimize search and analysis operations for speed and low resource consumption.

To enable managing indices with different mappings, analyzers, and replication settings. Leona is using so-called index sets which are an abstraction of all these settings.

Leona is maintaining an index alias per index set which is always pointing to the current write-active index from that index set. There is always exactly one index to which new messages are written until the configured rotation criterion (number of documents, index size, or index age) has been met.

A background task continuously checks if the rotation criterion of an index set has been met and a new index is created and prepared when that happens. Once the index is ready, the index alias is atomically switched to it. That means that all leona nodes can write messages into the alias without even knowing what the current write-active index of the index set is.

Index Set Configuration

Index sets have a variety of different settings related to how Leona will store messages into the Elasticsearch cluster.

  • Title: A descriptive name of the index set.

  • Description: A description of the index set for human consumption.

  • Index prefix: A unique prefix used for Elasticsearch indices managed by the index set. The prefix must start with a letter or number, and can only contain letters, numbers, _, -, and +.

  • Analyzer: The Elasticsearch analyzer for the index set.

  • Index shards: The default is 4.

  • Index replicas: The default is 0.

  • Max number of segments: The maximum of segments per Elasticsearch index after index optimization, see Segment Merging for details.

Keep the Index Ranges in Sync

Leona will take care of calculating index ranges automatically as soon as a new index has been created.

You can easily re-build the information yourself after manually deleting indices or doing other changes that might cause synchronization problems:

curl -XPOST http://127.0.0.1:9000/api/system/indices/ranges/rebuid

This will trigger a system job:

... Recalculating index ranges.
... Submitted SystemJob <9b64a9d0-dcac-11e6-97c3-6c4008b8fc28> [org.graylog2.indexer.ranges.RebuildIndexRange
... Recalculating index ranges for index set Default index set (graylog2_*): 5 indices affected

Last updated