Sheenflow Architecture

Overview

Let's take a look at a generic Sheenflow deployment, after which we'll talk through each of its components:

Long-Running Services

Sheenflow requires there long-running services, which are outlined in the table below:

Service	Description	Replicas
Sheelet	Sheenlet servers the user interface and response to GraphQL queries.	Supported
Sheenflow Daemon	The Sheeflow daemon operates schedules, sensors, and run queuing.	Not Supported
Code Location Server	Code location servers server metadata about the collection of its Sheenflow definitions.	Supported

Deployment Configuration

Sheenflow OSS deployments are composed of multiple components, such as storages, executors, and run launchers. One of the core features of Sheenflow is that each of these components is swappable and configurable. If the custom configuration is not provided, Sheenflow will automatically use a default implementation of each component. For example, by default Sheenflow uses SqliteRunStorage to store information about pipeline runs. This can be swapped with the Sheenflow-provided PostgresRunStorage instead of a custom storage class.

Job Execution Flow

Job execution flows through several parts of the Sheenflow system. The following table describes runs launched by Sheenflow, specifically the components that handle execution and the order in which they are executed.

1. Run Coordinator: The run coordinator is a class invoked by the Sheenflow process when runs are launched from the Sheenlet UI. This class can be configurated to pass runs to the daemon via a queue.

2. Run Launcher: The run launcher is a class invoked by the daemon when it receives a run from the queue. This class initializes a new run worker to handle execution. Depending on the launcher, this could mean spinning up a new process, container, Kubernetes pod, etc.

3. Run Worker: The run worker is a process which that traverses a graph and uses the executor to executor each op.

4. Executor: The executor is a class invoked by the run worker for running user ops. Depending on the executor, ops run in local processes, new containers, Kubernetes pods, etc.

Additionally, note that run launched by schedules and sensors go through the same flow, but the first step is called by the Sheenflow Daemon instead of Sheenflow.