Airbyte Based Sources

Airbyte is an open-source ETL framework. Jitsu supports Airbyte as of the connector backend (the other one being Singer and native connectors

This page describes how to configure Luden Server if it runs in standalone mode (without Configurator). If you deployed Jitsu along with Configurator, you can configure Airbyte connectors directly in the UI.

Understanding Airbyte Connectors

Each Airbyte connector is a standalone docker image. The connector implements Airbyte protocol. The protocol is CLI based: connector executor (Luden) feeds connector with a formatted JSON objects (see below) and parses the stdout which is a stream of JSON objects as well.

Configuration

The connectors' configuration should be placed under the sources section of configuration file. The structure is an extension of generic source configuration - type: airbyte indicates that the source should be handled by Airbyte executor

sources:
  <connector_id>:
    type: airbyte
    schedule: '...'                  # Cron expression or period notation (@daily or @hourly)
    destinations: [ ]                # List of destinations
    config:
      config: {}                     # Airbyte config object (see below)
      docker_image: 'airbyte/image'  # Airbyte connector image
      catalog: {}                    # Airbyte catalog object (see below).
                                     # Optional. If not provided, all streams
                                     # will be synchronized

Example:

sources:
  ...
  luden_airbyte_hubspot:
    type: airbyte
    destinations: [ "clickhouse_destination_id" ]
    schedule: '*/5 * * * *'
    config:
      config:
        credentials:
          api_key: '<HUBSPOT_API_KEY>'
        start_date: "2017-01-25T00:00:00Z"
      docker_image: source-hubspot
  airbyte_source_shopify:
    destinations: [ "clickhouse_destination_id" ]
    type: airbyte
    schedule: '@daily'
    config:
      config:
        api_password: '<SHOPIFY_API_PASSWORD>'
        shop: 'https://EXAMPLE.myshopify.com'
        start_date: "2020-10-10"
      docker_image: source-shopify

JSON configuration parameters such as config, catalog, state can be an object or a raw JSON or JSON string or path to local JSON file