Google BigQuery

Luden supports BigQuery as a destination. For more information about BigQuery see docs. BigQuery destination can work in stream and batch modes. In stream mode Luden uses BigQuery Streaming API. In batch mode Luden writes incoming events in formatted file on the Google Cloud Storage and creates a loading job to store data from GCP files into BigQuery.

Configuration

BigQuery destination config consists of the following schema:

destinations:
  my_bigquery:
    type: bigquery
    google:
      gcs_bucket: google_cloud_storage_bucket
      bq_project: big_query_project
      bq_dataset: big_query_dataset
      key_file: path_to_bqkey.json # or json string of key e.g. "{"service_account":...}"