Transferring data to Firebolt

This guide explains how to transfer data to Firebolt to store and further process information.

Concept

Firebolt is an Active Destination. After you set Firebolt as the Destination of a Datastream, data is transferred to Firebolt each time data is fetched for the Datastream. For more information, see Destination types.

You can assign multiple Destinations to a Datastream. For more information on possible limitations, see Assigning multiple Destinations to a Datastream.

Prerequisites

Before you complete the procedure in this guide, perform all of the following actions:

  • Find the address of the Firebolt database where you want to transfer data.

  • Set up an Amazon S3 bucket as Storage for Firebolt. For more information, see Setting up Storage for Data Extracts.

  • Find the region code of your S3 bucket. The region code determines which database to enter when setting up an Authorization to Firebolt. For more information on finding your S3 bucket region code, see the AWS documentation.

Procedure

To transfer data from a Datastream to Firebolt, follow these steps:

  1. Add Firebolt as a Destination to the Workspace which contains the Datastream or to one of its parent Workspaces.

  2. Assign the Firebolt Destination to the Datastream.

  3. Configure transfer settings.

Adding Firebolt as Destination

To add Firebolt as a Destination to a Workspace, follow these steps:

  1. Click the Transfer element and select the Workspace you work with in Connect, Enrich & Transfer.

  1. Click + Add.

  2. Click Firebolt.

  1. In the Authorization page, fill in the following fields:

    Database

    Specify the name of the Firebolt database where you want to transfer the data. Make sure the database shares the same region code as the S3 bucket that stores the data.

    Username

    The username of the Firebolt account.

    Password

    The password of the Firebolt account.

  2. Click Authorize.

  3. In the Configuration page, fill in the following fields:

    Name

    (Optional) Rename the Destination.

    S3 Storage

    Select the S3 Storage that stores the data. Leave this field empty to use the default Workspace storage settings.

  1. Click Create.

Assigning Firebolt as Destination

To assign the Firebolt Destination to a Datastream, follow these steps:

  1. Click the Connect element and select the Workspace you work with in Connect, Enrich & Transfer.

  1. Select the Connector Name Datastream.

  1. In the Destinations section, click + Add Destination.

  2. Click Assign Existing Destinations.

  1. Select the Firebolt checkbox in the list.

  2. Click Save.

Configuring transfer settings

To configure transfer settings, follow these steps:

  1. Click the Connect element and select the Workspace you work with in Connect, Enrich & Transfer.

  1. Select the Connector Name Datastream.

  1. In the Destinations section, find the Firebolt Destination in the list, and click on the right.

  2. Click Destination Settings.

  1. Fill in the following fields:

    Table name

    Specify the target in the Destination where to transfer data from the Datastream. The name can contain alphanumeric characters and underscores. For example, target_table.

    By default, Adverity saves data from each Datastream in a different table named {datastream_type}_{datastream_id} (for example, mailgun_83).

    You can specify the same target for several Datastreams. If a column is shared between Datastreams, Adverity performs a full outer join and concatenates values. If a column is not shared between Datastreams, Adverity writes null values in the relevant cells.

    Use placeholders to create unique, dynamic table names in the Destination. Use the following placeholders:

    Placeholder

    Description

    {app_label}

    The Datastream Type's short name.

    {datastream_id}

    The Datastream ID.

    {datastream_type}

    The Datastream Type.

    {extract_id}

    The Data Extract ID.

    {id}

    The Datastream ID.

    {meta[*]}

    Replace * with a metadata placeholder to use metadata in the name. For example, {meta[datastream_URI]} uses the Datastream URI as the name. For more information on metadata and placeholders, see Using placeholders.

    {name}

    The automatically generated filename of the Data Extract.

    {scheduled_day}

    The day from the start date of a date range for a scheduled data fetch.

    {scheduled_month}

    The month from the start date of a date range for a scheduled data fetch.

    {scheduled_year}

    The year from the start date of a date range for a scheduled data fetch.

    {upload_day}

    The day when the Data Extract is transferred to the Firebolt Destination.

    {upload_hour}

    The hour when the Data Extract is transferred to the Firebolt Destination.

    {upload_minute}

    The minute when the Data Extract is transferred to the Firebolt Destination.

    {upload_month}

    The month when the Data Extract is transferred to the Firebolt Destination.

    {upload_second}

    The second when the Data Extract is transferred to the Firebolt Destination.

    {upload_year}

    The year when the Data Extract is transferred to the Firebolt Destination.

    Mode

    Select what Adverity does when a table already exists in the Firebolt Destination. Select one of the following options:

    • Select Delete to delete all data in the exiting table in the Destination before populating the table with the most recent Data Extract.

    • Select Protected to make sure the transferred data is not deleted in future data transfers.

    Primary Index

    Enter the primary fields that tells Firebolt how to sort and index the data. By default, the primary field of dt_created is entered. Enter multiple primary indexes separated by commas. For more information on primary indexes, see the Firebolt documentation.

  2. Click Save.