Setting up destinations#
This guide provides a high-level overview of loading data into destinations in Adverity. For detailed steps on configuring a specific destination, see our individual destination guides.
Introduction#
A destination is a tool where you can send the data you have collected and transformed in Adverity for storage, analysis, or visualization. While this often refers to external tools like databases or business intelligence platforms, Adverity also provides internal destinations, such as Adverity Data Storage and Explore & Present.
Destinations allows you to use your harmonized and transformed data outside of Adverity. Multiple destinations can be added to a single datastream to ensure the data is availbale in all of the tools you need. When a destination is enabled for a datastream the data is automatically loaded into it with each fetch.
The general workflow involves creating and authorizing a destination, configuring its specific settings, assigning it to one or more datastreams, and enabling data loading.
Prerequisites#
Before you can load data into a destination, perform all of the following actions:
Create a datastream to collect data from a data source. For more information, see Creating a datastream.
Fetch data at least once using the datastream. For more information, see Manual, Smart, and Custom fetches.
Apply Data Mapping to the collected data to ensure its schema is harmonized. For more information, see Applying Data Mapping to a datastream.
Note
Some destinations require specific column names. Make sure to map your data to target fields with the correct names.
Configuring destination-level settings#
After creating the destination, you must configure its settings. This step determines how and where your data will be stored in the target system. The available options vary significantly depending on the destination type.
- Common settings for all destinations
Target Table/File Name: Specify a naming convention for the tables or files that Adverity creates in the destination. You can often use placeholders like
{datastream_name}or{date}to create dynamic names. For more information, see Using placeholders to name objects.
- Settings for file-based destinations
For destinations like Amazon S3, SFTP, and Azure Blob Storage, you can typically configure the following:
File Format: Choose the output format, such as CSV, JSON, or Apache Parquet.
Compression: Select a compression type like GZIP or leave it as None.
File Path: Define the specific folder or directory where files will be stored.
CSV file options: For CSV format, you can specify the Delimiter, Quote Character, and whether to Include Header.
- Settings for databases and data warehouses
For destinations like Snowflake, Google BigQuery, and PostgreSQL, you typically need to specify the storage location:
Database/Project ID: The name of the target database or project.
Schema: The schema where the tables will be created.
- Settings for BI and visualization tools
For destinations like Tableau, you need to define where the data source will be published:
Site: The name of the Tableau site.
Project: The destination project folder within Tableau.
Assigning a destination and configuring data loading#
Once the destination is configured, you must assign it to the datastream from which you want to load data. After assigning the destination, you can configure how data is loaded into it, and then enable the destination for the datastream.
To assign a destination, configure its loading settings, and enable it, follow these steps:
Assign the destination to a datastream.
You can assign a destination to both new and existing datastreams:
To assign a destination to a new datastream, select it in the final step of the datastream creation wizard.
To assign a destination to an existing datastream, follow these steps:
Navigate to the overview page of the datastream.
In the Load section, select your configured destination.
Configure the datastream loading settings.
Define how Adverity updates data in the destination. These overwrite options are crucial for managing data integrity and are configured for each datastream loading to the destination.
To configure these settings, navigate to the datastream’s Load section, find the destination, and click Actions > Destination settings.
Common overwrite options include:
Append: Adds new data as new rows to the existing table or file. In many database destinations, this is the default behavior if no other overwrite option is selected.
Truncate: Deletes all data from the target table before loading the new data extract. This is equivalent to a full overwrite.
Overwrite by Key Columns: Updates existing rows and inserts new ones based on unique key columns defined in Data Mapping. This provides a function similar to an “upsert” by replacing only the relevant data.
Conditional Overwrite: Provides more granular control by overwriting data based on specific criteria, such as a matching Date Range or Filename.
Enable data loading for the datastream.
In the datastream’s Load section, enable the toggle next to the destination’s name. A pop-up window will ask if you want to load previously collected data. Click Yes, load data to transfer all historical data, or click Skip to load only data from future fetches.
After loading is completed, your data will be available in your chosen destination.
As a result, data will be loaded into all assigned and enabled destinations from the datastream with each fetch.