Configuring advanced Data Mapping

This guide explains how to configure advanced Data Mapping for your datastreams.

Introduction

Apply Data Mapping to a datastream to map source fields to target fields that conform to your unified naming and formatting conventions in Adverity.

This guide explains how to configure advanced Data Mapping for your datastreams. For more information on configuring basic Data Mapping, see Applying Data Mapping to a datastream.

Changes to the Data Mapping that change the table structure of the data extract may not match the structure of your data in Adverity Data Storage and any destinations.

Changes made to the Data Mapping are not applied to data that has already been stored or transferred. To update the data in Adverity Data Storage and any destinations with the new Data Mapping, requeue the data extracts. For more information, see Requeueing a data extract.

Prerequisites

Before you complete the procedure in this guide, perform all of the following actions:

Create a datastream.

Fetch data for the datastream.

Mapping a source field to an existing target field

To harmonize your data, map the source fields in your data extract to target fields available in Adverity. Using the same target fields in multiple datastreams makes it easier to harmonize your data. To map a source field to a target field, follow these steps:

Select the workspace you work with in Adverity and then, in the platform navigation menu, click Datastreams.
Open the chosen datastream by clicking on its name.
In the top navigation panel, click Data Mapping.
In the row for the source field you want to map, in the Target fields column, type in the target field that you want to use and select it from the drop-down menu.

The Data Mapping changes are saved automatically.

Mapping a source field to a new target field

If the target field that you want to use does not appear in the drop-down menu, you can create a new target field with a name of your choice. This creates a new field in the data transferred to the destination. To create a new target field and map a source field in your datastream to your new field, follow these steps:

Select the workspace you work with in Adverity and then, in the platform navigation menu, click Datastreams.
Open the chosen datastream by clicking on its name.
In the top navigation panel, click Data Mapping.
In the row for the source field you want to map, in the Target field column, type in the name of the target field that you want to create and click + Create new.
In the Create new target field pop-up that opens, in the Name field, enter the name of the new target field. This name can only contain lower-case letters, numbers, and underscores.
(Optional) In the Display name field, enter a display name for the new target field. This name will be used for the target field in the Data Explorer and Dashboards pages.
In the Data type field, select the data type. For more information on data types, see Data types used in data harmonization.
(Optional) In the Length field, specify the maximum character length of the values in the new field. This option is only available if you select String in the Data type field.
(Optional) In the Currency field, select the currency in which the values in the new field are displayed. This options is only available if you select Currency in the Data type field.
In the Measure field, specify the mathematical function underlying the values in the new field. This option is only available if you select a numerical data type in the Data type field, and it only has an effect if you store data in Adverity Data Storage. For more information, see Measures used in data harmonization.
(Optional) In the Description field, enter a description for the new target field.
Click Create.

As a result, you have mapped your chosen source field to your newly created target field. This target field will now appear on the Data Dictionary page.

Mapping source fields to target fields in the data extract preview

You can also map a source field to a new or existing target field in a specific data extract in the data extract preview. To do this, follow these steps:

Preview the data extract containing the field you want to map.
In the table header, under the name of the source field that you want to map, perform one of the following actions:
- To map an unmapped source field to an existing target field, click on Not mapped and type in the name of the target field that you want to map.
- To change the target field to which a source field is already mapped, click on the existing target field and type in the name of the target field that you want to map.
- To map a source field to a new target field, click on Not mapped or the existing target field, then type in the name of the target field you want to create and click + Create new.
  
  In the Create new target field window, follow steps 5-12 above.

Setting key columns

Key columns uniquely identify a data set. Use key columns to ensure that data is correctly overwritten when you transfer data to the destination that you assigned to the datastream. For more information, see Configuring transfer settings.

You can only set dimension fields as key columns.

To set a field as a key column, follow these steps:

Select the workspace you work with in Adverity and then, in the platform navigation menu, click Datastreams.
Open the chosen datastream by clicking on its name.
In the top navigation panel, click Data Mapping.
Find the field in the list.
In the Key Column column, enable the toggle.

The Data Mapping changes are saved automatically.

After changing the key columns, you need to perform additional actions to transfer data to some destinations. For an example, see Advanced SQL Database tips.

To overwrite data in a destination based on key columns, select Key Columns in the destination configuration. For more information, see Configuring transfer settings.

Some destinations do not support overwriting data with key columns. For an example, see Advanced Google BigQuery tips.

Changing a field's internal data type

When you fetch a data extract, Adverity automatically detects and configures the internal data type for each field. The internal data type of a field is separate from the data type you set in Data Mapping.

To change a field's internal data type in a data extract, follow these steps:

Select the workspace you work with in Adverity and then, in the platform navigation menu, click Datastreams.
Open the chosen datastream by clicking on its name.
In the top navigation panel, click Data Extracts.
In the list, click the top hyperlinked data extract in the Name column.
Hold the pointer over one of the column headings, and then click the icon.
In the Columns section on the left hand side of the page, find the field in the list.
In the field's row, click the drop-down menu on the right.
Select the new data type for the field.

As a result, you have changed the internal data type of the selected field.

Requeueing a data extract

Changes made to the Data Mapping are not applied to data that has already been stored in Adverity Data Storage or transferred to an external destination. To apply the changes to the data in Adverity Data Storage and any destinations, you must requeue the data extract.

To do this, follow these steps:

Select the workspace you work with in Adverity and then, in the platform navigation menu, click Datastreams.
Open the chosen datastream by clicking on its name.
In the top navigation panel, click Data Extracts.
In the table, select the checkboxes for the data extracts you want to update with the Data Mapping changes.
In the Select an action drop-down menu above the data extracts, select one of the following options:
- Requeue extracts (for all destinations)
  
  Select this option to requeue the selected data extracts, store the data in Adverity Data Storage, and transfer the data to any assigned external destinations.
- Requeue extracts (for Adverity Data Storage)
  
  Select this option to requeue the selected data extracts and store the data in Adverity Data Storage.
- Requeue extracts (for [assigned destination])
  
  Select this option to requeue the selected data extracts and transfer the data to the named external destination.

As a result, this data extract will be imported again. Any changes made to the Data Mapping of this datastream will be applied to the data when it is stored in Adverity Data Storage or transferred to an external destination.