Configuring advanced Data Mapping

This guide explains how to configure advanced Data Mapping for your datastreams.

Introduction

Data Mapping maps source fields to target fields in a datastream to apply standard naming conventions. This enables you to easily compare similar data from different data sources. Only mapped fields can be loaded into Adverity Data Storage or a destination.

Adverity automatically maps source fields for which Default Data Mapping is defined.

This guide explains how to configure advanced Data Mapping for your datastreams. For more information on configuring basic Data Mapping, see Applying Data Mapping to a datastream.

Changes to the Data Mapping that change the table structure of the data extract may not match the structure of your data in Adverity Data Storage and any destinations.

Changes made to the Data Mapping are not applied to data that has already been loaded into Adverity Data Storage or destinations. To update the data in Adverity Data Storage and any destinations with the new Data Mapping, re-load the data extracts. For more information, see Re-loading a data extract.

Prerequisites

Before you complete the procedure in this guide, perform all of the following actions:

Create a datastream.

Fetch data for the datastream.

Mapping source fields

To harmonize your data, map the source fields in your data extract to target fields available in Adverity. Using the same target fields in multiple datastreams makes it easier to harmonize your data.

Mapping a source field to an existing target field

To map a source field to an existing target field, follow these steps:

Select the workspace you work with in Adverity and then, in the platform navigation menu, click Datastreams.
Open the chosen datastream by clicking on its name.
In the top navigation panel, click Data Mapping.
In the row for the source field you want to map, in the Target fields column, type in the target field that you want to use and select it from the drop-down menu.

The Data Mapping changes are saved automatically.

Mapping a source field to a new target field

If the target field that you want to use does not appear in the drop-down menu, you can create a new target field with a name of your choice. This creates a new field in the data loaded into Adverity Data Storage or external destination. To create a new target field and map a source field in your datastream to your new field, follow these steps:

Select the workspace you work with in Adverity and then, in the platform navigation menu, click Datastreams.
Open the chosen datastream by clicking on its name.
In the top navigation panel, click Data Mapping.
In the row for the source field you want to map, in the Target field column, type in the name of the target field that you want to create and click + Create new.
In the Create new target field pop-up that opens, in the Name field, enter the name of the new target field. This name can only contain lower-case letters, numbers, and underscores.
(Optional) In the Display name field, enter a display name for the new target field. This name will be used for the target field in the Data Explorer and Dashboards pages.
In the Data type field, select the data type. For more information on data types, see Data types used in data harmonization.
(Optional) In the Length field, specify the maximum character length of the values in the new field. This option is only available if you select String in the Data type field.
(Optional) In the Currency field, select the currency in which the values in the new field are displayed. This options is only available if you select Currency in the Data type field.
In the Measure field, specify the mathematical function underlying the values in the new field. This option is only available if you select a numerical data type in the Data type field, and it only has an effect if you load data into Adverity Data Storage. For more information, see Measures used in data harmonization.
(Optional) In the Description field, enter a description for the new target field.
Click Create.

As a result, you have mapped your chosen source field to your newly created target field. This target field will now appear on the Data Dictionary page.

Mapping source fields to target fields in the data extract preview

You can also map a source field to a new or existing target field in a specific data extract in the data extract preview. To do this, follow these steps:

Preview the data extract containing the field you want to map.
In the table header, under the name of the source field that you want to map, perform one of the following actions:
- To map an unmapped source field to an existing target field, click on Not mapped and type in the name of the target field that you want to map.
- To change the target field to which a source field is already mapped, click on the existing target field and type in the name of the target field that you want to map.
- To map a source field to a new target field, click on Not mapped or the existing target field, then type in the name of the target field you want to create and click + Create new.
  
  In the Create new target field window, follow steps 5-12 above.

Automatically mapping all source fields

To map all source fields in the data extract automatically, follow these steps:

Select the workspace you work with in Adverity and then, in the platform navigation menu, click Datastreams.
Open the chosen datastream by clicking on its name.
In the top navigation panel, click Data Mapping.
Below the table, click Map automatically.

The Data Mapping changes are saved automatically.

Adverity decides which target field to assign to each unmapped source field using the following information, in the following order:

Assign the target field defined in the Default Data Mapping for the source field.
If no Default Data Mapping is defined, assign the target field to which the source field is mapped most often in other datastreams.
If the source field is not mapped in another datastream, assign an existing target field with the same name as the source field.
If a target field fulfilling the above conditions does not exist, create and assign a new target field.

The new target field's name is the same as the source field's name, in lower case and with underscores instead of spaces. For example, the source field Ad Name will be mapped to a new target field ad_name.

Setting key columns

Key columns uniquely identify a data set. Use key columns to ensure that data is correctly overwritten when you load data into the destination that you assigned to the datastream. For more information, see Configuring data loading settings.

You can only set dimension fields as key columns.

To set a field as a key column, follow these steps:

Select the workspace you work with in Adverity and then, in the platform navigation menu, click Datastreams.
Open the chosen datastream by clicking on its name.
In the top navigation panel, click Data Mapping.
Find the field in the list.
In the Key Column column, enable the toggle.

The Data Mapping changes are saved automatically.

After changing the key columns, you need to perform additional actions to load data into some destinations. For an example, see Advanced SQL Database tips.

To overwrite data in a destination based on key columns, select Key Columns in the destination configuration. For more information, see Configuring data loading settings.

Some destinations do not support overwriting data with key columns. For an example, see Advanced Google BigQuery tips.

Changing a field's internal data type

When you fetch a data extract, Adverity automatically detects and configures the internal data type for each field. The internal data type of a field is separate from the data type you set in Data Mapping.

To change a field's internal data type in a data extract, follow these steps:

Select the workspace you work with in Adverity and then, in the platform navigation menu, click Datastreams.
Open the chosen datastream by clicking on its name.
In the top navigation panel, click Data Extracts.
In the list, click the top hyperlinked data extract in the Name column.
Hold the pointer over one of the column headings, and then click the icon.
In the Columns section on the left hand side of the page, find the field in the list.
In the field's row, click the drop-down menu on the right.
Select the new data type for the field.

As a result, you have changed the internal data type of the selected field.

Re-loading a data extract

Changes made to the Data Mapping are not applied to data that has already been loaded into Adverity Data Storage or an external destination. To apply the changes to the data in Adverity Data Storage and any destinations, you must re-load the data extract.

To do this, follow these steps:

Select the workspace you work with in Adverity and then, in the platform navigation menu, click Datastreams.
Open the chosen datastream by clicking on its name.
In the top navigation panel, click Data Extracts.
In the table, select the checkboxes for the data extracts you want to update with the Data Mapping changes.
In the Select an action drop-down menu above the data extracts, select one of the following options:
- Re-load into Adverity Data Storage and destinations
  
  Select this option to re-load the selected data extracts into Adverity Data Storage and any assigned external destinations.
- Re-load into Adverity Data Storage
  
  Select this option to re-load the selected data extracts into Adverity Data Storage.
- Re-load into [destination]
  
  Select this option to re-load the selected data extracts into the named external destination.

As a result, this data extract will be re-loaded. Any changes made to the Data Mapping of this datastream will be applied to the data when it is reloaded into Adverity Data Storage or an external destination.