Google BigQuery Destination reference

This reference explains in detail how to configure data transfer to Google BigQuery.

Prerequisites

Before you complete the procedure in this guide, perform all of the following actions:

Configuring data transfer to Google BigQuery

To configure data transfer to Google BigQuery, follow these steps:

  1. Click the Transfer element and select the Workspace you work with in Connect, Enrich & Transfer.

  1. Click the Google BigQuery Destination in the list.

  2. In the left navigation panel, click Schema Options.

  3. Fill in the following fields:

    Schema Mapping

    By default, Adverity applies the Schema Mapping conventions to the Data Extracts when transferring them to the Destination. For more information on Schema Mapping, see Harmonizing data.

    To transfer data to the Destination without applying the Schema Mapping conventions, clear this checkbox.

    Force String Type

    By default, Adverity automatically recognizes the data type of each field. Select this checkbox to convert all fields to the String data type.

  4. Click Save.

Google BigQuery does not support overwriting data in the Destination with key columns.

Troubleshooting: Data transfer to Google BigQuery fails after changing the table structure of the Data Extract

A data transfer to Google BigQuery may fail because of a change in the table structure of the Data Extract. To ensure the frictionless operation of production systems, Adverity only modifies existing tables in Google BigQuery if the changes are considered neutral, such as adding a new field to the table. Changes to the table structure of the Data Extract are not considered neutral. As a consequence, Adverity does not update the existing table in Google BigQuery resulting in failed data transfer.

The following actions change the table structure of the Data Extract:

  • You change the key columns in the Datastream's Schema Mapping. For more information on key columns, see Setting key columns.

  • You change the data type of a column in the Datastream's Schema Mapping. For more information on the data types used in Adverity, see Data types used in data harmonization.

  • You change the maximum character length of values in a column which contains String values.

To resolve this problem, follow these steps:

  1. In Google BigQuery, modify or drop the table that corresponds to the Datastream.

  2. In Adverity, transfer data from the Datastream to Google BigQuery.