Collecting data from Google Drive

This guide explains how to collect data from Google Drive. To learn how to collect data from a different data source, go back to the Available data sources in Adverity overview.

Creating a datastream to collect data from Google Drive

The basics of creating a datastream to collect data from any data source are explained in our guide to Creating a datastream. This guide contains information about the specific steps to create a datastream to fetch data from Google Drive.

Configuration: Choose the data you want to collect from Google Drive

The Google Drive connector has many optional fields to configure. This section covers only the mandatory fields to configure for the connector. For a complete description of the optional fields to configure, see Advanced Google Drive tips.

To choose what data to collect and customize the Google Drive datastream configuration, follow these steps:

1. (Optional) Rename your datastream.

2. In Source URL, enter the sub-folder where the data files are found in your Google Drive.

3. Click Test to check the authorization.

4. In File pattern, enter the name of the file to collect data from. To collect data from more than one file, specify a regular expression that matches the filenames. For more information on how to configure the File pattern field, see Advanced Google Drive tips.

What's next?

• Apply Data Mapping to your collected data to harmonize data collected from different sources in Adverity.

• Transform your data to meet your needs by creating and applying enrichments to your datastream.

• Load your data into Adverity Data Storage to send it to BI tools using data shares

• Load your data into an external destination of your choice

Advanced Google Drive tips

Configuring data collection from Google Drive

In the Settings tab of your Google Drive datastream overview, you can configure a number of additional settings:

Warnings enabled

Select this checkbox to display a warning in the datastream Overview page when no file is found on Google Drive.

The File Parsing section contains the following fields:

Parse

Select the data format used in the source files. The type of data format selected may cause additional fields to appear.

Delimiter

If CSV is selected in Parse, specify the character used to separate values in the CSV source files.

Quote Char

If CSV is selected in Parse, specify the character used to quote values with special characters in the CSV source files.

Quoting

If CSV is selected in Parse, specify when a quoting character should be added to the field values in the data extract. Choose from one of the following options:

• Select all to add quote characters to everything in the data extract, regardless of the field type.

• (Default) Select minimal to add quote characters only when required. For example, a quote character will be added to a field that contains either the Quote Char or Delimiter.

• Select none to ensure no quote characters are added to the data extract.

• Select nonnumeric to add quote characters to everything, except integer and float values.

Sheet

If Excel is selected in Parse, specify the name of the sheet within the Excel source files to import to Adverity.

Column offset

If Excel is selected in Parse, specify the number of columns that you do not want to import to Adverity from each Excel source file. For example, if the first column contains information that you do not want to import, specify 1 in this field.

Row offset

Specify the number of rows that you do not want to import to Adverity from each source file. For example, if the first row contains header information that you do not want to import, specify 1 in this field. This field is available for all parse types.

Skip initial space

If CSV is selected in Parse, select this checkbox to ignore any whitespace that follows the selected delimiter character.

Drive Type

Specify the type of the Google Drive.

The Source section contains the following fields:

Export type (for GoogleSheets)

In the drop-down menu, specify the data format used to export data to Google Sheets.

The File Matching Options section contains the following fields:

File pattern

Specify a regular expression that matches the file attachment from which to collect data. By default, File pattern is pre-populated with .* which collects data for any attached file.

Other examples of regular expressions for file patterns include:

• .*\.xlsx collects every file that ends with .xlsx.

• .*\.csv collects every file that ends with .csv.

• .*Display.*\.csv collects every file that contains the (case-sensitive) phrase Display anywhere in the file name and also ends with .csv.

Source encoding

Select the character encoding used in the source files. By default, the option Auto-detect is selected and will automatically detect the encoding of the source files.

Zip match

If you collect data from ZIP or GZIP files, specify a regular expression that matches the file names within the ZIP or GZIP containers. Leave this field empty to collect all files within the ZIP or GZIP container.

Archive password

If the ZIP or GZIP files require password access, provide the password. By default, Adverity uses the previously entered password, if available.

Keep filename

Select this checkbox to name data extracts using the same name as the corresponding source files. If you select this checkbox and the source filename stays the same between fetches, data in the corresponding data extract is overwritten.

The File Processing section contains the following fields:

Process all

By default, Adverity only processes the most recently uploaded files. Select this checkbox to process all files that match the criteria you specify.

Recursive

By default, Adverity only searches for source files in the folder that you specify. Select this checkbox to search in all the subfolders of the specified folder.

Concatenate files

Select this checkbox to combine data from all source files into a single data extract.

Delete source

Select this checkbox to delete the source files after Adverity has imported their data.

Move to

Specify the full path to the folder in Google Drive into which the source files will be moved after Adverity has imported the data.

Uploading files into Adverity

To upload a file into Adverity using the Google Drive datastream, follow these steps:

1. Select the workspace you work with in Adverity and then, in the platform navigation menu, click Datastreams.

2. Open the Google Drive datastream by clicking on its name.

3. In the top right corner of the page, click Upload.

4. Click Choose File, and select the file to upload.

5. (Optional) Select the Keep data in raw state checkbox to achieve the following goals:
   

   • Keep the data in its original form and do not apply any enrichments assigned to this datastream. For more information on enriching your data, see Enriching data in Adverity.

   • Fetch the data without loading it into the destination specified for this datastream. For more information on loading data into a destination, see Introduction to loading data into Adverity Data Storage and external destinations.

6. Click Upload.

As a result, the file is uploaded into Adverity and Adverity creates a data extract that includes the uploaded data.

Fetching files from Google Drive manually

To browse files in the Google Drive server and fetch them manually, follow these steps:

1. Select the workspace you work with in Adverity and then, in the platform navigation menu, click Datastreams.

2. Open the Google Drive datastream by clicking on its name.

3. In the top right corner of the page, click More .

4. Click Browser.

5. Select the files to fetch. If the file is in a folder, click on the folder name to open the folder and view the files.

6. Click Fetch selected files.

The fetch collects data from Google Drive which takes some time. The Overview page of the newly created datastream is now displayed. To preview the collected data, follow these steps:

1. In the All tasks tab, find the task at the top of the list, and click Show extracts.

2. Click the top hyperlink.

3. The data extract is displayed in a table containing the data that you have fetched.