Manual and scheduled fetches

This guide explains how to fetch data manually, view the collected data, and create and configure scheduled fetches.

Introduction

A fetch is the process of collecting data using a datastream. There are two types of fetches:

• Manual - start a single fetch manually

• Scheduled - schedule regular, automated fetches or schedule a one-time fetch to run at a specific time

Prerequisites

Before you complete the procedure in this guide, perform all of the following actions:

• Create a datastream.

Fetching data manually

After creating a datastream, fetch data manually to perform a single fetch that will collect data for the date range you choose. This is a single fetch and will not be repeated automatically.To fetch data manually, follow these steps:

1. Select the workspace you work with in Adverity and then, in the platform navigation menu, click Datastreams.

2. Open the chosen datastream by clicking on its name.

3. In the top right corner, click Fetch now.

4. In Choose date range, select the time period for which data is collected.

5. (Optional) Click Advanced options to adjust the following settings:

   Fetch type

   Select one of the following options:

   • Select Run a single fetch including data from start until end to fetch all the data within the selected time period.

   • Select Rerun scheduled fetches in the time period specified to rerun scheduled fetches within the selected time period.

   Keep data in raw state

   Select this checkbox to achieve the following goals:

   • Keep the data in its original form and do not apply any enrichment assigned to this datastream. For more information on enriching your data, see Introduction to the Enrichments page.

   • Fetch the data without sending it to the destination that you specified for this datastream. For more information on loading data into a destination, see Introduction to the Destinations page.

   Consolidate extract by fetch

   Select this checkbox to fetch the data into one data extract. If unchecked, Adverity creates a different data extract for each day.

6. (Optional) Define optional filters for this fetch. The filters available in this section depend on the chosen connector.

7. Click Fetch.

Viewing fetched data

To view the data you collected in your most recent fetch, follow these steps:

1. Select the workspace you work with in Adverity and then, in the platform navigation menu, click Datastreams.

2. Open the chosen datastream by clicking on its name.

3. In the All tasks tab, find the task at the top of the list, and click Show data extracts.

4. Click the top hyperlinked element.

As a result, you are taken to the Data extracts tab of the datastream overview, where you can see a table containing the data that you have fetched.

Here, you can perform a range of actions, including applying enrichments to your data extract, downloading your collected data, and configuring Data Mapping.

Scheduling data fetches

Create a scheduled data fetch to automatically collect data for a specific time period at a frequency of your choice. For example, you can schedule a weekly fetch to collect data from the previous week. This option means you do not have to run frequent manual fetches in order to collect the data you need.

Creating a scheduled fetch

To create a scheduled fetch, follow these steps:

1. Select the workspace you work with in Adverity and then, in the platform navigation menu, click Datastreams.

2. Open the chosen datastream by clicking on its name.

3. In the Scheduling section, click the toggle to enable scheduled fetches.

As a result, a default scheduled fetch is created.

Default scheduled fetches are performed once every day. A default scheduled fetch will collect data for the time period which is typical for your previous usage preferences, at a random time between 00:00 and 04:00 UTC (Coordinated Universal Time). This is because most data source APIs use UTC, which means that 00:00 UTC is the earliest time to import data for the previous day.

Configuring scheduled fetches

After creating a scheduled fetch, you can configure it to collect data for your required time period at your required frequency. To change the settings for a scheduled fetch, follow these steps:

1. Select the workspace you work with in Adverity and then, in the platform navigation menu, click Datastreams.

2. Open the chosen datastream by clicking on its name.

3. In the Scheduling section, choose one of the following options:

   • To create and configure a new scheduled fetch, enable the Scheduling toggle, and then click + Schedule fetch.

   • To configure an existing scheduled fetch, hover your mouse over the scheduled fetch you want to configure, and click Edit.

4. In the Schedule window, fill in the following fields:

   Frequency

   Choose how often the fetch runs. If you choose Once, the scheduled fetch will be performed at the scheduled date and time, but will not repeat. A week is considered to run from Monday to Sunday, inclusive.

   We refer to the day that the fetch is scheduled to run as the "scheduled date" in the following field descriptions.

   Time of day UTC

   Choose the time when the fetch runs on the scheduled date. Specify the time according to the UTC (Coordinated Universal Time) standard.

   Time range

   Choose the time period for which Adverity collects data. Time presets work according to UTC.

   • To fetch data for the same dates each time the scheduled fetch runs, select Custom time range.

   • To fetch all data available from the datastream each time the scheduled fetch runs, select Lifetime. This option is not available for all datastreams.

   All the other options are relative to the scheduled date. For example:

   • Current month fetches data from the first day of the month, up to and including the scheduled date.

   • Previous month fetches data from the first to the last day of the month before the month the fetch runs.

   • Last 30 days fetches data for the 30-day period preceding, but not including, the scheduled date.

   Offset to execution time

   (Optional) Specify the number of days with which Adverity shifts the scheduled date. A positive value shifts the scheduled date to an earlier date, a negative value shifts the scheduled date to a later date. This setting is not available if you select Custom time range in the Time range setting.

   For example, you specify Current month in the Time range field, and 5 days in the Offset to execution time field. When the scheduled fetch runs on 3 April, Adverity shifts the scheduled date 5 days earlier to 29 March. With the Current month setting, Adverity collects data from the first day of the month to the scheduled date, which means 1 - 29 March.

5. Click Apply.

Repeat the process above to configure multiple scheduled fetches which run independently of each other.

Video guide: Scheduling data fetches

This video guide explains how to set up and configure scheduled data fetches.