Manual and scheduled fetches
This guide explains how to fetch data manually, view the collected data, and create and configure scheduled fetches.
Introduction
A fetch is the process of collecting data using a datastream. There are two types of fetches:
-
Manual - start a single fetch manually
-
Scheduled - schedule regular, automated fetches or schedule a one-time fetch to run at a specific time
Prerequisites
Before you complete the procedure in this guide, perform all of the following actions:
Fetching data manually
After creating a datastream, fetch data manually to perform a single fetch that will collect data for the date range you choose. This is a single fetch and will not be repeated automatically.To fetch data manually, follow these steps:
-
Select the workspace you work with in Adverity and then, in the , click Datastreams.
-
In the top right corner, click Fetch now.
-
In Choose date range, select the time period for which data is collected.
-
(Optional) Click
Advanced options to adjust the following settings:
- Fetch type
-
Select one of the following options:
-
Select Run a single fetch including data from start until end to fetch all the data within the selected time period.
-
Select Rerun scheduled fetches in the time period specified to rerun scheduled fetches within the selected time period.
-
-
Keep data in raw state
-
Select this checkbox to achieve the following goals:
-
Keep the data in its original form and do not apply any enrichment assigned to this datastream. For more information on enriching your data, see Introduction to the Enrichments page.
-
Fetch the data without sending it to the destination that you specified for this datastream. For more information on loading data into a destination, see Introduction to the Destinations page.
-
-
Consolidate extract by fetch
-
Select this checkbox to fetch the data into one data extract. If unchecked, Adverity creates a different data extract for each day.
-
(Optional) Define optional filters for this fetch. The filters available in this section depend on the chosen connector.
-
Click Fetch.
Viewing fetched data
To view the data you collected in your most recent fetch, follow these steps:
-
Select the workspace you work with in Adverity and then, in the , click Datastreams.
-
In the All tasks tab, find the task at the top of the list, and click
Show data extracts.
-
Click the top hyperlinked element.
As a result, you are taken to the Data extracts tab of the datastream overview, where you can see a table containing the data that you have fetched.
Here, you can perform a range of actions, including applying enrichments to your data extract, downloading your collected data, and configuring Data Mapping.
Scheduling data fetches
Create a scheduled data fetch to automatically collect data for a specific time period at a frequency of your choice. For example, you can schedule a weekly fetch to collect data from the previous week. This option means you do not have to run frequent manual fetches in order to collect the data you need.
If the datastream for which you schedule a fetch becomes invalid, the scheduled fetch will be disabled. You will need to change the to a valid in order to schedule a new fetch for the affected datastream.
used for aCreating a scheduled fetch
To create a scheduled fetch, follow these steps:
-
Select the workspace you work with in Adverity and then, in the , click Datastreams.
-
In the Scheduling section, click the toggle to enable scheduled fetches.
As a result, a default scheduled fetch is created.
Default scheduled fetches are performed once every day. A default scheduled fetch will collect data for the time period which is typical for your previous usage preferences, at a random time between 00:00 and 04:00 UTC (Coordinated Universal Time). This is because most data source APIs use UTC, which means that 00:00 UTC is the earliest time to import data for the previous day.
Configuring scheduled fetches
After creating a scheduled fetch, you can configure it to collect data for your required time period at your required frequency. To change the settings for a scheduled fetch, follow these steps:
-
Select the workspace you work with in Adverity and then, in the , click Datastreams.
-
In the Scheduling section, choose one of the following options:
-
To create and configure a new scheduled fetch, enable the Scheduling toggle, and then click + Schedule fetch.
-
To configure an existing scheduled fetch, hover your mouse over the scheduled fetch you want to configure, and click
Edit.
-
-
In the Schedule window, fill in the following fields:
-
Frequency
-
Choose how often the fetch runs. If you choose Once, the scheduled fetch will be performed at the scheduled date and time, but will not repeat. A week is considered to run from Monday to Sunday, inclusive.
-
We refer to the day that the fetch is scheduled to run as the "scheduled date" in the following field descriptions.
-
Time of day UTC
-
Choose the time when the fetch runs on the scheduled date. Specify the time according to the UTC (Coordinated Universal Time) standard.
-
Time range
-
Choose the time period for which Adverity collects data. Time presets work according to UTC.
-
To fetch data for the same dates each time the scheduled fetch runs, select Custom time range.
-
To fetch all data available from the datastream each time the scheduled fetch runs, select Lifetime. This option is not available for all datastreams.
All the other options are relative to the scheduled date. For example:
-
Current month fetches data from the first day of the month, up to and including the scheduled date.
-
Previous month fetches data from the first to the last day of the month before the month the fetch runs.
-
Last 30 days fetches data for the 30-day period preceding, but not including, the scheduled date.
-
-
Offset to execution time
-
(Optional) Specify the number of days with which Adverity shifts the scheduled date. A positive value shifts the scheduled date to an earlier date, a negative value shifts the scheduled date to a later date. This setting is not available if you select Custom time range in the Time range setting.
For example, you specify Current month in the Time range field, and 5 days in the Offset to execution time field. When the scheduled fetch runs on 3 April, Adverity shifts the scheduled date 5 days earlier to 29 March. With the Current month setting, Adverity collects data from the first day of the month to the scheduled date, which means 1 - 29 March.
-
-
Click Apply.
Repeat the process above to configure multiple scheduled fetches which run independently of each other.
In the following example, assume your account runs on Pacific Standard Time (PST) and you want to collect today's data every 2 hours. Time presets in Adverity work on UTC. Therefore, selecting Today in the Time range field would mean that fetches conducted between 0:00 and 8:00 UTC would be skipped. These early morning fetches would not return any data from the source due to the discrepancy between the time zones.
If you fetched data at 2:00 UTC on July 7 using the Today setting, the fetch would attempt to collect data for the whole day of July 7. However, at the time of the fetch, the local time for the data source is 18:00 PST on July 6. For this reason, the fetch would attempt to collect data for a future date and would return no data.
To resolve this issue, select the following options:
-
In Frequency, select Every 2 hours.
-
In Time range, select Last 2 Days.
-
In Offset to execution time, select -1.
These settings instruct Adverity to fetch data for yesterday and today in UTC time. Until 8:00 UTC, it is effectively today and tomorrow in PST time. Consequently, the fetches return the most recent data, even during the critical hours of 0:00 to 8:00 UTC when the dates for the two timezones are different.
Video guide: Scheduling data fetches
This video guide explains how to set up and configure scheduled data fetches.