Manual and scheduled fetches

This guide explains how to fetch data manually, view the collected data, and create and configure scheduled fetches.

Introduction

A fetch is the process of collecting data using a datastream. There are two types of fetches:

  • Manual - start a single fetch manually

  • Scheduled - schedule regular, automated fetches or schedule a one-time fetch to run at a specific time

This video guide explains how to set up and configure scheduled data fetches.

Prerequisites

Before you complete the procedure in this guide, perform all of the following actions:

Fetching data manually

After creating a datastream, fetch data manually to perform a single fetch that will collect data for the date range you choose. This is a single fetch and will not be repeated automatically.To fetch data manually, follow these steps:

  1. Go to the Datastreams page.

  2. Open the chosen datastream by clicking on its name.

  3. In the top right corner, click Fetch now.

  4. In Choose date range, select the time period for which data is collected.

    If you select a long time period, Adverity automatically suggests splitting this fetch into smaller fetches to avoid errors and improve performance. You can change this in the Advanced options as described in the next step.

  5. (Optional) Click Advanced options to adjust the following settings:

    Fetch type

    Select one of the following options:

    • Select Run a single fetch including data from start until end to fetch all the data within the selected time period.

    • Select Rerun scheduled fetches in the time period specified to rerun scheduled fetches within the selected time period.

    Keep data in raw state

    Select this checkbox to achieve the following goals:

    • Keep the data in its original form and do not apply any transformation assigned to this datastream. For more information on enriching your data, see Introduction to the Transformations page.

    • Fetch the data without sending it to the destination that you specified for this datastream. For more information on loading data into a destination, see Introduction to the Destinations page.

    Consolidate extract by fetch

    Select this checkbox to fetch the data into one data extract. If unchecked, Adverity creates a different data extract for each day.

    Split into smaller fetches

    Select this checkbox to split a large fetch into multiple smaller fetches in order to avoid errors and improve performance. For example, if you are fetching data for a time period of six months, you can split this fetch into six one-month fetches.

    If you select this checkbox, in the Split by drop-down menu below, select the time period for the smaller fetches. You can choose one of the following options: days, weeks, months or quarters.

    Please note that splitting fetches into multiple smaller fetches will increase the number of API requests made. This may affect request limits for some data sources.

    For example, splitting a single six-month fetch into six one-month fetches will increase the number of API requests from 1 to 6.

  6. (Optional) Define optional filters for this fetch. The filters available in this section depend on the chosen connector.

  7. Click Fetch.

Viewing fetched data

To view the data you collected in your most recent fetch, follow these steps:

  1. Go to the Datastreams page.

  2. Open the chosen datastream by clicking on its name.

  3. In the All tasks tab, find the task at the top of the list, and click on the task overview.

As a result, you can see a table containing the data that you have fetched.

Here, you can perform a range of actions, including applying transformations to your data extract, downloading your collected data, and configuring Data Mapping.

Scheduling data fetches

Create a scheduled data fetch to automatically collect data for a specific time period at a frequency of your choice. For example, you can schedule a weekly fetch to collect data from the previous week. This option means you do not have to run frequent manual fetches in order to collect the data you need.

If the authorization used for a datastream for which you schedule a fetch becomes invalid, the scheduled fetch will be disabled. You will need to change the authorization to a valid authorization in order to schedule a new fetch for the affected datastream.

Creating a scheduled fetch

To create a scheduled fetch, follow these steps:

  1. Go to the Datastreams page.

  2. Open the chosen datastream by clicking on its name.

  3. In the Scheduling section, click the toggle to enable scheduled fetches.

As a result, a default scheduled fetch is created.

Default scheduled fetches are performed once every day. A default scheduled fetch will collect data for the time period which is typical for your previous usage preferences, at a random time between 00:00 and 04:00 UTC (Coordinated Universal Time). This is because most data source APIs use UTC, which means that 00:00 UTC is the earliest time to import data for the previous day.

Configuring scheduled fetches

Configuring a scheduled fetch

After creating a scheduled fetch, you can configure it to collect data for your required time period at your required frequency. To change the settings for a scheduled fetch, follow these steps:

  1. Go to the Datastreams page.

  2. Open the chosen datastream by clicking on its name.

  3. In the Scheduling section, choose one of the following options:

    • To create and configure a new scheduled fetch, enable the Scheduling toggle, and then click + Schedule fetch.

    • To configure an existing scheduled fetch, hover your mouse over the scheduled fetch you want to configure, and click Edit.

  4. In the Schedule window, fill in the following fields:

    Frequency

    Choose how often the fetch runs. If you choose Once, the scheduled fetch will be performed at the scheduled date and time, but will not repeat. A week is considered to run from Monday to Sunday, inclusive.

    We refer to the day that the fetch is scheduled to run as the "scheduled date" in the following field descriptions.

    Time of day (UTC)

    Choose the time when the fetch runs on the scheduled date. Specify the time according to the UTC (Coordinated Universal Time) standard.

    Time range

    Choose the time period for which Adverity collects data. Time presets work according to UTC.

    • To fetch data for the same dates each time the scheduled fetch runs, select Custom time range.

      If you select Custom time range and leave the Date to field empty, Adverity dynamically sets this field to Yesterday.

    • To fetch all data available from the datastream each time the scheduled fetch runs, select Lifetime. This option is not available for all datastreams.

    All the other options are relative to the scheduled date. For example:

    • Current month fetches data from the first day of the month, up to and including the scheduled date.

    • Previous month fetches data from the first to the last day of the month before the month the fetch runs.

    • Last 30 days fetches data for the 30-day period preceding, but not including, the scheduled date.

    Offset to execution time

    (Optional) Specify the number of days with which Adverity shifts the scheduled date. A positive value shifts the scheduled date to an earlier date, a negative value shifts the scheduled date to a later date. This setting is not available if you select Custom time range in the Time range setting.

  5. Click Apply.

Repeat the process above to configure multiple scheduled fetches which run independently of each other.

Example: Using a scheduled fetch to fetch the previous month's data

You can use a scheduled fetch to collect data each month for the previous month. To do so, configure your scheduled fetch in the Schedule window using these settings:

  • In Frequency, select Every month.

  • In Day of month, select 1.

  • In Time of day (UTC), select the time in UTC that corresponds to 23:59 in your time zone.

  • In Time range, select Current month.

  • In Offset to execution time, select 1.

As a result, each month, when your scheduled fetch runs on the first day of the month at 23:59, Adverity shifts the date 1 day earlier to 23:59 on the last day of the previous month. The Current month setting means that Adverity will fetch data for all days of the previous month.

For example, when this fetch runs at 23:59 on April 1, Adverity will collect data from March 1 to 23:59 on March 31.

Example: Using a scheduled fetch to collect data in a different time zone

Time presets in Adverity use UTC. In the following example, assume your account runs on Pacific Standard Time (PST) and you want to collect today's data every 2 hours.

If you select Today in the Time range field, no data will be collected in any fetches between 00:00 and 08:00 UTC due to the difference between the time zones.

For example, if you fetch data at 02:00 UTC on July 7, at the time of the fetch, the local time for the data source is 18:00 PST on July 6. The fetch would attempt to collect data for a future date and would return no data.

To solve this problem, configure your scheduled fetch in the Schedule window using these settings:

  • In Frequency, select Every 2 hours.

  • In Time range, select Last 2 Days.

  • In Offset to execution time, select -1.

As a result, Adverity will fetch data for yesterday and today in UTC time. Until 08:00 UTC, it is effectively today and tomorrow in PST time. This means that the scheduled fetches return the most recent data, even when the dates for the two time zones are different.