Manual, Smart, and Custom fetches

Manual, Smart, and Custom fetches#

This guide explains how to fetch data manually, view the collected data, and create and configure scheduled fetches.

Introduction#

A fetch is the process of collecting data using a datastream. There are three types of fetches:

  • Manual – start a single fetch manually

  • Smart – intelligently automates data collection by optimizing fetch timing, respecting API rate limits, and applying connector-specific attribution windows

  • Custom – set up regular, automated fetches or schedule a one-time fetch at a specific time, with precise control over schedule and time range

Smart Schedule uses advanced optimization to determine the best times for data collection, minimizing processing errors while ensuring data accuracy through appropriate attribution windows. This eliminates the need to manually calculate and manage complex scheduling parameters.

This video guide explains how to set up and configure scheduled data fetches.

Prerequisites#

Before you complete the procedure in this guide, perform all of the following actions:

Fetching data manually#

After creating a datastream, fetch data manually to perform a single fetch that will collect data for the date range you choose. This is a single fetch and will not be repeated automatically.To fetch data manually, follow these steps:

  1. Go to the Datastreams page.

  2. Open the chosen datastream by clicking on its name.

  3. In the top right corner, click Fetch now.

  4. In Choose date range, select the time period for which data is collected.

    Note

    If you select a long time period, Adverity automatically suggests splitting this fetch into smaller fetches to avoid errors and improve performance. You can change this in the Advanced options as described in the next step.

  5. (Optional) Click image1 Advanced options to adjust the following settings:

    Fetch type

    Select one of the following options:

    • Select Run a single fetch including data from start until end to fetch all the data within the selected time period.

    • Select Rerun scheduled fetches in the time period specified to rerun scheduled fetches within the selected time period.

    Keep data in raw state

    Select this checkbox to achieve the following goals:

    • Keep the data in its original form and do not apply any transformation assigned to this datastream. For more information on enriching your data, see Introduction to the Transformations page.

    • Fetch the data without sending it to the destination that you specified for this datastream. For more information on loading data into a destination, see Introduction to the Destinations page.

    Consolidate extract by fetch

    Select this checkbox to fetch the data into one data extract. If unchecked, Adverity creates a different data extract for each day.

    Split into smaller fetches

    Select this checkbox to split a large fetch into multiple smaller fetches in order to avoid errors and improve performance. For example, if you are fetching data for a time period of six months, you can split this fetch into six one-month fetches.

    If you select this checkbox, in the Split by drop-down menu below, select the time period for the smaller fetches. You can choose one of the following options: days, weeks, months or quarters.

    Warning

    Please note that splitting fetches into multiple smaller fetches will increase the number of API requests made. This may affect request limits for some data sources.

    For example, splitting a single six-month fetch into six one-month fetches will increase the number of API requests from 1 to 6.

  6. (Optional) Define optional filters for this fetch. The filters available in this section depend on the chosen connector.

  7. Click Fetch.

Viewing fetched data#

To view the data you collected in your most recent fetch, follow these steps:

  1. Go to the Datastreams page.

  2. Open the chosen datastream by clicking on its name.

  3. In the All tasks tab, find the task at the top of the list, and click on the task overview.

As a result, you can see a table containing the data that you have fetched.

Here, you can perform a range of actions, including applying transformations to your data extract, downloading your collected data, and configuring Data Mapping.

Smart and Custom fetches#

You can automate data collection using Smart or Custom fetches. Smart fetches use intelligent automation to optimize timing and handle dependencies, while Custom fetches allow you to define a precise schedule.

Warning

If the authorization used for a datastream for which you schedule a fetch becomes invalid, the scheduled fetch will be disabled. You will need to change the authorization to a valid authorization in order to schedule a new fetch for the affected datastream.

Note

The fetches will try to start at the time you selected, not considering dependent datastreams (except datastreams in a bundle), API rate limits, or availability of the resources.

Creating a scheduled fetch#

To create a scheduled fetch, follow these steps:

  1. Go to the Datastreams page.

  2. Open the chosen datastream by clicking on its name.

  3. In the Scheduling section, click the toggle to enable scheduled fetches.

As a result, a default scheduled fetch is created.

Default scheduled fetches are performed once every day. A default scheduled fetch will collect data for the time period which is typical for your previous usage preferences, at a random time between 00:00 and 04:00 UTC (Coordinated Universal Time). This is because most data source APIs use UTC, which means that 00:00 UTC is the earliest time to import data for the previous day.

Using Smart Schedule#

Smart Schedule provides the following benefits:

  • Automatically fetches new data and refreshes existing data with appropriate attribution windows.

  • Respects datastream dependencies, waiting for prerequisite datastreams to update first.

  • Optimizes fetch timing while respecting API rate limits to minimize errors.

  • Analyzes historical data to determine optimal fetch patterns.

  • Prioritizes fetches based on frequency (daily fetches take priority over weekly or monthly).

To enable Smart Schedule, follow these steps:

  1. Go to the Datastreams page.

  2. Open the chosen datastream by clicking on its name.

  3. In the Scheduling section, click the toggle to enable scheduled fetches.

  4. Configure the Smart Schedule settings:

    Fetch new data

    Select one of the following options:

    • Daily - After midnight, fetch data from yesterday

    • Weekly - Every Monday, fetch data from last week

    • Monthly - On the 1st day of the month, fetch data from the last month

    Refresh existing data

    Select the number of days to refresh. Smart Schedule automatically applies the recommended attribution window based on the connector. For example:

    • Facebook Ads fetches by default every day yesterday and the last 6 days

    • Google Ads fetches by default every day yesterday and the last 29 days

    • TikTok Ads - last 28 days

    • LinkedIn Ads - last 30 days

    • File/Datalake, Database category - None

    Load the data by (UTC)

    Set a specific time (in UTC) by which you want the data to be loaded for this datastream. Setting a “load by” time prioritizes this datastream, significantly increasing the chances that the scheduled data is delivered by the selected time.

    The default time used for all datastreams with Smart scheduling can be changed in the workspace admin settings.

  5. Click Apply.

Note

If you need more specific scheduling requirements than what Smart Schedule offers, you can use the custom scheduling option described in the Configuring a custom scheduled fetch section.

How Smart Schedule manages datastream dependencies#

When you enable Smart Schedule for datastreams that have dependencies, Smart Schedule automatically:

  • Identifies all direct dependencies between datastreams.

  • Ensures source datastreams are updated before dependent datastreams.

  • Checks that all required data ranges are available in source datastreams.

  • Triggers dependent datastream fetches only when all prerequisites are met.

This intelligent sequencing eliminates the need to manually coordinate fetch schedules between related datastreams, ensuring data consistency throughout your pipeline.

Applying Smart Schedule to multiple datastreams#

To apply Smart Schedule to multiple datastreams at once, follow these steps:

  1. Go to the Datastreams page.

  2. Select the checkboxes next to the datastreams you want to schedule.

  3. Click Schedule fetches.

  4. Configure the Smart Schedule settings as described above.

  5. Click Apply to selected.

This feature eliminates the need to configure each datastream individually, saving significant time when managing multiple data sources.

Configuring scheduled fetches#

Configuring a custom scheduled fetch#

After creating a scheduled fetch, you can configure it to collect data for your required time period at your required frequency. To change the settings for a scheduled fetch, follow these steps:

  1. Go to the Datastreams page.

  2. Open the chosen datastream by clicking on its name.

  3. In the Scheduling section, choose one of the following options:

    • In the Scheduling section, click the toggle to enable scheduled fetches.

    • To configure an existing scheduled fetch, hover your mouse over the scheduled fetch you want to configure, and click image2 Edit.

  4. In the scheduling popup, click Set custom scheduling.

  5. In the Set scheduling window, fill in the following fields:

    Frequency

    Choose how often the fetch runs. If you choose Once, the scheduled fetch will be performed at the scheduled date and time, but will not repeat. A week is considered to run from Monday to Sunday, inclusive.

    We refer to the day that the fetch is scheduled to run as the “scheduled date” in the following field descriptions.

    Time of day (UTC)

    Choose the time when the fetch runs on the scheduled date. Specify the time according to the UTC (Coordinated Universal Time) standard.

    Time range

    Choose the time period for which Adverity collects data. Time presets work according to UTC.

    • To fetch data for the same dates each time the scheduled fetch runs, select Custom time range.

      If you select Custom time range and leave the Date to field empty, Adverity dynamically sets this field to Yesterday.

    • To fetch all data available from the datastream each time the scheduled fetch runs, select Lifetime. This option is not available for all datastreams.

    All the other options are relative to the scheduled date. For example:

    • Current month fetches data from the first day of the month, up to and including the scheduled date.

    • Previous month fetches data from the first to the last day of the month before the month the fetch runs.

    • Last 30 days fetches data for the 30-day period preceding, but not including, the scheduled date.

    Offset to execution time

    (Optional) Specify the number of days with which Adverity shifts the scheduled date. A positive value shifts the scheduled date to an earlier date, a negative value shifts the scheduled date to a later date. This setting is not available if you select Custom time range in the Time range setting.

  6. Click Apply.

Repeat the process above to configure multiple scheduled fetches which run independently of each other.

Example: Using a scheduled fetch to fetch the previous month’s data#

You can use a scheduled fetch to collect data each month for the previous month. To do so, configure your scheduled fetch in the Schedule window using these settings:

  • In Frequency, select Every month.

  • In Day of month, select 1.

  • In Time of day (UTC), select the time in UTC that corresponds to 23:59 in your time zone.

  • In Time range, select Current month.

  • In Offset to execution time, select 1.

As a result, each month, when your scheduled fetch runs on the first day of the month at 23:59, Adverity shifts the date 1 day earlier to 23:59 on the last day of the previous month. The Current month setting means that Adverity will fetch data for all days of the previous month.

For example, when this fetch runs at 23:59 on April 1, Adverity will collect data from March 1 to 23:59 on March 31.

Example: Using a scheduled fetch to collect data in a different time zone#

Time presets in Adverity use UTC. In the following example, assume your account runs on Pacific Standard Time (PST) and you want to collect today’s data every 2 hours.

If you select Today in the Time range field, no data will be collected in any fetches between 00:00 and 08:00 UTC due to the difference between the time zones.

For example, if you fetch data at 02:00 UTC on July 7, at the time of the fetch, the local time for the data source is 18:00 PST on July 6. The fetch would attempt to collect data for a future date and would return no data.

To solve this problem, configure your scheduled fetch in the Schedule window using these settings:

  • In Frequency, select Every 2 hours.

  • In Time range, select Last 2 Days.

  • In Offset to execution time, select -1.

As a result, Adverity will fetch data for yesterday and today in UTC time. Until 08:00 UTC, it is effectively today and tomorrow in PST time. This means that the scheduled fetches return the most recent data, even when the dates for the two time zones are different.

Contents