Loading historical data to new destinations in Management API

This guide explains how to programmatically trigger a historical data load to destination tables using the Management API.

Introduction

When you add a new destination to an existing datastream, you can trigger a historical data load to backfill that destination table with data from previous data extracts. This allows you to automate the full deployment of a new destination — including the backfill step — as part of a CI/CD or scripted workflow, without requiring any interaction with the Adverity user interface.

Prerequisites

Before you complete the procedure in this guide, ensure you have the following:

• A Management API key with the destination:write scope. For more information, see Generating an API key in Adverity.

• A datastream with existing data extracts.

• At least one enabled destination binding on the datastream.

Discovering extract and binding IDs

Before triggering a historical load, you need to determine the IDs of the data extracts you want to requeue. You can retrieve these using the legacy extracts endpoint filtered by datastream ID.

To list data extracts for a specific datastream, follow these steps:

1. Create a GET request to the following endpoint:

   https://{{INSTANCE}}/api/extracts/?datastream_id={{DATASTREAM_ID}}
   

2. In the HTTP request header, include the parameter Authorization with value Bearer {{KEY}}.

3. Send the request.

Import the request example as raw text to your HTTP client (such as Postman). The cURL request example is the following:

curl --location --request GET 'https://{{INSTANCE}}/api/extracts/?datastream_id={{DATASTREAM_ID}}' \
--header 'Authorization: Bearer {{KEY}}'

In the response, locate the id values of the data extracts you want to backfill, as well as any binding id values if you want to target a specific destination binding.

Requeueing extracts sequentially

By default, the requeue endpoint processes data extracts in chronological order (sequential mode). This ensures that data is loaded in the correct temporal sequence.

To requeue data extracts sequentially, follow these steps:

1. Create a POST request to the following endpoint:

   https://{{INSTANCE}}/api/v1/extracts/requeue/
   

2. In the HTTP request header, include the parameter Authorization with value Bearer {{KEY}}.

3. In the HTTP request header, include the parameter Content-Type with value application/json.

4. In the HTTP request body, include the extract_ids parameter with the list of data extract IDs to requeue:

   {
       "extract_ids": [101, 102, 103]
   }
   

5. Send the request.

Import the request example as raw text to your HTTP client (such as Postman). The cURL request example is the following:

curl --location --request POST 'https://{{INSTANCE}}/api/v1/extracts/requeue/' \
--header 'Authorization: Bearer {{KEY}}' \
--header 'Content-Type: application/json' \
--data-raw '{"extract_ids": [101, 102, 103]}'

A successful request returns a 200 response with a confirmation message, the extract IDs in chronological order, and the total count:

{
    "message": "Dispatched requeue for 3 extract(s).",
    "extract_ids": [101, 102, 103],
    "extract_count": 3
}

Note

Sequential mode is the default behaviour. You can omit "sequential": true from the request body or include it explicitly — the result is the same.

Requeueing extracts in parallel

If the order in which data is loaded does not matter, you can process data extracts in parallel by setting sequential to false. This can improve throughput for large backfills.

To requeue data extracts in parallel, follow these steps:

1. Create a POST request to the following endpoint:

   https://{{INSTANCE}}/api/v1/extracts/requeue/
   

2. In the HTTP request header, include the parameter Authorization with value Bearer {{KEY}}.

3. In the HTTP request header, include the parameter Content-Type with value application/json.

4. In the HTTP request body, include the extract_ids parameter and set sequential to false:

   {
       "extract_ids": [101, 102],
       "sequential": false
   }
   

5. Send the request.

Import the request example as raw text to your HTTP client (such as Postman). The cURL request example is the following:

curl --location --request POST 'https://{{INSTANCE}}/api/v1/extracts/requeue/' \
--header 'Authorization: Bearer {{KEY}}' \
--header 'Content-Type: application/json' \
--data-raw '{"extract_ids": [101, 102], "sequential": false}'

A successful request returns a 200 response with the same structure as the sequential mode:

{
    "message": "Dispatched requeue for 2 extract(s).",
    "extract_ids": [101, 102],
    "extract_count": 2
}

Requeueing extracts to a specific destination binding

By default, the requeue endpoint loads data to all enabled destination bindings on the datastream. You can restrict the load to a specific binding by including the binding_ids parameter.

To requeue data extracts to a specific destination binding, follow these steps:

1. Create a POST request to the following endpoint:

   https://{{INSTANCE}}/api/v1/extracts/requeue/
   

2. In the HTTP request header, include the parameter Authorization with value Bearer {{KEY}}.

3. In the HTTP request header, include the parameter Content-Type with value application/json.

4. In the HTTP request body, include the extract_ids parameter and the binding_ids parameter with the ID of the target binding:

   {
       "extract_ids": [101],
       "binding_ids": [55]
   }
   

5. Send the request.

Import the request example as raw text to your HTTP client (such as Postman). The cURL request example is the following:

curl --location --request POST 'https://{{INSTANCE}}/api/v1/extracts/requeue/' \
--header 'Authorization: Bearer {{KEY}}' \
--header 'Content-Type: application/json' \
--data-raw '{"extract_ids": [101], "binding_ids": [55]}'

A successful request returns a 200 response scoped to the specified binding:

{
    "message": "Dispatched requeue for 1 extract(s).",
    "extract_ids": [101],
    "extract_count": 1
}

Error reference

The following table describes the error responses returned by the POST /api/v1/extracts/requeue/ endpoint:

Status code	Cause	Response message
422	One or more extract IDs do not exist or are not accessible with the provided API key.	Extracts not found or inaccessible: [<id>]
422	One or more binding IDs are not associated with the datastreams of the provided extracts.	Bindings not found or not associated with the provided extracts: [<id>]
422	More than 50 extract IDs were supplied in a single request.	Validation error listing extract_ids.
422	The extract_ids field is missing or empty.	Validation error listing extract_ids.
403	The API key does not have the destination:write scope.	Forbidden.