Configuring datastreams in Management API

Configuring datastreams in Management API#

This guide explains how to configure and edit datastreams with the Management API.

Listing datastreams of a specific type#

To list datastreams of a specific datastream type, follow these steps:

Create a GET request to the following endpoint:
```
https://{{INSTANCE}}/api/datastream-types/{{DATASTREAM_TYPE}}/datastreams/
```
Replace {{DATASTREAM_TYPE}} with the datastream type’s ID. For more information on determining a datastream type’s ID, see Determining a Datastream Type’s ID.
In the HTTP request header, include the parameter Authorization with one of the following values:
- Token {{KEY}} if you use a key generated with user credentials in Management API.
- Bearer {{KEY}} if you use a key generated in the Adverity user interface.
Send the request.

As a result, you obtain the list of datastreams of a specific datastream type.

Import the request example as raw text to your HTTP client (such as Postman). The cURL request example is the following:

curl --location --request GET 'https://{{INSTANCE}}/api/datastream-types/{{DATASTREAM_TYPE}}/datastreams/' \
--header 'Authorization: Token {{KEY}}'

Note

Replace Token with Bearer in the Authorization header if you are using a key generated in the Adverity user interface.

Example of JSON response:

{
    "count": 15,
    "next": null,
    "previous": null,
    "results": [
        {
            "id": {{DATASTREAM_ID}},
            "cron_type": "day",
            "cron_interval": 1,
            "cron_start_of_day": "00:30:00",
            "cron_interval_start": 1,
            "time_range_preset": 13,
            "delta_type": 2,
            "delta_interval": 1,
            "delta_interval_start": 1,
            "delta_start_of_day": "00:00:00",
            "datatype": "Staging",
            "creator": "jane@doe.com",
            "datastream_type_id": {{DATASTREAM_TYPE}},
            "absolute_url": "https://{{INSTANCE}}/adform/{{DATASTREAM_ID}}/adform-buyer/"
        }
    ]
}

The value in an id parameter of the response is a datastream’s ID.

Note

The datatype parameter reflects the state of the datastream’s destinations. A read returns "Live" if at least one destination is enabled, and "Staging" if all of them are disabled. You can also write datatype when you edit a datastream: setting "Live" enables all destinations, and setting "Staging" disables them.

Determining current datastream configuration#

To determine a datastream’s current configuration, follow these steps:

Create a GET request to the following endpoint:
```
https://{{INSTANCE}}/api/datastream-types/{{DATASTREAM_TYPE}}/datastreams/{{DATASTREAM_ID}}/
```
Replace {{DATASTREAM_ID}} with the datastream’s ID.
In the HTTP request header, include the parameter Authorization with one of the following values:
- Token {{KEY}} if you use a key generated with user credentials in Management API.
- Bearer {{KEY}} if you use a key generated in the Adverity user interface.
Send the request.

As a result, you determined the current datastream configuration.

Import the request example as raw text to your HTTP client (such as Postman). The cURL request example is the following:

curl --location --request GET 'https://{{INSTANCE}}/api/datastream-types/{{DATASTREAM_TYPE}}/datastreams/{{DATASTREAM_ID}}/' \
--header 'Authorization: Token {{KEY}}'

Note

Replace Token with Bearer in the Authorization header if you are using a key generated in the Adverity user interface.

Retrieving the assigned destinations#

By default, a datastream detail response does not include the destinations that the datastream delivers to. To include them, add the expand query parameter with the value destinations to either datastream detail endpoint:

GET /api/datastreams/{{DATASTREAM_ID}}/
GET /api/datastream-types/{{DATASTREAM_TYPE}}/datastreams/{{DATASTREAM_ID}}/

Both endpoints return the same assigned_destinations content for the same datastream.

To retrieve a datastream’s assigned destinations, follow these steps:

Create a GET request to one of the following endpoints:
```
https://{{INSTANCE}}/api/datastreams/{{DATASTREAM_ID}}/?expand=destinations
```
Replace {{DATASTREAM_ID}} with the datastream’s ID.
In the HTTP request header, include the parameter Authorization with one of the following values:
- Token {{KEY}} if you use a key generated with user credentials in Management API.
- Bearer {{KEY}} if you use a key generated in the Adverity user interface.
Send the request.

As a result, the response includes an assigned_destinations array that describes the destinations assigned to the datastream.

Import the request example as raw text to your HTTP client (such as Postman). The cURL request example is the following:

curl --location -g --request GET 'https://{{INSTANCE}}/api/datastreams/{{DATASTREAM_ID}}/?expand=destinations' \
--header 'Authorization: Token {{KEY}}'

Note

Replace Token with Bearer in the Authorization header if you are using a key generated in the Adverity user interface.

Each object in the assigned_destinations array contains the following fields:

Field	Description
`target_id`	The numeric ID of the destination.
`destination_name`	The name of the destination.
`destination_type`	The type of the destination, for example `bigquerytarget`.
`enabled`	Boolean. `true` if the binding is enabled, `false` if it is disabled.
`destination_settings`	An object with destination-specific settings. The fields depend on the destination type. For more information, see the table below.

Example of the assigned_destinations array in a response:

"assigned_destinations": [
    {
        "target_id": 123,
        "destination_name": "My BigQuery",
        "destination_type": "bigquerytarget",
        "enabled": true,
        "destination_settings": {
            "table_name": "analytics.spend",
            "write_disposition": "WRITE_TRUNCATE",
            "partition_by_date": true,
            "partition_column_name": "date",
            "expires_in": null,
            "overwrite_key_columns": false
        }
    }
]

The destination_settings object contains a curated set of settings that depends on the destination type:

destination type	`destination_settings` fields
SQL and Snowflake	`table_name`, `truncate`, `overwrite_key_columns`, `overwrite_datastream`, `overwrite_date_range_column_name`
BigQuery	`table_name`, `write_disposition` (`WRITE_APPEND`, `WRITE_TRUNCATE`, or `WRITE_EMPTY`), `partition_by_date`, `partition_column_name`, `expires_in`, `overwrite_key_columns`
Google Sheets	`filename`, `sheet`
Databricks and Delta Lake	`table_name`
Warehouse (managed by Adverity)	`ingestion_mode`, `row_date_column_name`, `overwrite_key_columns`, `datasource_label`, `overwrite_datastream`, `overwrite_filename`, `overwrite_date_range_column_name`, and workspace-mapping fields
Adverity Insights	`datasource_label`, `overwrite_key_columns`, `overwrite_datastream`, `overwrite_filename`, `overwrite_date_range_column_name`, and workspace-mapping fields
File and audience destinations	No settings. The `destination_settings` object is empty.
Any other destination type	No settings. The `destination_settings` object is empty.

Note

The assigned_destinations array includes disabled bindings ("enabled": false), but it does not include system-invalidated bindings. When you omit the expand parameter, the response does not include the assigned_destinations array and is otherwise unchanged.

Determining datastream configuration options#

To determine the options you can configure for a datastream, follow these steps:

Create an OPTIONS request to the following endpoint: Replace {{DATASTREAM_ID}} with the datastream’s ID.
```
https://{{INSTANCE}}/api/datastream-types/{{DATASTREAM_TYPE}}/datastreams/{{DATASTREAM_ID}}/
```
In the HTTP request header, include the parameter Authorization with one of the following values:
- Token {{KEY}} if you use a key generated with user credentials in Management API.
- Bearer {{KEY}} if you use a key generated in the Adverity user interface.
Send the request.

As a result, you obtained the list of options you can configure for the datastream.

Import the request example as raw text to your HTTP client (such as Postman). The cURL request example is the following:

curl --location --request OPTIONS 'https://{{INSTANCE}}/api/datastream-types/{{DATASTREAM_TYPE}}/datastreams/{{DATASTREAM_ID}}/' \
--header 'Authorization: Token {{KEY}}'

In the response, search for specific parameters and objects to edit your datastream configuration. If a specific object includes the parameter read_only with the value true, you cannot edit it. "read_only": false means that you can edit a particular object. Fields that are auth-specific (such as accounts, campaigns, or advertisers) include "is_auth_scoped": true, indicating that the available options depend on the authorization used.

Configuring a datastream#

To configure a datastream, follow these steps:

Create a PATCH request to the following endpoint:
```
https://{{INSTANCE}}/api/datastream-types/{{DATASTREAM_TYPE}}/datastreams/{{DATASTREAM_ID}}/
```
Replace {{DATASTREAM_ID}} with the datastream’s ID.
In the HTTP request header, include the parameter Authorization with one of the following values:
- Token {{KEY}} if you use a key generated with user credentials in Management API.
- Bearer {{KEY}} if you use a key generated in the Adverity user interface.
In the HTTP request header, include the parameter Content-Type with value application/json.
In the HTTP request body, include the names of the datastream configuration options as parameters. Specify a new value for each option.
Send the request.

As a result, you configured the datastream.

Import the request example as raw text to your HTTP client (such as Postman). The cURL request example is the following:

curl --location --request PATCH 'https://{{INSTANCE}}/api/datastream-types/{{DATASTREAM_TYPE}}/datastreams/{{DATASTREAM_ID}}/' \
--header 'Content-Type: application/json' \
--header 'Authorization: Token {{KEY}}' \
--data-raw '{"name": "NEW_VALUE"}'

Note

Replace Token with Bearer in the Authorization header if you are using a key generated in the Adverity user interface.

Listing options for filters#

When configuring filters in a datastream, you can only choose from a limited set of options. For example, in the Advertiser filter, you can only choose from a specific list of advertisers. The list is populated by advertisers about which the authorization you use to connect to the data source is authorized to collect data.

To obtain the list of options for a filter, given a datastream type and an authorization, follow these steps:

Determine the ID of the authorization you use to connect to the data source.
Determine the Datastream Type’s ID.
Determine the Datastream Type options.

In the response, search for the filter you want to configure. Fields that are auth-specific include "is_auth_scoped": true. For example, the result may contain the following information about the advertisers filter:

"advertisers": {
    "type": "multiple choice",
    "required": false,
    "read_only": false,
    "label": "Advertisers",
    "is_auth_scoped": true,
    "help_text": "If left blank, data for all advertisers available will be retrieved.",
    "lookup": {
        "url": "/adform/adformdatastream/advertisers/",
        "arguments": [
            "auth"
        ]
    }
},

Note the value of the url parameter in the response.

Create a GET request to the following endpoint:
```
https://{{INSTANCE}}/URL_FOUND_IN_RESPONSE_OF_PREVIOUS_STEP/?auth=CONNECTION_ID
```
Replace URL_FOUND_IN_RESPONSE_OF_PREVIOUS_STEP with the value of the url parameter from the previous step, and replace CONNECTION_ID with the ID of the authorization you use to connect to the data source.
In the HTTP request header, include the parameter Authorization with one of the following values:
- Token {{KEY}} if you use a key generated with user credentials in Management API.
- Bearer {{KEY}} if you use a key generated in the Adverity user interface.
Send the request.

As a result, you obtain the list of options for the filters, given this datastream type and authorization.

Import the request example as raw text to your HTTP client (such as Postman). The cURL request example is the following:

curl --location --request GET 'https://{{INSTANCE}}/adform/adformdatastream/advertisers/?auth=123' \
--header 'Authorization: Token {{KEY}}'

Note

The URL for this request is dynamic and depends on the connector. The example above is for the Adform connector. Replace Token with Bearer in the Authorization header if you are using a key generated in the Adverity user interface.

Example of JSON response:

{
    "err": "nil",
    "results": [
        {
            "id": 123,
            "text": "Advertiser1"
        },
        {
            "id": 124,
            "text": "Advertiser2"
        }
    ]
}

Configuring local data retention#

Adverity lets you control how long collected data is retained locally. You can set three retention behaviour flags and a retention policy on any existing datastream using the Management API.

Retention behaviour flags#

Three boolean fields control how Adverity handles empty and duplicate data during retention:

Field	Description
`remove_if_empty`	When `true`, Adverity removes the data extract if it contains no rows after processing.
`skip_empty_rows`	When `true`, Adverity skips empty rows when loading data.
`prune_instantly`	When `true`, Adverity removes expired data immediately rather than waiting for a scheduled cleanup job.

To set or update these flags, follow these steps:

Create a PATCH request to the following endpoint:
```
https://{{INSTANCE}}/api/datastreams/{{DATASTREAM_ID}}/
```
Replace {{DATASTREAM_ID}} with the datastream’s ID.
In the HTTP request header, include the parameter Authorization with one of the following values:
- Token {{KEY}} if you use a key generated with user credentials in Management API.
- Bearer {{KEY}} if you use a key generated in the Adverity user interface.
In the HTTP request header, include the parameter Content-Type with value application/json.

In the HTTP request body, include the retention behaviour flags you want to configure:

{
    "remove_if_empty": true,
    "skip_empty_rows": true,
    "prune_instantly": true
}

Send the request.

As a result, the retention behaviour flags are updated for the datastream.

Import the request example as raw text to your HTTP client (such as Postman). The cURL request example is the following:

curl --location --request PATCH 'https://{{INSTANCE}}/api/datastreams/{{DATASTREAM_ID}}/' \
--header 'Authorization: Token {{KEY}}' \
--header 'Content-Type: application/json' \
--data-raw '{"remove_if_empty": true, "skip_empty_rows": true, "prune_instantly": true}'

Note

If you have generated your API key in the Adverity user interface, replace Token with Bearer in the Authorization header.

Note

This endpoint (/api/datastreams/) is the generic datastream endpoint that accepts any datastream ID directly, without requiring the datastream type ID. It is different from the per-type endpoint (/api/datastream-types/{{DATASTREAM_TYPE}}/datastreams/) used for other datastream configuration options.

Retention policy#

The retention policy controls how long Adverity keeps collected data. Use retention_type to choose the policy and retention_number to specify the quantity when required.

Field	Value	Description
`retention_type`	`1`	Retain all data indefinitely. `retention_number` is not required.
`retention_type`	`2`	Retain the last N fetches. `retention_number` is required.
`retention_type`	`3`	Retain data for the last N days. `retention_number` is required.
`retention_type`	`4`	Retain the last N data extracts. `retention_number` is required.

Important

When retention_type is any value other than 1 (Retain all), also include retention_number in the request. Sending a retention_type that requires a number without providing retention_number — or sending "retention_number": null while a non-retain-all type is set — returns a 400 response.

To set the retention policy, include retention_type and, when required, retention_number in the same PATCH request body:

{
    "retention_type": 3,
    "retention_number": 30
}

Import the request example as raw text to your HTTP client (such as Postman). The cURL request example is the following:

curl --location --request PATCH 'https://{{INSTANCE}}/api/datastreams/{{DATASTREAM_ID}}/' \
--header 'Authorization: Token {{KEY}}' \
--header 'Content-Type: application/json' \
--data-raw '{"retention_type": 3, "retention_number": 30}'

Note

If you have generated your API key in the Adverity user interface, replace Token with Bearer in the Authorization header.

Note

These same retention fields (remove_if_empty, skip_empty_rows, prune_instantly, retention_type, retention_number) are also accepted when creating a datastream. Include them in the POST request body described in Creating datastreams.

Deleting a datastream#

To delete a datastream, follow these steps:

Create a DELETE request to the following endpoint:
```
https://{{INSTANCE}}/api/datastream-types/{{DATASTREAM_TYPE}}/datastreams/{{DATASTREAM_ID}}/
```
- Replace {{DATASTREAM_TYPE}} with the datastream type’s ID. For more information on finding a datastream type’s ID, see Determining a Datastream Type’s ID.
- Replace {{DATASTREAM_ID}} with the datastream’s ID.
In the HTTP request header, include the parameter Authorization with one of the following values:
- Token {{KEY}} if you use a key generated with user credentials in Management API.
- Bearer {{KEY}} if you use a key generated in the Adverity user interface.
Send the request.

As a result, you deleted the datastream.

Import the request example as raw text to your HTTP client (such as Postman). The cURL request example is the following:

curl --location --request DELETE 'https://{{INSTANCE}}/api/datastream-types/{{DATASTREAM_TYPE}}/datastreams/{{DATASTREAM_ID}}/' \
--header 'Authorization: Token {{KEY}}'

Note

Replace Token with Bearer in the Authorization header if you are using a key generated in the Adverity user interface.

Configuring datastreams in Management API

Contents

Configuring datastreams in Management API#

Listing datastreams of a specific type#

Determining current datastream configuration#

Retrieving the assigned destinations#

Determining datastream configuration options#

Configuring a datastream#

Listing options for filters#

Configuring local data retention#

Retention behaviour flags#

Retention policy#

Deleting a datastream#