Configuring datastreams in Management API¶

This guide explains how to configure and edit datastreams with the Management API.

See GET /api/datastreams/{id}/ and PATCH /api/datastream-types/{parent_lookup_type_id}/datastreams/{id}/ in the API reference.

Prerequisites¶

Before you complete the procedures in this guide, perform the following action:

Use a Connect API key with the following permissions:
- Datastream: Read-only (datastream:read) to list and view datastreams.
- Datastream: Write (datastream:write) to update or delete datastreams.
Picking Write for a permission also gives the key its Read-only access. Without the permission a request needs, Management API returns HTTP 403. You cannot change the permissions of a key, so to add one, create a new key. For more information, see Creating a Connect API key in Adverity.

Listing datastreams of a specific type¶

To list datastreams of a specific datastream type, follow these steps:

Create a GET request to the following endpoint:
```
https://{{INSTANCE}}/api/datastream-types/{{DATASTREAM_TYPE}}/datastreams/
```
Replace {{DATASTREAM_TYPE}} with the datastream type’s ID. For more information on determining a datastream type’s ID, see Determining a Datastream Type’s ID.
In the HTTP request header, include the parameter Authorization with value Bearer {{KEY}}.

Optionally, add query parameters to filter the results:

Parameter	Description
`auth`	One or more authorization IDs. Returns only the datastreams that use one of the given authorizations. Separate multiple values with commas, for example `auth=1,2,3`.
`creator`	The username of the datastream’s creator.
`ids`	One or more datastream IDs. Separate multiple values with commas, for example `ids=1,2,3`.
`name`	Filter by datastream name. Matches names that contain this value.
`stack_id`	The ID of the workspace the datastream belongs to.
`transformer`	The ID of a transformation assigned to the datastream.
`scheduled_deletion`	Filter by scheduled deletion state. Use `true` to return only the datastreams that are scheduled for deletion, or `false` to exclude them. Omit the parameter to return both.

Send the request.

As a result, you obtain the list of datastreams of a specific datastream type.

Note

These same query parameters, including auth, are also accepted by the generic GET /api/datastreams/ endpoint, which lists datastreams across datastream types.

Import the request example as raw text to your HTTP client (such as Postman). The cURL request example is the following:

curl --location --request GET 'https://{{INSTANCE}}/api/datastream-types/{{DATASTREAM_TYPE}}/datastreams/' \
--header 'Authorization: Bearer {{KEY}}'

Example of JSON response:

{
    "count": 15,
    "next": null,
    "previous": null,
    "results": [
        {
            "id": {{DATASTREAM_ID}},
            "uuid": "{{DATASTREAM_UUID}}",
            "cron_type": "day",
            "cron_interval": 1,
            "cron_start_of_day": "00:30:00",
            "cron_interval_start": 1,
            "time_range_preset": 13,
            "delta_type": 2,
            "delta_interval": 1,
            "delta_interval_start": 1,
            "delta_start_of_day": "00:00:00",
            "datatype": "Staging",
            "creator": "jane@doe.com",
            "datastream_type_id": {{DATASTREAM_TYPE}},
            "absolute_url": "https://{{INSTANCE}}/adform/{{DATASTREAM_ID}}/adform-buyer/"
        }
    ]
}

The value in an id parameter of the response is a datastream’s ID.

Note

The response also includes a uuid field for each datastream. The endpoints under /api/v1/ identify datastreams by this UUID instead of the integer ID. The generic GET /api/datastreams/ endpoint, which lists datastreams across datastream types, returns both id and uuid too. Either list lets you map integer IDs to the /api/v1/ endpoints without one request per datastream.

Note

The datatype parameter reflects the state of the datastream’s destinations. A read returns "Live" if at least one destination is enabled, and "Staging" if all of them are disabled. You can also write datatype when you edit a datastream: setting "Live" enables all destinations, and setting "Staging" disables them.

Determining current datastream configuration¶

To determine a datastream’s current configuration, follow these steps:

Create a GET request to the following endpoint:
```
https://{{INSTANCE}}/api/datastream-types/{{DATASTREAM_TYPE}}/datastreams/{{DATASTREAM_ID}}/
```
Replace {{DATASTREAM_ID}} with the datastream’s ID.
In the HTTP request header, include the parameter Authorization with value Bearer {{KEY}}.
Send the request.

As a result, you determined the current datastream configuration.

Import the request example as raw text to your HTTP client (such as Postman). The cURL request example is the following:

curl --location --request GET 'https://{{INSTANCE}}/api/datastream-types/{{DATASTREAM_TYPE}}/datastreams/{{DATASTREAM_ID}}/' \
--header 'Authorization: Bearer {{KEY}}'

Retrieving the assigned destinations¶

By default, a datastream detail response does not include the destinations that the datastream delivers to. To include them, add the expand query parameter with the value destinations to either datastream detail endpoint:

GET /api/datastreams/{{DATASTREAM_ID}}/
GET /api/datastream-types/{{DATASTREAM_TYPE}}/datastreams/{{DATASTREAM_ID}}/

Both endpoints return the same assigned_destinations content for the same datastream.

To retrieve a datastream’s assigned destinations, follow these steps:

Create a GET request to one of the following endpoints:
```
https://{{INSTANCE}}/api/datastreams/{{DATASTREAM_ID}}/?expand=destinations
```
Replace {{DATASTREAM_ID}} with the datastream’s ID.
In the HTTP request header, include the parameter Authorization with value Bearer {{KEY}}.
Send the request.

As a result, the response includes an assigned_destinations array that describes the destinations assigned to the datastream.

Import the request example as raw text to your HTTP client (such as Postman). The cURL request example is the following:

curl --location -g --request GET 'https://{{INSTANCE}}/api/datastreams/{{DATASTREAM_ID}}/?expand=destinations' \
--header 'Authorization: Bearer {{KEY}}'

Each object in the assigned_destinations array contains the following fields:

Field	Description
`target_id`	The numeric ID of the destination.
`destination_name`	The name of the destination.
`destination_type`	The type of the destination, for example `bigquerytarget`.
`enabled`	Boolean. `true` if the binding is enabled, `false` if it is disabled.
`destination_settings`	An object with destination-specific settings. The fields depend on the destination type. For more information, see the table below.

Example of the assigned_destinations array in a response:

"assigned_destinations": [
    {
        "target_id": 123,
        "destination_name": "My BigQuery",
        "destination_type": "bigquerytarget",
        "enabled": true,
        "destination_settings": {
            "table_name": "analytics.spend",
            "write_disposition": "WRITE_TRUNCATE",
            "partition_by_date": true,
            "partition_column_name": "date",
            "expires_in": null,
            "overwrite_key_columns": false
        }
    }
]

The destination_settings object contains a curated set of settings that depends on the destination type:

destination type	`destination_settings` fields
SQL and Snowflake	`table_name`, `truncate`, `overwrite_key_columns`, `overwrite_datastream`, `overwrite_date_range_column_name`
BigQuery	`table_name`, `write_disposition` (`WRITE_APPEND`, `WRITE_TRUNCATE`, or `WRITE_EMPTY`), `partition_by_date`, `partition_column_name`, `expires_in`, `overwrite_key_columns`
Google Sheets	`filename`, `sheet`
Databricks and Delta Lake	`table_name`
Warehouse (managed by Adverity)	`ingestion_mode`, `row_date_column_name`, `overwrite_key_columns`, `datasource_label`, `overwrite_datastream`, `overwrite_filename`, `overwrite_date_range_column_name`, and workspace-mapping fields
Adverity Insights	`datasource_label`, `overwrite_key_columns`, `overwrite_datastream`, `overwrite_filename`, `overwrite_date_range_column_name`, and workspace-mapping fields
File and audience destinations	No settings. The `destination_settings` object is empty.
Any other destination type	No settings. The `destination_settings` object is empty.

Note

The assigned_destinations array includes disabled bindings ("enabled": false), but it does not include system-invalidated bindings. When you omit the expand parameter, the response does not include the assigned_destinations array and is otherwise unchanged.

Determining datastream configuration options¶

To determine the options you can configure for a datastream, follow these steps:

Create an OPTIONS request to the following endpoint: Replace {{DATASTREAM_ID}} with the datastream’s ID.
```
https://{{INSTANCE}}/api/datastream-types/{{DATASTREAM_TYPE}}/datastreams/{{DATASTREAM_ID}}/
```
In the HTTP request header, include the parameter Authorization with value Bearer {{KEY}}.
Send the request.

As a result, you obtained the list of options you can configure for the datastream.

Import the request example as raw text to your HTTP client (such as Postman). The cURL request example is the following:

curl --location --request OPTIONS 'https://{{INSTANCE}}/api/datastream-types/{{DATASTREAM_TYPE}}/datastreams/{{DATASTREAM_ID}}/' \
--header 'Authorization: Bearer {{KEY}}'

In the response, search for specific parameters and objects to edit your datastream configuration. If a specific object includes the parameter read_only with the value true, you cannot edit it. "read_only": false means that you can edit a particular object. Fields that are auth-specific (such as accounts, campaigns, or advertisers) include "is_auth_scoped": true, indicating that the available options depend on the authorization used.

Configuring a datastream¶

To configure a datastream, follow these steps:

Create a PATCH request to the following endpoint:
```
https://{{INSTANCE}}/api/datastream-types/{{DATASTREAM_TYPE}}/datastreams/{{DATASTREAM_ID}}/
```
Replace {{DATASTREAM_ID}} with the datastream’s ID.
In the HTTP request header, include the parameter Authorization with value Bearer {{KEY}}.
In the HTTP request header, include the parameter Content-Type with value application/json.
In the HTTP request body, include the names of the datastream configuration options as parameters. Specify a new value for each option.
Send the request.

As a result, you configured the datastream.

Import the request example as raw text to your HTTP client (such as Postman). The cURL request example is the following:

curl --location --request PATCH 'https://{{INSTANCE}}/api/datastream-types/{{DATASTREAM_TYPE}}/datastreams/{{DATASTREAM_ID}}/' \
--header 'Content-Type: application/json' \
--header 'Authorization: Bearer {{KEY}}' \
--data-raw '{"name": "NEW_VALUE"}'

Listing options for filters¶

When configuring filters in a datastream, you can only choose from a limited set of options. For example, in the Advertiser filter, you can only choose from a specific list of advertisers. The list is populated by advertisers about which the authorization you use to connect to the data source is authorized to collect data.

To obtain the list of options for a filter, given a datastream type and an authorization, follow these steps:

Determine the ID of the authorization you use to connect to the data source.
Determine the Datastream Type’s ID.
Determine the Datastream Type options.

In the response, search for the filter you want to configure. Fields that are auth-specific include "is_auth_scoped": true. For example, the result may contain the following information about the advertisers filter:

"advertisers": {
    "type": "multiple choice",
    "required": false,
    "read_only": false,
    "label": "Advertisers",
    "is_auth_scoped": true,
    "help_text": "If left blank, data for all advertisers available will be retrieved.",
    "lookup": {
        "url": "/adform/adformdatastream/advertisers/",
        "arguments": [
            "auth"
        ]
    }
},

Note the value of the url parameter in the response.

Create a GET request to the following endpoint:
```
https://{{INSTANCE}}/URL_FOUND_IN_RESPONSE_OF_PREVIOUS_STEP/?auth=CONNECTION_ID
```
Replace URL_FOUND_IN_RESPONSE_OF_PREVIOUS_STEP with the value of the url parameter from the previous step, and replace CONNECTION_ID with the ID of the authorization you use to connect to the data source.
In the HTTP request header, include the parameter Authorization with value Bearer {{KEY}}.
Send the request.

As a result, you obtain the list of options for the filters, given this datastream type and authorization.

Import the request example as raw text to your HTTP client (such as Postman). The cURL request example is the following:

curl --location --request GET 'https://{{INSTANCE}}/adform/adformdatastream/advertisers/?auth=123' \
--header 'Authorization: Bearer {{KEY}}'

Note

The URL for this request is dynamic and depends on the connector. The example above is for the Adform connector.

Example of JSON response:

{
    "err": "nil",
    "results": [
        {
            "id": 123,
            "text": "Advertiser1"
        },
        {
            "id": 124,
            "text": "Advertiser2"
        }
    ]
}

Configuring local data retention¶

Adverity lets you control how long collected data is retained locally. You can set three retention behaviour flags and a retention policy on any existing datastream using the Management API.

Retention behaviour flags¶

Three boolean fields control how Adverity handles empty and duplicate data during retention:

Field	Description
`remove_if_empty`	When `true`, Adverity removes the data extract if it contains no rows after processing.
`skip_empty_rows`	When `true`, Adverity skips empty rows when loading data.
`prune_instantly`	When `true`, Adverity removes expired data immediately rather than waiting for a scheduled cleanup job.

To set or update these flags, follow these steps:

Create a PATCH request to the following endpoint:
```
https://{{INSTANCE}}/api/datastreams/{{DATASTREAM_ID}}/
```
Replace {{DATASTREAM_ID}} with the datastream’s ID.
In the HTTP request header, include the parameter Authorization with value Bearer {{KEY}}.
In the HTTP request header, include the parameter Content-Type with value application/json.

In the HTTP request body, include the retention behaviour flags you want to configure:

{
    "remove_if_empty": true,
    "skip_empty_rows": true,
    "prune_instantly": true
}

Send the request.

As a result, the retention behaviour flags are updated for the datastream.

Import the request example as raw text to your HTTP client (such as Postman). The cURL request example is the following:

curl --location --request PATCH 'https://{{INSTANCE}}/api/datastreams/{{DATASTREAM_ID}}/' \
--header 'Authorization: Bearer {{KEY}}' \
--header 'Content-Type: application/json' \
--data-raw '{"remove_if_empty": true, "skip_empty_rows": true, "prune_instantly": true}'

Note

This endpoint (/api/datastreams/) is the generic datastream endpoint that accepts any datastream ID directly, without requiring the datastream type ID. It is different from the per-type endpoint (/api/datastream-types/{{DATASTREAM_TYPE}}/datastreams/) used for other datastream configuration options.

Note

On this endpoint, stack, app_label, absolute_url, and next_run are read-only. Including them in a PATCH or PUT request body has no effect — Adverity ignores them rather than applying any value you send. To move a datastream to a different workspace, send stack_id with the target workspace’s ID instead.

Retention policy¶

The retention policy controls how long Adverity keeps collected data. Use retention_type to choose the policy and retention_number to specify the quantity when required.

Field	Value	Description
`retention_type`	`1`	Retain all data indefinitely. `retention_number` is not required.
`retention_type`	`2`	Retain the last N fetches. `retention_number` is required.
`retention_type`	`3`	Retain data for the last N days. `retention_number` is required.
`retention_type`	`4`	Retain the last N data extracts. `retention_number` is required.

Important

When retention_type is any value other than 1 (Retain all), also include retention_number in the request. Sending a retention_type that requires a number without providing retention_number — or sending "retention_number": null while a non-retain-all type is set — returns a 400 response.

To set the retention policy, include retention_type and, when required, retention_number in the same PATCH request body:

{
    "retention_type": 3,
    "retention_number": 30
}

Import the request example as raw text to your HTTP client (such as Postman). The cURL request example is the following:

curl --location --request PATCH 'https://{{INSTANCE}}/api/datastreams/{{DATASTREAM_ID}}/' \
--header 'Authorization: Bearer {{KEY}}' \
--header 'Content-Type: application/json' \
--data-raw '{"retention_type": 3, "retention_number": 30}'

Note

These same retention fields (remove_if_empty, skip_empty_rows, prune_instantly, retention_type, retention_number) are also accepted when creating a datastream. Include them in the POST request body described in Creating datastreams.

Deleting a datastream¶

To delete a datastream, follow these steps:

Create a DELETE request to the following endpoint:
```
https://{{INSTANCE}}/api/datastream-types/{{DATASTREAM_TYPE}}/datastreams/{{DATASTREAM_ID}}/
```
- Replace {{DATASTREAM_TYPE}} with the datastream type’s ID. For more information on finding a datastream type’s ID, see Determining a Datastream Type’s ID.
- Replace {{DATASTREAM_ID}} with the datastream’s ID.
In the HTTP request header, include the parameter Authorization with value Bearer {{KEY}}.
Send the request.

As a result, you deleted the datastream.

Import the request example as raw text to your HTTP client (such as Postman). The cURL request example is the following:

curl --location --request DELETE 'https://{{INSTANCE}}/api/datastream-types/{{DATASTREAM_TYPE}}/datastreams/{{DATASTREAM_ID}}/' \
--header 'Authorization: Bearer {{KEY}}'