Managing pulling mode and smart schedule in Management API

Managing pulling mode and smart schedule in Management API#

This guide explains how to list datastreams and manage their pulling mode — switching between smart schedule and custom schedule — using the Management API.

Both schedule types coexist: switching to smart schedule does not delete your custom schedule configuration, so you can switch back at any time. For information on configuring custom fetch schedules, see Scheduling fetches.

Warning

PATCH and PUT requests accept optional query parameters to limit the scope of the operation. If no filter parameters are included, the request applies to all datastreams accessible to the API key. Always include the datastream_id parameter (or another filter) when you intend to update a specific datastream.

Prerequisites#

Before you complete the procedures in this guide, perform all of the following actions:

Generate an API key with the appropriate scope:
- datastream:read to list datastreams.
- datastream:write to update datastream configuration.
For more information on generating API keys, see Authorizing to Management API.
Create a datastream and note its datastream ID. For more information, see Creating datastreams.

Listing datastreams#

To retrieve a list of datastreams, follow these steps:

Create a GET request to the following endpoint:
```
https://{{INSTANCE}}/api/v1/datastreams/
```
In the HTTP request header, include the parameter Authorization with value Bearer {{KEY}}.

Optionally, add query parameters to filter the results:

Parameter	Description
`datastream_id`	One or more datastream UUIDs. Repeat the parameter for multiple values.
`workspace_id`	One or more workspace UUIDs. Repeat the parameter for multiple values.
`workspace_scope`	Scope of the workspace filter. Accepted values: `ancestors`, `descendants`, `family`.
`enabled`	Filter by enabled state. Accepted values: `true`, `false`.
`active_pulling_mode`	Filter by active pulling mode. Accepted values: `smart`, `smart-bundle`, `schedule`.

Send the request.

Import the request example as raw text to your HTTP client (such as Postman). The cURL request example is the following:

curl --request GET 'https://{{INSTANCE}}/api/v1/datastreams/' \
--header 'Authorization: Bearer {{KEY}}'

The example response below shows one datastream with smart schedule active:

{
  "count": 1,
  "items": [
    {
      "id": "{{DATASTREAM_UUID}}",
      "name": "{{DATASTREAM_NAME}}",
      "enabled": true,
      "workspace": {
        "id": "{{WORKSPACE_UUID}}",
        "name": "{{WORKSPACE_NAME}}"
      },
      "pulling": {
        "active_mode": "smart",
        "smart": {
          "is_active": true,
          "frequency": "daily",
          "refresh_range": {
            "value": 7,
            "unit": "day"
          },
          "deadline": null
        },
        "smart_bundle": null
      }
    }
  ]
}

The pulling.active_mode field indicates which schedule type is currently active for the datastream. Possible values are:

smart — smart schedule is active.
smart-bundle — a smart bundle schedule is active (deprecated).
schedule — a custom fetch schedule is active.
null — no schedule is configured.

Enabling or disabling datastreams#

To enable or disable one or more datastreams, follow these steps:

Create a PATCH request to the following endpoint:
```
https://{{INSTANCE}}/api/v1/datastreams/
```
In the HTTP request header, include the parameter Authorization with value Bearer {{KEY}}.
In the HTTP request header, include the parameter Content-Type with value application/json.
Optionally, add query parameters to filter which datastreams are affected. See the filter parameters described in Listing datastreams.
In the HTTP request body, include the enabled parameter set to true or false.
Send the request.

As a result, the specified datastreams are enabled or disabled. The response contains the UUIDs of all datastreams whose state actually changed.

Import the request example as raw text to your HTTP client (such as Postman). The cURL request example is the following:

curl --request PATCH 'https://{{INSTANCE}}/api/v1/datastreams/?datastream_id={{DATASTREAM_UUID}}' \
--header 'Authorization: Bearer {{KEY}}' \
--header 'Content-Type: application/json' \
--data-raw '{"enabled": false}'

The example response is the following:

{
  "affected": {
    "datastream_ids": ["{{DATASTREAM_UUID}}"]
  }
}

Switching to smart schedule#

To switch one or more datastreams to smart schedule, follow these steps:

Create a PATCH request to the following endpoint:
```
https://{{INSTANCE}}/api/v1/datastreams/
```
In the HTTP request header, include the parameter Authorization with value Bearer {{KEY}}.
In the HTTP request header, include the parameter Content-Type with value application/json.
Optionally, add query parameters to filter which datastreams are affected. See the filter parameters described in Listing datastreams.

In the HTTP request body, include the following payload:

{
  "pulling": {
    "active_mode": "smart"
  }
}

Send the request.

As a result, smart schedule is activated for the targeted datastreams. The response contains the UUIDs of all datastreams whose pulling mode actually changed.

Import the request example as raw text to your HTTP client (such as Postman). The cURL request example is the following:

curl --request PATCH 'https://{{INSTANCE}}/api/v1/datastreams/?datastream_id={{DATASTREAM_UUID}}' \
--header 'Authorization: Bearer {{KEY}}' \
--header 'Content-Type: application/json' \
--data-raw '{"pulling": {"active_mode": "smart"}}'

The example response is the following:

{
  "affected": {
    "datastream_ids": ["{{DATASTREAM_UUID}}"]
  }
}

Switching to custom schedule#

To switch one or more datastreams to a custom schedule, follow these steps:

Note

A datastream must already have a custom fetch schedule configured before you can switch it to schedule mode. If any datastream in the targeted set has no fetch schedule, the request returns HTTP 400 and no changes are applied. To configure a custom schedule, see Scheduling fetches.

Create a PATCH request to the following endpoint:
```
https://{{INSTANCE}}/api/v1/datastreams/
```
In the HTTP request header, include the parameter Authorization with value Bearer {{KEY}}.
In the HTTP request header, include the parameter Content-Type with value application/json.
Optionally, add query parameters to filter which datastreams are affected. See the filter parameters described in Listing datastreams.

In the HTTP request body, include the following payload:

{
  "pulling": {
    "active_mode": "schedule"
  }
}

Send the request.

As a result, custom schedule is activated for the targeted datastreams. The response contains the UUIDs of all datastreams whose pulling mode actually changed.

Import the request example as raw text to your HTTP client (such as Postman). The cURL request example is the following:

curl --request PATCH 'https://{{INSTANCE}}/api/v1/datastreams/?datastream_id={{DATASTREAM_UUID}}' \
--header 'Authorization: Bearer {{KEY}}' \
--header 'Content-Type: application/json' \
--data-raw '{"pulling": {"active_mode": "schedule"}}'

The example response is the following:

{
  "affected": {
    "datastream_ids": ["{{DATASTREAM_UUID}}"]
  }
}

Configuring smart-pulling settings#

To set or update the smart-pulling configuration for one or more datastreams, follow these steps:

Create a PUT request to the following endpoint:

https://{{INSTANCE}}/api/v1/datastreams/pulling/smart/

In the HTTP request header, include the parameter Authorization with value Bearer {{KEY}}.
In the HTTP request header, include the parameter Content-Type with value application/json.
Optionally, add query parameters to filter which datastreams are affected. See the filter parameters described in Listing datastreams.

In the HTTP request body, include the smart-pulling configuration. The available parameters are:

Parameter	Required	Description
`is_active`	Yes	Whether smart schedule is active. Accepted values: `true`, `false`.
`frequency`	Yes	How often data is fetched. Accepted values: `daily`, `weekly`, `monthly`.
`refresh_range`	No	The rolling date window to refresh on each fetch. Omit to use the default range. Contains: `value` — integer number of units. `unit` — time unit. Accepted values: `day`, `week`, `month`.
`deadline`	No	The time by which the fetch must complete. Omit to use no deadline. Contains: `time` — time in `HH:MM:SS` format, rounded to the half hour (for example, `10:00:00` or `10:30:00`).

Send the request.

As a result, the smart-pulling configuration is applied to all targeted datastreams. The response contains the UUIDs of datastreams whose configuration actually changed.

Import the request example as raw text to your HTTP client (such as Postman). The cURL request example is the following:

curl --request PUT 'https://{{INSTANCE}}/api/v1/datastreams/pulling/smart/?datastream_id={{DATASTREAM_UUID}}' \
--header 'Authorization: Bearer {{KEY}}' \
--header 'Content-Type: application/json' \
--data-raw '{
    "is_active": true,
    "frequency": "daily",
    "refresh_range": {
        "value": 7,
        "unit": "day"
    },
    "deadline": {
        "time": "06:00:00"
    }
}'

The example response is the following:

{
  "affected": {
    "datastream_ids": ["{{DATASTREAM_UUID}}"]
  }
}

Managing pulling mode and smart schedule in Management API

Contents

Managing pulling mode and smart schedule in Management API#

Prerequisites#

Listing datastreams#

Enabling or disabling datastreams#

Switching to smart schedule#

Switching to custom schedule#

Configuring smart-pulling settings#