Collecting data in Management API

Collecting data in Management API#

This article documents how to collect data and retrieve a data extract with the Adverity Management API.

Prerequisites#

Before you complete the procedure in this guide, perform all of the following actions:

Fetching data#

There are two options on how to collect data from datastreams:

Required body parameters#

start

The first day of the date range for which to collect data (expressed as UTC).

end

The last day of the date range for which to collect data (expressed as UTC).

Optional body parameters#

callback

Include a URL that is called once the fetch has been completed. Callback information you receive at the specified URL includes the filename, creation date, datastream name, Stack name, file size, number of collected rows, number of collected columns, and fetch status. An example of the string you may include in the request body is the following: "callback":"https://CALLBACK_URL"

priority

To change the priority of a task, use the priority parameter in the request body. To specify where the new task is placed in the task queue, use one of the following:

  • Use "priority": "high" to assign the task a high priority. Adverity places the task at the top after tasks with high priority.

  • Use "priority": "default" to assign the task the default priority set in your Adverity instance. This is usually high priority.

  • Use "priority": "low" to assign the task a low priority. Adverity places the task at the end of the task queue.

  • Use "priority": "backdata" for background data updates. Adverity will perform the task when nothing else is running.

Running a fetch#

By running a single fetch, you create one data extract with data in the defined date range from start to end. To run a single fetch of the data from a datastream, follow these steps:

  1. Create a POST request to the following endpoint:

    https://{{INSTANCE}}/api/datastreams/{{DATASTREAM_ID}}/fetch_fixed/

  2. In the HTTP request header, include the parameter Authorization with one of the following values:

    • Token {{KEY}} if you use a key generated with user credentials in Management API.

    • Bearer {{KEY}} if you use a key generated in the Adverity user interface.

  3. In the HTTP request header, include the parameter Content-Type with value application/json.

  4. In the HTTP request body, include the date range of your fetch in the following format:

    {
    "start":"YYYY-MM-DDTHH:MM:SSZ",
    "end":"YYYY-MM-DDTHH:MM:SSZ"
}

  5. (Optional) In the HTTP request body, include the parameter priority with value {{PRIORITY}}.

  6. Send the request.

Example of a valid response:

{
    "status": "ok",
    "message": "Fetch scheduled.",
    "start": "YYYY-MM-DDTHH:MM:SSZ",
    "end": "YYYY-MM-DDTHH:MM:SSZ",
    "jobs": [
        {
            "id": 123,
            "url": "https://{{INSTANCE}}/api/jobs/123/"
        }
    ]
}

Import the request example as raw text to your HTTP client (such as Postman). The cURL request example is the following:

curl --location --request POST 'https://{{INSTANCE}}/api/datastreams/{{DATASTREAM_ID}}/fetch_fixed/' \
--header 'Content-Type: application/json' \
--header 'Authorization: Token {{KEY}}' \
--data-raw '{
    "start":"YYYY-MM-DDTHH:MM:SSZ",
    "end":"YYYY-MM-DDTHH:MM:SSZ",
    "priority":"{{PRIORITY}}"
}'

Note

If you have generated your API key in the Adverity user interface, replace Token with Bearer in the Authorization header.

Note

Fetches triggered via the Management API are labeled the same as scheduled fetches. To distinguish between a manual fetch (from the UI or API) and a scheduled fetch, check the datastream_extract_scheduled field in the extract metadata. If the time is 00:00:00, the fetch was manual. If it shows a specific time (for example, 03:10:00), the fetch was scheduled.

Note

To configure the date range of your fetch and schedule jobs to periodically trigger fetches, see Scheduling fetches.

Triggering scheduled fetch jobs#

To execute all scheduled fetch jobs within the selected date range, follow these steps:

  1. Create a POST request to the following endpoint:

    https://{{INSTANCE}}/api/datastreams/{{DATASTREAM_ID}}/fetch/

  2. In the HTTP request header, include the parameter Authorization with one of the following values:

    • Token {{KEY}} if you use a key generated with user credentials in Management API.

    • Bearer {{KEY}} if you use a key generated in the Adverity user interface.

  3. In the HTTP request header, include the parameter Content-Type with value application/json.

  4. In the HTTP request body, include the date range of your fetch in the following format:

    {
    "start":"YYYY-MM-DDTHH:MM:SSZ",
    "end":"YYYY-MM-DDTHH:MM:SSZ"
}

  5. Send the request.

Import the request example as raw text to your HTTP client (such as Postman). The cURL request example is the following:

curl --location --request POST 'https://{{INSTANCE}}/api/datastreams/{{DATASTREAM_ID}}/fetch/' \
--header 'Authorization: Token {{KEY}}' \
--header 'Content-Type: application/json' \
--data-raw '{
    "start":"YYYY-MM-DDTHH:MM:SSZ",
    "end":"YYYY-MM-DDTHH:MM:SSZ"
}'\

}'

Note

If you have generated your API key in the Adverity user interface, replace Token with Bearer in the Authorization header.

Note

Scheduled fetch jobs within the selected date range are different from scheduling, where you can set up periodic fetch jobs. For more information, see Scheduling fetches.

Downloading the most recent data extract#

To download the most recent data extract of a specific datastream in CSV format, follow these steps:

  1. Create a GET request to the following endpoint:

    https://{{INSTANCE}}/api/datastreams/{{DATASTREAM_ID}}/extracts/most-recent/download/

  2. In the HTTP request header, include the parameter Authorization with one of the following values:

    • Token {{KEY}} if you use a key generated with user credentials in Management API.

    • Bearer {{KEY}} if you use a key generated in the Adverity user interface.

  3. Send the request.

Import the request example as raw text to your HTTP client (such as Postman). The cURL request example is the following:

curl --location --request GET 'https://{{INSTANCE}}/api/datastreams/{{DATASTREAM_ID}}/extracts/most-recent/download/' \
--header 'Authorization: Token {{KEY}}'

Note

If you have generated your API key in the Adverity user interface, replace Token with Bearer in the Authorization header.

Note

By default, you download the data extract in CSV format. To download the data extract in XLSX or JSON format, use the following endpoints:

  • For XLSX format:

    https://{{INSTANCE}}/api/datastreams/{{DATASTREAM_ID}}/extracts/most-recent/download/?fileformat=xlsx

  • For JSON format:

    https://{{INSTANCE}}/api/datastreams/{{DATASTREAM_ID}}/extracts/most-recent/download/?fileformat=json

Downloading a specific data extract#

Obtaining extract ID#

To download a specific data extract, you need to know the extract ID. To obtain the extract ID, follow these steps:

  1. Create a GET request to the following endpoint:

    https://{{INSTANCE}}/api/extracts/?datastream_id={{DATASTREAM_ID}}

  2. In the HTTP request header, include the parameter Authorization with one of the following values:

    • Token {{KEY}} if you use a key generated with user credentials in Management API.

    • Bearer {{KEY}} if you use a key generated in the Adverity user interface.

  3. Send the request.

You can also filter extracts by creation date using the created_from and created_to query parameters (both expressed as UTC):

https://{{INSTANCE}}/api/extracts/?datastream_id={{DATASTREAM_ID}}&created_from=YYYY-MM-DDTHH:MM:SS&created_to=YYYY-MM-DDTHH:MM:SS

The cURL request example is the following:

curl --location --request GET \
  'https://{{INSTANCE}}/api/extracts/?datastream_id={{DATASTREAM_ID}}&created_from=YYYY-MM-DDTHH:MM:SS&created_to=YYYY-MM-DDTHH:MM:SS' \
  --header 'Authorization: Token {{KEY}}'

Note

If you include only created_from, the response returns all extracts created on or after that datetime. Include created_to to set an upper bound. All timestamps in the Management API are in UTC.

Note

There are more ways in which you can retrieve information about fetches and fetch IDs. For more examples of possible API requests, see the Postman collection.

In the response, find the ID of the fetch job (first occurrence of the parameter id). In the procedure described below, use this ID as the {{EXTRACT_ID}} to retrieve the data extract.

Example of a part of a valid response:

{
    "count": null,
    "next": null,
    "previous": null,
    "results": [
        {
            "id": 123123,
            "created": "YYYY-MM-DDTHH:MM:SSZ",
            "datastream": {
                "id": 123,
                "stack": {
                    "id": 123,
                    "name": "name",
                    "slug": "name",
(...)

The first ID marked in red is the {{EXTRACT_ID}} needed for the procedure described in the following section. Use the JSONPath $.results[*].id to filter for all data extract IDs in your response. The first ID is the ID of the latest fetch. The last ID is the ID of the oldest fetch.

Import the request example as raw text to your HTTP client (such as Postman). The cURL request example is the following:

curl --location --request GET 'https://{{INSTANCE}}/api/extracts/?datastream_id={{DATASTREAM_ID}}' \
--header 'Authorization: Token {{KEY}}'

Note

If you have generated your API key in the Adverity user interface, replace Token with Bearer in the Authorization header.

Downloading a specific data extract#

To download a specific data extract using the {{EXTRACT_ID}}, follow these steps:

  1. Create a GET request to the following endpoint:

    https://{{INSTANCE}}/api/extracts/{{EXTRACT_ID}}/download/

  2. In the HTTP request header, include the parameter Authorization with one of the following values:

    • Token {{KEY}} if you use a key generated with user credentials in Management API.

    • Bearer {{KEY}} if you use a key generated in the Adverity user interface.

  3. Send the request.

Import the request example as raw text to your HTTP client (such as Postman). The cURL request example is the following:

curl --location --request GET 'https://{{INSTANCE}}/api/extracts/{{EXTRACT_ID}}/download/' \
--header 'Authorization: Token {{KEY}}'

Note

If you have generated your API key in the Adverity user interface, replace Token with Bearer in the Authorization header.

Note

By default, you download the data extract in CSV format. To download the data extract in XLSX or JSON format, use the following endpoints:

  • For XLSX format:

    https://{{INSTANCE}}/api/extracts/{{EXTRACT_ID}}/download/?fileformat=xlsx

  • For JSON format:

    https://{{INSTANCE}}/api/extracts/{{EXTRACT_ID}}/download/?fileformat=json
Contents