Managing storage in Management API

Managing storage in Management API#

This guide explains how to manage storage entries using the Management API. In Adverity, storage entries appear under Settings → Storage and are used to stage data extracts before loading them into a destination such as Google BigQuery.

Prerequisites#

Before you complete the procedure in this guide, perform all of the following actions:

  • Ensure your API key belongs to a user with the Administrator workspace role. Only users with the Administrator role can create, update, or delete storage entries. Users without this role can only list storage entries.

  • Create a workspace and determine its WORKSPACE_ID. For more information, see Configuring workspaces in Management API.

  • For Google Cloud Storage entries, create a Google Cloud Storage authorization in Adverity and determine its AUTH_ID. For more information, see Creating and configuring authorizations in Management API.

Note

These endpoints require the Bearer scheme for the Authorization header.

Listing storage entries#

To list all storage entries, follow these steps:

  1. Create a GET request to the following endpoint:

    https://{{INSTANCE}}/api/storage/

  2. In the HTTP request header, include the parameter Authorization with the value Bearer {{KEY}}.

  3. Send the request.

Import the request example as raw text to your HTTP client (such as Postman). The cURL request example is the following:

curl --location --request GET 'https://{{INSTANCE}}/api/storage/' \
--header 'Authorization: Bearer {{KEY}}'

Creating a storage entry#

To create a new storage entry, follow these steps:

  1. Create a POST request to the following endpoint:

    https://{{INSTANCE}}/api/storage/

  2. In the HTTP request header, include the parameter Authorization with the value Bearer {{KEY}}.

  3. In the HTTP request header, include the parameter Content-Type with value application/json.

  4. In the HTTP request body, include the following parameters and values:

    name

    Use a unique name for the storage entry. The name must be unique across the entire tenant.

    stack

    Use the WORKSPACE_ID of the workspace this storage entry belongs to.

    url

    Use the full storage URL including the scheme. For example, gcs://my-bucket/exports or s3://my-bucket/path. Supported schemes: gcs, s3, azure, sftp, ftp, ftp+passive, smb.

    auth

    (Optional, Google Cloud Storage only) Use the AUTH_ID of the Google Cloud Storage authorization.

    Example of the request body:

    {
   "name": "NEW_STORAGE_NAME",
   "stack": {{WORKSPACE_ID}},
   "url": "gcs://my-bucket/exports",
   "auth": {{AUTH_ID}}
}

  5. Send the request.

A successful request returns HTTP 201 and the created record, including its id. Save this id if you intend to configure bulk loading for a Google BigQuery destination. For more information, see Enabling bulk loading for Google BigQuery in Management API.

Import the request example as raw text to your HTTP client (such as Postman). The cURL request example is the following:

curl --location --request POST 'https://{{INSTANCE}}/api/storage/' \
--header 'Authorization: Bearer {{KEY}}' \
--header 'Content-Type: application/json' \
--data-raw '{
    "name": "NEW_STORAGE_NAME",
    "stack": {{WORKSPACE_ID}},
    "url": "gcs://my-bucket/exports",
    "auth": {{AUTH_ID}}
}'

Updating a storage entry#

To update a storage entry, follow these steps:

  1. Create a PATCH request to the following endpoint:

    https://{{INSTANCE}}/api/storage/{{STORAGE_ID}}/

  2. In the HTTP request header, include the parameter Authorization with the value Bearer {{KEY}}.

  3. In the HTTP request header, include the parameter Content-Type with value application/json.

  4. In the HTTP request body, include the parameters you want to update.

  5. Send the request.

Import the request example as raw text to your HTTP client (such as Postman). The cURL request example is the following:

curl --location --request PATCH \
'https://{{INSTANCE}}/api/storage/{{STORAGE_ID}}/' \
--header 'Authorization: Bearer {{KEY}}' \
--header 'Content-Type: application/json' \
--data-raw '{"name": "UPDATED_STORAGE_NAME"}'

Deleting a storage entry#

To delete a storage entry, follow these steps:

  1. Create a DELETE request to the following endpoint:

    https://{{INSTANCE}}/api/storage/{{STORAGE_ID}}/

  2. In the HTTP request header, include the parameter Authorization with the value Bearer {{KEY}}.

  3. Send the request.

Import the request example as raw text to your HTTP client (such as Postman). The cURL request example is the following:

curl --location --request DELETE \
'https://{{INSTANCE}}/api/storage/{{STORAGE_ID}}/' \
--header 'Authorization: Bearer {{KEY}}'

Troubleshooting#

Symptom

Likely cause and action

403 Forbidden on POST, PATCH, or DELETE

Your API key belongs to a user without the Administrator workspace role. Use a key from an Administrator account.

400 “Auth is only valid for Google Cloud Storage.”

The auth field was set on a storage entry whose URL does not use the gcs:// scheme. Remove the auth field or change the URL to a gcs:// address.

400 mentioning name uniqueness

The name value is already used by another storage entry in the tenant. storage entry names must be unique across the entire tenant.
Contents