Configuring advanced datastream settings

This guide explains how to configure advanced settings for datastreams.

Configure advanced data collection settings for a datastream to specify how Adverity collects data from the datastream.

This video guide explains what local data retention is and how to configure related settings.

Configuring advanced data collection settings

To configure advanced data collection settings for a datastream, follow these steps:

1. Go to the Datastreams page.

2. Open the chosen datastream by clicking on its name.

3. In the top navigation panel, click Local Data Retention.

4. Fill in the following fields:

   Extract Filenames

   Select one of the following options:

   • Unique by fetch - choose this option to create one data extract for all data collected during a fetch.

   • Unique by day - choose this option to create a different data extract for each day of the time range specified in a fetch. The date is determined by the column set as a Key date column for the datastream. Use this option to ensure that the data collected by the datastream is distinct and there are no duplicate rows.

   Key Date Column

   Select the date column in your datastream.

   Retention type

   Specify how Adverity retains fetched data. Choose one of the following options:

   • To retain all data indefinitely, select Retain All.

   • To retain data for a specific number of fetches, select Retain N fetches and specify the number of fetches in the Retention Number field.

   • To retain data for a specific number of data extracts, select Retain N extracts and specify the number of data extracts in the Retention Number field.

   • To retain data for a specific number of days after it is fetched, select Retain N days and specify the number of days in the Retention Number field.

   Remove if empty

   Check this box to automatically delete data extracts without any rows when the fetch runs.

   Prune instantly

   Deleted data extracts are retained for a grace period by default. Check this box to permanently erase deleted data extracts without the usual grace period.

5. Click Save.

6. In the top navigation panel, click Settings.

7. Fill in the following fields:

   These settings are hidden by default. To show these settings, select the Show "Share with child workspaces" and "Fetch data sequentially" checkbox in the workspace settings.

   Fetch data sequentially (for advanced users only)

   Fetch jobs are executed simultaneously by default. Check this box to execute fetch jobs in sequential order, one after another.

   Share with children

   By default, child workspaces cannot access the datastreams of their parent workspace. Check this box to make this datastream available for all child workspaces of the workspace you currently use.

8. In the Monitors section, select the Add status column checkbox to include the status of the data quality monitors assigned to this datastream in the data extract.

   This section is available only if you have access to Adverity Data Quality Suite.

   When this checkbox is selected, Adverity adds a new column to the data extract for each of the assigned monitors with the severity set to Warning. Status columns are named following this convention: dt_monitor_{{monitor’s name}}_status.

   The new column includes the monitor's status for each data row out of the following options:

   • passed

   • failed

9. Click Save.