Working with universal monitors

This guide explains how to use universal monitors.

Introduction

What are monitors and what can they do?

A data monitor is an automated data quality check that is performed each time you fetch data. With data monitors, you can find anomalies in your data more easily.

For an overview of your data monitors, see Introduction to the Data Quality page.

What types of monitors are there in Adverity?

There are two types of data monitors in Adverity:

• Universal monitors

  Universal monitors detect data anomalies and flag potential issues across all data sources. They perform general data quality checks, such as finding duplicate rows in your data.

  By default, universal monitors raise a warning when an anomaly is detected. You can edit a universal monitor to trigger an error for specific datastreams.

• Custom monitors

  Custom monitors allow you to define custom rules that your data must meet.

  For more information, see Working with custom monitors.

How do I use universal monitors?

To use universal monitors, enable them on the Administration page of the root workspace. The enabled universal monitors will be applied to all datastreams on your Adverityinstance. Additionally, you can change universal monitors configuration on a datastream level.

Using universal monitors

Currently, you can use universal data monitors to detect duplicate rows in your data.

Detecting duplicate data

Duplicate data in your data extracts can cause inaccurate data analysis. When the duplication monitor is enabled for a datastream, Adverity will automatically detect duplicate data in this datastream and alert you so that you can remove it before any problems occur.

If Adverity detects one or more rows containing identical data in a data extract that you have collected, a Data Quality warning will appear in the Enrich & Monitor section of the task in the datastream overview.

To check the details about the detected duplicates, see Viewing details of the data quality issues in your data extract. If necessary, remove the duplicate data from your data extract before loading the data into Adverity Data Storage or external destinations.

Enabling duplicate data checks

To switch duplicate data detection on or off for all datastreams, follow these steps:

1. Select the root workspace and then, in the platform navigation menu, click Administration.

2. Select or deselect the Enable checking for duplicate rows in extracts checkbox.

   This setting is available to users with Administrator permissions in the root workspace of your organization, and will be applied to all child workspaces of the root workspace.

Managing the duplication monitor for a specific datastream

You can perform the following actions with the duplication monitor on a datastream level:

Edit the duplication monitor

By default, the duplication monitor triggers a warning when duplicates are detected. To change this setting for a specific datastream so that it triggers an error instead, follow these steps:

1. Select the workspace you work with in Adverity and then, in the platform navigation menu, click Datastreams.

2. Open the chosen datastream by clicking on its name.

3. In the Monitors subsection of the datastream overview, find the duplication monitor.

4. Click Edit duplication monitor.

5. Select the issue type that should be triggered if the monitor detects duplicates from the following options:

   Trigger an error

   Select this option to raise an error and stop processing the data.

   Trigger a warning

   Select this option to raise a warning and continue processing the data.

6. Click Apply.

Deactivate the duplication monitor

To deactivate the duplication monitor for a datastream, follow these steps:

1. Select the workspace you work with in Adverity and then, in the platform navigation menu, click Datastreams.

2. Open the chosen datastream by clicking on its name.

3. In the Monitors subsection of the datastream overview, find the duplication monitor.

4. Disable the toggle next to the duplication monitor.