Working with custom monitors

This guide explains how to create, edit, and delete custom monitors from the Data Quality page and work with the custom monitors assigned to a datastream.

Introduction

What are monitors and what can they do?

A data monitor is an automated data quality check that is performed each time you fetch data. With data monitors, you can find anomalies in your data more easily.

For an overview of your data monitors, see Introduction to the Data Quality page.

What types of monitors are there in Adverity?

There are two types of data monitors in Adverity:

  • Universal monitors

    Universal monitors detect data anomalies and flag potential issues across all data sources.

    For more information, see Working with universal monitors.

  • Custom monitors

    Custom monitors allow you to define custom rules that your data must meet. Custom monitors can be used to ensure that the data you fetch is valid and matches your specific requirements. You can tell Adverity to raise a datastream issue (error or warning) if a custom monitor detects an anomaly.

    For example, you can use a custom monitor to ensure that the ID column is not empty and tell Adverity to display a datastream error if the incoming data does not match the rule.

How do I use custom monitors?

To use custom monitors, create a custom monitor. After the monitor is assigned to a datastream, every time you fetch data using the datastream, the collected data is checked to match the rules defined in the monitor.

Limitations

Custom monitors cannot be created for the fields with the following data types:

  • duration

  • formula

  • json

Creating a custom monitor

To create a new custom monitor, follow these steps:

  1. Go to the Data Quality page.

  2. In the top right corner of the page, click + Add custom monitor.

  3. In the Assigned for section, select the datastreams to which the monitor will be assigned. You must select at least one datastream.

  4. In the Validation rules section, define the rules that your data must meet. You can use the predefined rules, configure a custom rule, or combine different types of rules.

    When creating or editing a custom monitor assigned to multiple datastreams, only the fields available in all of the assigned datastreams are listed in this drop-down.

    Use the Data preview section below to review a data extract fetched from your datastream.

    Click + Add rule to add an additional rule to your monitor.

    You can add the following rules to your custom monitor:

    Empty values

    Select the fields that must contain no empty values.

    List of values

    Select a field that must contain only values from a list and enter the list of expected values.

    Date format

    Select the fields that must contain a specific date format and select the expected format.

    Custom template

    Use the following fields to configure custom rules. You can create combinations of custom rules using + Or and +And operators.

    Field

    Select a field from your data extract for which you want to define a rule.

    Operator

    Select a condition that the values of the selected field must meet.

    To define the rule using a regular expression, select the has structure (regex) option and then enter your regular expression in the Value field.

    Value

    Enter the field's value to define the rule.

  5. Select a datastream issue to be raised if the collected data does not meet the rules defined in the previous step. These datastream issues are displayed in the datastream overview - for more information, see Viewing datastream issues.

    Trigger an error

    Select this option to raise an error and stop processing the data.

    Trigger a warning

    Select this option to raise a warning and continue processing the data.

  6. In the Monitor name section, enter the name for your custom monitor.

  7. Click Apply.

As a result, a custom monitor has been created and assigned to the selected datastreams. Every time you fetch data using the datastreams, the collected data will be checked using the rules defined in the monitor.

Editing a custom monitor

To edit a custom monitor, follow these steps:

  1. Go to the Data Quality page.

  2. In the Custom monitors section, click on the name of the custom monitor you want to edit.

  3. Make the changes to the monitor.

  4. Click Apply.

As a result, the custom monitor configuration has been updated and will be used for all assigned datastreams.

Deleting a custom monitor

To delete a custom monitor, follow these steps:

  1. Go to the Data Quality page.

  2. Find the custom monitor which you want to delete.

  3. Click Select an action in the monitor's row.

  4. Click Delete monitor.

  5. In the confirmation dialog, click Delete monitor.

As a result, the custom monitor has been deleted.

Managing custom monitors assigned to a datastream

You can perform the following actions with the custom monitors assigned to the datastream:

Edit an assigned monitor

To edit an assigned monitor, follow these steps:

The monitor's changes will affect all datastreams to which it is assigned.

  1. Go to the Datastreams page.

  2. Open the chosen datastream by clicking on its name.

  3. In the Monitors subsection of the datastream overview, hover over the monitor you want to edit.

  4. Click Edit monitor.

  5. Make the changes to the monitor.

    When editing a custom monitor from the datastream, you cannot change the list of datastreams to which the monitor is assigned.

  6. Click Apply.

Unassign a monitor

To unassign a monitor from a datastream, follow these steps:

  1. Go to the Datastreams page.

  2. Open the chosen datastream by clicking on its name.

  3. In the Monitors subsection of the datastream overview, hover over the monitor you want to unassign.

  4. Click Remove monitor.

Include the monitor's status in the data extract

To include the status of the monitor assigned to the datastream in the data extract, update the datastream settings. Only the status of the monitors with severity set to Warning can be included. For more information, see Configuring advanced datastream settings.

Custom monitor use cases

Custom monitors are extremely versatile and can help you improve the quality of your collected data in many different ways. The list below provides examples to give you some ideas about how to use custom monitors with your data:

Check whether your data contains values from a specific list

For example, make sure that values are not mistyped or missing by creating a custom monitor to check that each value in the column Brand is one of a defined list of brand names.

Check that specific fields are not empty

For example, make sure that customers have provided contact details by creating a custom monitor to check that the Phone number field is not empty.

Check for errors in your data

For example, make sure that quantity values are valid by creating a custom monitor to check that no values in the Orders field are negative numbers.

Check that conditions are met for specific combinations of data

Make sure that specific combinations of field values are met by creating custom monitors to check these combinations, as described in the following examples:

  • Make sure all your data is complete, e.g. if the Product status field for a product is discontinued, the Date discontinued field for that product is not empty.

  • Make sure your organization is following up on leads, e.g. if the Contacted field for a lead is set to contacted, the Contact date field is not empty.

  • Make sure that active promotions have a valid discount code, e.g. if a promotion is active, then the Discount code field must not be empty, and the code must begin with 2024.

  • Make sure that users are of legal age to subscribe to your content, e.g. if a customer signs up to a product that requires users to be over 18, then the value in the Age field for that customer must be at least 18.