Scaling datastreams#
This guide explains how to scale your datastreams in Adverity.
Introduction#
Expand your data operations and make changes to your data collection setup at scale. Having many datastreams and changing data needs comes with repetitive tasks for multiple datastreams. In Adverity, you can perform these tasks efficiently for all affected datastreams at once.
Fetching data manually for multiple datastreams#
To start a manual fetch for multiple datastreams, follow these steps:
Go to the Datastreams page.
Select the datastreams for which you want to start a fetch.
Above the datastreams list, click Fetch now.
In the pop-up window, configure the fetch parameters. For more information, see Fetching data manually.
Click Fetch.
Enabling and disabling schedules for multiple datastreams#
To enable or disable schedules for multiple datastreams, follow these steps:
Go to the Datastreams page.
Select the datastreams for which you want to enable or disable schedules.
Above the datastreams list, in the Schedule
dropdown, click Enable/Disable schedule.
Enabling and disabling all destinations for multiple datastreams#
To enable or disable all destinations for multiple datastreams, follow these steps:
Go to the Datastreams page.
Select the datastreams for which you want to enable or disable all destinations.
Above the datastreams list, in the Destination
dropdown, click Enable/Disable destinations.
Premium scalability actions#
Prerequisites#
Only users with Administrator permissions can perform premium scalability actions.
Renaming multiple datastreams#
To rename multiple datastreams in the selected workspace, follow these steps:
Go to the Datastreams page.
Select the datastreams you want to rename.
Above the datastreams list, click Rename.
Select and configure a renaming rule from the following options:
Note
All rules are case-sensitive.
Find and replace
To remove found matches, leave the Replacement field empty.
Add prefix
Add suffix
(Optional) Click + Add another rule to create a more complex renaming rule. Multiple renaming rules are applied sequentially from top to bottom.
Click Save.
Cloning multiple datastreams#
You can clone multiple datastreams at once within the same workspace or to a different workspace. During multiple datastream cloning, you can:
re-use, clone, or replace authorizations
re-use or clone transformations
rename datastreams
Schedules, Data Mapping, datastream settings, and local data retention settings are preserved for the cloned datastreams.
If you want to clone datastreams to a new workspace, you can clone the workspace. For more information, see Cloning workspaces.
Limitations#
Cloning of destinations and Bundle datastreams is not supported.
Renaming and modifying transformations is not supported.
Cloning multiple datastreams#
To clone multiple datastreams, follow these steps:
Go to the Datastreams page.
Select the datastreams you want to clone.
Above the datastreams list, click
Clone.In the Workspace selection step, select the workspace to which you want to clone the selected datastreams.
Click Next.
In the Authorizations step, adjust the authorizations used by the selected datastreams. Click
Change to edit which authorization is used by a datastream.You can set the authorization used by each datastream to one of these options, if applicable:
Re-use the authorization from the parent workspace
Clone the original authorization
Use an authorization from the target workspace
Use an authorization from other workspaces
Click Next.
In the Transformations step, select one of these options:
Include transformations
Select this option to clone selected datastreams including all or some assigned transformations. If the transformation belongs to the parent workspace, it will be re-used. If the transformation belongs to the source workspace and cannot be re-used in the new workspace, it will be cloned.
Ignore transformations
Select this option to clone selected datastreams without any assigned transformations.
(Optional) If you selected Include transformations in the previous step, configure the list of assigned transformations for each of the selected datastreams. The order of the assigned transformations is preserved.
Click Next.
Review the summary of the datastreams to be cloned.
(Optional) To rename the cloned datastreams, click
Rename datastreams.
For more information, see Renaming datastreams.Click Clone
.
What’s next?#
Once the datastream cloning is complete, you may need to perform the following actions to finish your setup:
Customize datastream configurations, for example, change the selected accounts.
Rename cloned authorizations.
Rename or adjust cloned transformations.
Fetch necessary historical data.
Assign monitors. For more information, see Introduction to the Data Quality page.
Assign destinations. For more information, see Assigning a destination to multiple datastreams.
Enable schedules. For more information, see Enabling and disabling schedules for multiple datastreams.
Assigning a destination to multiple datastreams#
Limitations#
Only destinations that do not require additional configuration on the datastream level can be assigned to multiple datastreams at once. You can assign the following destinations to multiple datastreams:
Azure Blob
Azure Synapse
Databricks
Databricks (SQL)
File
File OneDrive
Firebolt
Google BigQuery
Google Sheets
LinkedIn Matched Audiences
Microsoft SQL Server
SAP Data Warehouse Cloud
SAP Marketing Cloud
Snapchat Audiences
Snowflake
SQL Database
Assigning a destination to multiple datastreams#
To assign a destination to multiple datastreams, follow these steps:
Go to the Datastreams page.
Select the datastreams to which you want to assign a destination.
Above the datastreams list, in the Destination
dropdown, click Add destination.Select the destination that you want to assign and click Save.
The destination will be assigned to selected datastreams.
(Optional) Disable the destination for specific datastreams.
(Optional) Load data previously collected in the selected datastreams to the newly assigned destination. Data will be loaded only when the destination is enabled.