Transferring data to File

This guide explains how to transfer data to File to store and further process information.

Introduction

Use the File destination to transfer data to generic file servers accessible via FTP, FTPS, SFTP, or SMB.

You can assign multiple destinations to a datastream. For more information on possible limitations, see Assigning multiple destinations to a datastream.

Prerequisites

Before you complete the procedure in this guide, perform all of the following actions:

• Create a datastream whose data you want to transfer to File. For more information on creating a datastream, see Collecting data from different data sources.

• Ensure you have login details to the destination with the following permissions:

  • Read, write, and delete files and folders.

  • List folders.

Procedure

To transfer data from a datastream to File, follow these steps:

1. Add File as a destination to the workspace which contains the datastream or to one of its parent workspaces.

2. Assign the File destination to the datastream.

3. Configure transfer settings.

Adding File as a destination

To add File as a destination to a workspace, follow these steps:

1. Select the workspace you work with in Adverity and then, in the platform navigation menu, click Destinations.

2. Click + Add Destination.

3. Click File.

4. Click Setup a new Authorization.

5. Click Next.

6. Select one of the following options:

   • To connect to the file server with your username and password, click File, and then log in with your credentials.

   • To connect to the file server with an SFTP public key, click SFTP Public Key. If you are using an RSA key file to authorize the connection, ensure that the key is in OpenSSH format. On the next page, enter your username, select the public key type, upload the private key file, and click Authorize.

7. In the Configuration page, fill in the following fields:

   Name

   (Optional) Rename the destination.

   Destination URL

   In the drop-down on the left, select the file server type. In the text field in the middle, enter the base URL of the file server. In the text field on the right, enter the path to the folder where to transfer data. Click Test to check the authorization.

   Output format

   Select the data format that Adverity uses to transfer data to the destination.

   When you transfer data to AVRO file format, select AVRO to use the null codec, or AVRO (deflate) to use the deflate codec. For more information on codecs, see the Apache documentation.

   For more information on advanced configuration settings, see Advanced File destination configuration.

8. Click Create.

Assigning File as a destination

To assign the File destination to a datastream, follow these steps:

1. Select the workspace you work with in Adverity and then, in the platform navigation menu, click Datastreams.

2. Select the chosen datastream.

3. In the Transfer section, click + Assign destination.

4. Click Assign existing destinations.

5. Select the File checkbox in the list.

6. Click Save.

Configuring transfer settings

To configure transfer settings, follow these steps:

1. Select the workspace you work with in Adverity and then, in the platform navigation menu, click Datastreams.

2. Select the chosen datastream.

3. In the Destinations section, find the File destination in the list, and click on the right.

4. Click Destination settings.

5. Fill in the following fields:

   Filename

   Specify the target file in the destination where to transfer data from the datastream. The name can contain alphanumeric characters and underscores. For example, target_file.

   To transfer data to sub-folders within the folder defined in the Destination URL field, specify a file path. For example, folder1/target_file.

   By default, Adverity saves data from each datastream in a different file named {datastream_type}_{datastream_id}_{scheduled_year}_{scheduled_month}_{scheduled_day}.

   You can specify the same target file for several datastreams. If a column is shared between datastreams, Adverity performs a full outer join and concatenates values. If a column is not shared between datastreams, Adverity writes null values in the relevant cells.

   Use placeholders to create unique, dynamic file names in the destination. Use the following placeholders:

   Placeholder	Description
   {app_label}	The Datastream Type's short name.
   {datastream_id}	The datastream ID.
   {datastream_type}	The Datastream Type.
   {extension}	The file extension of the data extract.
   {extract_id}	The data extract ID.
   {id}	The datastream ID.
   {meta[*]}	Replace * with a metadata placeholder to use metadata in the file name. For example, {meta[datastream_URI]} uses the datastream URI as the file name. For more information on metadata and placeholders, see Using placeholders.
   {name}	The automatically generated filename of the data extract.
   {scheduled_day}	The day when the data fetch was scheduled to run.
   {scheduled_month}	The month when the data fetch was scheduled to run.
   {scheduled_year}	The year when the data fetch was scheduled to run.
   {upload_day}	The day when the data extract is transferred to the File destination.
   {upload_hour}	The hour when the data extract is transferred to the File destination.
   {upload_minute}	The minute when the data extract is transferred to the File destination.
   {upload_month}	The month when the data extract is transferred to the File destination.
   {upload_second}	The second when the data extract is transferred to the File destination.
   {upload_year}	The year when the data extract is transferred to the File destination.

6. Click Save.