Collecting data using Web Connect

This guide documents the configuration options of Web Connect in Adverity.

Introduction

What is Web Connect?

Web Connect is a universal connector in Adverity. With Web Connect, you can set up a connection to a data source API that does not have an available connector.

How do I configure a Web Connect datastream?

To set up a Web Connect datastream you need to define how Adverity should send and process the API requests exchanged with the data source API. The configuration may include the following:

Defining how the API requests should be authorized. For more information, see Setting up an authorization to Web Connect.
Configuring the API requests that should be sent to collect data. For more information, see Request.
Specifying the pagination parameters used by the API when sending data. For more information, see Pagination Options.
Configuring how to process the API response and extract the data you need. For more information, see Response.

Authorization methods

When you connect to an API using Web Connect you may need to create an authorization.

Depending on the API requirements, you may need to enter your credentials in the request URL, header or body instead of using an authorization in Adverity. In this case, select Continue without authorization in the Authorization step of the datastream wizard.

For more information on configuring Web Connect authorizations, see Setting up an authorization to Web Connect.

Configuring data collection with Web Connect

This section explains how to configure data collection with Web Connect.

You may not need to configure all fields for the specific API and use case.

Request

Configure the fields in this section to define the request sent to the API.

URL

Enter the URL of the API, including the required parameters and values.

You can add the following parameters to the URL:

Start and end date parameters

You can set start and end date parameters to define the fetch date range in the URL or request body. You can set the date parameters in the following two ways:
- {start} and {end} placeholders
  
  When using the {start} and {end} placeholders, you can change the fetch date range using the date range selector as in other connectors.
  
  For example, if your API uses the date_from parameter to define the start of the date range and the date_to parameter to define the end of the date range, add the following expression to the URL to configure the fetch date range:
```
date_from={start}&date_to={end}
```
  Your API may use a different date format than Adverity. To use a custom date format supported by your API, configure the Date format field below.
- Static values
  
  Alternatively, you can use static values to define the date range for which to collect data.
  
  For example, to collect data from October 1, 2023 to October 5, 2023 with the date_from and date_to API parameters, add the following expression to the URL:
```
date_from="2023-10-01"&date_to="2023-10-05"
```
  Make sure to enter the dates in the date format supported by your API.
Authorization token

If you have an authorization token for the API, you can add it to the request URL.

If you are using an authorization to generate a token, you can add it to the request URL using the {YOUR_TOKEN} placeholder.

For example, if the URL uses the token parameter, add the following expression to the URL:
```
token={YOUR_TOKEN}
```
Value tables

To send multiple requests for a list of parameter values, use a value table in the request URL. This is typically used for lists of IDs, where you would need to collect data for multiple IDs. Adverity sends a separate request for each value in the value table.

For example, to use the value table with the name valuetablename, add the following expression to the URL:
```
{vt(valuetablename)}
```
In this expression, vt signifies that the placeholder in curly brackets is a value table.

For more information on automatically populating value tables, see Automatically populate a value table.
Other static parameters

You can add any other static parameters to the URL required by the API.

For example, to collect data from the following endpoint https://api.application.com including an authentication token generated by your authorization and using the date range selected through the datastream interface, configure the URL in the following way:

Request method

Select either the GET or POST HTTP request method, and configure the request.

For the GET request method, you can configure the header in JSON format.

For the POST request method, you can select the request content type, configure the header in JSON format, and enter an HTTP request body in the selected content type.

Request content type

Select the content type for the POST request required by your API.
Request headers

Enter the parameters required in the header in JSON format.

For example, your request header may look like this:
```
{
	"api-key":"abcdef",
	"Accept":"application/stream+json"
}
```
Request body

In the request body, configure what data to collect if it is required by the API.

You can set start and end date parameters to define the fetch date range. The parameters can be set to static values or use the {start} and {end} placeholders. For more information, see Start and end date parameters.

For example, your request body may look like this:
```
{
	"categories":[
		"brandName",
		"eventCode"
	],
	"metrics":[
		"clickCount",
		"impressionCount"
	],
	"currency":"USD",
	"dates":{
		"eventDateStart":"{start}",
		"eventDateEnd":"{end}"
	},
	"sort":{
		"eventCode":"ASC"
	},
	"summaryOnly":false
}
```

Date format

Configure the date format for the {start} and {end} placeholders to match the API requirements.

Most APIs use the ISO8601 format (YYYY-MM-DD). If your API uses a different format, for example DD/MM/YYYY or MMDDYYYY, configure it in this field.

To define the date format, use the following placeholders:

%m - a month as a two-digit number, from 01 to 12. This matches the mm format.
%d - a day as a two-digit number, from 01 to 31. This matches the dd format.
%Y - a year as a four-digit number. This matches the yyyy format.

For example, to set the date format to DD/MM/YYYY, enter %d/%m/%Y into the Date format field.

For more information on available formats, see Unix reference.

Navigation URLs

Include a list of URLs that are called before fetching the data. Use this option if you need to perform a request before you fetch data.

Enter the list as "request method - URL" pairs.

Pagination Options

Configure the fields in this section to set the pagination parameters supported or required by the API. Pagination defines how the API sends the requested data.

What is API pagination?

API pagination divides the responses into several pages to send large amounts of data more efficiently. For example, instead of 10,000 lines of JSON, the API sends 10 pages with 1,000 lines of JSON on each page. To retrieve all data from an API, all pages are fetched one after another. These pages are then combined into one data extract. Adverity's Web Connect can fetch all pages for various API pagination options.

For a more general explanation of pagination options, see Everything You Need to Know About API Pagination.

Configuring pagination with Web Connect

Select a pagination option supported by your API from the following options:

Page & PageSize Pagination

With this pagination type, the API splits your data into pages and sends them one by one. You can define the page size.