Collecting data from Mailgun

This guide explains how to collect data from Mailgun. To learn how to collect data from a different data source, go back to the Available data sources in Adverity overview.

Limitations

Collecting data from Mailgun comes with the following limitations:

The Mailgun account price plan determines how long data is retained for.

If you choose to use Adverity's Mailgun account, the data retention is as follows:
- Emails (including the message body and file attachments) are stored on Mailgun's servers for 3 days. If the email is not fetched within 3 days of being sent, it is deleted.
- A log of all email activity (known as Events) is kept for 15 days.
If you choose to use your own Mailgun account, refer to the Mailgun price plan for data retention limits on your account.

Prerequisites

Before you start collecting data from Mailgun, perform all of the following actions:

Ensure you have a valid API key to access your Mailgun account. For more information, see the Mailgun API documentation. If you do not have a valid API key, contact your Account manager.

Creating a datastream to collect data from Mailgun

The basics of creating a datastream to collect data from any data source are explained in our guide to Creating a datastream. This guide contains information about the specific steps to create a datastream to fetch data from Mailgun.

Authorization: Allow Adverity to access Mailgun

To set up a new authorization to fetch data from Mailgun, follow these steps:

In the Authorization step of the datastream wizard, click Access Mailgun using your credentials.
Click Next.

Choose the Mailgun server for which you want to set up an authorization.
In Private API key, enter your API key. For more information, see the Mailgun user interface.
In Recipient pattern, enter a recipient email pattern, followed by your domain name, e.g. address@yourdomain.com.

Configuration: Choose the data you want to collect from Mailgun

The Mailgun connector has many optional fields to configure. This section covers only the mandatory fields to configure for the connector. For a complete description of the optional fields to configure, see Advanced Mailgun tips.

To choose what data to collect and customize the Mailgun datastream configuration, follow these steps:

(Optional) Rename your datastream.

In File Pattern, enter the filename of the email attachment from which to collect data. You can collect data on a file directly attached to an email and files linked in the email body.
- To collect a file attached to the email, enter the filename of the attachment. To collect data from more than one file attachment, specify a regular expression that matches the filenames.
- To collect an attachment that is linked in the body of an email, enter the link text or the button label you need to click to collect the data. For more information on how to extract files linked in the email body, see Collecting files linked in the email body.

For information on configuring other Mailgun fields, see Advanced Mailgun tips.

Verify your Mailgun email address

You may need to verify your Mailgun email address to start collecting data.

You can find the email with a verification link or code sent by your data source in the events section of the datastream. For more information, see Viewing email body.

What's next?

Apply Data Mapping to your collected data to harmonize data collected from different sources in Adverity.
Transform your data to meet your needs by creating and applying enrichments to your datastream.
Load your data into Adverity Data Storage to send it to BI tools using data shares
Load your data into an external destination of your choice

Advanced Mailgun tips

Configuring data collection from Mailgun

In the Settings tab of your Mailgun datastream overview, you can configure a number of additional settings:

Auto fetch

Select this checkbox to automatically fetch data from Mailgun each time an email is received by the email address displayed in the Recipient field. This checkbox is selected by default.

To avoid duplication of data, when Auto fetch is enabled, scheduled fetches are disabled for this datastream.

In the Recipient field, Adverity displays the email address associated with the Mailgun account. The data included in all emails received by this Recipient email address is included in the Mailgun data extracts.

For more information on the recipient email address, see Configuring the recipient email address pattern.

The File Matching Options section contains the following fields:

File pattern

Specify a regular expression that matches the file attachment from which to collect data. By default, File pattern is pre-populated with .* which collects data for any attached file.

Other examples of regular expressions for file patterns include:

.*\.xlsx collects every file that ends with .xlsx.
.*\.csv collects every file that ends with .csv.
.*Display.*\.csv collects every file that contains the (case-sensitive) phrase Display anywhere in the file name and also ends with .csv.

It is also possible to collect files that are linked in the email body. For more information on how to collect these files, see Collecting files linked in the email body.

Zip match

If you collect data from ZIP or GZIP files, specify a regular expression that matches the file names within the ZIP or GZIP containers. Leave this field empty to collect all files within the ZIP or GZIP container.

Archive password

If the ZIP or GZIP files require password access, provide the password. By default, Adverity uses the previously entered password, if available.

The File Parsing section contains the following fields:

Parse

Select the data format used in the source files. The type of data format selected may cause additional fields to appear.

Source encoding

Select the character encoding used in the source files. By default, the option Auto-detect is selected and will automatically detect the encoding of the source files.

Delimiter

If CSV is selected in Parse, specify the character used to separate values in the CSV source files.

Quote Char

If CSV is selected in Parse, specify the character used to quote values with special characters in the CSV source files.

Quoting

If CSV is selected in Parse, specify when a quoting character should be added to the field values in the data extract. Choose from one of the following options:

Select all to add quote characters to everything in the data extract, regardless of the field type.
(Default) Select minimal to add quote characters only when required. For example, a quote character will be added to a field that contains either the Quote Char or Delimiter.
Select none to ensure no quote characters are added to the data extract.
Select nonnumeric to add quote characters to everything, except integer and float values.

Sheet

If Excel is selected in Parse, specify the name of the sheet within the Excel source files to import to Adverity.

Column offset

If Excel is selected in Parse, specify the number of columns that you do not want to import to Adverity from each Excel source file. For example, if the first column contains information that you do not want to import, specify 1 in this field.

Row offset

Specify the number of rows that you do not want to import to Adverity from each source file. For example, if the first row contains header information that you do not want to import, specify 1 in this field. This field is available for all parse types.

Skip initial space

If CSV is selected in Parse, select this checkbox to ignore any whitespace that follows the selected delimiter character.

The Advanced Settings section contains the following fields:

Keep Filename

Select this checkbox to name data extracts using the same name as the corresponding source files. If you select this checkbox and the source filename stays the same between fetches, data in the corresponding data extract is overwritten.

Processing

Choose how files are collected. Select one of the following options:

Fetch email attachments only
Fetch email attachments and extract URLs from email plain text body
Fetch email attachments and extract URLs from email HTML body

Depending on which option you select, you may need to configure the File Pattern field above. For more information, see Collecting files linked in the email body.

Missing Attachment Warning

Select this checkbox to display a warning when Adverity fetches data for the Mailgun datastream and there are no file attachments that match your criteria.

Mailgun events

Mailgun events display information about all emails fetched by the Mailgun datastream. The Mailgun events page displays the following information:

If the email on the Mailgun server has been fetched or not.
The timestamp of when the email was fetched.
The email subject. The subject is displayed as a hyperlink. Click on this hyperlink to go to a separate page that displays the body of the email. For more information, see Viewing email body.
The email address of the sender.
How many attachments are attached to the email. This number is displayed as a hyperlink. Click on this hyperlink to go to a separate page that displays the details of the attachment. For more information, see Viewing attachment details.

Viewing Mailgun events

To view Mailgun events, follow these steps:

Select the workspace you work with in Adverity and then, in the platform navigation menu, click Datastreams.
Open the Mailgun datastream by clicking on its name.
In the top right corner of the page, click More .
Click Events.

Checking Mailgun for uncollected data

To check if the Mailgun server has data which has not yet been collected by Adverity, follow these steps:

Select the workspace you work with in Adverity and then, in the platform navigation menu, click Datastreams.
Open the Mailgun datastream by clicking on its name.
In the top right corner of the page, click More .
Click Events.
In the top right corner of the page, click Update.

As a result, a table will appear in the Events page where each row represents an email which have been received by the Mailgun server but has yet to be collected by Adverity.

Manually collecting email data

To manually fetch email data from Mailgun, follow these steps:

Select the workspace you work with in Adverity and then, in the platform navigation menu, click Datastreams.
Open the Mailgun datastream by clicking on its name.
In the top right corner of the page, click More .
Click Events.
Select the checkboxes of the emails for which you want to collect data.
Click Fetch selected events.

As a result, you are navigated back to the datastream overview page where you can view the progress of the data collection for the selected emails.

Viewing attachment details

To view the details of email attachments, follow these steps:

Select the workspace you work with in Adverity and then, in the platform navigation menu, click Datastreams.
Open the Mailgun datastream by clicking on its name.
In the top right corner of the page, click More .
Click Events.
Find the email with attachments to view in more detail.
Click on the number displayed as a hyperlink in the Attachments column.

As a result, you are navigated to a separate page that displays the details of the email attachment. These details include:

Content-type - This displays the file type and format. For example, image/png.
Filename - This displays the name of the attached file.
Size - This displays the size of the file in bytes.

Viewing email body

To view the email body, follow these steps:

Select the workspace you work with in Adverity and then, in the platform navigation menu, click Datastreams.
Open the Mailgun datastream by clicking on its name.
In the top right corner of the page, click More .
Click Events.
Find the email for which you want to view the email body.
Click on the email's subject displayed as a hyperlink in the Subject column.

As a result, you are navigated to a separate page that displays the body of the email which includes the following:

Date displays the exact date on which that email was received.
Body Plain displays the message body text as plain text.
Body HTML displays the message body text in a HTML format.

Uploading files into Adverity

To upload a file into Adverity using the Mailgun datastream, follow these steps:

Select the workspace you work with in Adverity and then, in the platform navigation menu, click Datastreams.
Open the Mailgun datastream by clicking on its name.
In the top right corner of the page, click Upload.
Click Choose File, and select the file to upload.
(Optional) Select the Keep data in raw state checkbox to achieve the following goals:
- Keep the data in its original form and do not apply any enrichments assigned to this datastream. For more information on enriching your data, see Enriching data in Adverity.
- Fetch the data without loading it into the destination specified for this datastream. For more information on loading data into a destination, see Introduction to loading data into Adverity Data Storage and external destinations.
Click Upload.

As a result, the file is uploaded into Adverity and Adverity creates a data extract that includes the uploaded data.

Collecting files linked in the email body

The email body can contain links or buttons to files. For example, an email may contain a button labeled Download which you need to click to collect the files.

To collect files linked in the email body, follow these steps:

Select the workspace you work with in Adverity and then, in the platform navigation menu, click Datastreams.
Open the Mailgun datastream by clicking on its name.
In the top navigation panel, click Settings.
In File pattern, specify a regular expression that matches the link text or enter the name of the button label. For example, to collect files linked from button labeled Download, enter Download.
In the Advanced Settings section, in the Processing field, select one of the following:
- If the email is formatted in plain text, select Fetch email attachments and extract URLs from email plain text body
- If the email is formatted in HTML, select Fetch email attachments and extract URLs from email HTML body

Click Save.

Changing the Mailgun server

When you change the authorization and the Mailgun server for an existing Mailgun datastream, the recipient e-mail address is automatically updated. If you then change the authorization again and select the original Mailgun server, the original recipient e-mail address is restored automatically. You can find the recipient e-mail address in the Recipient field.

For example, if you change the authorization and the server from Mailgun (US) to Mailgun (EU), the recipient e-mail address is automatically updated. If you then change the server back from Mailgun (EU) to Mailgun (US), the original e-mail address is restored automatically.

Configuring the recipient email address pattern

You can define the recipient email address pattern for each authorization. To configure it, follow these steps:

Go to the Authorization Center.
Find the authorization you want to update.
In the Recipient pattern field, define the email address pattern you want to use with one of these options:
- Use the {address} variable to create a unique email address for each datastream created with this authorization. The {address} variable will be replaced with a unique combination of digits for each datastream.
  
  For example, you can set the recipient pattern as follows:
  
  {address}@yourdomain.com
- Use a custom name for the recipient address. In this case, each datastream using this authorization will use the same recipient address.
  
  For example, you can set the recipient pattern as follows:
  
  {{CUSTOM-NAME}}@yourdomain.com.
Click Save

You can also define the recipient address pattern when creating a new authorization.