Collecting data from Mailgun
This guide explains how to collect data from Mailgun. To learn how to collect data from a different data source, go back to the Available data sources in Adverity overview.
Limitations
Collecting data from Mailgun comes with the following limitations:
-
The Mailgun account price plan determines how long data is retained for.
If you choose to use Adverity's Mailgun account, the data retention is as follows:
-
Emails (including the message body and file attachments) are stored on Mailgun's servers for 3 days. If the email is not fetched within 3 days of being sent, it is deleted.
-
A log of all email activity (known as Events) is kept for 15 days.
If you choose to use your own Mailgun account, refer to the Mailgun price plan for data retention limits on your account.
-
Prerequisites
Before you start collecting data from Mailgun, perform all of the following actions:
-
Ensure you have a valid API key to access your Mailgun account. For more information, see the Mailgun API documentation. If you do not have a valid API key, contact your Account manager.
Creating a datastream to collect data from Mailgun
The basics of creating a datastream to collect data from any data source are explained in our guide to Creating a datastream. This guide contains information about the specific steps to create a datastream to fetch data from Mailgun.
Authorization: Allow Adverity to access Mailgun
Before creating an Adverity access to Mailgun, perform the following actions:
to give-
Find and copy your webhook signing key in the Mailgun user interface.
-
Generate and copy a private API key in the Mailgun user interface.
For more information about finding and generating these keys, see the Mailgun documentation.
To set up a new Mailgun, follow these steps:
to fetch data from-
In the Authorization step of the datastream wizard, click Access Mailgun using your credentials.
-
Click Next.
-
Choose the Mailgun server for which you want to set up an .
-
In Webhook signing key, enter your webhook signing key.
-
In Private API key, enter your API key.
-
In Recipient pattern, enter a recipient email pattern, followed by your domain name, e.g.
address@yourdomain.com
.
-
Click Authorize.
Configuration: Choose the data you want to collect from Mailgun
The Mailgun connector has many optional fields to configure. This section covers only the mandatory fields to configure for the connector. For a complete description of the optional fields to configure, see
To choose what data to collect and customize the Mailgun datastream configuration, follow these steps:
-
(Optional) Rename your datastream.
-
In File Pattern, enter the filename of the email attachment from which to collect data. You can collect data on a file directly attached to an email and files linked in the email body.
-
To collect a file attached to the email, enter the filename of the attachment. To collect data from more than one file attachment, specify a regular expression that matches the filenames.
-
To collect an attachment that is linked in the body of an email, enter the link text or the button label you need to click to collect the data. For more information on how to extract files linked in the email body, see Collecting files linked in the email body.
-
For information on configuring other Mailgun fields, see Advanced Mailgun tips.
Verify your Mailgun email address
You may need to verify your Mailgun email address to start collecting data.
You can find the email with a verification link or code sent by your data source in the events section of the datastream. For more information, see Viewing email body.
What's next?
-
Apply Data Mapping to your collected data to harmonize data collected from different sources in Adverity.
-
Transform your data to meet your needs by creating and applying enrichments to your datastream.
-
Load your data into Explore & Present to visualize your data in Adverity
-
Load your data into an external destination of your choice
Advanced Mailgun tips
Configuring data collection from Mailgun
In the Settings tab of your Mailgun datastream overview, you can configure a number of additional settings:
-
Auto fetch
-
Select this checkbox to automatically fetch data from Mailgun each time an email is received by the email address displayed in the Recipient field. This checkbox is selected by default.
To avoid duplication of data, when Auto fetch is enabled, scheduled fetches are disabled for this datastream.
-
In the Recipient field, Adverity displays the email address associated with the Mailgun account. The data included in all emails received by this Recipient email address is included in the Mailgun data extracts.
For more information on the recipient email address, see Configuring the recipient email address pattern.
The File Matching Options section contains the following fields:
-
File pattern
-
Specify a regular expression that matches the file attachment from which to collect data. By default, File pattern is pre-populated with
.*
which collects data for any attached file. -
Other examples of regular expressions for file patterns include:
-
.*\.xlsx
collects every file that ends with .xlsx. -
.*\.csv
collects every file that ends with .csv. -
.*Display.*\.csv
collects every file that contains the (case-sensitive) phrase Display anywhere in the file name and also ends with .csv.
-
-
It is also possible to collect files that are linked in the email body. For more information on how to collect these files, see Collecting files linked in the email body.
-
Zip match
-
If you collect data from ZIP or GZIP files, specify a regular expression that matches the file names within the ZIP or GZIP containers. Leave this field empty to collect all files within the ZIP or GZIP container.
-
Archive password
-
If the ZIP or GZIP files require password access, provide the password. By default, Adverity uses the previously entered password, if available.
The File Parsing section contains the following fields:
-
Parse
-
Select the data format used in the source files. The type of data format selected may cause additional fields to appear.
-
Source encoding
-
Select the character encoding used in the source files. By default, the option Auto-detect is selected and will automatically detect the encoding of the source files.
-
Delimiter
-
If CSV is selected in Parse, specify the character used to separate values in the CSV source files.
-
Quote Char
-
If CSV is selected in Parse, specify the character used to quote values with special characters in the CSV source files.
-
Quoting
-
If CSV is selected in Parse, specify when a quoting character should be added to the field values in the data extract. Choose from one of the following options:
-
Select all to add quote characters to everything in the data extract, regardless of the field type.
-
(Default) Select minimal to add quote characters only when required. For example, a quote character will be added to a field that contains either the Quote Char or Delimiter.
-
Select none to ensure no quote characters are added to the data extract.
-
Select nonnumeric to add quote characters to everything, except integer and float values.
-
-
Sheet
-
If Excel is selected in Parse, specify the name of the sheet within the Excel source files to import to Adverity.
-
Column offset
-
If Excel is selected in Parse, specify the number of columns that you do not want to import to Adverity from each Excel source file. For example, if the first column contains information that you do not want to import, specify 1 in this field.
-
Row offset
-
Specify the number of rows that you do not want to import to Adverity from each source file. For example, if the first row contains header information that you do not want to import, specify 1 in this field. This field is available for all parse types.
-
Skip initial space
-
If CSV is selected in Parse, select this checkbox to ignore any whitespace that follows the selected delimiter character.
The Advanced Settings section contains the following fields:
-
Keep Filename
-
Select this checkbox to name data extracts using the same name as the corresponding source files. If you select this checkbox and the source filename stays the same between fetches, data in the corresponding data extract is overwritten.
-
Processing
-
Choose how files are collected. Select one of the following options:
-
Fetch email attachments only
-
Fetch email attachments and extract URLs from email plain text body
-
Fetch email attachments and extract URLs from email HTML body
Depending on which option you select, you may need to configure the File Pattern field above. For more information, see Collecting files linked in the email body.
-
-
Missing Attachment Warning
-
Select this checkbox to display a warning when Adverity fetches data for the Mailgun datastream and there are no file attachments that match your criteria.
Mailgun events
Mailgun events display information about all emails fetched by the Mailgun datastream. The Mailgun events page displays the following information:
-
If the email on the Mailgun server has been fetched or not.
-
The timestamp of when the email was fetched.
-
The email subject. The subject is displayed as a hyperlink. Click on this hyperlink to go to a separate page that displays the body of the email. For more information, see Viewing email body.
-
The email address of the sender.
-
How many attachments are attached to the email. This number is displayed as a hyperlink. Click on this hyperlink to go to a separate page that displays the details of the attachment. For more information, see Viewing attachment details.
Viewing Mailgun events
To view Mailgun events, follow these steps:
-
Go to the Datastreams page.
-
In the top right corner of the page, click More .
-
Click Events.
Checking Mailgun for uncollected data
To check if the Mailgun server has data which has not yet been collected by Adverity, follow these steps:
-
Go to the Datastreams page.
-
In the top right corner of the page, click More .
-
Click Events.
-
In the top right corner of the page, click Update.
As a result, a table will appear in the Events page where each row represents an email which have been received by the Mailgun server but has yet to be collected by Adverity.
Manually collecting email data
To manually fetch email data from Mailgun, follow these steps:
-
Go to the Datastreams page.
-
In the top right corner of the page, click More .
-
Click Events.
-
Select the checkboxes of the emails for which you want to collect data.
-
Click Fetch selected events.
As a result, you are navigated back to the datastream overview page where you can view the progress of the data collection for the selected emails.
Viewing attachment details
To view the details of email attachments, follow these steps:
-
Go to the Datastreams page.
-
In the top right corner of the page, click More .
-
Click Events.
-
Find the email with attachments to view in more detail.
-
Click on the number displayed as a hyperlink in the Attachments column.
As a result, you are navigated to a separate page that displays the details of the email attachment. These details include:
-
Content-type - This displays the file type and format. For example, image/png.
-
Filename - This displays the name of the attached file.
-
Size - This displays the size of the file in bytes.
Viewing email body
To view the email body, follow these steps:
-
Go to the Datastreams page.
-
In the top right corner of the page, click More .
-
Click Events.
-
Find the email for which you want to view the email body.
-
Click on the email's subject displayed as a hyperlink in the Subject column.
As a result, you are navigated to a separate page that displays the body of the email which includes the following:
-
Date displays the exact date on which that email was received.
-
Body Plain displays the message body text as plain text.
-
Body HTML displays the message body text in a HTML format.
Uploading files into Adverity
To upload a file into Adverity using the Mailgun datastream, follow these steps:
-
Go to the Datastreams page.
-
In the top right corner of the page, click Upload.
-
Click Choose File, and select the file to upload.
-
(Optional) Select the Keep data in raw state checkbox to achieve the following goals:
-
Keep the data in its original form and do not apply any enrichments assigned to this datastream. For more information on enriching your data, see Enriching data in Adverity.
-
Fetch the data without loading it into the destination specified for this datastream. For more information on loading data into a destination, see Loading data into destinations.
-
-
Click Upload.
As a result, the file is uploaded into Adverity and Adverity creates a data extract that includes the uploaded data.
Collecting files linked in the email body
The email body can contain links or buttons to files. For example, an email may contain a button labeled Download which you need to click to collect the files.
To collect files linked in the email body, follow these steps:
-
Go to the Datastreams page.
-
In the Settings.
, click -
In File pattern, specify a regular expression that matches the link text or enter the name of the button label. For example, to collect files linked from button labeled Download, enter
Download
. -
In the Advanced Settings section, in the Processing field, select one of the following:
-
If the email is formatted in plain text, select Fetch email attachments and extract URLs from email plain text body
-
If the email is formatted in HTML, select Fetch email attachments and extract URLs from email HTML body
-
-
Click Save.
Changing the Mailgun server
When you change the datastream, the recipient e-mail address is automatically updated. If you then change the again and select the original Mailgun server, the original recipient e-mail address is restored automatically. You can find the recipient e-mail address in the Recipient field.
and the Mailgun server for an existing MailgunFor example, if you change the Mailgun (US) to Mailgun (EU), the recipient e-mail address is automatically updated. If you then change the server back from Mailgun (EU) to Mailgun (US), the original e-mail address is restored automatically.
and the server fromConfiguring the recipient email address pattern
You can define the recipient email address pattern for each
. To configure it, follow these steps:-
Go to the Authorization Center.
-
Find the
you want to update. -
In the Recipient pattern field, define the email address pattern you want to use with one of these options:
-
Use the
{address}
variable to create a unique email address for each datastream created with this . The{address}
variable will be replaced with a unique combination of digits for each datastream.For example, you can set the recipient pattern as follows:
{address}@yourdomain.com
-
Use a custom name for the recipient address. In this case, each datastream using this will use the same recipient address.
For example, you can set the recipient pattern as follows:
{{CUSTOM-NAME}}@yourdomain.com
.
-
-
Click Save
You can also define the recipient address pattern when creating a new
.