Use Case: Lotame Data Exchange

OVERVIEW

This page will describe the process for a Lotame Data Exchange partner to send an inbound batch file of data into Lotame for processing.


Lotame supports three file transfer options for our clients for batch file processing: AWS S3 buckets, Google Cloud Storage, and sFTP transfers.

FILE SIZE NOTICE

If you expect your files to be larger than 5GB, please contact support@lotame.com for assistance.

For S3 transfers, Lotame will look for files daily on an S3 bucket provided by partners. 

Lotame supports retrieving files using our Lotame External ARN. To access, please perform the following on your S3 bucket.

  • Allow access to the Lotame ARN of arn:aws:iam::311305431179:role/LotameS2SExternalRole-us-east-1

  • Provide the following AWS S3 actions to the above ARN:

    • s3:ListBucket

    • s3:GetObject

    • s3:DeleteObject

    • s3:PutObject

An example S3 bucket policy clause is below as an example:

{
  "Sid": "AllowLotameRead",
  "Effect": "Allow",
  "Principal": {
    "AWS": "arn:aws:iam::311305431179:role/LotameS2SExternalRole-us-east-1"
  },
  "Action": [
    "s3:ListBucket",
    "s3:GetObject",
    "s3:DeleteObject",
    "s3:PutObject"
  ],
  "Resource": [
    "arn:aws:s3:::<your bucket name>",
    "arn:aws:s3:::<your bucket name>/<base path for Lotame files>/*"
  ]
}

Lotame requires partners to provide the following details once the above has been completed:

  • The S3 bucket name where Lotame will retrieve the files.

  • As noted below, a directory structure is based on the root of the s3 bucket directory you provided above.

    • <clientDir1>/<clientDir2> Your Technical Account Manager will coordinate the naming of these two directories. Once created, they are static.
    • <clientDir1>/<clientDir2>/archive: Once processed, the data files are moved to this static folder.
    • <clientDir1>/<clientDir2>/YYYYMMDD: This directory should be created daily by the process on your side that deposits the day's files.

Lotame supports retrieving files from your Google Cloud Storage bucket. To support, please provide Lotame with the following:

  • Storage Bucket Name & Path
  • Google Cloud Project ID
  • Service Account HMAC Key
    • Permissions on the account that allows "Storage Object Admin"

Please share the HMAC key securely. Reach out to your Lotame account representative or support@lotame.com for options.

sFTP Transfer

Client-Hosted Server

To host the files on your sFTP server, complete the following:

  • Lotame will provide you with our SSH public key, which you must apply to your server's trusted keys.
  • Provide Lotame with the DNS or IP address of the sFTP server.
  • Provide Lotame with the username that will allow our SSH key to log in.
  • Create a directory structure as noted below based on the root of the username's path.
    • <clientDir1>/<clientDir2> Your Technical Account Manager will coordinate naming these two directories. Once created, they are static.
    • <clientDir1>/<clientDir2>/archive: Once processed, the data files are moved to this static folder.
    • <clientDir1>/<clientDir2>/YYYYMMDD: This directory should be created daily by the process on your side that deposits the day's files.

Lotame Hosted Server

Lotame provides an sFTP server for file transfer. When you provide an SSH public key (in RSA format), Lotame will provide the sFTP user account and directory path to drop the files. 

sFTP processing requires a specific directory format, as described below.

  • <clientDir> Your Technical Account Manager will coordinate the naming of this directory. Once created, the directory is static.
  • <clientDir>/<YYYYMMDD>: This directory should be created daily by the process on your side that deposits the day's files..

Required Data Files

Regardless of the choices made on transfer type and file type above, Lotame requires the following three files to process segments into the platform successfully. 

Segment Membership File

This is the main data file with a static filename of segmentmembership.json.gz. This file is GZIP encoded to be space efficient and minimize transfer timings and associated transfer costs.

Each row in the file is a JSON object for an ID. The ID should be unique in the file. The JSON object must be on a single row without line breaks. Details of the JSON object are in the next section of this document.

The Done file has a filename of YYYYMMDD.done (the .done extension is case-sensitive). This zero-length file should be created when all processing on your end is complete and ready for Lotame to ingest the data. Processing on our end will poll for the .done file before ingesting the Segment Membership file.

This ZIP file at the top of the document has examples of each file for your reference when sending behaviors to Lotame.

This file has a static filename of segmentmembership.json.gz.md5. This plain-text file only contains the MD5 checksum of the Segment Membership file discussed above. An example terminal command to create the file is:

md5sum segmentmembership.json.gz > segmentmembership.json.gz.md5

This file is optional to send for extra validation that your segmentmembership.json.gz.md5 file was successfully transferred. If used, Lotame will validate the MD5 checksum against the segmentmembership.json.gz.md5 file before processing. If you choose to use it, please inform your account manager so we can enable this extra processing.

Segment Membership File Specifications

Core JSON Object

Key Data Type Required? Description
userid String Yes The id of the user.
Note
- A userID should only appear once (one row) in a given file
- IDs must only contain alphanumeric characters.
idtype String Yes Allows Lotame to identify the ID type of the userid (PID, IDFA, RIDA, etc)
Refer to the User ID Type section further down this document for the complete list of supported ID types.

Note: The idtype value is case-insensitive.
segments Array of Strings Yes List of behaviors that you want to associate with the userid.
ips array of ip objects No The sightings data for the userID. See the below table for IPs object details
remove String No Segments to remove from the userid. It is specifically applicable when sending delta files daily.

IPs Object Details

Key Data Type Required? Description
ip String Yes IP4 address. (IPv6 not currently supported)
timestamps Array of Strings Yes The epoch timestamp in which the segment was attributed to the profile. Must be in UTC seconds.

Supported ID Types

Lotame supports the following ID types to be passed in the idtype field in the Data Membership file:

Web IDs
IDType Description
PID Lotame Profile IDs
TPID Third-Party IDs
Mobile IDs
IDType Description
IDFA Apple Identifier for Advertising
GAID Google Advertising ID
Connected TV (CTV) IDs
IDType Description
RIDA Roku TV and Streaming Devices
TVOS Apple TV
CHRM Android TV / Google Chromecast Devices
AFAI Amazon Fire Stick / TV Devices
SMSG Samsung TVs
MSAI Microsoft Advertiser ID
SONY Sony TV
LGTV LG TV
FTCH Fetch TV
UUID Generic CTV ID. Use this type if the CTV ID manufacturer does not match the above types.
Declared IDs
IDType Description
EMAIL Plain-Text Email (ex: email@test.com)
EMAIL_SHA256 Email address SHA-256 encrypted
(ex: 79c7558b8890ef49f29a022b7704edeeafa4af9bd90f1771e87dbf5db7ce1f03)
Note: Emails and SHA256 Emails must be submitted to a Lotame-owned AWS S3 bucket.

Use Cases

Monetizing Data with Behaviors

Below are examples of a single line of JSON within your segmentmembership file. The row must contain a valid user ID, ID type, and segment IDs. Lotame supports receiving multiple idtype values. Files should contain only the ID type agreed upon for the directory. Please consult your Lotame representative if you're unsure.

{"userid":"e1ec57335975afb3eb81c898d09cf9b9","idtype":"PID","segments": ["1","2abc","123www"]}
{"userid":"2v2ab-13b1-4350-be7f-fd21d0p2b","idtype":"GAID","segments":["pp3456","7654cdef"]}
{"userid":"1a2fbcab-13b1-4350-be7f-fd21d0924f4b","idtype":"IDFA","segments":["lmnop3234"]}
{"userid":"e480a48a23eb8c6b82401048c3456945a7020ad0470402219bcec770d43b406e","idtype":"PANO","segments": ["rockmusic"]}
{"userid":"email@test.com","idtype":"EMAIL","segments":["vbn432","mju6yh"]}
{"userid":"79c7558b8890ef49f29a022b7704edeeafa4af9bd90f1771e87dbf5db7ce1f03","idtype":"EMAIL_SHA256","segments":["lkj899","ert345"]}

Monetizing with Behavior & Sightings Information

Partners including IP address & timestamp with their data should follow the example below. The row must contain a valid user ID, ID type, segment ID, IP address and timestamp. Lotame requires the most recent IP address and timestamp for each user ID.

{"userid":"e1ec57335975afb3eb81c898d09cf9b9","idtype":"PID","segments":["a1234","b5678"],"ips":[{"ip":"10.255.255.182","timestamps":["1533149952","1533150020"]},{"ip":"10.255.100.222","timestamps":["1533140052"]}]}
{"userid":"38094715-24da-4d3d-afc8-92a3456b4489","idtype":"AFAI","segments":["ccc890","l45ask"],"ips":[{"ip":"10.255.255.111","timestamps":["1533147852","1533150360"]}]}