Use Case: Lotame Data Exchange

OVERVIEW

This article describes the operational process for a Lotame Data Exchange partner to send an inbound batch file of data into Lotame for ingestion and processing.

Supported File Transfer Mechanisms

Lotame supports three file transfer options for batch file processing: Amazon AWS S3 Transfer, Google Cloud Storage, and sFTP transfers.

FILE SIZE NOTICE: If you expect your files to be larger than 5GB, please contact support@lotame.com for assistance.

EMAIL NOTICE If you are sending plain-text or SHA-256 encoded emails, then it is required you send your file to a Lotame hosted AWS S3 bucket.

AWS S3 Transfer

Partner Hosted S3 Bucket

For AWS S3 transfers, Lotame will look for files daily on a partner-hosted S3 bucket. 

Lotame supports retrieving files using our External ARN. To grant Lotame access, please perform the following on your S3 bucket.

  • Allow access to the Lotame ARN of arn:aws:iam::311305431179:role/LotameS2SExternalRole-us-east-1

  • Provide the following AWS S3 actions to the above ARN:

    • s3:ListBucket

    • s3:GetObject

    • s3:DeleteObject

    • s3:PutObject

Example of an S3 bucket policy clause:

{
  "Sid": "AllowLotameRead",
  "Effect": "Allow",
  "Principal": {
    "AWS": "arn:aws:iam::311305431179:role/LotameS2SExternalRole-us-east-1"
  },
  "Action": [
    "s3:ListBucket",
    "s3:GetObject",
    "s3:DeleteObject",
    "s3:PutObject"
  ],
  "Resource": [
    "arn:aws:s3:::<your bucket name>",
    "arn:aws:s3:::<your bucket name>/<base path for Lotame files>/*"
  ]
}

Lotame requires partners to provide the following details once the above has been completed:

  • The S3 bucket name where Lotame will retrieve the files.

  • As noted below, a directory structure is based on the root of the s3 bucket directory you provided above.

    • <clientDir1>/<clientDir2> Your Technical Account Manager will coordinate the naming of these two directories. Once created, they are static.
    • <clientDir1>/<clientDir2>/archive: Once processed, the data files are moved to this static folder.
    • <clientDir1>/<clientDir2>/YYYYMMDD: This directory should be created daily by the process on your side that deposits the day's files.

Lotame Hosted S3 Bucket

When a Lotame-hosted S3 Bucket is used, partners are required to provide the following:

  • ARN that will be used to access the bucket

Lotame will return:

  • AWS S3 bucket name

  • A directory structure under that bucket that looks like the following

    • <clientDir1>/<clientDir2> Your Technical Account Manager will coordinate the naming of these two directories. Once created, they are static.
    • <clientDir1>/<clientDir2>/YYYYMMDD: This directory should be created daily by the process on your side that deposits the day's files.

Google Cloud Storage

Lotame supports retrieving files from your Google Cloud Storage bucket. To support, please provide Lotame with the following:

  • Storage Bucket Name & Path
  • Google Cloud Project ID
  • Service Account HMAC Key
    • Permissions on the account that allows "Storage Object Admin"

Please share the HMAC key securely. Reach out to your Lotame account representative or support@lotame.com for options.

sFTP Transfer

Client-Hosted Server

To host the files on your sFTP server, complete the following:

  • Lotame will provide you with our SSH public key, which you must apply to your server's trusted keys.
  • Provide Lotame with the DNS or IP address of the sFTP server.
  • Provide Lotame with the username that will allow our SSH key to log in.
  • Create a directory structure as noted below based on the root of the username's path.
    • <clientDir1>/<clientDir2> Your Technical Account Manager will coordinate naming these two directories. Once created, they are static.
    • <clientDir1>/<clientDir2>/archive: Once processed, the data files are moved to this static folder.
    • <clientDir1>/<clientDir2>/YYYYMMDD: This directory should be created daily by the process on your side that deposits the day's files.

Lotame Hosted Server

Lotame provides an sFTP server for file transfer. When you provide an SSH public key (in RSA format), Lotame will provide the sFTP user account and directory path to drop the files. 

sFTP processing requires a specific directory format, as described below.

  • <clientDir> Your Technical Account Manager will coordinate the naming of this directory. Once created, the directory is static.
  • <clientDir>/<YYYYMMDD>: This directory should be created daily by the process on your side that deposits the day's files..

Required Data Files

Regardless of the choices made on transfer type and file type above, Lotame requires the following three files to process segments into the platform successfully. 

Segment Membership File

This is the main data file with a static filename of segmentmembership.json.gz. This file is GZIP encoded to be space efficient and minimize transfer timings and associated transfer costs.

Each row in the file is a JSON object for an ID. The ID should be unique in the file. The JSON object must be on a single row without line breaks. Details of the JSON object are in the next section of this document.

Done File

The Done file has a filename of YYYYMMDD.done (the .done extension is case-sensitive). This zero-length file should be created when all processing on your end is complete and ready for Lotame to ingest the data. Processing on our end will poll for the .done file before ingesting the Segment Membership file.

This ZIP file at the top of the document has examples of each file for your reference when sending behaviors to Lotame.

Checksum File (Optional)

This file has a static filename of segmentmembership.json.gz.md5. This plain-text file only contains the MD5 checksum of the Segment Membership file discussed above. An example terminal command to create the file is:

md5sum segmentmembership.json.gz > segmentmembership.json.gz.md5

This file is optional to send for extra validation that your segmentmembership.json.gz.md5 file was successfully transferred. If used, Lotame will validate the MD5 checksum against the segmentmembership.json.gz.md5 file before processing. If you choose to use it, please inform your account manager so we can enable this extra processing.

Segment Membership File Specifications

Core JSON Object

Key Data Type Required? Description
userid String Yes The id of the user.
Note: 
- A userID should only appear once (one row) in a given file
- IDs must only contain alphanumeric characters.
idtype String Yes Allows Lotame to identify the ID type of the userid (PID, IDFA, RIDA, etc)
Refer to the User ID Type section further down this document for the complete list of supported ID types.

Note: The idtype value is case-insensitive.
segments Array of Strings Yes List of behaviors that you want to associate with the userid.
ips array of ip objects No The sightings data for the userID. See the below table for IPs object details
remove String No Segments to remove from the userid. It is specifically applicable when sending delta files daily.

IPs Object Details

Key Data Type Required? Description
ip String Yes IP4 address. (IPv6 not currently supported)
timestamps Array of Strings Yes The epoch timestamp in which the segment was attributed to the profile. Must be in UTC seconds.

Supported ID Types

Lotame supports the following ID types to be passed in the idtype field in the Segment Membership file:

Web IDs
IDType Description
PID Lotame Profile IDs
TPID Third-Party IDs
Mobile IDs
IDType Description
IDFA Apple Identifier for Advertising
GAID Google Advertising ID
Connected TV (CTV) IDs
IDType Description
RIDA Roku TV and Streaming Devices
TVOS Apple TV
CHRM Android TV / Google Chromecast Devices
AFAI Amazon Fire Stick / TV Devices
SMSG Samsung TVs
MSAI Microsoft Advertiser ID
SONY Sony TV
LGTV LG TV
FTCH Fetch TV
UUID Generic CTV ID. Use this type if the CTV ID manufacturer does not match the above types.
Declared IDs
IDType Description
EMAIL Plain-Text Email (ex: email@test.com)
NOTE: Plain-text email support is in private beta.
EMAIL_SHA256 Email address SHA-256 encrypted
(ex: 79c7558b8890ef49f29a022b7704edeeafa4af9bd90f1771e87dbf5db7ce1f03)
Note: Emails and SHA256 Emails must be submitted to a Lotame-owned AWS S3 bucket.

Examples

Monetizing Data with Behaviors

Below are examples of a single line of JSON within the Segment Membership file. Each row must contain a valid userid, idtype, and segment set. Lotame supports receiving multiple idtype values. Files should contain only the idtypes agreed upon for the directory. 

{"userid":"e1ec57335975afb3eb81c898d09cf9b9","idtype":"PID","segments": ["1","2abc","123www"]}
{"userid":"2v2ab-13b1-4350-be7f-fd21d0p2b","idtype":"GAID","segments":["pp3456","7654cdef"]}
{"userid":"1a2fbcab-13b1-4350-be7f-fd21d0924f4b","idtype":"IDFA","segments":["lmnop3234"]}
{"userid":"e480a48a23eb8c6b82401048c3456945a7020ad0470402219bcec770d43b406e","idtype":"PANO","segments": ["rockmusic"]}
{"userid":"email@test.com","idtype":"EMAIL","segments":["vbn432","mju6yh"]}
{"userid":"79c7558b8890ef49f29a022b7704edeeafa4af9bd90f1771e87dbf5db7ce1f03","idtype":"EMAIL_SHA256","segments":["lkj899","ert345"]}

Monetizing with Behavior & Sightings Information

Partners including IP address & timestamp with their data should follow the example below. Each row must contain a valid userid, idtype, segments , IP address and timestamp. Lotame requires the most recent IP address and timestamp for each user ID.

{"userid":"e1ec57335975afb3eb81c898d09cf9b9","idtype":"PID","segments":["a1234","b5678"],"ips":[{"ip":"10.255.255.182","timestamps":["1533149952","1533150020"]},{"ip":"10.255.100.222","timestamps":["1533140052"]}]}
{"userid":"38094715-24da-4d3d-afc8-92a3456b4489","idtype":"AFAI","segments":["ccc890","l45ask"],"ips":[{"ip":"10.255.255.111","timestamps":["1533147852","1533150360"]}]}

Best Practices

Following these best practices ensures your data is eligible for monetization in the Lotame Data Marketplace while minimizing data storage and hosting fees.

ID Syncing

  • Technical documentation and guidance for syncing ID’s with Lotame can be found here: 

    • Cookie Syncing - If initiating a user sync pixel on your inventory to sync cookie ID’s with Lotame, we recommend the sync pixel be fired once per user every 30 days. 

    • Panorama ID Retrieval - If initiating Lotame’s Sync.JS tag to GET the Panorama ID, Lotame automatically throttles the call based on information we have stored in localStorage. Lotame recommends firing the Sync.JS tag on every page load. 

Batch File Delivery

  • General

    • Lotame recommends batch files are separated by ID type (Panorama ID, MAID, Email/HEM, Cookie, CTV) when possible

    • Do not provide batch files larger than 5GB. If your file is larger than 5GBs, we recommend splitting the files into smaller sizes

  • Segment Membership Files

    • Lotame recommends that partners send incremental files daily in addition to ensuring that every active user is delivered at least once every 30 days.

      • When uploading a user for full refresh files, please include the user's comprehensive segment list. This allows Lotame to add and remove segments on the user based on what was previously provided

    • You must communicate your segment batch file configuration to the Lotame support team to ensure data is processed correctly

  • Sightings Files

    • While it is possible to include sightings data in segment batch files, Lotame recommends sending sightings data in batch separate and independent of segment data

    • A best practice is to send a daily batch file that includes activity against an ID for the previous day

Previous Use Case: Sending Sightings Data Only Next Supported File Transfer Mechanisms

Content aside

Categories

View all topics
  • 5 mths agoLast active
  • 363Views
  • 1 Following