Use Case: Lotame Data Exchange

updated 2 wk ago

OVERVIEW

This article describes the operational process for a Lotame Data Exchange partner to send an inbound batch file of data into Lotame for ingestion and processing.

Supported File Transfer Mechanisms

Lotame supports three file transfer options for batch file processing: Amazon AWS S3 Transfer, Google Cloud Storage, and sFTP transfers.

EMAIL NOTICE If you are sending plain-text or SHA-256 encoded emails, then it is required you send your file to a Lotame hosted AWS S3 bucket.

AWS S3 Transfer

Partner Hosted S3 Bucket

For AWS S3 transfers, Lotame will look for files daily on a partner-hosted S3 bucket.

Lotame supports retrieving files using our External ARN. To grant Lotame access, please perform the following on your S3 bucket.

Allow access to the Lotame ARN of arn:aws:iam::311305431179:role/LotameS2SExternalRole-us-east-1
Provide the following AWS S3 actions to the above ARN:
- s3:ListBucket
- s3:GetObject
- s3:DeleteObject
- s3:PutObject

Example of an S3 bucket policy clause:

{
  "Sid": "AllowLotameRead",
  "Effect": "Allow",
  "Principal": {
    "AWS": "arn:aws:iam::311305431179:role/LotameS2SExternalRole-us-east-1"
  },
  "Action": [
    "s3:ListBucket",
    "s3:GetObject",
    "s3:DeleteObject",
    "s3:PutObject"
  ],
  "Resource": [
    "arn:aws:s3:::<your bucket name>",
    "arn:aws:s3:::<your bucket name>/<base path for Lotame files>/*"
  ]
}

Lotame requires partners to provide the following details once the above has been completed:

The S3 bucket name where Lotame will retrieve the files.
As noted below, a directory structure is based on the root of the s3 bucket directory you provided above.
- <clientDir1>/<clientDir2> Your Technical Account Manager will coordinate the naming of these two directories. Once created, they are static.
- <clientDir1>/<clientDir2>/archive: Once processed, the data files are moved to this static folder.
- <clientDir1>/<clientDir2>/YYYYMMDD: This directory should be created daily by the process on your side that deposits the day's files.

Lotame Hosted S3 Bucket

When a Lotame-hosted S3 Bucket is used, partners are required to provide the following:

ARN that will be used to access the bucket

Lotame will return:

AWS S3 bucket name
A directory structure under that bucket that looks like the following
- <clientDir1>/<clientDir2> Your Technical Account Manager will coordinate the naming of these two directories. Once created, they are static.
- <clientDir1>/<clientDir2>/YYYYMMDD: This directory should be created daily by the process on your side that deposits the day's files.

Google Cloud Storage

Lotame supports retrieving files from your Google Cloud Storage bucket. To support, please provide Lotame with the following:

Storage Bucket Name & Path
Google Cloud Project ID
Service Account HMAC Secret & Access Keys
- Permissions on the account that allows "Storage Object Admin"

Please share the HMAC keys securely. Reach out to your Lotame account representative or support@lotame.com for options.

sFTP Transfer

Client-Hosted Server

To host the files on your sFTP server, complete the following:

Apply the Lotame SSH key below to your server's trusted keys.

ssh-rsa AAAAB3NzaC1yc2EAAAABIwAAAQEA2xLuBkss5vuD86YPGPc5/YMtN5w6jRc0dcDye13nQijWgasx/64Iz4yG0adaGomkXmrbSFzy5ViK1cq561/XGXuYkjqAd2EHf2su9VFv5kfRnRtGwsfLQWrwXAdyq+CfKrc5QIH9e63DioYEC8vXoQEWYMhdlohRPxcceExdTv5PsffMCDmYjDIYTqJPO0/D2lSokJfLcQcgdkXq8jeTFXqgdEhegLaYhccOqBVrlJQyXMM5eYRrh+nGFxgOrTu02tnOXXClZbE5ysgs6ch7CLOjCBoF+COeUYpaAW+OM+SjWNVvUQC4zDRr+paIC3YKK/SWmPCixdzevGIYwdHBwQ== support@lotame.com

Provide Lotame with the DNS or IP address of the sFTP server.
Provide Lotame with the username that will allow our SSH key to log in.
Create a directory structure as noted below based on the root of the username's path.
- <clientDir1>/<clientDir2> Your Technical Account Manager will coordinate naming these two directories. Once created, they are static.
- <clientDir1>/<clientDir2>/archive: Once processed, the data files are moved to this static folder.
- <clientDir1>/<clientDir2>/YYYYMMDD: This directory should be created daily by the process on your side that deposits the day's files.

Lotame Hosted Server

Lotame provides an sFTP server for file transfer. When you provide an SSH public key (in RSA format), Lotame will provide the sFTP user account and directory path to drop the files.

Hostname - booth.crwdcntrl.net
Port - standard (22)

sFTP processing requires a specific directory format, as described below.

<clientDir> Your Technical Account Manager will coordinate the naming of this directory. Once created, the directory is static.
<clientDir>/<YYYYMMDD>: This directory should be created daily by the process on your side that deposits the day's files..

Required Data Files

Regardless of the choices made on transfer type and file type above, Lotame requires the following three files to process segments into the platform successfully.

Segment Membership File

This is the main data file with a static filename of segmentmembership.json.gz. This file is GZIP encoded to be space efficient and minimize transfer timings and associated transfer costs.

Each row in the file is a JSON object for an ID. The ID should be unique in the file. The JSON object must be on a single row without line breaks. Details of the JSON object are in the next section of this document.

Done File

The Done file has a filename of YYYYMMDD.done (the .done extension is case-sensitive). This zero-length file should be created when all processing on your end is complete and ready for Lotame to ingest the data. Processing on our end will poll for the .done file before ingesting the Segment Membership file.

This ZIP file at the top of the document has examples of each file for your reference when sending behaviors to Lotame.

Checksum File (Optional)

This file has a static filename of segmentmembership.json.gz.md5. This plain-text file only contains the MD5 checksum of the Segment Membership file discussed above. An example terminal command to create the file is:

md5sum segmentmembership.json.gz > segmentmembership.json.gz.md5

This file is optional to send for extra validation that your segmentmembership.json.gz.md5 file was successfully transferred. If used, Lotame will validate the MD5 checksum against the segmentmembership.json.gz.md5 file before processing. If you choose to use it, please inform your account manager so we can enable this extra processing.

Segment Membership File Specifications

Core JSON Object

Key	Data Type	Required?	Description
userid	String	Yes	The id of the user. Note: - A userID should only appear once (one row) in a given file - IDs must only contain alphanumeric characters.
idtype	String	Yes	Allows Lotame to identify the ID type of the userid (PID, IDFA, RIDA, etc) Refer to the User ID Type section further down this document for the complete list of supported ID types. Note: The idtype value is case-insensitive.
segments	Array of Strings	Yes	List of behaviors that you want to associate with the userid.
consent	Boolean	Yes, unless tcfconsent key is passed	Instructs Lotame as to whether you have obtained consent for the user/record. - true = yes, consent has been provided - false = no, consent has not be provided Note: - If blank, false, or set to anything other than true, Lotame will skip the user record. - Alternatively, you can use the tcfconsent key for EMEA users.
tcfconsent	String	Yes, unless consent key is passed	For EMEA users, you can provide the TCF consent string that was obtained along with the collected data included in the record Note: - If blank or required vender IDs do not have proper consent, Lotame will ignore the user record - Alternatively, you can use the consent key
ips	array of ip objects	No	The sightings data for the userID. See the below table for IPs object details
remove	String	No	Segments to remove from the userid. It is specifically applicable when sending delta files daily.

IPs Object Details

Key	Data Type	Required?	Description
ip	String	Yes	IP4 address. (IPv6 not currently supported)
timestamps	Array of Strings	Yes	The epoch timestamp in which the segment was attributed to the profile. Must be in UTC seconds.

Supported ID Types

Lotame supports the following ID types to be passed in the idtype field in the Segment Membership file:

Web IDs
IDType	Description
PID	Lotame Profile IDs
TPID	Third-Party IDs NOTE: TPIDs are client-specific and require Lotame approval and implementation before use.
Mobile IDs
IDType	Description
IDFA	Apple Identifier for Advertising
GAID	Google Advertising ID
Connected TV (CTV) IDs
IDType	Description
RIDA	Roku TV and Streaming Devices
TVOS	Apple TV
CHRM	Android TV / Google Chromecast Devices
AFAI	Amazon Fire Stick / TV Devices
SMSG	Samsung TVs
MSAI	Microsoft Advertiser ID
SONY	Sony TV
LGTV	LG TV
FTCH	Fetch TV
UUID	Generic CTV ID. Use this type if the CTV ID manufacturer does not match the above types.
Declared IDs
IDType	Description
EMAIL	Plain-Text Email (ex: email@test.com) NOTE: Plain-text email support is in private beta.
EMAIL_SHA256	Email address SHA-256 encrypted (NOTE - lowercase and remove whitespace before hashing) (ex: 79c7558b8890ef49f29a022b7704edeeafa4af9bd90f1771e87dbf5db7ce1f03)
Note: Emails and SHA256 Emails must be submitted to a Lotame-owned AWS S3 bucket.

Examples

Monetizing Data with Behaviors

Below are examples of a single line of JSON within the Segment Membership file. Each row must contain a valid userid, idtype, and segment set. Lotame supports receiving multiple idtype values. Files should contain only the idtypes agreed upon for the directory.

{"userid":"e1ec57335975afb3eb81c898d09cf9b9","idtype":"PID","segments": ["1","2abc","123www"],"consent":true}
{"userid":"2v2ab-13b1-4350-be7f-fd21d0p2b","idtype":"GAID","segments":["pp3456","7654cdef"],"consent":true}
{"userid":"1a2fbcab-13b1-4350-be7f-fd21d0924f4b","idtype":"IDFA","segments":["lmnop3234"],"consent":true}
{"userid":"e480a48a23eb8c6b82401048c3456945a7020ad0470402219bcec770d43b406e","idtype":"PANO","segments": ["rockmusic"],"consent":true}
{"userid":"email@test.com","idtype":"EMAIL","segments":["vbn432","mju6yh"],"tcfconsent":"CPS0IbiPS0IbiABAMBFRACCsAKgAAAPAAIYgAvwAwAAgAEAC-AAxEAA"}
{"userid":"79c7558b8890ef49f29a022b7704edeeafa4af9bd90f1771e87dbf5db7ce1f03","idtype":"EMAIL_SHA256","segments":["lkj899","ert345"],"consent":true}

Monetizing with Behavior & Sightings Information

Partners including IP address & timestamp with their data should follow the example below. Each row must contain a valid userid, idtype, segments , IP address and timestamp. Lotame requires the most recent IP address and timestamp for each user ID.

{"userid":"e1ec57335975afb3eb81c898d09cf9b9","idtype":"PID","segments":["a1234","b5678"],"ips":[{"ip":"10.255.255.182","timestamps":["1533149952","1533150020"]},{"ip":"10.255.100.222","timestamps":["1533140052"]}],"tcfconsent":"CPS0IbiPS0IbiABAMBFRACCsAKgAAAPAAIYgAvwAwAAgAEAC-AAxEAA"}
{"userid":"38094715-24da-4d3d-afc8-92a3456b4489","idtype":"AFAI","segments":["ccc890","l45ask"],"ips":[{"ip":"10.255.255.111","timestamps":["1533147852","1533150360"]}],"consent":true}

Best Practices

Following these best practices ensures your data is eligible for monetization in the Lotame Data Marketplace while minimizing data storage and hosting fees.

ID Syncing

Technical documentation and guidance for syncing ID’s with Lotame can be found here:
- Cookie Syncing - If initiating a user sync pixel on your inventory to sync cookie ID’s with Lotame, we recommend the sync pixel be fired once per user every 30 days.
- Panorama ID Retrieval - If initiating Lotame’s Sync.JS tag to GET the Panorama ID, Lotame automatically throttles the call based on information we have stored in localStorage. Lotame recommends firing the Sync.JS tag on every page load.

Batch File Delivery

General
- Lotame recommends batch files are separated by ID type (Panorama ID, MAID, Email/HEM, Cookie, CTV) when possible
Segment Membership Files
- Lotame recommends that partners send incremental files daily in addition to ensuring that every active user is delivered at least once every 30 days.
  - When uploading a user for full refresh files, please include the user's comprehensive segment list. This allows Lotame to add and remove segments on the user based on what was previously provided
- You must communicate your segment batch file configuration to the Lotame support team to ensure data is processed correctly
Sightings Files
- While it is possible to include sightings data in segment batch files, Lotame recommends sending sightings data in batch separate and independent of segment data
- A best practice is to send a daily batch file that includes activity against an ID for the previous day

Previous Use Case: Sending Sightings Data Only Next Supported File Transfer Mechanisms