Use Case: Lotame Data Exchange
OVERVIEW
This article describes the operational process for a Lotame Data Exchange partner to send an inbound batch file of data into Lotame for ingestion and processing.
Supported File Transfer Mechanisms
Lotame supports three file transfer options for batch file processing: Amazon AWS S3 Transfer, Google Cloud Storage, and sFTP transfers.
EMAIL NOTICE If you are sending plain-text or SHA-256 encoded emails, then it is required you send your file to a Lotame hosted AWS S3 bucket.
AWS S3 Transfer
Partner Hosted S3 Bucket
For AWS S3 transfers, Lotame will look for files daily on a partner-hosted S3 bucket.
Lotame supports retrieving files using our External ARN. To grant Lotame access, please perform the following on your S3 bucket.
Allow access to the Lotame ARN of arn:aws:iam::311305431179:role/LotameS2SExternalRole-us-east-1
Provide the following AWS S3 actions to the above ARN:
s3:ListBucket
s3:GetObject
s3:DeleteObject
s3:PutObject
Example of an S3 bucket policy clause:
{
"Sid": "AllowLotameRead",
"Effect": "Allow",
"Principal": {
"AWS": "arn:aws:iam::311305431179:role/LotameS2SExternalRole-us-east-1"
},
"Action": [
"s3:ListBucket",
"s3:GetObject",
"s3:DeleteObject",
"s3:PutObject"
],
"Resource": [
"arn:aws:s3:::<your bucket name>",
"arn:aws:s3:::<your bucket name>/<base path for Lotame files>/*"
]
}
Lotame requires partners to provide the following details once the above has been completed:
The S3 bucket name where Lotame will retrieve the files.
As noted below, a directory structure is based on the root of the s3 bucket directory you provided above.
- <clientDir1>/<clientDir2> Your Technical Account Manager will coordinate the naming of these two directories. Once created, they are static.
- <clientDir1>/<clientDir2>/archive: Once processed, the data files are moved to this static folder.
- <clientDir1>/<clientDir2>/YYYYMMDD: This directory should be created daily by the process on your side that deposits the day's files.
Lotame Hosted S3 Bucket
When a Lotame-hosted S3 Bucket is used, partners are required to provide the following:
ARN that will be used to access the bucket
Lotame will return:
- AWS S3 bucket name
A directory structure under that bucket that looks like the following
- <clientDir1>/<clientDir2> Your Technical Account Manager will coordinate the naming of these two directories. Once created, they are static.
- <clientDir1>/<clientDir2>/YYYYMMDD: This directory should be created daily by the process on your side that deposits the day's files.
Google Cloud Storage
Lotame supports retrieving files from your Google Cloud Storage bucket. To support, please provide Lotame with the following:
- Storage Bucket Name & Path
- Google Cloud Project ID
- Service Account HMAC Secret & Access Keys
- Permissions on the account that allows "Storage Object Admin"
Please share the HMAC keys securely. Reach out to your Lotame account representative or support@lotame.com for options.
sFTP Transfer
Client-Hosted Server
To host the files on your sFTP server, complete the following:
- Apply the Lotame SSH key below to your server's trusted keys.
ssh-rsa AAAAB3NzaC1yc2EAAAABIwAAAQEA2xLuBkss5vuD86YPGPc5/YMtN5w6jRc0dcDye13nQijWgasx/64Iz4yG0adaGomkXmrbSFzy5ViK1cq561/XGXuYkjqAd2EHf2su9VFv5kfRnRtGwsfLQWrwXAdyq+CfKrc5QIH9e63DioYEC8vXoQEWYMhdlohRPxcceExdTv5PsffMCDmYjDIYTqJPO0/D2lSokJfLcQcgdkXq8jeTFXqgdEhegLaYhccOqBVrlJQyXMM5eYRrh+nGFxgOrTu02tnOXXClZbE5ysgs6ch7CLOjCBoF+COeUYpaAW+OM+SjWNVvUQC4zDRr+paIC3YKK/SWmPCixdzevGIYwdHBwQ== support@lotame.com
- Provide Lotame with the DNS or IP address of the sFTP server.
- Provide Lotame with the username that will allow our SSH key to log in.
- Create a directory structure as noted below based on the root of the username's path.
- <clientDir1>/<clientDir2> Your Technical Account Manager will coordinate naming these two directories. Once created, they are static.
- <clientDir1>/<clientDir2>/archive: Once processed, the data files are moved to this static folder.
- <clientDir1>/<clientDir2>/YYYYMMDD: This directory should be created daily by the process on your side that deposits the day's files.
Lotame Hosted Server
Lotame provides an sFTP server for file transfer. When you provide an SSH public key (in RSA format), Lotame will provide the sFTP user account and directory path to drop the files.
- Hostname - booth.crwdcntrl.net
- Port - standard (22)
sFTP processing requires a specific directory format, as described below.
- <clientDir> Your Technical Account Manager will coordinate the naming of this directory. Once created, the directory is static.
- <clientDir>/<YYYYMMDD>: This directory should be created daily by the process on your side that deposits the day's files..
Required Data Files
Regardless of the choices made on transfer type and file type above, Lotame requires the following three files to process segments into the platform successfully.
Segment Membership File
This is the main data file with a static filename of segmentmembership.json.gz. This file is GZIP encoded to be space efficient and minimize transfer timings and associated transfer costs.
Each row in the file is a JSON object for an ID. The ID should be unique in the file. The JSON object must be on a single row without line breaks. Details of the JSON object are in the next section of this document.
Done File
The Done file has a filename of YYYYMMDD.done (the .done extension is case-sensitive). This zero-length file should be created when all processing on your end is complete and ready for Lotame to ingest the data. Processing on our end will poll for the .done file before ingesting the Segment Membership file.
This ZIP file at the top of the document has examples of each file for your reference when sending behaviors to Lotame.
Checksum File (Optional)
This file has a static filename of segmentmembership.json.gz.md5. This plain-text file only contains the MD5 checksum of the Segment Membership file discussed above. An example terminal command to create the file is:
md5sum segmentmembership.json.gz > segmentmembership.json.gz.md5
This file is optional to send for extra validation that your segmentmembership.json.gz.md5 file was successfully transferred. If used, Lotame will validate the MD5 checksum against the segmentmembership.json.gz.md5 file before processing. If you choose to use it, please inform your account manager so we can enable this extra processing.
Segment Membership File Specifications
Core JSON Object
Key | Data Type | Required? | Description |
userid | String | Yes | The id of the user. Note: - A userID should only appear once (one row) in a given file - IDs must only contain alphanumeric characters. |
idtype | String | Yes | Allows Lotame to identify the ID type of the userid (PID, IDFA, RIDA, etc) Refer to the User ID Type section further down this document for the complete list of supported ID types. Note: The idtype value is case-insensitive. |
segments | Array of Strings | Yes | List of behaviors that you want to associate with the userid. |
consent | Boolean | Yes, unless tcfconsent key is passed | Instructs Lotame as to whether you have obtained consent for the user/record. - true = yes, consent has been provided - false = no, consent has not be provided Note: - If blank, false, or set to anything other than true, Lotame will skip the user record. - Alternatively, you can use the tcfconsent key for EMEA users. |
tcfconsent | String | Yes, unless consent key is passed | For EMEA users, you can provide the TCF consent string that was obtained along with the collected data included in the record Note: - If blank or Lotame’s vender ID (95) does not have proper consent, Lotame will ignore the user record - Alternatively, you can use the consent key |
ips | array of ip objects | No | The sightings data for the userID. See the below table for IPs object details |
remove | String | No | Segments to remove from the userid. It is specifically applicable when sending delta files daily. |
IPs Object Details
Key | Data Type | Required? | Description |
ip | String | Yes | IP4 address. (IPv6 not currently supported) |
timestamps | Array of Strings | Yes | The epoch timestamp in which the segment was attributed to the profile. Must be in UTC seconds. |
Supported ID Types
Lotame supports the following ID types to be passed in the idtype field in the Segment Membership file:
Web IDs | |
IDType | Description |
PID | Lotame Profile IDs |
TPID | Third-Party IDs NOTE: TPIDs are client-specific and require Lotame approval and implementation before use. |
Mobile IDs | |
IDType | Description |
IDFA | Apple Identifier for Advertising |
GAID | Google Advertising ID |
Connected TV (CTV) IDs | |
IDType | Description |
RIDA | Roku TV and Streaming Devices |
TVOS | Apple TV |
CHRM | Android TV / Google Chromecast Devices |
AFAI | Amazon Fire Stick / TV Devices |
SMSG | Samsung TVs |
MSAI | Microsoft Advertiser ID |
SONY | Sony TV |
LGTV | LG TV |
FTCH | Fetch TV |
UUID | Generic CTV ID. Use this type if the CTV ID manufacturer does not match the above types. |
Declared IDs | |
IDType | Description |
Plain-Text Email (ex: email@test.com) NOTE: Plain-text email support is in private beta. |
|
EMAIL_SHA256 | Email address SHA-256 encrypted (NOTE - lowercase and remove whitespace before hashing) (ex: 79c7558b8890ef49f29a022b7704edeeafa4af9bd90f1771e87dbf5db7ce1f03) |
Note: Emails and SHA256 Emails must be submitted to a Lotame-owned AWS S3 bucket. |
Examples
Monetizing Data with Behaviors
Below are examples of a single line of JSON within the Segment Membership file. Each row must contain a valid userid, idtype, and segment set. Lotame supports receiving multiple idtype values. Files should contain only the idtypes agreed upon for the directory.
{"userid":"e1ec57335975afb3eb81c898d09cf9b9","idtype":"PID","segments": ["1","2abc","123www"],"consent":true}
{"userid":"2v2ab-13b1-4350-be7f-fd21d0p2b","idtype":"GAID","segments":["pp3456","7654cdef"],"consent":true}
{"userid":"1a2fbcab-13b1-4350-be7f-fd21d0924f4b","idtype":"IDFA","segments":["lmnop3234"],"consent":true}
{"userid":"e480a48a23eb8c6b82401048c3456945a7020ad0470402219bcec770d43b406e","idtype":"PANO","segments": ["rockmusic"],"consent":true}
{"userid":"email@test.com","idtype":"EMAIL","segments":["vbn432","mju6yh"],"tcfconsent":"CPS0IbiPS0IbiABAMBFRACCsAKgAAAPAAIYgAvwAwAAgAEAC-AAxEAA"}
{"userid":"79c7558b8890ef49f29a022b7704edeeafa4af9bd90f1771e87dbf5db7ce1f03","idtype":"EMAIL_SHA256","segments":["lkj899","ert345"],"consent":true}
Monetizing with Behavior & Sightings Information
Partners including IP address & timestamp with their data should follow the example below. Each row must contain a valid userid, idtype, segments , IP address and timestamp. Lotame requires the most recent IP address and timestamp for each user ID.
{"userid":"e1ec57335975afb3eb81c898d09cf9b9","idtype":"PID","segments":["a1234","b5678"],"ips":[{"ip":"10.255.255.182","timestamps":["1533149952","1533150020"]},{"ip":"10.255.100.222","timestamps":["1533140052"]}],"tcfconsent":"CPS0IbiPS0IbiABAMBFRACCsAKgAAAPAAIYgAvwAwAAgAEAC-AAxEAA"}
{"userid":"38094715-24da-4d3d-afc8-92a3456b4489","idtype":"AFAI","segments":["ccc890","l45ask"],"ips":[{"ip":"10.255.255.111","timestamps":["1533147852","1533150360"]}],"consent":true}
Best Practices
Following these best practices ensures your data is eligible for monetization in the Lotame Data Marketplace while minimizing data storage and hosting fees.
ID Syncing
Technical documentation and guidance for syncing ID’s with Lotame can be found here:
Cookie Syncing - If initiating a user sync pixel on your inventory to sync cookie ID’s with Lotame, we recommend the sync pixel be fired once per user every 30 days.
Panorama ID Retrieval - If initiating Lotame’s Sync.JS tag to GET the Panorama ID, Lotame automatically throttles the call based on information we have stored in localStorage. Lotame recommends firing the Sync.JS tag on every page load.
Batch File Delivery
General
Lotame recommends batch files are separated by ID type (Panorama ID, MAID, Email/HEM, Cookie, CTV) when possible
Segment Membership Files
Lotame recommends that partners send incremental files daily in addition to ensuring that every active user is delivered at least once every 30 days.
When uploading a user for full refresh files, please include the user's comprehensive segment list. This allows Lotame to add and remove segments on the user based on what was previously provided
You must communicate your segment batch file configuration to the Lotame support team to ensure data is processed correctly
Sightings Files
While it is possible to include sightings data in segment batch files, Lotame recommends sending sightings data in batch separate and independent of segment data
A best practice is to send a daily batch file that includes activity against an ID for the previous day
Content aside
Categories
-
- Activation Channel Partners
-
- Inbound to Spherical
- Overview
- Use Case: ID Mapping
- Use Case: Sending Behaviors to Spherical
- Use Case: Sending Sightings Data Only
- Use Case: Lotame Data Exchange
- Supported File Transfer Mechanisms
- Supported ID Types
- How to Create and Share Public SSH Keys
- Passing User Consent
- Onboarding - File Specifications
- Manually Uploading Onboarding Files to Lotame S3
- Outbound from Spherical
- Inbound to Spherical
- Data Collection with Lightning Tag
- Administrative API
- User ID Syncing
- Single Sign-On (SSO)
- Lotame Panorama ID—Prebid, Google ESP & Amazon CxM
- 11 mths agoLast active
- 707Views
- 1 Following