1

Server to Server Technical Specs - Inbound to Lotame - Batch File Processing

Overview

This document describes Lotame’s server-to-server specs for delivering data to the Lotame platform in batch. The standard use-case is to send behavioral data tied to select userIDs into the Lotame platform. Partners and clients can optionally add additional objects to the file to enhance the data including behavior recency, userID sightings data, country code and more. 

The main components of Lotame’s Inbound Batch specs include:

  1. File Server Location - where the files will be stored

  2. File Names - The names for the three required files 

  3. File Format - The required file format

The main use-cases of Lotame’s Inbound Batch specs include:

  1. Send behavioral data tied to userIDs into the Lotame platform
  2. Send userID and sightings data (IP + timestamp) to graph userID to Lotame Panorama IDs ™
  3. Send a mapping file of Lotame userIds (pids, maids, ctv ids) tied to your own userID (tpid)
     

Supported File Transfer Mechanisms

For batch file processing, Lotame supports two types of file transfer options for our clients: AWS S3 buckets and sFTP server.

All AWS or sFTP key transfers will be handled in a secure way to minimize access to the keys. Lotame by default will use a direct-access only Google Doc, but we can work with other options should your company provide it. Please do not share keys in a non-secure way such as in an email.

AWS S3 Transfer

For S3 transfers, Lotame will look for files daily on an S3 bucket provided by the client. 

Lotame requires clients to provide the following:

  1. AWS Secret Key. Please provide it securely, coordinate with your Technical Account Manager for options.
  2. AWS Bucket Name
  3. AWS access permissions to download and move files. The move is for archiving processed files to an archive directory.
  4. A directory structure as noted below based on the root of the S3 bucket.

<clientDir1>/<clientDir2> = Your Technical Account Manager will coordinate with naming for these two directories. Once created, they are static.

<clientDir1>/<clientDir2>/archive = Once processed, data is move to this static folder.

<clientDir1>/<clientDir2>/YYYYMMDD = This is the directory that should be created daily by the process on your side that then deposits the day's files within.

sFTP Transfer

Lotame Hosted sFTP Server

Lotame provides an sFTP server for file transfer. When clients provide an SSH public key, Lotame will return the sFTP user account and directory path to drop the files. 

sFTP processing requires a specific directory format, as described below.

<clientDir>/<YYYYMMDD> = <clientDir> value will be provided by Lotame. This is static and will not change. <YYYYMMDD> value is the date of the file contents. Your upload process must create this directory and deposit the files.

Client Hosted sFTP Server

In the case where the client wants to host the files on their sFTP server, we can support that by doing the following:

  1. Lotame will provide you with our SSH public key which you will need to apply to your server.
  2. Provide Lotame the DNS or IP address of the sFTP server.
  3. Provide Lotame the username that will allow our SSH key to login.
  4. A directory structure as noted below based at the root of the username's path.

<clientDir1>/<clientDir2> = Your Technical Account Manager will coordinate with naming for these two directories. Once created, they are static.

<clientDir1>/<clientDir2>/archive = Once processed, data is move to this static folder.

<clientDir1>/<clientDir2>/YYYYMMDD = This is the directory that should be created daily by the process on your side that then deposits the day's files within.

Data File Types

Lotame supports receiving files as comprehensive or deltas from the prior file. Comprehensive each day contains every segment for each ID in the file. Delta files contain only segments that have been added or removed from the ID since the prior file. Please discuss with your Lotame Technical Account Manager options on which file type works best for your integration.

Note: Delta file types require a periodic comprehensive file sent to ensure our segments do not get out of sync. The default is 30 days, but you can work with your Lotame Technical Account Manager to decide a different timeframe that is best for your business needs.

Data File Specifications

Regardless of the choices made on transfer type and file type above, Lotame requires three files each day for us to process your segments into the Lotame platform successfully. 

  • The segment membership file. This file is the main file containing segments for all IDs to import on that day.
    • Filename: segmentmembership.json.gz
  • A file containing the MD5 hash of the contents of the above file. This file allows Lotame to validate the completeness of the segment membership file's download to Lotame servers for processing.
    • Filename: segmentmembership.json.gz.md5
  • A 'done' file signals to Lotame that all processing on your side is complete, and we are approved to start importing the data.
    • Filename: YYYYMMDD.done

Segment Membership File

This file has a static filename of segmentmembership.json.gz. This file should be a gzip encoded file to be file space efficient and minimize transfer timings. 

Each row in the file is a JSON object for an ID. The ID should be unique in the file. The JSON object must be on a single row without linebreaks. Details of the JSON object are below.

Supported ID Types

Supported ID types to be passed in the idtype field are as follows.

Universal IDs
IDType Description
PANO Lotame Panorama ID ™
Web IDs
IDType Description
PID Lotame Profile IDs
TPID Third Party IDs
Mobile IDs
IDType Description
IDFA Apple Identifier for Advertising
GAID Google Advertising ID
Connected TV (CTV) IDs
IDType Description
RIDA Roku TV and Streaming Devices
TVOS Apple TV
CHRM Android TV / Google Chromecast Devices
AFIA Amazon Fire Stick / TV Devices
SMSG Samsung TVs
Note: To see our full list of supported CTV IDs, please see our CTV Overview page.
Customer IDs
Lotame also supports sending in customer specific ID types. Please work with your Lotame Account Manager to learn more about this and whether it is right for your implementation.

Use Case 1: Sending Behaviors to Lotame

When sending behaviors to Lotame, we support the below fields in order to pass appropriate information on the userid you pass.

Key Data Type Required? Description
userid String Yes The id of the user.
Note
- A userID, should only appear once (one row) in a given file
- IDs must only contain alphanumeric characters.
idtype String Yes Allows Lotame to identify the id type of the userid (PID, IDFA, RIDA, etc)
Refer to the User ID Type section for the full list of supported ID types.
segments Array of Objects Yes Behaviors that you want to associate with the userid. If needed, you can pass sighting recency by passing a time stamp inside the object.
Refer to the below “Segment Object Details” table showing how to pass these values.

Note You can specify the recency of the behavior by including a timestamp inside the object.
country String No Two-character ISO country code that allows Lotame to apply a country behavior to the UserID.
ips array of ip objects No The sightings data for the userID. See below table for ips object details
remove String No Segments to remove from the userid. Specifically useful when sending delta files daily.

Segment Object Details

Key Data Type Required? Description
name String Yes Behavior that you want to associate with the userid.
It is preferred segments are alphanumeric with no special characters.
timestamp Integer No The epoch timestamp in which the segment was attributed to the profile. Must be in UTC seconds.

IPs Object Details

Key Data Type Required? Description
ip String Yes IP4 address. (IPv6 not currently supported)
timestamps Array of Strings Yes The epoch timestamp in which the segment was attributed to the profile. Must be in UTC seconds.

Example: Passing in Behaviors Only:

{"userid":"e1ec57335975afb3eb81c898d09cf9b9","idtype":"pid","segments": ["1","2","arts & music"]}
{"userid":"e480a48a23eb8c6b82401048c3456945a7020ad0470402219bcec770d43b406e","idtype":"pano","segments": ["rockmusic"]}
{"userid":"1a2fbcab-13b1-4350-be7f-fd21d0924f4b","idtype":"idfa","segments":["rockmusic"]}
{"userid":"2v2fbcab-13b1-4350-be7f-fd21d0924p2b","idtype":"gaid","segments":["football","sports"]}

Example: Passing Behaviors and a Country

{"userid":"1a2fbcab-13b1-4350-be7f-fd21d0924f4b","idtype":"idfa","country":"US","segments":["rockmusic"]}

Example: Adding Recency to a Behavior:

{"userid":"e1ec57335975afb3eb81c898d09cf9b9","idtype":"pid","segments":[{"name":"rockmusic","timestamp":1614274012},{"name":"jazz","timestamp":1614014782}]}

Example: Adding Sightings information to a Behavior:

{"userid":"e1ec57335975afb3eb81c898d09cf9b9","idtype":"pid","segments":[{"name":"rockmusic","timestamp":1614274012},{"name":"jazz","timestamp":1614014782}],"ips":[{"ip":"10.255.255.182","timestamps":["1533149952","1533150020"]},{"ip":"10.255.100.222","timestamps":["1533140052"]}]}

Example: Remove Behavior from a UserID:

{"userid":"e1ec57335975afb3eb81c898d09cf9b9","idtype":"pid","segments": ["1","2","arts & music"],"remove": ["rock music"]}

Example: Kitchen Sink of all above as we support all options in the same file:

{"userid":"e1ec57335975afb3eb81c898d09cf9b9","idtype":"pid","segments": ["1","2","arts & music"]}
{"userid":"e480a48a23eb8c6b82401048c3456945a7020ad0470402219bcec770d43b406e","idtype":"pano","segments": ["rockmusic"]}
{"userid":"1a2cab-13b1-4350-be7f-fd21d0924f4c","idtype":"idfa","segments":["rockmusic"]}
{"userid":"2v2ab-13b1-4350-be7f-fd21d0p2b","idtype":"gaid","segments":["football","sports"]}
{"userid":"1a2fbcab-13b1-4350-be7f-fd924f4b","idtype":"idfa","country":"US","segments":["rockmusic"]}
{"userid":"38094715-24da-4d3d-afc8-92a3456b4489","idtype":"afai","segments":[{"name":"rockmusic","timestamp":1614274012},{"name":"jazz","timestamp":1614014782}]}
{"userid":"e1ec57335975afb3eb81c898d09cf9b9","idtype":"pid","segments":[{"name":"rockmusic","timestamp":1614274012},{"name":"jazz","timestamp":1614014782}],"ips":[{"ip":"10.255.255.182","timestamps":["1533149952","1533150020"]},{"ip":"10.255.100.222","timestamps":["1533140052"]}]}

Use Case 2: UserID Sightings Data Only

Clients can deliver their sightings data to Lotame’s graph, so that their userIDs can be connected to the Lotame Panorama ID ™.  

For this use-case, Lotame accepts all ID Types - Customer IDs, MAIDs, CTV IDs - except TPIDs and PIDs

Key Data Type Required? Description
userid String Yes The id of the user.
Note
- A userID, should only appear once (one row) in a given file
- IDs must only contain alphanumeric characters.
idtype String Yes Allows Lotame to identify the id type of the userid (IDFA, RIDA, etc)
Refer to the User ID Type section for the full list of supported ID types.
ips array of ip objects Yes The sightings data for the userID. See below table for ips object details

IPs Object Details

Key Data Type Required? Description
ip String Yes IP4 address. (IPv6 not currently supported)
timestamps Array of Strings Yes The epoch timestamp in which the segment was attributed to the profile. Must be in UTC seconds.

Example:

{"userid":"1a2fbcab-13b1-4350-be7f-fd21d0924f4b","idtype":"idfa","ips":[{"ip":"10.255.255.182","timestamps":["1533149952","1533150020"]},{"ip":"10.255.100.222","timestamps":["1533140052"]}]}
{"userid":"79742805f2724a9f85fca160eec219c0","idtype":"tvos","ips":[{"ip":"10.255.255.37","timestamps":["1624564854","1624263894"]},{"ip":"10.255.100.222","timestamps":["1624553738"]}]}

How often should I send sightings data to Lotame?

As a best practice, when clients see activity on a userID, they should send:

  • Min. - 1 timestamp per customer ID + IP per day, 
  • Max. - 12 timestamps per customer ID + IP per hour. 

Use Case 3: ID Mapping File

Send Lotame a mapping file server-side to allow Lotame to map your TPID to a Lotame ID (PID, MAIDs (IDFA, GAID), CTV IDs (RIDA, TVOS, etc)).  This allows Lotame to connect their userIDs to your TPID for user linking and server-side data transfer.

Key Data Type Required? Description
userid String Yes The id of the user. (This would be Lotame userIDs - PIDs, MAIDs or CTV IDs. NOT TPIDs)
Note
- A userID, should only appear once (one row) in a given file
- IDs must only contain alphanumeric characters.
idtype String Yes Allows Lotame to identify the id type of the userid (PID, IDFA, RIDA, etc)
Refer to the User ID Type section for the full list of supported ID types.
tpid String Yes The Third Party ID (TPID) to sync with the UserID.

Example:

{"userid":"e1ec57335975afb3eb81c898d09cf9b9","idtype":"pid","tpid": "abcdefghijklmnop123456789"}

MD5 Hash File

This file has a static filename of segmentmembership.json.gz.md5. This plain-text file only contains the MD5 hash of the gzip encoded Segment Membership file discussed above.

Done File

The Done file has a filename of YYYYMMDD.done. This zero-length file should be created when all processing on your end is complete and ready for Lotame to import the data. Processing on our end will poll for this file before beginning to import the Segment Membership file.

Example Files

This ZIP file at the top of the document has examples of each file for your reference when sending behaviors to Lotame.

Upvote1 Follow
  • 1 Upvotes
  • 8 days agoLast active
  • 739Views
  • 1 Following
Log In