Server to Server Technical Specs - Inbound to Spherical - Batch File Processing

updated 9 days ago

sample-files-sending-behaviors-to-lotame-in-batch

OVERVIEW

This article contains a comprehensive overview of the batch method to deliver data to Spherical, including the mechanisms, the file types, and specifications for those file formats. This will allow to to deliver behavioral data tied to selected userIDs collected by your systems or servers. Additionally, you may optionally enhance the data including behavior recency, userID sightings data, country code and more.

The main components of Lotame’s Inbound Batch process that are covered in this article include:

Determine the mechanism for Spherical to pick up your files
Determine which option to use for updating your data: Comprehensive or Delta
Create the 3 required files, following the appropriate specification
Learn about example use-cases

Supported File Transfer Mechanisms

For batch file processing, the Spherical platform supports two types of file transfer options for our clients: AWS S3 buckets and sFTP server.

All AWS or sFTP key transfers will be handled in a secure way to minimize access to the keys. Spherical by default will use a direct-access only Google Doc, but we can work with other options should your company provide it. Please do not share keys in a non-secure way such as in an email.

FILE SIZE NOTICE

If you expect your files to be larger than 5GB, please reach out to support@lotame.com for some extra assistance.

AWS S3 Transfer

For S3 transfers, Spherical will look for files daily on an S3 bucket provided by the client.

Spherical supports retrieving files using our Lotame External ARN. To support this, please implement a bucket policy on your s3 bucket that allows:

Access to the Lotame ARN of arn:aws:iam::311305431179:role/LotameS2SExternalRole-us-east-1
For the following S3 actions:
- s3:ListBucket
- s3:GetObject
- s3:DeleteObject
- s3:PutObject

An example S3 bucket policy clause is below as an example:

{
  "Sid": "AllowLotameRead",
  "Effect": "Allow",
  "Principal": {
    "AWS": "arn:aws:iam::311305431179:role/LotameS2SExternalRole-us-east-1"
  },
  "Action": [
    "s3:ListBucket",
    "s3:GetObject",
    "s3:DeleteObject",
    "s3:PutObject"
  ],
  "Resource": [
    "arn:aws:s3:::<your bucket name>",
    "arn:aws:s3:::<your bucket name>/<base path for Lotame files>/*"
  ]
}

Lotame requires clients to provide the following:

The S3 bucket name where Spherical will retrieve the files.
A directory structure, as noted below based on the s3 bucket directory you provided above.

<clientDir1>/<clientDir2> = Your Technical Account Manager will coordinate with naming for these two directories. Once created, they are static.

<clientDir1>/<clientDir2>/archive = Once processed, data is moved to this static folder.

<clientDir1>/<clientDir2>/YYYYMMDD = This is the directory that should be created daily by the process on your side that then deposits the day's files within.

sFTP Transfer

Lotame Hosted Server

Lotame provides an sFTP server for file transfer. When clients provide an SSH public key (in RSA format), Lotame will provide the sFTP user account and directory path to drop the files.

Hostname - booth.crwdcntrl.net
Port - standard (22)

sFTP processing requires a specific directory format, as described below.

<clientDir>/<YYYYMMDD> = <clientDir> value will be provided by Lotame. This is static and will not change. <YYYYMMDD> value is the date of the file contents. Your upload process must create this directory and deposit the files.

Client-Hosted Server

In the case where the client wants to host the files on their sFTP server, we can support that by doing the following:

Lotame will provide you with our SSH public key which you will need to apply to your server.
Provide Lotame the DNS or IP address of the sFTP server.
Provide Lotame the username that will allow our SSH key to login.
Create a directory structure as noted below based at the root of the username's path.

<clientDir1>/<clientDir2> = Your Technical Account Manager will coordinate with naming for these two directories. Once created, they are static.

<clientDir1>/<clientDir2>/archive = Once processed, data is move to this static folder.

<clientDir1>/<clientDir2>/YYYYMMDD = This is the directory that should be created daily by the process on your side that then deposits the day's files within.

Data File Types

Lotame supports receiving files as comprehensive or deltas from the prior file. Comprehensive each day contains every segment for each ID in the file. Delta files contain only segments that have been added or removed from the ID since the prior file. Please discuss with your Lotame Technical Account Manager options on which file type works best for your integration.

Note: Delta file types require a periodic comprehensive file sent to ensure our segments do not get out of sync. The default is 30 days, but you can work with your Lotame Technical Account Manager to decide a different timeframe that is best for your business needs.

Data File Specifications

Regardless of the choices made on transfer type and file type above, Lotame requires three files each day for us to process your segments into the Spherical platform successfully.

The segment membership file. This file is the main file containing segments for all IDs to import on that day.
- Filename: segmentmembership.json.gz
A file containing the MD5 hash of the contents of the above file. This file allows Lotame to validate the completeness of the segment membership file's download to Lotame servers for processing.
- Filename: segmentmembership.json.gz.md5
A 'done' file signals that all processing on your side is complete, and we are approved to start importing the data.
- Filename: YYYYMMDD.done

Segment Membership File

This file has a static filename of segmentmembership.json.gz. This file should be a gzip encoded file to be file space efficient and minimize transfer timings.

Each row in the file is a JSON object for an ID. The ID should be unique in the file. The JSON object must be on a single row without line breaks. Details of the JSON object are below.

Supported ID Types

Supported ID types to be passed in the idtype field are as follows.

Universal IDs
IDType	Description
PANO	Lotame Panorama ID ™
Declared IDs
IDType	Description
EMAIL	Plain-Text Email
Web IDs
IDType	Description
PID	Lotame Profile IDs
TPID	Third Party IDs
Mobile IDs
IDType	Description
IDFA	Apple Identifier for Advertising
GAID	Google Advertising ID
Connected TV (CTV) IDs
IDType	Description
RIDA	Roku TV and Streaming Devices
TVOS	Apple TV
CHRM	Android TV / Google Chromecast Devices
AFAI	Amazon Fire Stick / TV Devices
SMSG	Samsung TVs
Note: To see our full list of supported CTV IDs, please see our CTV Overview page.
Customer IDs
Spherical also supports sending in customer specific ID types. Please work with your Lotame Account Manager to learn more about this and whether it is right for your implementation.

MD5 Hash File

This file has a static filename of segmentmembership.json.gz.md5. This plain-text file only contains the MD5 hash of the gzip encoded Segment Membership file discussed above.

Done File

The Done file has a filename of YYYYMMDD.done. This zero-length file should be created when all processing on your end is complete and ready for Spherical to import the data. Processing on our end will poll for this file before beginning to import the Segment Membership file.

This ZIP file at the top of the document has examples of each file for your reference when sending behaviors to Spherical.

For a walk-through tutorial on building each of these file formats for testing, please see this document.

Sample Use Cases

The main use-cases for Inbound to Spherical Batch transfers include:

Send behavioral data tied to userIDs into the Spherical platform
Send userID and sightings data (IP + timestamp) to graph userID to Lotame Panorama IDs ™
Send a mapping file of Lotame userIds (pids, maids, ctv ids) tied to your own userID (tpid)

Use Case 1: Sending Behaviors to Spherical

When sending behaviors to Spherical, we support the below fields in order to pass appropriate information on the userid you pass.

Key	Data Type	Required?	Description
userid	String	Yes	The id of the user. Note: - A userID, should only appear once (one row) in a given file - IDs must only contain alphanumeric characters.
idtype	String	Yes	Allows Spherical to identify the id type of the userid (PID, IDFA, RIDA, etc) Refer to the User ID Type section for the full list of supported ID types. Note The idtype value is case-insensitive.
segments	Array of Objects	Yes	Behaviors that you want to associate with the userid. If needed, you can pass sighting recency by passing a time stamp inside the object. Refer to the below “Segment Object Details” table showing how to pass these values. Note You can specify the recency of the behavior by including a timestamp inside the object.
country	String	No	Two-character ISO country code that allows Spherical to apply a country behavior to the UserID.
ips	array of ip objects	No	The sightings data for the userID. See below table for ips object details
remove	String	No	Segments to remove from the userid. Specifically useful when sending delta files daily.

Segment Object Details

Key	Data Type	Required?	Description
name	String	Yes	Behavior that you want to associate with the userid. It is preferred segments are alphanumeric with no special characters.
timestamp	Integer	No	The epoch timestamp in which the segment was attributed to the profile. Must be in UTC seconds.

IPs Object Details

Key	Data Type	Required?	Description
ip	String	Yes	IP4 address. (IPv6 not currently supported)
timestamps	Array of Strings	Yes	The epoch timestamp in which the segment was attributed to the profile. Must be in UTC seconds.

Example: Passing in Behaviors Only:

{"userid":"e1ec57335975afb3eb81c898d09cf9b9","idtype":"pid","segments": ["1","2","arts & music"]}
{"userid":"e480a48a23eb8c6b82401048c3456945a7020ad0470402219bcec770d43b406e","idtype":"pano","segments": ["rockmusic"]}
{"userid":"1a2fbcab-13b1-4350-be7f-fd21d0924f4b","idtype":"idfa","segments":["rockmusic"]}
{"userid":"johndoe@test.com","idtype":"email","segments":["football","sports"]}

Example: Passing Behaviors and a Country

{"userid":"1a2fbcab-13b1-4350-be7f-fd21d0924f4b","idtype":"idfa","country":"US","segments":["rockmusic"]}

Example: Adding Recency to a Behavior:

{"userid":"e1ec57335975afb3eb81c898d09cf9b9","idtype":"pid","segments":[{"name":"rockmusic","timestamp":1614274012},{"name":"jazz","timestamp":1614014782}]}

Example: Adding Sightings information to a Behavior:

{"userid":"e1ec57335975afb3eb81c898d09cf9b9","idtype":"pid","segments":["rockmusic","jazz"],"ips":[{"ip":"10.255.255.182","timestamps":["1533149952","1533150020"]},{"ip":"10.255.100.222","timestamps":["1533140052"]}]}

Example: Remove Behavior from a UserID:

{"userid":"e1ec57335975afb3eb81c898d09cf9b9","idtype":"pid","segments": ["1","2","arts & music"],"remove": ["rock music"]}

Example: Kitchen Sink of all above as we support all options in the same file:

{"userid":"e1ec57335975afb3eb81c898d09cf9b9","idtype":"pid","segments": ["1","2","arts & music"]}
{"userid":"e480a48a23eb8c6b82401048c3456945a7020ad0470402219bcec770d43b406e","idtype":"pano","segments": ["rockmusic"]}
{"userid":"mattsmitch22@test.com","idtype":"email","segments":["rockmusic"]}
{"userid":"2v2ab-13b1-4350-be7f-fd21d0p2b","idtype":"gaid","segments":["football","sports"]}
{"userid":"1a2fbcab-13b1-4350-be7f-fd924f4b","idtype":"idfa","country":"US","segments":["rockmusic"]}
{"userid":"38094715-24da-4d3d-afc8-92a3456b4489","idtype":"afai","segments":[{"name":"rockmusic","timestamp":1614274012},{"name":"jazz","timestamp":1614014782}]}
{"userid":"e1ec57335975afb3eb81c898d09cf9b9","idtype":"pid","segments":[{"name":"rockmusic","timestamp":1614274012},{"name":"jazz","timestamp":1614014782}],"ips":[{"ip":"10.255.255.182","timestamps":["1533149952","1533150020"]},{"ip":"10.255.100.222","timestamps":["1533140052"]}]}

Use Case 2: UserID Sightings Data Only

Clients can deliver their sightings data to Lotame’s graph, so that their userIDs can be connected to the Lotame Panorama ID ™.

For this use-case, Spherical accepts all ID Types - Customer IDs, MAIDs, CTV IDs - except TPIDs and PIDs

Key	Data Type	Required?	Description
userid	String	Yes	The id of the user. Note: - A userID, should only appear once (one row) in a given file - IDs must only contain alphanumeric characters.
idtype	String	Yes	Allows Spherical to identify the id type of the userid (IDFA, RIDA, etc) Refer to the User ID Type section for the full list of supported ID types. Note The idtype value is case-insensitive.
ips	array of ip objects	Yes	The sightings data for the userID. See below table for ips object details