Server to Server Technical Specs - Outbound from Spherical - Batch File Processing

OVERVIEW

This article contains the technical specs for sending data out of the Spherical platform using supported file transfer mechanisms.

AWS S3 Transfer

Lotame supports delivering files to your AWS S3 bucket using our Lotame External ARN. To support, please implement a bucket policy on your s3 bucket that allows:

  • Access to the Lotame ARN of arn:aws:iam::311305431179:role/LotameS2SExternalRole-us-east-1

  • For the following S3 actions:

    • s3:ListBucket

    • s3:GetObject

    • s3:PutObject

An example S3 bucket policy clause is below as an example:

{
  "Sid": "AllowLotameWrite",
  "Effect": "Allow",
  "Principal": {
    "AWS": "arn:aws:iam::311305431179:role/LotameS2SExternalRole-us-east-1"
  },
  "Action": [
    "s3:ListBucket",
    "s3:GetObject",
    "s3:PutObject"
  ],
  "Resource": [
    "arn:aws:s3:::<your bucket name>",
    "arn:aws:s3:::<your bucket name>/<base path for Lotame files>/*"
  ]
}

Lotame requires clients to provide the following:

  • AWS Bucket Name.

  • A directory structure, as noted below based on the s3 bucket directory you provided above.

    • Either <YYYYMMDDHH>/<clientDir> or <clientDir>/<YYYYMMDDHH>. 

    • The <clientDir> value will be provided by Lotame, and it is static and will not change. <YYYYMMDDHH> value is the GMT date + hour of the file contents and will be created by our upload process for each file delivery.

Google Cloud Storage

Lotame supports delivering files to your Google Cloud Storage bucket. To support, please provide Lotame with the following:

  1. Storage Bucket Name & Path

  2. Google Cloud Project ID

  3. Service Account HMAC Secret & Access Keys

    1. Permissions on the account that allows Storage Object Creator and Storage Object Viewer. 

Please share the HMAC keys securely. Reach out to your Lotame account representative for options.

sFTP Transfer

Lotame Hosted sFTP Server

Lotame provides an sFTP server for file transfer. When clients provide an SSH public key, Lotame will return the sFTP user account and directory path to retrieve the files.  

The directory structure is <YYYYMMDDHH>/<clientDir> or <clientDir>/<YYYYMMDDHH>.  The <clientDir> value will be provided by Lotame and it is static and will not change. <YYYYMMDDHH> value is the GMT date + hour of the file contents and will be created by our upload process for each file delivery.

Client Hosted sFTP Server

In the case where you want the files sent to your own sFTP server, complete the following steps: 

  1. Apply the Lotame SSH key below to your server's trusted keys.
    1.   
      ssh-rsa AAAAB3NzaC1yc2EAAAABIwAAAQEA2xLuBkss5vuD86YPGPc5/YMtN5w6jRc0dcDye13nQijWgasx/64Iz4yG0adaGomkXmrbSFzy5ViK1cq561/XGXuYkjqAd2EHf2su9VFv5kfRnRtGwsfLQWrwXAdyq+CfKrc5QIH9e63DioYEC8vXoQEWYMhdlohRPxcceExdTv5PsffMCDmYjDIYTqJPO0/D2lSokJfLcQcgdkXq8jeTFXqgdEhegLaYhccOqBVrlJQyXMM5eYRrh+nGFxgOrTu02tnOXXClZbE5ysgs6ch7CLOjCBoF+COeUYpaAW+OM+SjWNVvUQC4zDRr+paIC3YKK/SWmPCixdzevGIYwdHBwQ== support@lotame.com
  2. Provide Lotame the DNS or IP address of the sFTP server.
  3. Provide Lotame the username that will allow our SSH key to login.
  4. Create a directory as noted below based at the root of your username's path.

<clientDir>= Your Technical Account Manager will coordinate with naming of this directory. Once created, they are static. 

<clientDir>/<YYYYMMDDHH> is the GMT date + hour of the file contents and will be created by our upload process for each file delivery.

Delivery Modes

Lotame supports two delivery modes - comprehensive and incremental. 

Regardless of which mode is used, the delivered files will always include IDs that had activity for the day.

Comprehensive

When comprehensive mode is selected, Lotame will include the user's complete list of audiences each time the user ID is included in a file. We recommend these best practices when using comprehensive mode delivery:

  1. Keep only the latest full record for each ID.

  2. Purge data for an ID 30 days after receipt.

Incremental

When incremental mode is selected, Lotame will only include an ID in a file when:

  1. An ID qualifies for a new audience that has not been delivered in the previous 30 days, or

  2. The system determines it should “refresh” an active ID, in which case the record will include all active audiences for the ID

The default “refresh” is once every 30 days, but you can work with your Lotame representative to determine if a different time frame is available for your business needs.

We recommend these best practices when using incremental mode delivery:

  1. We do not identify the comprehensive records when in incremental mode, so you cannot deterministically action differently on an incremental record vs. a comprehensive record.

  2. Discard data received more than 30 days ago and aggregate all data that remains as the complete set of audiences for an ID.

Data File Specifications

Lotame sends 5 files for the Outbound Batch processing.

  1. Audience Membership file which contains all the UserIDs and their associated audiences.
  2. Audience Metadata file which contains a mapping of audience ID to human-readable description of that audience.
  3. Audience Membership MD5 checksum file to be used to validate the correctness of the Audience Membership file delivery.
  4. Audience Metadata MD5 checksum file to be used to validate the correctness of the Audience Metadata file delivery.
  5. A DONE file to denote to your processing that Lotame is done with file uploads for that day.

Audience Membership File

This file has a static filename of audiencemembership.tsv.gz. This file is a gzip encoded file to be file space efficient and minimize transfer timings. 

The file is tab-separated between the two data points of UserID and Audiences. The UserID are unique in the file.

Key Data Type Description
UserID String UserID identifier to use in the Spherical platform. UserIDs have a a prefix followed by colon to identify the type of identifier passed. Allowed identifier types are listed below.
Audiences Comma Separated List of Strings Comma separated list of audiences to link to the UserID

Supported UserID Identifiers

Universal IDs
Identifier Prefix Description
PANO Lotame Panorama ID ™
Web IDs
Identifier Prefix Description
{no prefix or colon} Lotame Profile IDs (pid)
{no prefix or colon} Third Party IDs (tpid)
Mobile IDs
Identifier Prefix Description
IDFA Apple Identifier for Advertising
GAID Google Advertising ID
Connected TV IDs (OTT) IDs
Identifier Prefix Description
RIDA Roku TV and Streaming Devices
TVOS Apple TV
CHRM Android TV / Google Chromecast Devices
AFIA Amazon Fire TV Devices
SMSG Samsung TVs
HIP4 IPv4 Address
Customer IDs
Spherical also supports exporting your customer-specific ID types you have sent into the system. Please work with your Lotame Account Manager to learn more about this and whether it is right for your implementation.

To see our full list of supported CTV IDs, please see our CTV Overview page.

Lotame Profile IDs (PID)/ Third Party IDs (TPID), Mobile Advertising Device IDs and CTV (OTT) IDs can be combined in the same file, or separated into multiple files if requested.  

  • CTV (OTT) file can be further broken down into multiple files by each device type (Roku, Apple TV etc) on request.

  • Third Party IDs (TPID) and Lotame Profile IDs (PID) cannot be combined in the same file.  

Example File

e1ec57335975afb3eb81c898d09cf9b9    395338,285289
cac945ff3946e16842af8c0951e7b9ee    395338
PANO:0bed52425b1b65dec4cf2373119b16d53938b85871267f094f2ea13a224849e5    365147,395338,296412
PANO:ac3382286397f7297242867dbd2216d539381bc632e61fafd9ba38aebf4a4a30    285289,395338,375412
IDFA:C87CB4B6-87A1-42E3-A994-B3734A0D4F69    395338,375412
IDFA:E860367A-E08C-406F-9472-C57A916A2889    285289,395338
GAID:000958bf-4c0d-4c68-9ff5-d6bbb1b08739    358945,285289
GAID:2v2fbcab-13b1-4350-be7f-fd21d0924p2b    395338

Audience Metadata File

This file has a static filename of metadata.json. 

The JSON file is contains an audience object which contains an array of mapping objects. The format of the mapping object is denoted below.

Key Data Type Description
id Integer Lotame Audience ID value found in the above Audience Membership file
name String Human-readable description of the Audience ID

Example File

{
  "audience":[
    {
      "id":123,
      "name":"Sports Lovers"
    },
    {
      "id":234,
      "name": "Exposed to Campaign X"
    }
  ]
}

Audience Membership MD5 Hash File

This file has a static filename of audiencemembership.tsv.gz.md5. This plain-text file only contains the MD5 hash of the gzip encoded Audience Membership. Use the MD5 hash to validate the completeness of the uploaded Audience Membership file.

Audience Metadata MD5 Hash File

This file has a static filename of metadata.json.md5. This plain-text file contains the MD5 hash of the gzip encoded Audience Metadata file. Use the MD5 hash to validate the completeness of the uploaded Audience Metadata file.

Done File

The Done file has a filename of YYYYMMDDHH.done. This zero-length file will be created when all uploads by Lotame are complete for the day. Please have your processing wait on this file to exist before loading the day's files.

Content aside

Categories

View all topics
  • 1 Upvotes
  • 3 yrs agoLast active
  • 3175Views
  • 1 Following