AWS S3
Push vehicle data updates directly into your AWS S3 buckets for scalable storage and analysis.
With the AWS S3 integration it's possible for you to get vehicle data updates pushed directly to your AWS S3 buckets. This tutorial will help you get started step by step.
Introduction to AWS S3
AWS S3 is a versatile object storage service offered by Amazon Web Services. It's possible for you to manage data buckets in your own AWS account and then configuring your data applications in our console to push data to your buckets. Using this method for data delivery works for many use-cases:
When you want to collect data from your fleet evaluation purposes without any technical effort.
When you are using Big Data platforms that have existing plugins for AWS S3, buckets can serve as excellent middle steps in your data pipeline.
If you are scaling your data usage to very large fleets, S3 can keep up with large volumes of push data at a low computing cost.
When slight latency in the data delivery is not an issue. Efficiency is more important than real-time updates.
Continue to work with data payloads based on the Auto API JSON schema.
File and object structure
All messages that are published are divided into different topics. By looking at the topic structure you can determine the type of data that is being sent and for which vehicle.
s3_uri: s3://<bucket-name>/<app-id>/<auto-api-version>/<year>/<month>/<day>/<time>-<hash>.json
breakdown:
bucket_name: The name of the bucket that you created
app_id: Your unique App ID e.g. 9FBBEDED80595912588FF4FF
auto_api_level: level13 # latest version of the Auto API
year: The year when the JSON file was created
month: The month when the JSON file was created
day: The day when the JSON file was created
time: The specific time when the JSON file was created
hash: A hash to make the file name unique
Here is the JSON format that is used for every file that is stored. As each file contains many data updates coming from different vehicles, all messages are encapsulated in a JSON-array.
Our system is designed to guarantee at-least-once delivery of each message. Please consider the possibility that a message_id is delivered more than once.
Here's an example of the JSON that would be sent for odometer data:
To deliver the data, we are using a native batching mechanism that is provided by AWS. This means that each file has a maximum of 1'000 messages and contains push messages from the last 5 minutes. If the maximum amount of messages is reached before 5 minutes, the file is simply stored and the next file is created right away. You can expect to receive many files in your buckets, however considering the nature of working with buckets and objects in AWS this is not an issue when processing the data for further use.
DATA PAYLOAD
All data objects, that hold the specific car data payload, are formatted according to the Auto API JSON schema. A detailed breakdown of each property is found in our JSON schema spec.
Create an S3 Bucket
The first thing you will need to do is to create an S3 bucket in your AWS account. If you have no AWS account, you can sign up here. Once logged in, follow these steps:
Choose "Amazon S3" to open the specific service page
Click the "Create bucket" button
Fill in a name of your bucket, which needs to be globally unique at AWS
Select "EU (Ireland) eu-west-1" as the AWS Region
All other options on the bucket creation page can remain as already selected by default. Make sure that "Block all public access" is checked as you want to keep your bucket secure.
Click "Create bucket" to finalise the creation
AWS REGION
Note that data can currently only be delivered to buckets in the eu-west-1 region.
Example of a created S3 bucket:
In the bucket details page, click on the "Permissions" tab
Scroll down to the "Bucket policy" and click edit and enter the following JSON. Just mind that you need to replace the
<BUCKET_NAME>placeholders with the name of the bucket that you just created. The<PRINCIPAL>placeholder is set according to if you are configuring a sandbox application or live data application.
Once you have created your bucket, you will need to update it's permissions policy to allow us to push data to it:
Sandbox principal:
arn:aws:iam::238742147684:role/service-role/production-sandbox-kafka-s3-sink-01-role-kzzbjkvyLive data principal:
arn:aws:iam::238742147684:role/service-role/production-live-kafka-s3-sink-01-role-zytmbiq9
Policy:
Configure Project
Select the project you have created.
Choose "API Certificates" then "Streaming" in the left menu section.
Choose "AWS S3 Bucket" as your choice of push data delivery.
Click "Save and Update"
Enter the name of the S3 bucket you created in AWS and hit "Save". As all bucket names are globally unique in AWS, you just need to enter the name and not any ARN or URI.
Once you have saved we will verify that the connection is working by storing a test file in your bucket. Only if the test push succeeds will the S3 configuration be saved. Feel free to delete the test file after the verification.
Receiving data
You will now get any new data being generated by your vehicles pushed to your bucket. If you are working in the sandbox environment, you can easily try it out in the simulator by changing a few data properties. Just make sure your application has the permission to receive the specific data.
HEADS UP
As the S3 batch operations are running 5 minutes each, expect the first data to come in five minutes after a push message has been published for your application.
You will now get vehicle data delivered to you for all vehicles that have approved access grants. To receive an access grant, have a look at the OAuth2 or the Fleet Clearance guide depending on your specific application.
Last updated
Was this helpful?