Ctrlk
For the complete documentation index, see llms.txt. This page is also available as Markdown.

AWS S3

Push vehicle data updates directly into your AWS S3 buckets for scalable storage and analysis.

With the AWS S3 integration it's possible for you to get vehicle data updates pushed directly to your AWS S3 buckets. This tutorial will help you get started step by step.

Introduction to AWS S3

AWS S3 is a versatile object storage service offered by Amazon Web Services. It's possible for you to manage data buckets in your own AWS account and then configuring your data applications in our console to push data to your buckets. Using this method for data delivery works for many use-cases:

  • When you want to collect data from your fleet evaluation purposes without any technical effort.

  • When you are using Big Data platforms that have existing plugins for AWS S3, buckets can serve as excellent middle steps in your data pipeline.

  • If you are scaling your data usage to very large fleets, S3 can keep up with large volumes of push data at a low computing cost.

  • When latency in the data delivery is not an issue. Efficiency is more important than real-time updates.

  • Continue to work with data payloads based on the Auto API JSON schema.

File and object structure

All messages that are published are divided into different topics. By looking at the topic structure you can determine the type of data that is being sent and for which vehicle.

s3_uri: s3://<bucket-name>/<app-id>/<auto-api-version>/<year>/<month>/<day>/<time>-<hash>.json

breakdown:
  bucket_name: The name of the bucket that you created
  app_id: Your unique App ID e.g. 9FBBEDED80595912588FF4FF
  auto_api_level: level13 # latest version of the Auto API
  year: The year when the JSON file was created
  month: The month when the JSON file was created
  day: The day when the JSON file was created
  time: The specific time when the JSON file was created
  hash: A hash to make the file name unique

Here is the JSON format that is used for every file that is stored. As each file contains many data updates coming from different vehicles, all messages are encapsulated in a JSON-array.

Our system is designed to guarantee at-least-once delivery of each message. Please consider the possibility that a message_id is delivered more than once.

Here's an example of the JSON that would be sent for odometer data:

To deliver the data, we are using a native batching mechanism that is provided by AWS. This means that each file has a maximum of 1'000 messages and contains push messages from the last 5 minutes. If the maximum amount of messages is reached before 5 minutes, the file is simply stored and the next file is created right away. You can expect to receive many files in your buckets, however considering the nature of working with buckets and objects in AWS this is not an issue when processing the data for further use.

DATA PAYLOAD

All data objects, that hold the specific car data payload, are formatted according to the Auto API JSON schema. A detailed breakdown of each property is found in our JSON schema spec.

Create an S3 Bucket

The first thing you will need to do is to create an S3 bucket in your AWS account. If you have no AWS account, you can sign up here. Once logged in, follow these steps:

  1. Choose "Amazon S3" to open the specific service page

  2. Click the "Create bucket" button

  3. Fill in a name of your bucket, which needs to be globally unique at AWS

  4. Select "EU (Ireland) eu-west-1" as the AWS Region

  5. All other options on the bucket creation page can remain as already selected by default. Make sure that "Block all public access" is checked as you want to keep your bucket secure.

  6. Click "Create bucket" to finalise the creation

AWS REGION

Note that data can currently only be delivered to buckets in the eu-west-1 region.

Example of a created S3 bucket:

  1. In the bucket details page, click on the "Permissions" tab

  2. Scroll down to the "Bucket policy" and click edit and enter the following JSON. Just mind that you need to replace the <BUCKET_NAME> placeholders with the name of the bucket that you just created. The <PRINCIPAL>placeholder is set according to if you are configuring a sandbox application or live data application.

Once you have created your bucket, you will need to update it's permissions policy to allow us to push data to it:

  • Sandbox principal: arn:aws:iam::238742147684:role/service-role/production-sandbox-kafka-s3-sink-01-role-kzzbjkvy

  • Live data principal: arn:aws:iam::238742147684:role/service-role/production-live-kafka-s3-sink-01-role-zytmbiq9

Policy:

Configure Project

  1. Select the project you have created.

  2. Choose "API Certificates" then "Streaming" in the left menu section.

  3. Choose "AWS S3 Bucket" as your choice of push data delivery.

  4. Click "Save and Update"

  5. Enter the name of the S3 bucket you created in AWS and hit "Save". As all bucket names are globally unique in AWS, you just need to enter the name and not any ARN or URI.

Once you have saved we will verify that the connection is working by storing a test file in your bucket. Only if the test push succeeds will the S3 configuration be saved. Feel free to delete the test file after the verification.

Receiving data

You will now get any new data being generated by your vehicles pushed to your bucket. If you are working in the sandbox environment, you can easily try it out in the simulator by changing a few data properties. Just make sure your application has the permission to receive the specific data.

HEADS UP

As the S3 batch operations are running 5 minutes each, expect the first data to come in five minutes after a push message has been published for your application.

You will now get vehicle data delivered to you for all vehicles that have approved access grants. To receive an access grant, have a look at the OAuth2 or the Fleet Clearance guide depending on your specific application.

PreviousAWS KinesisNextAzure Blob Storage

Last updated

Was this helpful?