Getting started with Amazon Kinesis

Our Amazon Kinesis integration enables you to direct data from your vehicles to your own Kinesis stream. This tutorial will help you get started step by step.

Table of Contents

Introduction to Kinesis

Amazon Kinesis is a scalable, real-time data streaming service designed to handle large volumes of data generated by applications, devices, or events. It enables the collection, processing, and analysis of streaming data with low latency. It allows you to:

  • Move from batch-based to real-time, event-driven data processing.
  • Ingest, store, and analyze continuous streams of data through a fully managed service.
  • Process high-throughput data payloads from your vehicles.
  • Continue to work with data payloads based on the Auto API JSON schema.

Message structure

V2 MESSAGE FORMAT

Important: Note that our Kinesis feed is using version 2 message format, which is different from version 1 that our other streaming options are currently using. Version 2 includes combined property updates in a single message.

Here is the JSON format that is used for every message that is pushed to your stream. All data objects, that hold the specific car data payload, are formatted according to the Auto API JSON v2 schema. A breakdown of each property is found in the Auto API schema.

{
   "message_id": {Unique Message ID: string},
   "version": 2,
   "vin": {Vehicle Identification Number: string},
   "capability": {Auto API Capability: string},
   "property": {Auto API Property: string},
   "data": {Auto API Schema: objects}
}

Our system is designed to guarantee at-least-once delivery of each message. Please consider the possibility that a message_id is delivered more than once.

Here's an example of the JSON that would be sent for the odometer data:

{
    "version": 2,
    "message_id": "B10BB67D6B4D2EBAB49FD3F81D41111C2AC0D6E0816DE47F87DCDD1388F3D911",
    "vin": "VFXXXXXXXXXXXXXXX",
    "data": {
        "diagnostics": {
            "odometer": [
                {
                    "data": {
                        "value": 2050,
                        "unit": "kilometers"
                    },
                    "timestamp": "2021-10-06T13:59:25.648943Z"
                }
            ],
            "speed": [
                {
                    "data": {
                        "value": 42,
                        "unit": "kilometers_per_hour"
                    },
                    "timestamp": "2021-10-06T13:59:25.648943Z"
                },
                {
                    "data": {
                        "value": 55,
                        "unit": "kilometers_per_hour"
                    },
                    "timestamp": "2021-10-06T13:59:28.788943Z"
                }
            ]
        },
        "race": {
            "vehicle_moving": [
                {
                    "data": "moving",
                    "timestamp": "2021-10-06T13:59:25.648943Z"
                }
            ]
        }
    }
}

Create a Kinesis Stream

The first thing you will need to do is to create a Kinesis stream in your AWS account. If you have no AWS account, you can sign up here. Once logged in, follow these steps:

  1. Choose "Amazon Kinesis" to open the specific service page
  2. Click the "Create data stream" button
  3. Fill in a name of your data stream
  4. All options on the kinesis creation page can remain as already selected by default
  5. Click "Create stream" to finalise the creation

Once you have created your kinesis stream, you will need to update it's permissions policy to allow us to stream data to it:

  1. In the kinesis details page, click on the "Data stream sharing" tab
  2. Scroll down to the "Resource policy" and click edit and enter the following JSON.

SANDBOX VS. LIVE

Just mind that you need to replace the <YOUR_KINESIS_STREAM_ARN> placeholders with the name of the stream that you just created. The <PRINCIPAL> placeholder is set according to if you are configuring a sandbox application or live data application.

Sandbox principal: arn:aws:iam::238742147684:user/xv-janus-kinesis-sink-production-sandbox
Live data principal: arn:aws:iam::238742147684:user/xv-janus-kinesis-sink-production-live

{
    "Version": "2012-10-17",
    "Statement": [
        {
            "Sid": "StreamWriteStatementID",
            "Effect": "Allow",
            "Principal": {
                "AWS": "<PRINCIPAL>"
            },
            "Action": [
                "kinesis:DescribeStreamSummary",
                "kinesis:ListShards",
                "kinesis:PutRecord",
                "kinesis:PutRecords"
            ],
            "Resource": "<YOUR_KINESIS_STREAM_ARN>"
        }
    ]
}

Configure Project

In your project view, configure your app to push data to your Kinesis stream. In the High Mobility console:

  1. Select the project you have created.
  2. Choose "API Certificates" then "Streaming" in the left menu section.
  3. Choose "Amazon Kinesis" as your choice of push data delivery.
  4. Click "Save and Update"
  5. Enter the ARN of your Kinesis stream that you created in AWS and hit "Save".

Delivery Method

Note that you can only have one specific streaming data delivery methodat any point of time.

Once you have saved we will verify that the connection is working by storing a test message in your bucket. The shard ID and sequence ID will be listed for you to verify. Only if the test push succeeds will the Kinesis configuration be saved.

Receiving data

You will now get any new data being generated by your vehicles pushed to your stream. If you are working in the sandbox environment, you can easily try it out in the simulator by changing a few data properties. Just make sure your application has the permission to receive the specific data.

You will now get vehicle data delivered to you for all vehicles that have approved access grants. To receive an access grant, have a look at the Fleet Clearance guide.