Azure Blob Storage

Push vehicle data updates directly into your Azure Blob for scalable storage and analysis.

With the Azure Blob integration it's possible for you to get vehicle data updates pushed directly to your Blob container. This tutorial will help you get started step by step.

Introduction to Azure Blob Storage

Azure Blob Storagearrow-up-right is a versatile object storage service offered by Azure. It's possible for you to manage data container in your own Azure account and then configuring your data applications in our console to push data to your container. Using this method for data delivery works for many use-cases:

  • When you want to collect data from your fleet evaluation purposes without any technical effort.

  • When you are using Big Data platforms that have existing plugins for Azure Blob Storage, containers can serve as excellent middle steps in your data pipeline.

  • If you are scaling your data usage to very large fleets, Blob Storage can keep up with large volumes of push data at a low computing cost.

  • When slight latency in the data delivery is not an issue. Efficiency is more important than real-time updates.

  • Continue to work with data payloads based on the Auto API JSON schema.

File and object structure

All messages that are published are divided into different topics. By looking at the topic structure you can determine the type of data that is being sent and for which vehicle.

azure_blog_path: <your-storage-name>.blob.core.windows.net/<container-name>/<application-id>/<auto-api-version>/<year>/<month>/<day>/<time>-<hash>.jsonl

breakdown:
  your-storage-name: The name of the storage that you created
  app_id: Your unique App ID e.g. bdd34602-531f-4b2c-8576-3ab6e635dbcd
  auto_api_level: level13 # latest version of the Auto API
  year: The year when the JSON file was created
  month: The month when the JSON file was created
  day: The day when the JSON file was created
  time: The specific time when the JSON file was created
  hash: A hash to make the file name unique

Here is the JSONLarrow-up-right format that is used for every file that is stored. As each file contains many json objects each separated by new line.

circle-info

DATA PAYLOAD

Each line of this file is a data object that holds vehicle data points. The data object conforms to the Streaming Schema V2.

circle-info

Our system is designed to guarantee at-least-once delivery of each message. Please consider the possibility that a message_id is delivered more than once.

Setting up Azure Blob Storage

Create a Storage

The first thing you will need to do is to create a Storage account in your Azure account. If you have no Azure account, you can sign up herearrow-up-right. Once logged in, follow these steps:

  1. Go to "Create a resource" -> "Marketplace" and create a storage account show the page to create storage account in azure

  2. Visit the new resource and choose "Access Control (IAM)" and in the top menu, choose "Add" and then "Role Assignment" in the dropdown show the "Access Control" menu in the storage account page

  3. Choose the role "Storage Blob Data Contributor" search for Storage Blob Data Contributor and select that in the resource

  4. In "Members", start typing to select "highmobility-azure-sink" and hit save

  5. Now you can go to the "Properties" tab in the left menu, and copy the container URLcopy container url

Grant High Mobility

As the next step, you need to grant HighMobility access to your Storage Blob, in order to do that you need to find your "Azure Account Tenant ID" and visit the following link to grant your consent.

You need to provide different High Mobility client id for live and sandbox instances:

  • Live instance client id: 9d643e86-bae8-4a2f-a260-a6450e8360e0

  • Sandbox instance client id: c3856a66-06a4-4ae8-821b-3ef9ae3d87cc

circle-exclamation

Configuring your application

Now go back to the High Mobility console, choose your project ›› streaming ›› Azure Sink and enter the URL of your container.

If everything works well, we push a file to your storage and you can verify it. In case of error, double check the instruction, make sure you have granted us access (via oauth url provided).

If you still have issue, please contact us on Slackarrow-up-right or get in touch with our supportarrow-up-right.

Our Azure Blob storage offers great flexibility and configurability. As shown in the screenshot below, you can fine-tune the frequency at which data is flushed into their bucket. This allows you to decide for yourself the balance between data delivery time and data delivery costs, since each push to Azure Blob Storage incurs a cost.

circle-exclamation

Last updated

Was this helpful?