IoT Azure Pipeline
In this article I build an end to end solution for using Arduino to capture environmental readings, publishing to Azure, then storing and visualizing with Python.
Adding an 18650 battery with JST jack makes this a portable solution for monitoring environmental conditions.
Solution architecture sketch:
Solution components are:
- Arduino telemetry capture
- Azure IoT Hub
- Azure Functions for moving data
- CosmosDB for storing data
- Jupyter Notebook for preparing & publishing visualizations
- Datapane for publishing visualization
Arduino Telemetry Capture
I used an Arduino 1010 Wifi and Env Shield to capture data. This ran constantly in my office, I am primarily interested in the temperature and light level, to see how both vary through the day. The Env Shield captures UV and moisture information but I don’t store or visualize those readings.
The repository for this program is here.
The solution relies on secrets stored in a file named “arduino_secrets.h”. This contains information for connecting to wifi and our MQTT broker.
Your SSID and PASS refer to your wifi network’s name or SSID and password. The other information you’ll need to login to the Azure Portal and create an IoT Hub, name it whatever you want and put it in the region that makes sense for you, otherwise there isn’t too much to configure here. Once created, navigate to the resource and look for the Hostname, this is your broker:
Next we need to provision a device, look for “IoT Devices” along the left side and select “New”;
Name the device, leave all other settings as default; authentication as “Symmetric key” and leave “Auto-generate keys” checked, “Connect this device to an IoT Hub” enabled and save.
Your device ID is the SECRET_DEVICEID.
Now we’ll use the device name and your hostname to build the string for SECRET_USERNAME;
<your-hostname>-devices.net/<your-device>/?api-version=2018–06–30
So one example would be:
myarduino.azure-devices.net/ard1010wifi/?api-version=2018–06–30
Finally we need the SECRET_PASSWORD. We need to generate a SAS Token. You can generate this a few ways but the easiest is to use Visual Studio Code with the Azure IoT Tools plugin. Once installed you’ll need to login to Azure using VS Code, then in the Explorer pane you’ll be able to select Azure IoT Hub, locate your Hub, the Device you created, right click and Generate SAS Token;
Set the expiration in hours, the key is output and copied to your clipboard. You can use it exactly as it is for the SECRET_PASSWORD, just enclose it in quotes. Your code should include “SharedAccessSignature sr=”, do not omit this.
Now let’s talk a bit more about the Arduino program. It publishes a message every minute with readings including temperature, humidity, pressure and illuminance.
First, we connect to our MQTT broker;
Then we read our sensors and use the mqttClient to publish data to Azure. Each reading is added to our JSON Document, which is then serialized and published as an MQTT message.
The program’s loop simply connects to the MQTT broker, publishes messages and waits one minute;
There are other ways to perform the waiting, some solutions take a mills reading, tracking how long the Arduino has been online and dividing it to run every so many seconds. I’ve removed that in favor of delay() because millis will cause overflows and crashes after being online for too long.
Once you upload the program, you should see the messages start rolling into your IoT Hub in the Azure Portal!
Event Hub to CosmosDB
IoT Hub isn’t a persistence or storage solution, we only get access to the messages for a day on the free tier, that means we need to write a solution to pull our readings and store them. For that we’ll use an Azure Function which will run every 3 minutes, read the messages and write them to a CosmosDB where we can store them for reporting.
We can’t actually read from the IoT Hub using a Function, so first we are going to output the files to an Azure Blob account. To set this up we need to navigate back to our IoT Hub and setup Message Routing;
Click “Add”, then name your route and “Add endpoint” and select “Storage”. Most of the defaults can be left, you’ll need to “Pick a container” and create a new storage account if you don’t already have one. This is temporary storage that we will remove the files from using our Function.
The sliders for batch frequency and chunk size window determine how frequently information is written to storage, the defaults are reasonable. Remember the account name, we’ll need its storage key later.
CosmosDB
Next, we need to make a CosmosDB instance if you don’t already have one you can use. Return to the Azure Portal and create one, Cosmos has a lot more options than IoT Hub, I’d recommend;
If you don’t have any cosmos instances you should be eligible for a Free Tier Discount. Once you’ve created a Cosmos instance, we need to add containers. Cosmos containers are like database instances, each is a distinct namespace that we can query.
Click “New Container” in the top left, you may need to click it again after it takes you to the next screen. The first container I call ‘temps’, ‘readings’ might be more appropriate but what’s done is done;
Your partition key is required and I opted to use something that would reliably change, if you hover over the information tip you see this is used to distribute the data so it can be queried.
Azure Function
Now we need to deploy another Azure resource, a Function App. This will execute our code to read from IoT Hub and write to the CosmosDB container we just created.
Azure Functions are extremely inexpensive, paying by the fraction of a second for compute, and we only need a few seconds every 3 minutes, making this solution very cost effective.
The code we’ll deploy to our Function is here.
This code involves a few notable things, the first is the “function.json” which determines how this code is invoked (a timer) and what frequency (every 3 minutes).
You can edit the “schedule” to run more or less frequently.
The real work for this program is done in “__init__.py”. This is the program that will read from the blob storage, parse the readings and write them back to Cosmos.
This sets up our keys and configuration, the secrets are managed as environmental variables, configured in the Function properties. You’ll need to change the container and database names to fit what you configured when setting up your CosmosDB.
Function Secrets
Managing secrets in your Azure Function involves navigating to the Function App and its Configuration. Each secret is accessed using “os.environ.get” with the secret name, to add a new one simply navigate to your Function, Configuration and click “New application setting”, the Name will correspond to the string used to access, the Value is the actual key or secret.
You can see mine are named “blobstring”, “cosmoskey” and “cosmosurl”. To obtain these we’ll need to navigate to the blob account we used we used when saving information from IoT Hub. Navigate to the storage account and select Access Keys;
Select “Show keys” and copy the “Key” value for one of the keys, it can be key1 or key2, they’re identical but allow you to rotate keys independently. This string will be the “Value” for your “blobstring” secret in your Azure Function.
Next we need the Cosmos information. Navigate to your CosmosDB instance and select “Keys” on the left ribbon. Copy the “URI”, this is your “cosmosurl” value, then copy either the “PRIMARY KEY” or “SECONDARY KEY”. Much like with the Storage Account, it doesn’t matter which. This is your “cosmoskey” Value.
Now we have all of the secrets we need and can deploy our Function app.
