Get Started

Last updated: Dec 31th, 2016

Purpose

From the beginning of time, it seems, the community has viewed Microsoft as anti-OpenSource. The purpose of this project is multi-fold. The first goal was to deliver a solution by Microsoft at it's public Ignite event that was built using OpenSource technologies. Therefore, instead of using some flavor of .NET and/or Windows IoT Core, I chose to use offerings such as Node.js, Angular 2.0 and MongoDB.

The second purpose of this project was to facilitate the adoption of Microsoft Azure by enterprise community members who were already engaged in OpenSource projects. I wished to accomplish this by demonstrating the ease of moving the applications from an environment such as Heroku to an Azure cloud platform such as Service Fabric and to swap the data repository from MongoDB to DocumentDB. The objective was to perform these "migrations" with as little configuration changes as possible.

Finally, instead of talking about this process through concepts delivered by PowerPoint, I wanted to provide an application based on a real-world scenario. The application could be utilized by the community as a starter package in which others could contribute or use for their own projects. The scenario, in our case, is managing and tracking assets.

Download

There are two downloads options available.

Complete Source

The downloadable archive has the source code for the web application and the IoT client application.

Download Zip
Raspbian Only

The Raspbian image is a pre-configured, 8GB image for the Raspberry Pi 3 Model B.

Note: The image requires Win32DiskImager which can be found here. Imaging instructions can be found on the Raspberry Pi website.

Download Raspbian Image

 

Prerequisites

While the applications may run in other environments, the demo at Ignite was based on the prerequisites for the two components or this project:

Web Application
  • Heroku (and/or Azure)
  • MongoDB and/or DocumentDB
  • Azure IoT Hub
  • Azure Stream Analytics
Client Application

Installation - Basics

The purpose of this section is to help you prepare the required platforms for hosting the web application and client (Raspberry Pi) application. Keep in mind that the ultimate goal of this project was to demonstrate Microsoft's adoption of OpenSource and Azure's ability to host it. Therefore, the steps below concerning MongoDB and Heroku are completely optional and are denoted as such.

Download Source

Download the project using git git clone https://github.com/a11smiles/IoT-Asset-Management.git.

Once the project is downloaded, there are two separate folders:

  • /pi - contains the client application deployed to a Raspberry Pi
  • /webapp - the Node.js/Angular 2.0 web site

MongoDB (Optional)

For demonstration purposes, I used mlab.com to host my sample main repository. This repository is responsible for storing application users and managed assets. If you don't currently have an available MongoDB database (including locally hosted) and would like to test it's capabilities, mlab.com is a great, free choice.

Let's go ahead and create a sample database for our application so, first, go create a free account and confirm your email address. Now, follow the steps below to create your database and access credentials.

  1. On the dashboard, click "Create New".

  2. For creating a new deployment, select a "Single-node" plan within a "Sandbox" environment.

  3. Enter a database name and click "Create new MongoDB deployment".
  4. You'll then be returned to your deployments dashboard, showing your newly created database.
  5. Click on your new database.
  6. The details page should not have any "Collections" listed for your database, obviously, because you've just created the database. We do not have to create one manually now as our collections will be automatically created with our first documents.
  7. We do need to create an account by which to connect to our database. NOTE: We do not connect to our database using our mlab.com account. Instead we add individual users and their role to our specific databases.
  8. Click on the "Users" tab and click "Add database user."

  9. In the popup dialog, enter a database username and password. Leave "Make read-only" unchecked.
    For me, I entered the following information:

    Username: testadmin
    Password: testpassword
  10. Once I created the user and by observing the configuration at the top of the database configuration page, I can now connect to my hosted MongoDB by using the following URI:

    mongodb://testadmin:testpassword@ds019946.mlab.com:19946/iot-assets

NOTE

Your URI may be different depending on what you used for your username, password, database name and the hostname for the machine that's hosting your database. But, it should look similar to mine.

Heroku (Optional)

As many enterprises are regularly using Heroku for hosting their Node.js applications, I, again, wanted to do the same. As stated earlier, this step is optional.

In order to host our application we'll need to create an "app" in Heroku so, first, go create a free account and confirm your email address. Now, follow the steps below to setup the hosting environment for our Node.js application.

  1. On the dashboard, click "New", then "Create new app".

  2. Give your application a name and click "Create App".

NOTE

Whatever you name your application will be the hostname within your app's URI. (i.e. http://my-app-name.herokuapp.com)

Azure DocumentDB

DocumentDB is Microsoft Azure's NoSQL database platform. It will serve two purposes within our application.

First, DocumentDB, if we choose to use it instead of MongoDB, will serve as our main repository to hold our application's users and assets. Because Microsoft Azure provides a MongoDB connector for DocumentDB, it is very easy to switch between the two databases.

DocumentDB will also be the repository that holds all of our location data for our Bluetooth LE (RadBeacon) devices. Azure's IoT Hub coupled with Stream Analytics will pass the data into DocumentDB.

MongoDB Connector for DocumentDB

Currently, the MongoDB connector for Azure's DocumentDB is in preview. Thus, there is some reduced functionality by using DocumentDB for the main database. This can affect performance and security. Until the connector has greater compatibility and enters "General Availability," it is recommended that MongoDB is used for the main repository.

If you haven't done so already, create an Azure account at https://azure.microsoft.com/. Then, login to your portal to begin.

  1. Once you've logged into your Azure portal, on the left you will find see a menu (it could be collapsed). Click on the to create a new resource.
  2. In the "Search the marketplace" box, search for DocumentDB. You will be presented with two options in the search assist as you see in the below image.

DocumentDB vs. DocumentDB - Protocol Support for MongoDB

As stated in the above warning, DocumentDB - Protocol Support for MongoDB offers support to use DocumentDB with native MongoDB client libraries. If you'd like to use DocumentDB for the primary repository - or even test its functionality - then you'll need to create a DocumentDB with support for the MongoDB protocol. If you'd prefer to stick with MongoDB as the primary repository, then choose DocumentDB without support for the MondDB. Below are the options for each type of DocumentDB repository.

DocumentDB - IoT Repository Only

Follow these steps to set up the repository that is only used by the reporting IoT devices.

  1. Select "DocumentDB (NoSQL)". NOTE: You may have to choose this twice.
  2. In the resulting pane, click "Create"
  3. You'll then need to complete the template options to configure your DocumentDB instance.
    • ID - A globally unique name that defines your DocumentDB's namespace (URL).
    • NoSQL API - Choose "DocumentDB"
    • Subscription - Choose an active Azure subscription
    • Resource Group - Choose to either create a new resource group or select a pre-existing one
    • Location - Choose a location that's closest to you.
  4. Click "Create"

DocumentDB - IoT & Primary Repository

You should only follow these instructions if you wish the test DocumentDB's support for the MongoDB protocol for the application's primary repository.

  1. Select "DocumentDB (NoSQL) - Protocol Support for MongoDB". NOTE: You may have to choose this twice.
  2. In the resulting pane, click "Create"
  3. You'll then need to complete the template options to configure your DocumentDB instance.
    • ID - A globally unique name that defines your DocumentDB's namespace (URL).
    • NoSQL API (if available) - Choose "MongoDB".
    • Subscription - Choose an active Azure subscription.
    • Resource Group - Choose to either create a new resource group or select a pre-existing one.
    • Location - Choose a location that's closest to you.
  4. Click "Create"

DocumentDB Collections

The DocumentDB database you created in the previous section will host our various document collections.


Pi Messages Collection

The Pi Messages collection is responsible for storing all of our incoming messages (i.e. bluetooth beacon locations) from our Raspberry Pi's.

  1. In Azure, open your DocumentDB pane.

  2. In the top of the pane, click on "Add Collection"
  3. In the Add Collection pane, enter the following information:
    • Collection Id - The name of your document collection. Enter raspberry.
    • Pricing Tier - Choose a pricing tier for your collection. In production, this will need to be pretty high in order to have the ability to process and store incoming messages. In test, you can choose a smaller instance size.
    • Database - Choose "Create New" and enter messages.
  4. Click "OK"

You now have the messages document repository created.


Web Application Collection (Optional)

The web application collection is optional and should only be used if you 1) do not wish to use MongoDB as your main repository; and 2) you've created your DocumentDB with the MongoDB protocol option.

  1. In Azure, open your DocumentDB pane.
  2. In the top of the pane, click on "Add Collection"
  3. In the Add Collection pane, enter the following information:
    • Collection Id - The name of your document collection. Enter assets.
    • Pricing Tier - Choose a pricing tier for your collection. This isn't a high performing database, so a smaller instance is sufficient.
    • Database - Choose "Create New" and enter app.
  4. Click "OK"

Repeat the above steps to create also collection with an ID of users and use the database you just created, named app.

IoT Hub

The IoT Hub serves as the receiving endpoint for all incoming messages from the Pi. Unlike a typical web service, with IoT Hub, things like scalability and failover are baked in. IoT Hub can handle 100,000+ requests/sec.

  1. In Azure, create a new IoT Hub resource.
  2. In the add IoT Hub pane, enter a name for your IoT Hub (e.g. NodeHub).
  3. With the exception of the Resource Group and Location, you can accept the defaults. This tutorial won't cover the details regarding scaling, units and partitions.
  4. Click "Create"

It may take a little bit to create the IoT Hub, so be patient.

Once the IoT Hub creation process has finished, we need to define an access policy so that our Stream Analytics job can connect and read data.

  1. While in the Iot Hub pane (open it if it's not open already), click on "Shared access policies".
  2. In the Shared access policies pane, click "Add" on the top menu.
  3. Name the access policy (e.g. iothubowner) and check all four permissions.
  4. Click "Create".

Stream Analytics

Stream analytics acts as a middleware processor for our data. It, itself, really doesn't do anything to our data. Instead, it relies on other functionality within Azure. Stream analytics simply gives us the mechanism to "hook up" Azure components to our middleware.

  1. In Azure, create a new Stream Analytics Job resource.
  2. From a creation perspective, the only things required are a name (e.g. iot-assets) and the resource group/location. Go ahead and fill those in.
  3. Click "Create" when finished.

As with the IoT Hub, it may take a few minutes to create the Stream Analytics job. Once it has been created, we need to wire the necessary Azure features into the pipeline. There are three major parts to this exercise: 1) the Inputs; 2) the Query; and, 3) the Outputs. See the below image.

Let's configure each one of these. Before we begin, make sure that your job isn't running. It shouldn't be because we've not yet defined inputs and outputs, but verify in case you've already completed these steps and you're attempting the modify the settings. Also, we'll need to configure the inputs and outputs first before we complete the query.


Inputs
  1. While in the Stream Analytics job pane, click on "Inputs" in the either the side menu or the tile.
  2. In the Input pane, click "Add" on the top menu.
  3. For the new input:
    • Enter an alias (e.g. iot-hub-input)
    • Source Type - Data stream
    • Source - IoT hub
    • Subscription - your subscription
    • IoT Hub - NodeHub (or whatever you named it above)
    • Endpoint - Messaging
    • Shared access policy name - iothubowner (or whatever you named it above)
    • Consumer group - $Default
    • Event serialization format - JSON
    • UTF-8
  4. Click "Create"

Outputs
  1. While in the Stream Analytics job pane, click on "Outputs" in the either the side menu or the tile.
  2. In the Input pane, click "Add" on the top menu.
  3. For the new input:
    • Enter an alias (e.g. docdb-output)
    • Sink - DocumentDB
    • Subscription - your subscription
    • Account id - iot-assets (or whatever you named it above)
    • Database - messages (or whatever you named it above)
    • Collection name pattern - raspberry
    • Document id - ***leave blank***
  4. Click "Create"

Query
  1. While in the Stream Analytics job pane, click on "Query" in the either the side menu or the tile.
  2. In the query editor pane, enter the following:

    SELECT
        *
    INTO
        [docdb-output]
    FROM
        [iot-hub-input]
                                                
  3. Click "Save" on the top menu and close the pane.

You can now start the Stream Analytics job by clicking on "Start" on the Stream Analytics pane.

Installation - Raspberry Pi

This section is to assist you with setting up and configuring your Raspbian Pi. The setup is based on a Raspberry Pi 3 Model B. If you downloaded the pre-configured image from above (e.g. not building from a virgin Raspbian Pi image), then you may skip this section and move to the configuration section for the actual client applications.

Node and NPM

The Raspbian Jessie image comes pre-installed with Node v0.10.29. However, NPM is not installed. In order to download dependencies, we're going to need to: 1) upgrade Node; and, 2) install NPM.

  1. Download package lists to check for available upgrades

    sudo apt update

  2. Install Node (v4.5.0) and NPM (v2.15.19)

    wget https://nodejs.org/dist/v4.5.0/node-v4.5.0-linux-armv7l.tar.gz
    sudo mv node-v4.5.0-linux-armv7l.tar.gz /opt
    cd /opt
    sudo tar -xzf node-v4.5.0-linux-arm7l.tar.gz
    sudo mv node-v4.5.0-linux-arm7l nodejs
    sudo rm node-v4.5.0-linux-arm7l.tar.gz
    sudo rm /usr/bin/node
    sudo ln -s /opt/nodejs/bin/node /usr/bin/node
    sudo ln -s /opt/nodejs/bin/npm /usr/bin/npm

iBeacon Location Packages

The iBeacon location packages allows your Pi to communicate with iBeacon devices and understand their specific payload.

sudo apt-get install bluez-hcidump

Bluez Libraries

The Bluez libraries are a dependency library written in C++ that provides connectivity to Bluetooth LE (low-energy) devices.

  1. Install Dependencies

    sudo apt-get install -y libusb-dev libdbus-1-dev libglib2.0-dev libudev-dev libical-dev libreadline-dev

  2. Download Libraries

    wget http://www.kernel.org/pub/linux/bluetooth/bluez-5.41.tar.xz
    sudo tar -xvf bluez-5.41.tar.xz
    cd bluez-5.41


  3. Compile & Install

    sudo ./configure
    sudo make
    sudo make install


  4. Setup Bluez Service

    systemctl status bluetooth

    You'll likely see the bluez service is loaded but not active:

    ● bluetooth.service - Bluetooth service
       Loaded: loaded (/lib/systemd/system/bluetooth.service; disabled)
       Active: inactive (dead)
         Docs: man:bluetoothd(8)


    You can manually start the bluez service by running the following command:

    sudo systemctl start bluetooth

    After running the above, run the status command again and you should see the service is now active:

    ● bluetooth.service - Bluetooth service
       Loaded: loaded (/lib/systemd/system/bluetooth.service; disabled)
       Active: active (running) since Mon 2016-02-29 04:56:27 UTC; 1s ago
         Docs: man:bluetoothd(8)
     Main PID: 715 (bluetoothd)
       Status: "Running"
       CGroup: /system.slice/bluetooth.service
               └─715 /usr/local/libexec/bluetooth/bluetoothd


    You can stop the service by running:

    sudo systemctl start bluetooth

Automatically Starting the Bluetooth Service on Boot

If you want the bluez serice to run automatically when the Raspberry Pi boots. To enable the bluez service to run at boot, execute the following command:

sudo systemctl enable bluetooth

If you ever need to disable it, substitute enable with disable on the previous command.

Enable Bluetooth LE

The final step to installation is enable the bluetooth low energy features in bluez. These features are disabled by default because they are still under development and have been placed behind an experimental flag.

  1. To enable the bluez experimental features like LE, you need to modify the service configuration.

    sudo nano /lib/systemd/system/bluetooth.service

  2. You should see a configuration file similary to the following:

    [Unit]
    Description=Bluetooth service
    Documentation=man:bluetoothd(8)
    ConditionPathIsDirectory=/sys/class/bluetooth

    [Service]
    Type=dbus
    BusName=org.bluez
    ExecStart=/usr/local/libexec/bluetooth/bluetoothd
    NotifyAccess=main
    #WatchdogSec=10
    #Restart=on-failure
    CapabilityBoundingSet=CAP_NET_ADMIN CAP_NET_BIND_SERVICE
    LimitNPROC=1

    [Install]
    WantedBy=bluetooth.target
    Alias=dbus-org.bluez.service


  3. To enable the experimental features, add --experimental to the ExecStart line:

    [Unit]
    Description=Bluetooth service
    Documentation=man:bluetoothd(8)
    ConditionPathIsDirectory=/sys/class/bluetooth

    [Service]
    Type=dbus
    BusName=org.bluez
    ExecStart=/usr/local/libexec/bluetooth/bluetoothd --experimental
    NotifyAccess=main
    #WatchdogSec=10
    #Restart=on-failure
    CapabilityBoundingSet=CAP_NET_ADMIN CAP_NET_BIND_SERVICE
    LimitNPROC=1

    [Install]
    WantedBy=bluetooth.target
    Alias=dbus-org.bluez.service


  4. Save the file and exit the editor by pressing Ctrl-o, enter, then Ctrl-x.
  5. Now tell systemd to reload its configuration files by running:

    sudo systemctl daemon-reload

  6. If the bluez service was previously running, you'll want to restart it by running:

    sudo systemctl restart bluetooth

    Or, if the service wasn't running you can start it with the command above.
  7. Once the bluez service is running, you can check that the experimental features are enabled by running the status command again. This time, you should see the --experimental flag set on the service:

    ● bluetooth.service - Bluetooth service
       Loaded: loaded (/lib/systemd/system/bluetooth.service; disabled)
       Active: active (running) since Mon 2016-02-29 04:56:27 UTC; 1s ago
         Docs: man:bluetoothd(8)
     Main PID: 715 (bluetoothd)
       Status: "Running"
       CGroup: /system.slice/bluetooth.service
               └─715 /usr/local/libexec/bluetooth/bluetoothd --experimental


Configuration

Now that the web application has been downloaded and the Raspberry Pi dependencies have been installed, we need to configure both applications. The good news is that this section is very simple - the tedious requirements have been completed.

Web Application

To setup up the web application, you will need to set a few simple variables.

Find and open the file /webapp/config.js. In this file, you will find all of the application's variables. The idea when creating the application was to provide a single point of configuration instead of having to modify multiple files. Then, change the options as you prefer.

  • port - defaults to the applications's environment-configured port or 8080
  • repo - the repository type to use (e.g. 'mongodb' or 'documentdb')
  • mongodb - the connection string of your mongodb instance
  • documentdb - the connection string of your documentdb instance
  • secret - a random, secret string that is used to encrypt JSON web tokens
  • host - URI for your documentdb instance
  • masterKey - 'password' for your documentdb instance

Raspberry Pi

Configuring the Raspberry Pi is really a simple, two-step process. First, we need to provision the Pi in our Azure tenant. Then, we'll need to configure the monitoring application with our Pi's provisioned Id.


Registering the Device

Find and open the file /pi/registerDevice.js. There are 2 lines that you will need to change.

  1. On line 3, you will see device.deviceId = 'mtc-pi01';. Name your Pi device by changing mtc-pi01 to a name you prefer.
  2. Line 5 contains the connection string to your Azure's IoT Hub instance. While you could build this manually, the easiest option is to navigate to your IoT Hub instance in Azure, click on the Shared Access Policy that you created above (e.g. iothubowner) and copy either the Connection string—primary key or Connection string—secondary key.
  3. Once you've performed both steps, save the file and exit to the bash prompt.
  4. Run the script: node ./registerDevice.js
  5. Upon a successful run of the script, you will see two things printed to the console:
    1. Device id
    2. Device key
  6. The device id should be the same as what you entered onto line 3 in step 1. Copy both, the device id and key
  7. That specific Pi has now be registered with IoT Hub.

Configure the Listener

Find and open the file /pi/index.js. There is one line that you will need to change.

  1. Line 5 contains the connection string to your Azure's IoT Hub instance for that specific device. Like the connection string from the previous section, it is divided into three components and separated by semi-colons:
    • Hostname - the host name for you IoT Hub instance (this should be the same as what you copy and pasted as the host name in the previous script - e.g NodeHub.azure-devices.net)
    • DeviceId - the device id that you copied from the output in the previous step (also what you entered as the device id in the previous script)
    • SharedAccessKey - the device key that you copied from the output in the previous step
  2. Once you've created the connection string (remember, that all three sections should be concatenated and separated by a semi-colon), save the file and exit to the bash prompt.
  3. Run the script: node ./index.js
  4. The script will now run in the background, monitor any incoming iBeacon transmittals and push those up to your IoT Hub instance.

RadBeacon Dot(s)

You can now set up multiple RadBeacon Dot(s) to communicate with your Pi. In order to do so, you'll need to download the RadBeacon application from your phone's marketplace. Once the configuration application has been downloaded, follow the instructions below.

  1. Open the RadBeacon application on your phone. You should see an empty list.
  2. Hold down the on/off switch on the Dot for 3 seconds to put in Dot into configuration mode. You should see a green light blink twice.
  3. Refresh your phone's screen and you should see the Dot listed. (NOTE: Your ID will be different.)
  4. Select the listed Dot to open the configuration screen.
  5. On the configuration screen, you'll see a UUID. The default UUID is the same for all Dots. You'll need to generate a new one.
  6. You can either type one in manually or select the "Generate UUID" link on the screen.
  7. Optionally, you can increase/decrease the advertising rate and transmit power.
    • Advertising Rate - how frequently the Dot broadcasts a pulse or heartbeat.
    • Transmit Power - increase the distance the Dot will broadcast. Increasing power will allow the Pi to "pick up" the Dot from longer distances, but decreases the Dot's battery life and increases percentage of error. Decreasing power will reduce the transmission radius of the Dot, but increases the battery life and minimizes the margin of error.
  8. Once you've completed the above steps, choose "Actions" from the menu, then select "Apply" to save your settings.
  9. Reboot your Dot (turn it off, then back on) to allow it to start transmitting.

Deployment & Last Steps

All configuration steps have been completed. You are now ready to deploy the web application and start monitoring bluetooth beacons.

Heroku

While you can also deploy the web application to Azure, Heroku is a nice choice for testing as it offers a completely free tier for Node.js applications. You should have already created the application in Heroku in the steps above. Deploying the project is not very difficult. You can do it via Heroku Git, GitHub or Dropbox. The default setup in Heroku is using Heroku Git, so that's what we'll use.

One thing to note is that we don't want to push all files in the repository to Heroku, just the contents of the webapp subdirectory. While we could technically split the folder into a new repository, that may not be our best option if we want to keep all files in the same repository. So, instead, we'll simply copy the subfolder to it's own directory and push that single directory to Heroku.

Finally, instead of using pure Git commands, Heroku provides a nice CLI that makes things super easy for us.

  1. Visit Heroku to download and install the Heroku CLI. (You may be required to reboot.)
  2. Copy the webapp folder to another folder that is outside of the parent project folder. (Because there are many different OS's, I'll let you decide how you want to do that.)
  3. Initialize the folder as a Git repository

    git init

  4. Login to Heroku using the CLI

    heroku login

  5. Add Heroku as a remote repository using the CLI, replacing <your app name> with the name of the application you created in Heroku above

    heroku git:remote -a <your_app_name>

  6. Heroku requires a special file to know how to run your application:
    1. In the webapp home directory, create a file named Procfile (case-sensitive).
    2. Add the following code to the first line of the file and save the file.

      web: node server.js

  7. Now, let's commit the application to Heroku by entering the following commands.

    git add .
    git commit -m "initial commit"
    git push


iBeacon Monitoring

The Raspberry Pi('s) need to monitor for iBeacon broadcasts and transmit that data up to Azure. Because we've already configured the Pi('s) earlier, all we need to do is start the monitoring service.

  1. In the application folder (e.g. pi), run the following command

    node ./index.js

Once the iBeacons start transmitting, you'll need to add an asset to the application and enter the UUID of the iBeacon for the asset's UUID to start seeing data in the web application.