Record Metrics in Keystore Service

A common need when managing a fleet of devices is to track metrics and error logs. This guide demonstrates how to use West Connectivity Keystore service to collect metrics and expose those metrics from a custom API endpoint.

Prerequisites

There are 2 ways to follow this guide.

A. Use a Connector

In this scenario you only need a West Connectivity Connector and you don't need an Application. You will update the code on the Connector directly.

However you will need to add the following services to it: * Keystore * Webservice

Learn how to on the service management page.

B. Use an Application

In this case you will need:

An empty application
A Connector
With a test device (or Curl) to send data
To link the above Application and Connector

Collect Metrics

A first step for monitoring your device fleet is to collect information from devices. To keep things simple, you can collect two types of data about incoming messages from devices.

The number of messages per day
The contents of the last 10 messages

West Connectivity Device2 service provide an event that gives you an opportunity to respond to incoming device data. When handling data from the devices you may perform data conversion, store data, or send alerts. You may modify the way your application responds to data by editing the handler for the event in West Connectivity web-based code editor or by uploading code using the command line tool.

For the purpose of this guide, you can use the web-based editor. To modify the event handler, click on the Services tab of your application, select Products, and select the code tab. The code found here is executed when any devices with product types associated with the application send data. You can now add code to store metrics to the Keystore service.

Increment a Metric Counter

To make a daily counter, you can use incrby command. This command adds one to the "dailyCount" key, setting it to 1 if the key does not already exist. It does so automatically, so you will not miss any incoming data points.

Keystore.command({ command = "incrby", key = "dailyCount", args = {1}})

To only keep the count for the current day, you can take advantage of the expire command which will automatically handle the value expiration at the given time.

-- Tomorrow midnight timestamp in seconds
local expiration = math.floor(1 + os.time()/86400) * 86400

-- Set the expiration time for next day midnight
Keystore.command({ command = "expireat", key = "dailyCount", args = {expiration} })

Log the Last Messages

To log message content from your devices, you can update the Device2 service event handler as follows.

First get the message from the device contained in the 'payload' field of the event. The payload is an array in case the device sent historical data. Here we will use the first one.

NOTE: In most cases, you would want to also filter on event.identity rather than log all messages.

local message = to_json(event.payload[0])

Next you will push the message to a list in the Keystore service using the lpush command. This will allow you to see the last few device messages, which can be useful for debugging.

Keystore.command({ command = "lpush", key = "logs", args = {message}})

Keep only the last log items with the trimming command by specifying a range of data to keep.

NOTE: The list index starts at 0. Therefore, in order to keep the last 10 logs, the target list range index is 0 to 9.

Keystore.command({ command = "ltrim", key = "logs", args = {0, 9}})

All Together

Put it together to get the complete Event-Handler script.

Below script goes in different service depending if you choose to use an Connector or an Application as follow:

Connector -> Use the Device2 Service
Application -> Use the dynamic <Connector> Service

In both case the event is event.

local message = to_json(event.payload[0])

-- Increment the counter
Keystore.command({ command = "incrby", key = "dailyCount", args = {1}})

-- Tomorrow midnight timestamp in seconds
local expiration = math.floor(1 + os.time()/86400) * 86400

-- Set the expiration time for next day midnight
Keystore.command({ command = "expireat", key = "dailyCount", args = {expiration} })

-- Push the new message to the logs list, not in lua the first element start at index 1
Keystore.command({ command = "lpush", key = "logs", args = {message}})

-- Only keep the last 10 logs, the trim command requires a range of index to keep (starting with 0).
Keystore.command({ command = "ltrim", key = "logs", args = {0, 9}})

Expose Metrics

The easiest way to expose our collected data is to create an API route endpoint. Endpoints can be created with the "+" button on the Routes tab of your application. When you create a new route, you will be prompted to specify the path and HTTP method for the route. Create a new route in your application with the Method GET and the Path /metrics.

Once the route is created, you can write an endpoint script which runs when your API route endpoint is called. You can learn more about endpoint scripts on the Webservice reference page.

Exposing the count only requires you get the counter value store in the Keystore service with the get operation.

  local dailyCount = Keystore.get({ key = "dailyCount" })

Logs can use the lrange command which return a range of elements from the list. Note that {0, -1} indicates you want everything from the most recent log entry (0) to the oldest log entry (-1).

  local logs = Keystore.command({ command = "lrange", key = "logs", args = {0, -1}})

To make the endpoint return the values of dailyCount and logs, you can simply return them in a Lua table.

return { dailyCount = dailyCount.value, logs = logs.value }

This works because endpoint scripts return a 200 HTTP status code by default, and Lua tables are automatically transformed into JSON response bodies. However, you could also set these manually by setting response.code and response.body instead.

All Together

Here is the API route endpoint code in its entirety:

-- Get the dailyCount value
local dailyCount = Keystore.get({ key = "dailyCount" })

-- Get the last n logs, -1 means all items
local logs = Keystore.command({ command = "lrange", key = "logs", args = {0, -1}})

-- Reponse a 200 HTTP response with a JSON body object
return { dailyCount = dailyCount.value, logs = logs.value }

Test It

Generate Data from your Device

To test your scripts, you need to send data from your device to West Connectivity. Here is how to do that:

Create a Product or use an existing one from the West Connectivity portal and note its product ID.
If you create a new Product, be sure to add it to the Products configuration under the Services tab of the application.
Add a resource under the Definition tab. For example, you could create a resource with the Alias "message" and Data format "string".
Create a device on the Devices tab and note its identity.
Activate the device to get the device key (CIK). A simple HTTP request on the provisioning API will do. Normally this step is done by a device, but see below for how to do that using curl. (Be sure to substitute the items in brackets with your values.)

$ curl https://<product_id>.m2.diagnostic.westpharma.com/provision/activate \
  -H "Content-Type: application/x-www-form-urlencoded; charset=utf-8" \
  -d "vendor=<product_id>&model=<product_id>&sn=<identity>"

The Token for the device is returned. See the Provision API documentation for more about the activation step.

With the CIK in hand you are ready to write data. If you do not have a physical device, you can simulate one by sending data using the write API. See below for how to do that using curl. Again, be sure to substitute the items in brackets with your values.

$ curl https://<product_id>.m2.diagnostic.westpharma.com/api:v1/stack/alias \
  -H Authorization: Token <Token> \
  -H "Content-Type: application/x-www-form-urlencoded; charset=utf-8" \
  -d "<resource_alias>=<value>"

Here is an example, assuming a product ID of m0a6fwvjl7x8ncdi and device ID of 001.

$ curl https://m0a6fwvjl7x8ncdi.m2.diagnostic.westpharma.com/provision/activate -H "Content-Type: application/x-www-form-urlencoded; charset=utf-8" -d "vendor=m0a6fwvjl7x8ncdi&model=m0a6fwvjl7x8ncdi&sn=001"
029c56108afffc2507e3ffb0aeed105dd6c39922

$ curl https://m0a6fwvjl7x8ncdi.m2.diagnostic.westpharma.com/api:v1/stack/alias -H "X-Exosite-CIK: 029c56108afffc2507e3ffb0aeed105dd6c39922" -H "Content-Type: application/x-www-form-urlencoded; charset=utf-8" -d "message=hello world"

Get metrics

You can now retrieve your metrics data by opening the /metrics endpoint you defined in your web browser. For example, if your application name was my-iot-solution, then you would go to this address:

https://my-iot-solution.apps.diagnostic.westpharma.com/metrics

Your metrics data looks like this:

{
  "dailyCount": "1",
  "logs": [ "hello world" ]
}

You should now be able to use the West Connectivity Keystore service to track product metrics for your connected devices.