Monday 30 November 2015

Displaying real-time IoT data in IBM DOORS (Doors Web Access)

There are many ways in which we could imagine surfacing IoT data in the IBM DOORS interface.

One way that I would propose is to extend the rich hover so that you could see IoT data in real-time, displayed when you hover over an IBM DOORS link-end point. Imagine being an IBM RQM user looking at a defect and being able to hover over a link to a requirement and see the history of a thing displayed.

Extending the rich hover to display IoT data in IBM DOORS Web Access


Extending the rich hover to display IBM DOORS IoT data in IBM RQM


Showing display of real-time data in hover over



As part of the technical investigation I looked into the implementation of a JavaScript client which listens for data from a device in the IoT Foundation. And then display that real-time data in the UI. The functioning graphing component is shown in the examples above. I will blog about the JavaScript / IoT implementation in a future post. 



Sunday 29 November 2015

Moving DOORS9 towards real-time IoT sensors

Last time I shown how easy it is to use historian from your IoT enabled device data in DOORS using IoT Foundation HTTP API. I thought 'it cannot be that hard to show real-time data', I know there are no MQTT callbacks in DXL, no JSON parser (sure we can write one, but what's the point?).

Sounds challenging. So let's do it!

Let's play!

If you have spare 15 minutes please have a look at wonderful presentation by Dr John Cohn (@johncohnvt) - "The importance of play".

Let's create a very simple HTML static page you can open in any (recent!) browser and which will connect to IoT Foundation.

Very (and I mean it) simple HTML page

The page will have just two p elements, one for static info, and one used dynamically from JavaScript. It will additionally include 3 scripts; jquery, paho and MQTT connection helper class.
The entire page code is simply:
<!DOCTYPE html>
<html>
    <head>
        <meta http-equiv="Content-Type" content="text/html; charset=utf-8" />
        <script type="text/javascript" src="jquery.min.js"></script>
        <title>MQTT listener</title>
    </head>

    <body onContextMenu="return false">
        <p>Please do not close</p>
        <p id="pStatus"></p>
    </body>
        <script type="text/javascript" src="realtime.js"></script>
        <script type="text/javascript" src="mqttws31.js"></script>
        <script type="text/javascript">
          var realtime = new Realtime(org, apikey, apitoken, device_type, device_id);
        </script>
</html>

MQTT connection

As you know IoT devices communicate over MQTT, lightweight device to device protocol implemented in many languages. Among which there are many JavaScript implementations available, which you can find at http://mqtt.org/tag/javascript. Personally I'm using Paho because it is very easy to use with IBM Bluemix services.

Using Paho is straight forward, you need to create new Messaging.Client object providing host, port and clientId.
For newly created client you should provide two message callbacks:
  • onMessageArrived - function(msg)
  • onConnectionLost - function(e)
client = new Messaging.Client(hostname, 8883,clientId);
client.onMessageArrived = function(msg) {
  var topic = msg.destinationName;
  var payload = JSON.parse(msg.payloadString);
  data = payload.d;
};

client.onConnectionLost = function(e){
  console.log("Connection Lost at " + Date.now() + " : " + e.errorCode + " : " + e.errorMessage);
  this.connect(connectOptions);
}

Then you construct connection options object which will be passed to connect function of Messaging.Client. This will start connection with IoT MQTT Broker.
You have to specify:
  • timeout - number
  • useSSL - boolean
  • userName - string - as usual this will be API Key for your organization
  • password - string - and API Token  
  • onSuccess - function() - executed when connection was successful
  • onFailure - function(e) - execeuted on failure
var connectOptions = new Object();
connectOptions.keepAliveInterval = 3600;
connectOptions.useSSL=true;
connectOptions.userName=api_key;
connectOptions.password=auth_token;

connectOptions.onSuccess = function() {
  console.log("MQTT connected to host: "+client.host+" port : "+client.port+" at " + Date.now());
  $('#pStatus').text("MQTT connected to host: "+client.host+" port : "+client.port);
  self.subscribeToDevice();
}

connectOptions.onFailure = function(e) {
  console.log("MQTT connection failed at " + Date.now() + "\nerror: " + e.errorCode + " : " + e.errorMessage);
  $('#pStatus').text("MQTT  connection failed at " + Date.now() + "\nerror: " + e.errorCode + " : " + e.errorMessage);
}

console.log("about to connect to " + client.host);
$("#pStatus").text("about to connect to "+client.host);

client.connect(connectOptions);

Once you establish connection with IoT MQTT Broker you should subscribe to some topic.

var self = this;
// Subscribe to the device when the device ID is selected.
this.subscribeToDevice = function(){
  var subscribeOptions = {
    qos : 0,
    onSuccess : function() {
      console.log("subscribed to " + subscribeTopic);
    },
    onFailure : function(){
      console.log("Failed to subscribe to " + subscribeTopic);
      console.log("As messages are not available, visualization is not possible");
    }
  };
  
  if(subscribeTopic != "") {
    console.log("Unsubscribing to " + subscribeTopic);
    client.unsubscribe(subscribeTopic);
  }
  subscribeTopic = "iot-2/type/" + deviceType + "/id/" + deviceId + "/evt/+/fmt/json";
  client.subscribe(subscribeTopic,subscribeOptions);
} 

DOORS9 HTML window

Now it's time to display this very simple page in DOORS 9:
bool onB4Navigate(DBE dbe,string URL,frame,body){return true}
void onComplete(DBE dbe, string URL){}
bool onError(DBE dbe, string URL, string frame, int error){return true};
void onProgress(DBE dbe, int percentage){return true}

Module refreshModuleReff = null
void moduleRefreshCB(DBE x) {
  if (!null refreshModuleReff) {
    refresh refreshModuleReff
  }
}

string surl="c:\\iot\\realtime.html"
DB iotUIdb = create("do not close")
DBE iotUI=htmlView(iotUIdb, 100, 50, surl, onB4Navigate, onComplete, onError, onProgress)
DBE t = timer(iotUIdb, 1, moduleRefreshCB, "ping")
startTimer(t) 
realize iotUIdb 

You probably noticed I'm using timer functionality (page 48 in DXL Reference Manual), that's because layout DXL is run when the module is being refreshed, redrawn, etc. This simple code ensures it is refreshed every second.
There is a lot of space for improvement in above DXL; you can add a toggle to check if user wants refresh, you can add noError/lastError block before calling refresh refreshModuleReff. 

Now save above code as %doors_home%/lib/dxl/startup/iot.dxl and restart DOORS. This will create a top level window which can be accessed from any DXL context in your DOORS 9 session.

Layout DXL

Finally we can add some worker code. This will be using hidden perm which has following syntax:
string hidden(DBE, string)

string property = obj."accel_property"
if (property != "none") {
  noError()
  refreshModuleReff = current Module
  string strval = hidden(iotUI, "realtime." property "()")
  string err = lastError
  if (!null err) { halt }
  real val = realOf strval

  //enable blow if your Module is similar to the one from last post
  //obj."iot_data" = val
  display strval
} 

I put whole code from "MQTT Connection" chapter into realtime.js and created additional helper methods to get required
 
this.yaw = function() {
  if (data != null) return data.yaw;
  return -101;
}
this.pitch = function() {
  if (data != null) return data.pitch;
  return -101;
}
this.roll = function() {
  if (data != null) return data.roll;
  return -101;
}
var data = null;


Now the module is displaying real-time data from a device connected to IoT Foundation. It has 1 second delay, maybe you want to synchronize data object when it's retrieved from realtime.js.

Well afterwards I didn't need to write a JSON parser in DXL!

Conclusions

DOORS reading IoT real-time data is easy and possible! It was even simpler then I thought initially ;)

Remember in this blog I'm just giving examples what can be done, it's up to you to extend it with unsubscribe, change device type and device depending on current module. There are many thing you might want to add.

Thursday 19 November 2015

DOORS IoT data example

In my last post I demonstrated how easy it is to get data from IoT Foundation. But many of you might thing, why? Well I just wanted to show how easy it is to report on IoT data using pure old DOORS 9 but there was no fun in the preview post, just boring numbers. 

Example Scenario

Let's have a hypothetical example where we measure yaw/pitch/roll and have following rules:
  1. Measurable parameters of an Android Device shall not exceed expected maximum values
  2. Each report of a value exceeding 0.77 of maximum should be indicated Orange
  3. Each report of a value exceeding 0.88 of maximum should be indicated Red

So we need something to read data from IoT device, storage for a maximum value and some nice indicators.

Module Attributes

I just extended my module with extra attributes:
  • iot_data (real) - doesn't affect change bar, doesn't affect change dates and doesn't generate history
  • range_max (real) - a regular attribute of type range

Having add those I can update my layout DXL column to render color depending on current value:

if (findPlainText(re, ":", l, o, false)) {
  string ss = re[l+1:length(re) -3]
  real r = realOf ss
  obj."iot_data" = r
  real m = obj."range_max"
  real rm = r/m
  DBE cnv = getCanvas
  realBackground(cnv, realColor_Green)
  if (rm >= 0.77 && rm < 0.88) {
    realBackground(cnv, realColor_Orange)
    realColor(cnv, realColor_White)
  }
  else if (rm >= 0.88) {
    realBackground(cnv, realColor_Red)
    realColor(cnv, realColor_White)
  }
  else {
    realColor(cnv, realColor_Black)
  }
  display ss
}

Code isn't perfect, could be faster, but that's not the point here ;)

Now let's have some fun with canvas DXL. Add new DXL column and set its DXL to:

DBE canvas = getCanvas
if (canvas==null) halt

int rh = 50 du
int rw = 100 du

setHeight rh
setWidth rw

int normalize(real  x, max) {
  real  i = 180.0+ x/max * 180.0
  return intOf i
}

real x = (obj."iot_data")
real m = (obj."range_max")

if (m == 0.0) halt

int v = normalize(x, m)
int margin = 1

realColor(canvas, realColor_Green)

if (v >= 280 && v < 320) {
  realColor(canvas, realColor_Orange)
  margin = 2
}
else if (v >= 320) {
  realColor(canvas, realColor_Red)
  margin = 5
}

ellipse(canvas, 0, 0, rw, 2*rh)
realColor(canvas, realColor_White)
ellipse(canvas, margin, margin, rw-2*margin, (2*rh)-2*margin)

realColor(canvas, realColor_Black)
polarLine(canvas, rw/2, rh-1, rh, v)
Resulting View will show us (almost) real-time events and warning from our device.

DOORS 9 View with almost real-time IoT data warnings

Remember this could be a group of devices or a single device. It all depends what you want to measure.

Conclusions

IoT Foundation and DOORS 9 are really cool and really powerful! Hope to show you some more fun stuff soon!

Tuesday 17 November 2015

DOORS in IoT World

Last time I shown a quick example how to use HTTP request perms (DOORS functions) to get data from Cloudant database.
Today let's go a little step further and try to get more recent data.

Before you will continue with DOORS connection to IoT Foundation API please have a look at IoT Foundation Starter application. There you will learn how to create a simple IoT device, deploy a IoTStarterApp to your mobile, and start talking IoT! That's something I'm not covering in this blog (yet).

IoT connectivity

IoT devices communicate over Machine 2 Machine connectivity protocol - MQTT Message Queue Telemetry Transport. This protocol is extremely lightweight protocol much lighter than HTTP. MQTT becomes more and more popular, not only in IoT world but also in mobile data exchange, enven Facebook Messenger uses MQTT.

DOORS and IoT connectivity

MQTT protocol requires message broker and message client. From DOORS client point of view IBM IoT Foundation can be a message broker, but what with MQTT client? Well at the moment I do not see easy/best solution, one can try writing a MQTT client in DXL, or using some external library OLE interface. However MQTT messages are great for a real time data which can feed IoT Real-Time Insights or IoT Foundation.

OOTB DOORS can consume a historic data. In order to get a historic data from your device you can use IBM IoT Foundation HTTP API and you can do it using HTTP perms available in DOORS.

You can get events for:
  • every device from your organization - so all device types, all devices.
  • all devices of selected device types in your organization 
  • a single device
Each of possible historic data queries has a number of filters available.

Authorization 

IoT Foundation HTTP API uses a Basic Authentication in combination with HTTPS. You need to create  API key for your IoT service.

IoT API Key generation

When authenticating API Key is used as user name and its associated Auth Token as the password. Those should be provided in HTTP Authorization header in all requests.

Following DXL will add set authorization header:
  HttpHeader h = create
  string auth
  toBase64_(apiKey":"apiToken, auth)
  auth = auth[0:length(auth) -2]
  add(h, "Authorization", "Basic "auth)

toBase64_ is a perm which will convert a string into its Base64 representation. It it adds a newline character to the resulting string thus is it removed before use.

Historic query

Since I'm going to use my script in Layout DXL I'm not going to query for devices or their types. Let's assume I know already all these information (or it can be stored in module level attributes) so I will focus on a single query on specific device. From documentation I know I need to send a GET request (with empty body) to a URL:

https://internetofthings.ibmcloud.com/api/v0001/historian/${org}/${type}/${id}?filter

where:
${org} is organization
${type} - device type
${id} is a device ID

filter will let me narrow down response:
top - selects a number of top most events - selecting top=1 is almost equivalent to getting data in real time ;)
start/end - interval of historical data to query at
evt_type - narrows down selected events
sumarize and sumarize_typeString - allows performing aggregate functions of selected events.

My layout DXL will select an average from last 100 'accel' events over a selected attribute:
Buffer buf = create
buf = "https://internetofthings.ibmcloud.com/api/v0001/historian/" org"/" deviceType "/" deviceId "?top=100&evt_type=accel&summarize={"attr"}"

'accel' is one of the events sent by IoTStartApp if you use different IoT device, then please change that accordingly.

Proposed usage scenario

The scenario here is that you have requirement(s) for an Android device that is being measured in the real world. Those measurements may mean that error messages are passed back and ultimately you would want to report against these from your original requirements. The attribute "accel_property" in the following code is the key that identifies the thing and measurement that you are interested in (and is relevant to the requirement).

DXL IoT column in DOORS module


Whole DXL is:
Buffer getData(string apiKey, apiToken, org, deviceType, deviceId, eventId, attr)
{
  Buffer buf = create
  buf = "https://internetofthings.ibmcloud.com/api/v0001/historian/" org"/" deviceType "/" deviceId "?top=1&evt_type=" eventId "&summarize={"attr"}"
  HttpHeader h = create
  string auth
  toBase64_(apiKey":"apiToken, auth)
  auth = auth[0:length(auth) -2]
  add(h, "Authorization", "Basic "auth)
  HttpResponse resp = httpRequest(HttpGet, tempStringOf buf, null, h)
  delete h
  delete buf

  if (!null resp && resp.isOk)
  {
    HttpBody b = resp.body
    Buffer respBuf = create
    respBuf += b.value

    delete resp
    return respBuf
  }
  else {
    if (!null resp) {
      display "error getting response " resp.code""
      delete resp
    }
    else {
      display "connection error"
    }
  }

  return null
}

if (!null obj) {
  string val = obj."accel_property"
  if (val != "none") {
    Buffer b = getData("API KEY", "Auth Token", "org", "type", "id", "accel", val)

    if (!null b)  {
      string re = stringOf b
      int l,o
      if (findPlainText(re, ":", l, o, false)) {
        display val " last value " re[l+1:length(re) -3]
      }
      delete b
    }
  }
}

As you can see I'm using any fancy JSON parser, my query returns a single variable so it can be easily extracted from a string. 
I used an extra enum attribute I created for my module so each of my requirements can select different variable from query.

Conclusion

DOORS can be really easily connected to IoT Foundation and access its data. With help of htmlView DBE one can add an extra DXL window displaying IoT data in Real-Time.
There is a lot DOORS can do for you!

Bonus lines of DXL for Real-Time data

"Turn your mobile phone into an IoT device" is an interesting extension to the article on m2m I showed you in the very beginning of this post. In compare to preview post on IoT starter application it has "Step 6. Create a Bluemix app to visualize sensor data". Where you can see how to deploy IoT sample NodeJS application which will visualize real-time or historic data from your organization. Once you deploy this application to your Bluemix account run following DXL for a simple HTML viewer:

void iotDataView(string srcUrl)
{
    DB dlg = create("IoT real-time data", styleCentered)
    DBE html = htmlView(dlg, 1000, 800, srcUrl, onHTMLBeforeNavigate, onHTMLDocComplete, onHTMLError, onHTMLProgress)
    show dlg
}
iotDataView "http://(your_application_name).mybluemix.net"

This will create windows similar to the following:

Sunday 1 November 2015

DXL and Cloudant documents

It was a very busy month and I didn't have much time to write something interesting. Today I decided to do a quick tutorial how to use HTTP perms.

You can read more on HTTP Perms in DOORS DXL Reference

Reading Cloudant documents

We know DOORS integration is not only outgoing to outside World, sometimes we need to get data into DOORS. Today I will show you a very short DXL script which will let you get data from external database. Because of ease of use and simple connectivity I decided to connect to Cloudant.

If you are not familiar with Cloudant, have a quick look at its Learning Center. In short Cloudant is a NoSQL database-as-a-service (DaaS) widely used with Bluemix Mobile and Cloud applications. But let's not jump into my future post and let me show you how can you read Cloudant documents from DOORS DXL.

As you know Cloudant uses HTTP requests and my script will focus on DXL HTTP requests.

Buffer getData(string username, password, account, db, doc)
{
  Buffer buf = create
  buf = "https://" username ":" password "@" account ".cloudant.com/" db "/" doc
 
  HttpHeader h = create
  HttpResponse resp = HttpRequest(HttpGet, tempStringOf buf, null, h)
  delete h
  delete buf

  if (!null resp && resp.isOk)
  {
    HttpBody b = resp.body
    Buffer respBuf = create
    respBuf += b.value
   
    delete resp
    return respBuf 
  }
  else {
    if (!null resp) {
      print "error getting response " resp.code""
      delete resp
    }
    else {
      print "connection error"
    }
  }

  return null
}

Buffer b = getData("", "", "ae4582eb-c21f-4c1a-92b3-8da8589eea0c-bluemix", "doors9test", "aee5f8c1be606328ddedef1a546cb7ca")

if (!null b)  {
  print tempStringOf b
  delete b
}


I set read permissions on this Database so everyone should be able to get following JSON:

{"_id":"aee5f8c1be606328ddedef1a546cb7ca","_rev":"1-f44612cc036915a9f12f1afb7ba2fc37","title":"DOORS 9 connection test","description":"You shall read this ;)"}

Writing documents back to Cloudant


If you would like to PUT something to Cloudnant database you can use the same HttpRequest but this time you need to set HttpHeader Content-Type. This is required buy HttpRequest with HttpPut verb, otherwise an error will be reported. Setting header values is done with add perm:

HttpHeader h = create
add(h, "Content-Type", "application/json")  // minimum required header
add(h, "Accept", "application/json")

setValue perm is used to set HTTP body:
HttpBody body = create
Buffer buf = create
buf = {"_id":"aee5f8c1be606328ddedef1a546cb7ca","_rev":"1-f44612cc036915a9f12f1afb7ba2fc37"}
setValue(body, buf)
HttpResponse resp = HttpRequest(HttpPut, url, body, h)
That's how you can simply put and get data from the Cloudant!

Soon I will do some more fun stuff with DXL, Bluemix and IoT but for now...

Happy Halloween!