DOORS 9 direct IoT support - build-in challenge #1

Monday morning is a good time for reflection and quick chat with coworkers before throwing yourself into work cycle.
In this way I had quick chat session with my manager yesterday. That ended up with challenge#1 (DOORS and IoT) and later on I had another quick chat my friend from DNG team, that ended up with challenge #2 (DOORS 9 itself).

Since challenge #1 is about IoT let's leave details of the other one for another post- but I like the idea so I will definitely post something in that subject.

Challenge #1

It was simply; "DXL triggers and IoT."

Well in my last post I used external tool written in NodeJS to publish DOORS 9 TRS feed to IoTF broker.

You need a full DOORS Web Access stack with some extra settings (as TRS is not enabled by default) to be able to get TRS feed which you will consume with your external application before posting that to IoT Foundation broker.

DWA Stack used in last post

There are no events in TRS so the last example was querying for changes every 10 seconds. That's far from real-time in IoT terms.

DOORS 9 Triggers

Have a look at DXL Reference Manual if you are not familiar with concept of triggers in DOORS 9.

Triggers are as close to real-time events as it can be in RM tool. Yeah but DOOR 9 doesn't have MQTT client.

Does it have to have it?

Well NO!

Looking at Internet of Things documentation, you can find topic on HTTP(S) Connectivity for Devices. If you read my previous post you should be familiar with HttpRequest. So your trigger DXL have to do is send a HTTP POST request to
<target server: org_id.internetofthings.ibmcloud.com>/api/v0002/device/types/{DeviceType}/devices/{DeviceID}/events/{eventID}

Looking at the URL format you can guess some event values will be set automatically:

{

"device_type": {DeviceType},

"device_id": {DeviceID},

"evt_type": {eventID},

"timestamp":

{

"$date": 1450194956425

}

All you need to add is a evt itself.

So your entire trigger DXL will look like

Module m = current Module
Buffer msg = create
msg = ""
msg += "\"id\": \"" getResourceURL m"\", "
msg += "\"content\": \"" name m "\", "
msg += "\"group\": \"Open\""
msg += "}"

string ioturl = "https://org_id.internetofthings.ibmcloud.com/api/v0002/device/types/DeviceType/devices/DeviceID/events/trigger_update"

HttpHeader h = create
string auth = ""
toBase64_("use-token-auth:YOUR_TOKEN", auth)
auth = auth[0:length(auth) -2]
add(h, "Authorization", "Basic "auth)
add(h, "Content-Type", "application/json")

HttpBody b = create
setValue(b, msg)

HttpResponse resp = HttpRequest(HttpPost, ioturl, b, h)
delete b

if (!null resp && resp.isOk)
{
// no one really needs to see it
  print "got it " (!null resp ? resp.code "" : "null") "\n"
  delete resp
}

delete h
delete msg

That's all you need to do publish events in real-time to IoTF!

Conclusion

Again HTTP Request proved to be a very powerful and useful perm. Using it the initial stack was reduced to very simple form:

Final stack in DOORS to IoTF communication

DOORS 9 has a potential in it, you just need to know how to use it.

DOORS reporting to IoT

So far we had an example how to use DOORS to read IoT historian data. Now it's time to publish something back.

There's no problem sending some useless data, the problem is to make sense out if it. So let's think on some 'real World' usage.

"Would you like to know when your most favorite module was modified?" 

Why not!

This could look like this:

If you read my previous posts you know we have a Tracked Resource Set (TRS) support in DOORS Web Access.
When enabled (as it's not by default) TRS "tells" us about all Modification, Creation and Deletion changes in DOORS Database.

So we need to modify a TRS reader to its new purpose.

I decided I won't need a front end for my TRS translation service so I rewrote the app.js:

/* jshint node:true */
var trs = require('./trs')
var superagent = require('superagent');
var iotf = require("ibmiotf");

//this is used to hide self-signed certificate errors
process.env.NODE_TLS_REJECT_UNAUTHORIZED = "0";

// IoTF setup
var deviceClientConfig = {
  org: 'quickstart',
  type: 'mytype',
  id: '001122334455',
  'auth-method':'token',
  'auth-token': 'secret'
};

var deviceClient = new iotf.IotfDevice(deviceClientConfig );
deviceClient.connect();
deviceClient.on("connect", function () {
 console.log("Connected to IoTF");
});

var user1 = null;
function updateTRS() {
 if (user1 == null) {
  user1 = superagent.agent();
  user1
    .post('https://DOORS_SERVER:8443/dwa/j_acegi_security_check')
    .type('form')
    .send({ j_username: 'DOORS_USER', j_password: 'PASSWORD'})
    .end(function(err, res) {
    })
 }
 else{
  trs.TRSUpdate(user1, deviceClient, function(err) {
    // just ping I'm alive
    console.log(". " +err); // one might want to check if err is defined
  });
 }
}

//query DWA for TRS changes
setInterval(updateTRS, 10*1000); //every 10 seconds
updateTRS()

Simple and quick. Now changes to trs.js which as you can see now takes additional deviceClient parameter.

Changes are really obvious:

1 . First publishing method

function publishTRS(deviceClient, data) {
   //publishing event using the user-defined quality of service
   var myQosLevel=2

   deviceClient.publish("trs_update","json",'{"d" : { "trs" : '+JSON.stringify(data)+' }}', myQosLevel);
   console.log("send data ->" + util.inspect(data));
}

2. Modified function parameters.
Note changes are no longer added to array. Instead those are send directly to IoTF

< function parseChangeLogs(clResults, client, last) {
---
> function parseChangeLogs(clResults, last) {
>     var changes = [];
97c88
< publishTRS(client, {start:dt, id:o.object, content:object, group:grp});
---
> changes.push({start:dt, id:o.object, content:object, group:grp});
108c99
< last(err);
---
> last(changes);
169c159
< exports.TRSUpdate = function(usr, client, last) {
---
> exports.TRSUpdate = function(usr, last) {
176c166
< parseChangeLogs(results, client, last);
---
> parseChangeLogs(results, last);

Now your TRS example is ready to talk to IoTF! Time to prepare IoTF application to use this feed.

Internet of Things Foundation

Go to you Bluemix dashboard and add new IoTF starter application. This will create for you a simple Node-Red application. Remember to add some credentials so only you will have access to your dashboard.

Once ready navigate to you IoTF dashboard


Here we will add a new device type to you IoT world.

New IoTF Device Type

Select Devices, Device Types and Create Type (on the bottom of the page)

The wizard will guide you to provide

1. Name and description
   
2. Define template
   
3. Define template
   

Save the type and proceed to add new Device

IoTF Device

Select Devices, Create Device (again on the bottom of the page)
You will have similar wizard. If you select Type defined earlier some fields will be populated.

1. Device ID
   
2. Security
   
3. Summary
   
   Please note credentials are NOT recoverable!

Finally Node-Red end

Once your device is ready, you can work on your Node Red consumer. 

Here I will just indicate how it might look like. I'm not going to write your business logic ;) I'm just showing an example what you can do.

So your full example would might look like:

Mine is just showing it works so you can try and do what you want.

I had to configure my IoT input:

Prepare my "business logic"

And observe the output if warning is set to true:

Conclusion

In those really simple steps we made a DOORS Web Access Tracked Resource Set a s Thing!

We just 'thingified' DOORS! 

IoT data in JavaScript

Introduction

In my previous post Displaying real-time IoT data in IBM DOORS, I displayed IoT data in the DOORS Web Access hover over. This involved an investigation into how I would access IBM Foundation IoT data from JavaScript. My initial thoughts were this would be very easy, I will just use the Paho Js utility, though it didn't end up as straight forward as anticipated. Therefore the aim of this post is to make the process a little easier and highlight the gotchas I encountered.

DOORS 9 IoT Report

Let's think about following user story:
"As a Requirement Engineer I would like to know how many errors my devices report"

In DOORS 9 module one can visualize this as:

So we would like to get a number of errors of given type reported by each of devices.

Device Error Reports in Sample App

First I need a device which can send some error codes. The IoT Starter Application mentioned in one of previous post publishes three kind of messages

• touchmove - user interaction with screen
• accel - orientation of the device and its position
• text - a text provided by user

I extended that application to be able to send error codes.

 

Pressing "Send Error" button and selecting error code sends a MQTT message to IoTF broker.

Typical error message looks like;

{

device_id: "doors9"

evt_type: "error"

timestamp:

{

$date: 1449059192155

}

evt: 

{

errorCode: "20"

}

}

For a compiled version of this application follow this link.
Now I could start updating my module.

Design for Analytics

Requirements in my DOORS module have a "ErrorCode" integer attribute which links a requirement to a error code reported by my device.

Additionally I'm using DOORS module level attributes to store values I do not need hardcoded in my DXL. Those are:

• Authorization (string)- so I do not need to calculate Base64 on each call
• DeviceType 
  (string) - which of my devices this module describes
• Organization (string) - my Bluemix organisation name

With all information in place I can write some simple layout DXL (which can be convert to attribute DXL later on to improve performance)

Layout DXL 

I want a specific type of event from all devices of a given type from my organization. So I need to use a historic query for a device type:

buf = "https://internetofthings.ibmcloud.com/api/v0001/historian/" org"/" deviceType "?evt_type=" eventId

This will return a JSON string with a list of events. If you do not have a JSON parser ready you can try to parse this data with Regular Expressions. Please remember this is a very simple example and in real world one shall not attempt to parse JSON with regular expressions.

My main worker code looks like:

if (!null obj) {
  int ival = obj."ErrorCode"
  if (ival == 0) halt
 
  string val = obj."ErrorCode"
  Module m = module obj
  string auth = m."Authorization"
  string dtype = m."DeviceType"
  string org = m."Organization"
 
  if (!null m && auth != "" && org != "" && dtype != "")
  {
    Buffer b = getData(auth, org, dtype, "error")

    if (!null b)  {
      string s = stringOf b
      Regexp re = regexp2 "\"device_id\":\"([^\"]*)\"[^}]+.[^}]+{\"errorCode\":\"([^\"]*)\""
      int i = 0
      string device = "", code =""
      //  temporary skip to hold names of devices which reported
      Skip erSkp = createString
      int allErrors = 0
      int numDevices = 0
      while (!null s && re s && i<100) {  // i is just a guard, we know there is no more then a 100 results in one page
        device = s[match 1]
        code = s[match 2]
        int ireported = 0
        // if code matches attribute value
        if (code == val)
        {
          allErrors++ // icrease number of errors
         
          if (!find(erSkp, device, ireported)) {
            put(erSkp, device, 1)
            numDevices++
          }
          else {
            ireported++
            put(erSkp, device, ireported, true)
          }
        }
        s = s[end 0 +1:]
        i++
      } // while
      // clean up
      delete b
     
      // report
      if (allErrors != 0) {
        for ireported in erSkp do {
          device = (string key erSkp)
          displayRich "Device with Id {\\b "device "} reported an issue " (ireported == 1 ? "once" : ireported" times")
        }
      }
     
      delete erSkp
    } // null b
  } // module setup
} // !null obj

You can find full DXL here.

Conclusion

Above layout DXL works fine when there're not so many devices. Once there will be more of them we no longer want to see how many times each device reported given issue. Thus the DXL could be rewritten to show something like:

As you saw in this example I'm using layout DXL but I think for a better understanding and feedback, one should consider writing a utility DXL.
Maybe that utility could provide its own UI for easier navigation?

In example above there's no need to send a HttpRequest for each object... It is enough to make one call per 100 (max page) events returned by query and write a little more complex Skip management. That however would require to make one top level Skip, but I'm sure you all know how to do it.