Displaying real-time IoT data in IBM DOORS (Doors Web Access)

There are many ways in which we could imagine surfacing IoT data in the IBM DOORS interface.

One way that I would propose is to extend the rich hover so that you could see IoT data in real-time, displayed when you hover over an IBM DOORS link-end point. Imagine being an IBM RQM user looking at a defect and being able to hover over a link to a requirement and see the history of a thing displayed.

Extending the rich hover to display IoT data in IBM DOORS Web Access

Extending the rich hover to display IBM DOORS IoT data in IBM RQM

Showing display of real-time data in hover over

As part of the technical investigation I looked into the implementation of a JavaScript client which listens for data from a device in the IoT Foundation. And then display that real-time data in the UI. The functioning graphing component is shown in the examples above. I will blog about the JavaScript / IoT implementation in a future post. 

DWA headless authentication from Node.JS service

In my previous posts I was using dwaOauth.js to perform OAuth dance with DOORS Web Access. This was using DWA provided login page which required user to provide his DOORS credentials.

Then I showed you TRS feed reader which was using the same dwaOauth module. But it was very inconvenient to start reader and then manually authenticate with DWA. I would expect my server to connect to remote server itself and then just view the changes in the feed.

It's time to separate view from model and make them two separate applications.

Form authentication

With aid of superagent library one can perform form authentication with DWA and then use a agent to handle further TRS calls:

user1 = superagent.agent();
user1
  .post('https://your.dwa.server:port/dwa/j_acegi_security_check')
  .type('form')
  .send({ j_username: 'doors_user', j_password: 'password'})
  .end(function(err, res) {
    if (err != null) {
      console.log(err)
      user1 = null
    }
    else if (res.status != 200) {
      console.log(res.status)
    }
    else {
      console.log('user logged in to DWA')
      trs.TRSUpdate(user1, storeTRS);
    }
  })

Now user1 is authenticated and can be used to get data from DWA:

user1
  .get('https://your.dwa.server:port/dwa/rm/trs/changelog')
  .set('Accept', 'text/turtle')
  .buffer(true)
  .end(function(err, res) {
      // use res.text for plain text response from DWA
}

If you do not specify .buffer(true) res.text will be an empty string!

TRS sample usage

If you read my last two posts you should be able to parse DWA TRS data.

Here I want to show two possible user cases using TRS data.

Below examples are using visjs.org to display graphs.

Change "Counter"

So simply show the amount of each type of changes per minute (day, week)

 

Data input

Each point has following format:

{
  "x": dt, //date and time of the event
  "group":grp, // one of [module,project,object][Creation,Deletion,Modification]
  "y": i // number of group operations per time interval (minute)
}

Browser code

Very simple Graph2d initialization and usage.

    <script>
        var groups = new vis.DataSet([
            {id:'objectModification', content:'Objects Modified', value:0},
            /* ....  */
            ]);
        var items = new vis.DataSet([]);
        var container = document.getElementById('visualization');
        var timeline = new vis.Graph2d(container);
     
        timeline.setItems(items);
        timeline.setGroups(groups);
        timeline.setOptions({
            drawPoints: {style:'circle'},
            interpolation: false,
            defaultGroup: 'ungrouped',
            legend: true
          });
 
        var socket = io();
        socket.on('trsevent', function(msg){
            items.add(msg);
        });
    </script>

Server code

Below code goes trough changes from latest to last parsed and calculates data for each minute which is present TRS feed.

Parser assumes following format of the change id:
urn:rational::1-dbid-[M|P|O-\d+]-00000525:hostname:port:YYYY-MM-DD:HH:mm:ss.nnn:xx
and it is not parsing other ids (like users:rational::...)

trs.js updated parser:

function parseChangeLogs(clResults, last) {
  var store = clResults[1];
  var lastID = clResults[0] || '';
  
  // check for store
  if (typeof store !== 'undefined') {
    var changesHead = store.find(store.find(null,
                        trs.changes,
                        null)[0].object,
                   trs.first,
                   null)[0];
    if (typeof changesHead !== 'undefined') {
      if (changesHead.object != clResults[0]) {
        
        // remember head
        cache.putCache('trs_lastChangeLog', changesHead.object);
        
        // try to get last trs.order based on lastId
        var lastOrder = (lastID != '') ? store.find(lastID, trs.order, null)[0].object : '"0"^^http://www.w3.org/2001/XMLSchema#integer';
        var iLastOrder = parseInt(n3.Util.getLiteralValue(lastOrder));

        // process changes from the newest to last parsed (or last in list)
        var rest = changesHead.subject;
        var changes = [];

        var order = Number.MAX_VALUE;
        async.whilst(
            function () { return rest != trs.nil && order > iLastOrder},
            // get ordered list or changes
            function (next) {
              var o = store.find(rest, trs.first, null)[0];
              if (typeof o !='undefined') {
                
                // o [_b:xx, trs.first, urn:rational::1-dbid-M-00000525:hostname:8443:2015-09-03:14:45:58.971:39]
                var timearr = o.object.split(':');
                if (timearr[0] == 'urn') {
                  var d = timearr[6].split('-');
                  var dt = new Date(d[0], d[1]-1, d[2], timearr[7], timearr[8], timearr[9]);
                  dt.setSeconds(0);
                  
                  // update order
                  order = parseInt(n3.Util.getLiteralValue( store.find(o.object, trs.order, null)[0].object) );
                  
                  // timearr[3] |= 1-dbid-M-00000525
                  //        |= 1-dbid-O-9-00000525    
                  var tmp = timearr[3].split('-');
                  var target = (tmp[2] == "M") ? 'module' : ((tmp[2] == "P") ? 'project' : 'object');
                  
                  if (target != '') {
                    // type -> group
                    var type = store.find(o.object, trs.type, null)[0].object;
                    var grp = target + type.split('#')[1];
                    
                    // push
                    var bDone = false;
                    for (var idx in changes) {
                      if (changes[idx].x.getTime() == dt.getTime() &&
                        changes[idx].x.getDate() == dt.getDate() && 
                        changes[idx].group == grp)
                      {
                        changes[idx].y++;
                        console.log('=>'+grp);
                        bDone = true;
                      }
                    }
                    if (!bDone) {
                      changes.push({x:dt, group:grp, y:1});
                    }
                  }
                }
                
                rest = store.find(rest, trs.rest, null)[0].object;
              } else {
                rest = trs.nil;
              }
              next();
            },
            function (err) {
              last(changes);
            });
      }
    }
  }
}

As you can see all data processing is done on server side, browser just displays those points. Same approach is used in second example

What, when, how

A timeline graph showing when item (project, module or object) was either created, modified or deleted:


Above is an iframe so you could try Timeline. Data behind this iframe is a static data used to show possible usage.

Browser code

index.ejs is similar to previous one but this time is using a Timeline graph:

<script>
        var groups = new vis.DataSet([
            {id:'Modification', value:0},
            {id:'Creation', value:1},
            {id:'Deletion', value:2}
            ]);
        var items = new vis.DataSet([]);
        var container = document.getElementById('visualization');
        var timeline = new vis.Timeline(container);

        timeline.setItems(items);
        timeline.setGroups(groups);
        timeline.setOptions({
            groupOrder: function (a, b) {
              return a.value - b.value;
            },
            editable: false,
            type: 'point'
          });

        var socket = io();
        socket.on('trsevent', function(msg){
            items.add(msg);
        });
</script>

Server code

Below is just a working part of parser:

async.whilst(
  function () { return rest != trs.nil && order > iLastOrder},
  // get ordered list or changes
  function (next) {
      var o = store.find(rest, trs.first, null)[0];
      if (typeof o !='undefined') {
        
        // o [_b:xx, trs.first, urn:rational::1-dbid-M-00000525:hostname:8443:2015-09-03:14:45:58.971:39]
        var timearr = o.object.split(':');
        if (timearr[0] == 'urn') {
          var d = timearr[6].split('-');
          var dt = new Date(d[0], d[1]-1, d[2], timearr[7], timearr[8], timearr[9]);
          
          // update order
          order = parseInt(n3.Util.getLiteralValue( store.find(o.object, trs.order, null)[0].object) );
          
          // timearr[3] |= 1-dbid-M-00000525
          //         |= 1-dbid-O-9-00000525    
          var tmp = timearr[3].split('-');
          var object = '';
          
          if (tmp[2] == "M") {
            var obj = store.find(o.object, trs.changed, null)[0].object;
            if (obj.indexOf('view') == -1) {
              object += 'Module ' + tmp[tmp.length-1];
            }
          } else if (tmp[2] == "P") {
            // Object
            object = 'Project ' + tmp[tmp.length-1];
          } else {
            // Object
            object = 'Module ' + tmp[tmp.length-1] + " object " + tmp[3];
          }
          
          if (object != '') {
            // type -> group
            var type = store.find(o.object, trs.type, null)[0].object;
            var grp = type.split('#')[1];
            
            // push
                  changes.push({start:dt, id:o.object, content:object, group:grp});
          }
        }
        
          rest = store.find(rest, trs.rest, null)[0].object;
      } else {
          rest = trs.nil;
      }
      next();
  },
  function (err) {
      last(changes);
  }
);

Same assumptions for change id.

Data passed to Timeline looks like:
{
  "start": time of event,
  "id": id of event (urn:rational...)
  "content": what to show
  "group": modification, deletion or creation
}

Conclusions

TRS gives us ordered list of changes to the resources in working set. Those are ordered from newest to the latest change. One can simply show them as a list of changes, to generate some kind of report, or a graph.

One might want to request some additional information using DXL services and show those changes in browser (see below), store them, etc. Possibilities are endless.

Showing a change made to the object with aid of DXL service and Timeline item templates: