Showing posts with label trs. Show all posts
Showing posts with label trs. Show all posts

Friday, 11 December 2015

DOORS reporting to IoT

So far we had an example how to use DOORS to read IoT historian data. Now it's time to publish something back.

There's no problem sending some useless data, the problem is to make sense out if it. So let's think on some 'real World' usage.


"Would you like to know when your most favorite module was modified?" 


Why not!

This could look like this:
Someone is playing with my module!


If you read my previous posts you know we have a Tracked Resource Set (TRS) support in DOORS Web Access.
When enabled (as it's not by default) TRS "tells" us about all Modification, Creation and Deletion changes in DOORS Database.

So we need to modify a TRS reader to its new purpose.

I decided I won't need a front end for my TRS translation service so I rewrote the app.js:

/* jshint node:true */
var trs = require('./trs')
var superagent = require('superagent');
var iotf = require("ibmiotf");

//this is used to hide self-signed certificate errors
process.env.NODE_TLS_REJECT_UNAUTHORIZED = "0";

// IoTF setup
var deviceClientConfig = {
  org: 'quickstart',
  type: 'mytype',
  id: '001122334455',
  'auth-method':'token',
  'auth-token': 'secret'
};

var deviceClient = new iotf.IotfDevice(deviceClientConfig );
deviceClient.connect();
deviceClient.on("connect", function () {
 console.log("Connected to IoTF");
});

var user1 = null;
function updateTRS() {
 if (user1 == null) {
  user1 = superagent.agent();
  user1
    .post('https://DOORS_SERVER:8443/dwa/j_acegi_security_check')
    .type('form')
    .send({ j_username: 'DOORS_USER', j_password: 'PASSWORD'})
    .end(function(err, res) {
    })
 }
 else{
  trs.TRSUpdate(user1, deviceClient, function(err) {
    // just ping I'm alive
    console.log(". " +err); // one might want to check if err is defined
  });
 }
}

//query DWA for TRS changes
setInterval(updateTRS, 10*1000); //every 10 seconds
updateTRS()


Simple and quick. Now changes to trs.js which as you can see now takes additional deviceClient parameter.

Changes are really obvious:

1 . First publishing method

function publishTRS(deviceClient, data) {
   //publishing event using the user-defined quality of service
   var myQosLevel=2

   deviceClient.publish("trs_update","json",'{"d" : { "trs" : '+JSON.stringify(data)+' }}', myQosLevel);
   console.log("send data ->" + util.inspect(data));
}

2. Modified function parameters.
Note changes are no longer added to array. Instead those are send directly to IoTF

< function parseChangeLogs(clResults, client, last) {
---
> function parseChangeLogs(clResults, last) {
>     var changes = [];
97c88
< publishTRS(client, {start:dt, id:o.object, content:object, group:grp});
---
> changes.push({start:dt, id:o.object, content:object, group:grp});
108c99
< last(err);
---
> last(changes);
169c159
< exports.TRSUpdate = function(usr, client, last) {
---
> exports.TRSUpdate = function(usr, last) {
176c166
< parseChangeLogs(results, client, last);
---
> parseChangeLogs(results, last);

Now your TRS example is ready to talk to IoTF! Time to prepare IoTF application to use this feed.

Internet of Things Foundation

Go to you Bluemix dashboard and add new IoTF starter application. This will create for you a simple Node-Red application. Remember to add some credentials so only you will have access to your dashboard.

Once ready navigate to you IoTF dashboard
IoTF dashboard link

Here we will add a new device type to you IoT world.

New IoTF Device Type

Select Devices, Device Types and Create Type (on the bottom of the page)

The wizard will guide you to provide

  1. Name and description
  2. Define template
  3. Define template

Save the type and proceed to add new Device


IoTF Device

Select Devices, Create Device (again on the bottom of the page)
You will have similar wizard. If you select Type defined earlier some fields will be populated.

  1. Device ID
  2. Security
  3. Summary
    Please note credentials are NOT recoverable!

Finally Node-Red end


Once your device is ready, you can work on your Node Red consumer. 
Here I will just indicate how it might look like. I'm not going to write your business logic ;) I'm just showing an example what you can do.

So your full example would might look like:

Mine is just showing it works so you can try and do what you want.

I had to configure my IoT input:

Prepare my "business logic"

And observe the output if warning is set to true:


Conclusion

In those really simple steps we made a DOORS Web Access Tracked Resource Set a s Thing!

We just 'thingified' DOORS! 

Monday, 14 September 2015

TRS in DOORS Next Generation

DOORS Next Generation has two TRS specifications implemented TRS and TRS 2.
With very small changes to the my previous posts you can read TRS1 feed from DNG.

Well if you remember my first post you will notice that my dwaOauth.js works just fine with DNG. So that little module gives you not only access to DWA protected resources but you can reuse it with DNG or other CLM servers.
Remember there were small changes done in order to make it working with most recent Node.js. You can read about those changes in my OSLC TRS listener.

DNG setup

You will need to prepare your DNG to provide TRS. Have a look at Register applications as TRS providers for the Lifecycle Query Engine in IBM Knowledge Center.

Usually DNG is ready to provide TRS feed, all you need is to assign a license to user who would be used to read it.

Also you will need to check permissions on /lqe/web/admin/permissions

Now yow are ready to rad TRS from DOORS Next Generation.

Changes in trs.js

There are very little changes in function getChangelogFromDWA
> var previous = '/rm/trs';
var previous = '/dwa/rm/trs/changelogs'

and
> if (previous == '/rm/trs') {
if (previous == '/dwa/rm/trs/changelog') {

So we just changes the starting point for triple feed.

You probably will notice the difference in TRS changelog id in DNG feed. It ses uuid and it looks something like:
urn:uuid:cdaf9dae-ad2d-4434-87a7-aea2488182d6

Thus we need to modify parseChangeLogs function parser:

async.whilst(
  function () { return rest != trs.nil && order > iLastOrder},
  // get ordered list or changes
  function (next) {
      var o = store.find(rest, trs.first, null)[0];
      if (typeof o !='undefined') {
        
        var timearr = o.object.split(':');
        if (timearr[0] == 'urn') {
            // update order
            order = parseInt(n3.Util.getLiteralValue( store.find(o.object, trs.order, null)[0].object) );
            idx++;
            //
            //  DO SOMETHING USEFUL HERE
            //
        }
        
          rest = store.find(rest, trs.rest, null)[0].object;
      } else {
          rest = trs.nil;
      }
      next();
  },
  function (err) {
    console.log("read " + idx + " changelog items");
      last(changes);
  });

DWA has change id has a useful format so one doesn't need further requests to gets some basic details of the change occurred but it is easy enough to write some callback functions getting changes from DNG. Of course one should not relay on format of the id as it is not standardized, defined nor promised not to change. 

Conclusion

Well TRS is TRS ;) with little changes to my inital TRS posts you can consume DWA, DNG or even Bugzilla TRS feed. In near future I will explain how to modify this parser for TRS 2.0.
Stay tuned!

Tuesday, 8 September 2015

TRS sample usage

If you read my last two posts you should be able to parse DWA TRS data.

Here I want to show two possible user cases using TRS data.

Below examples are using visjs.org to display graphs.

Change "Counter"

So simply show the amount of each type of changes per minute (day, week)

 

Data input

Each point has following format:
{
  "x": dt, //date and time of the event
  "group":grp, // one of [module,project,object][Creation,Deletion,Modification]
  "y": i // number of group operations per time interval (minute)
}

Browser code

Very simple Graph2d initialization and usage.
    <script>
        var groups = new vis.DataSet([
            {id:'objectModification', content:'Objects Modified', value:0},
            /* ....  */
            ]);
        var items = new vis.DataSet([]);
        var container = document.getElementById('visualization');
        var timeline = new vis.Graph2d(container);
     
        timeline.setItems(items);
        timeline.setGroups(groups);
        timeline.setOptions({
            drawPoints: {style:'circle'},
            interpolation: false,
            defaultGroup: 'ungrouped',
            legend: true
          });
 
        var socket = io();
        socket.on('trsevent', function(msg){
            items.add(msg);
        });
    </script>

Server code

Below code goes trough changes from latest to last parsed and calculates data for each minute which is present TRS feed.

Parser assumes following format of the change id:
urn:rational::1-dbid-[M|P|O-\d+]-00000525:hostname:port:YYYY-MM-DD:HH:mm:ss.nnn:xx
and it is not parsing other ids (like users:rational::...)

trs.js updated parser:
function parseChangeLogs(clResults, last) {
  var store = clResults[1];
  var lastID = clResults[0] || '';
  
  // check for store
  if (typeof store !== 'undefined') {
    var changesHead = store.find(store.find(null,
                        trs.changes,
                        null)[0].object,
                   trs.first,
                   null)[0];
    if (typeof changesHead !== 'undefined') {
      if (changesHead.object != clResults[0]) {
        
        // remember head
        cache.putCache('trs_lastChangeLog', changesHead.object);
        
        // try to get last trs.order based on lastId
        var lastOrder = (lastID != '') ? store.find(lastID, trs.order, null)[0].object : '"0"^^http://www.w3.org/2001/XMLSchema#integer';
        var iLastOrder = parseInt(n3.Util.getLiteralValue(lastOrder));

        // process changes from the newest to last parsed (or last in list)
        var rest = changesHead.subject;
        var changes = [];

        var order = Number.MAX_VALUE;
        async.whilst(
            function () { return rest != trs.nil && order > iLastOrder},
            // get ordered list or changes
            function (next) {
              var o = store.find(rest, trs.first, null)[0];
              if (typeof o !='undefined') {
                
                // o [_b:xx, trs.first, urn:rational::1-dbid-M-00000525:hostname:8443:2015-09-03:14:45:58.971:39]
                var timearr = o.object.split(':');
                if (timearr[0] == 'urn') {
                  var d = timearr[6].split('-');
                  var dt = new Date(d[0], d[1]-1, d[2], timearr[7], timearr[8], timearr[9]);
                  dt.setSeconds(0);
                  
                  // update order
                  order = parseInt(n3.Util.getLiteralValue( store.find(o.object, trs.order, null)[0].object) );
                  
                  // timearr[3] |= 1-dbid-M-00000525
                  //        |= 1-dbid-O-9-00000525    
                  var tmp = timearr[3].split('-');
                  var target = (tmp[2] == "M") ? 'module' : ((tmp[2] == "P") ? 'project' : 'object');
                  
                  if (target != '') {
                    // type -> group
                    var type = store.find(o.object, trs.type, null)[0].object;
                    var grp = target + type.split('#')[1];
                    
                    // push
                    var bDone = false;
                    for (var idx in changes) {
                      if (changes[idx].x.getTime() == dt.getTime() &&
                        changes[idx].x.getDate() == dt.getDate() && 
                        changes[idx].group == grp)
                      {
                        changes[idx].y++;
                        console.log('=>'+grp);
                        bDone = true;
                      }
                    }
                    if (!bDone) {
                      changes.push({x:dt, group:grp, y:1});
                    }
                  }
                }
                
                rest = store.find(rest, trs.rest, null)[0].object;
              } else {
                rest = trs.nil;
              }
              next();
            },
            function (err) {
              last(changes);
            });
      }
    }
  }
}

As you can see all data processing is done on server side, browser just displays those points. Same approach is used in second example

What, when, how

A timeline graph showing when item (project, module or object) was either created, modified or deleted:


Above is an iframe so you could try Timeline. Data behind this iframe is a static data used to show possible usage.

Browser code

index.ejs is similar to previous one but this time is using a Timeline graph:
<script>
        var groups = new vis.DataSet([
            {id:'Modification', value:0},
            {id:'Creation', value:1},
            {id:'Deletion', value:2}
            ]);
        var items = new vis.DataSet([]);
        var container = document.getElementById('visualization');
        var timeline = new vis.Timeline(container);

        timeline.setItems(items);
        timeline.setGroups(groups);
        timeline.setOptions({
            groupOrder: function (a, b) {
              return a.value - b.value;
            },
            editable: false,
            type: 'point'
          });

        var socket = io();
        socket.on('trsevent', function(msg){
            items.add(msg);
        });
</script>

Server code

Below is just a working part of parser:
async.whilst(
  function () { return rest != trs.nil && order > iLastOrder},
  // get ordered list or changes
  function (next) {
      var o = store.find(rest, trs.first, null)[0];
      if (typeof o !='undefined') {
        
        // o [_b:xx, trs.first, urn:rational::1-dbid-M-00000525:hostname:8443:2015-09-03:14:45:58.971:39]
        var timearr = o.object.split(':');
        if (timearr[0] == 'urn') {
          var d = timearr[6].split('-');
          var dt = new Date(d[0], d[1]-1, d[2], timearr[7], timearr[8], timearr[9]);
          
          // update order
          order = parseInt(n3.Util.getLiteralValue( store.find(o.object, trs.order, null)[0].object) );
          
          // timearr[3] |= 1-dbid-M-00000525
          //         |= 1-dbid-O-9-00000525    
          var tmp = timearr[3].split('-');
          var object = '';
          
          if (tmp[2] == "M") {
            var obj = store.find(o.object, trs.changed, null)[0].object;
            if (obj.indexOf('view') == -1) {
              object += 'Module ' + tmp[tmp.length-1];
            }
          } else if (tmp[2] == "P") {
            // Object
            object = 'Project ' + tmp[tmp.length-1];
          } else {
            // Object
            object = 'Module ' + tmp[tmp.length-1] + " object " + tmp[3];
          }
          
          if (object != '') {
            // type -> group
            var type = store.find(o.object, trs.type, null)[0].object;
            var grp = type.split('#')[1];
            
            // push
                  changes.push({start:dt, id:o.object, content:object, group:grp});
          }
        }
        
          rest = store.find(rest, trs.rest, null)[0].object;
      } else {
          rest = trs.nil;
      }
      next();
  },
  function (err) {
      last(changes);
  }
);

Same assumptions for change id.

Data passed to Timeline looks like:
{
  "start": time of event,
  "id": id of event (urn:rational...)
  "content": what to show
  "group": modification, deletion or creation
}

Conclusions

TRS gives us ordered list of changes to the resources in working set. Those are ordered from newest to the latest change. One can simply show them as a list of changes, to generate some kind of report, or a graph.

One might want to request some additional information using DXL services and show those changes in browser (see below), store them, etc. Possibilities are endless.

Showing a change made to the object with aid of DXL service and Timeline item templates: