DWA headless authentication from Node.JS service

In my previous posts I was using dwaOauth.js to perform OAuth dance with DOORS Web Access. This was using DWA provided login page which required user to provide his DOORS credentials.

Then I showed you TRS feed reader which was using the same dwaOauth module. But it was very inconvenient to start reader and then manually authenticate with DWA. I would expect my server to connect to remote server itself and then just view the changes in the feed.

It's time to separate view from model and make them two separate applications.

Form authentication

With aid of superagent library one can perform form authentication with DWA and then use a agent to handle further TRS calls:

user1 = superagent.agent();
user1
  .post('https://your.dwa.server:port/dwa/j_acegi_security_check')
  .type('form')
  .send({ j_username: 'doors_user', j_password: 'password'})
  .end(function(err, res) {
    if (err != null) {
      console.log(err)
      user1 = null
    }
    else if (res.status != 200) {
      console.log(res.status)
    }
    else {
      console.log('user logged in to DWA')
      trs.TRSUpdate(user1, storeTRS);
    }
  })

Now user1 is authenticated and can be used to get data from DWA:

user1
  .get('https://your.dwa.server:port/dwa/rm/trs/changelog')
  .set('Accept', 'text/turtle')
  .buffer(true)
  .end(function(err, res) {
      // use res.text for plain text response from DWA
}

If you do not specify .buffer(true) res.text will be an empty string!

TRS in DOORS Next Generation

DOORS Next Generation has two TRS specifications implemented TRS and TRS 2.
With very small changes to the my previous posts you can read TRS1 feed from DNG.

Well if you remember my first post you will notice that my dwaOauth.js works just fine with DNG. So that little module gives you not only access to DWA protected resources but you can reuse it with DNG or other CLM servers.
Remember there were small changes done in order to make it working with most recent Node.js. You can read about those changes in my OSLC TRS listener.

DNG setup

You will need to prepare your DNG to provide TRS. Have a look at Register applications as TRS providers for the Lifecycle Query Engine in IBM Knowledge Center.

Usually DNG is ready to provide TRS feed, all you need is to assign a license to user who would be used to read it.

Also you will need to check permissions on /lqe/web/admin/permissions

Now yow are ready to rad TRS from DOORS Next Generation.

Changes in trs.js

There are very little changes in function getChangelogFromDWA

> var previous = '/rm/trs';

<  var previous = '/dwa/rm/trs/changelogs'

and

> if (previous == '/rm/trs') {

<  if (previous == '/dwa/rm/trs/changelog') {

So we just changes the starting point for triple feed.

You probably will notice the difference in TRS changelog id in DNG feed. It ses uuid and it looks something like:

urn:uuid:cdaf9dae-ad2d-4434-87a7-aea2488182d6

Thus we need to modify parseChangeLogs function parser:

async.whilst(
  function () { return rest != trs.nil && order > iLastOrder},
  // get ordered list or changes
  function (next) {
      var o = store.find(rest, trs.first, null)[0];
      if (typeof o !='undefined') {
        
        var timearr = o.object.split(':');
        if (timearr[0] == 'urn') {
            // update order
            order = parseInt(n3.Util.getLiteralValue( store.find(o.object, trs.order, null)[0].object) );
            idx++;
            //
            //  DO SOMETHING USEFUL HERE
            //
        }
        
          rest = store.find(rest, trs.rest, null)[0].object;
      } else {
          rest = trs.nil;
      }
      next();
  },
  function (err) {
    console.log("read " + idx + " changelog items");
      last(changes);
  });

DWA has change id has a useful format so one doesn't need further requests to gets some basic details of the change occurred but it is easy enough to write some callback functions getting changes from DNG. Of course one should not relay on format of the id as it is not standardized, defined nor promised not to change. 

Conclusion

Well TRS is TRS ;) with little changes to my inital TRS posts you can consume DWA, DNG or even Bugzilla TRS feed. In near future I will explain how to modify this parser for TRS 2.0.

Stay tuned!

TRS sample usage

If you read my last two posts you should be able to parse DWA TRS data.

Here I want to show two possible user cases using TRS data.

Below examples are using visjs.org to display graphs.

Change "Counter"

So simply show the amount of each type of changes per minute (day, week)

 

Data input

Each point has following format:

{
  "x": dt, //date and time of the event
  "group":grp, // one of [module,project,object][Creation,Deletion,Modification]
  "y": i // number of group operations per time interval (minute)
}

Browser code

Very simple Graph2d initialization and usage.

    <script>
        var groups = new vis.DataSet([
            {id:'objectModification', content:'Objects Modified', value:0},
            /* ....  */
            ]);
        var items = new vis.DataSet([]);
        var container = document.getElementById('visualization');
        var timeline = new vis.Graph2d(container);
     
        timeline.setItems(items);
        timeline.setGroups(groups);
        timeline.setOptions({
            drawPoints: {style:'circle'},
            interpolation: false,
            defaultGroup: 'ungrouped',
            legend: true
          });
 
        var socket = io();
        socket.on('trsevent', function(msg){
            items.add(msg);
        });
    </script>

Server code

Below code goes trough changes from latest to last parsed and calculates data for each minute which is present TRS feed.

Parser assumes following format of the change id:
urn:rational::1-dbid-[M|P|O-\d+]-00000525:hostname:port:YYYY-MM-DD:HH:mm:ss.nnn:xx
and it is not parsing other ids (like users:rational::...)

trs.js updated parser:

function parseChangeLogs(clResults, last) {
  var store = clResults[1];
  var lastID = clResults[0] || '';
  
  // check for store
  if (typeof store !== 'undefined') {
    var changesHead = store.find(store.find(null,
                        trs.changes,
                        null)[0].object,
                   trs.first,
                   null)[0];
    if (typeof changesHead !== 'undefined') {
      if (changesHead.object != clResults[0]) {
        
        // remember head
        cache.putCache('trs_lastChangeLog', changesHead.object);
        
        // try to get last trs.order based on lastId
        var lastOrder = (lastID != '') ? store.find(lastID, trs.order, null)[0].object : '"0"^^http://www.w3.org/2001/XMLSchema#integer';
        var iLastOrder = parseInt(n3.Util.getLiteralValue(lastOrder));

        // process changes from the newest to last parsed (or last in list)
        var rest = changesHead.subject;
        var changes = [];

        var order = Number.MAX_VALUE;
        async.whilst(
            function () { return rest != trs.nil && order > iLastOrder},
            // get ordered list or changes
            function (next) {
              var o = store.find(rest, trs.first, null)[0];
              if (typeof o !='undefined') {
                
                // o [_b:xx, trs.first, urn:rational::1-dbid-M-00000525:hostname:8443:2015-09-03:14:45:58.971:39]
                var timearr = o.object.split(':');
                if (timearr[0] == 'urn') {
                  var d = timearr[6].split('-');
                  var dt = new Date(d[0], d[1]-1, d[2], timearr[7], timearr[8], timearr[9]);
                  dt.setSeconds(0);
                  
                  // update order
                  order = parseInt(n3.Util.getLiteralValue( store.find(o.object, trs.order, null)[0].object) );
                  
                  // timearr[3] |= 1-dbid-M-00000525
                  //        |= 1-dbid-O-9-00000525    
                  var tmp = timearr[3].split('-');
                  var target = (tmp[2] == "M") ? 'module' : ((tmp[2] == "P") ? 'project' : 'object');
                  
                  if (target != '') {
                    // type -> group
                    var type = store.find(o.object, trs.type, null)[0].object;
                    var grp = target + type.split('#')[1];
                    
                    // push
                    var bDone = false;
                    for (var idx in changes) {
                      if (changes[idx].x.getTime() == dt.getTime() &&
                        changes[idx].x.getDate() == dt.getDate() && 
                        changes[idx].group == grp)
                      {
                        changes[idx].y++;
                        console.log('=>'+grp);
                        bDone = true;
                      }
                    }
                    if (!bDone) {
                      changes.push({x:dt, group:grp, y:1});
                    }
                  }
                }
                
                rest = store.find(rest, trs.rest, null)[0].object;
              } else {
                rest = trs.nil;
              }
              next();
            },
            function (err) {
              last(changes);
            });
      }
    }
  }
}

As you can see all data processing is done on server side, browser just displays those points. Same approach is used in second example

What, when, how

A timeline graph showing when item (project, module or object) was either created, modified or deleted:


Above is an iframe so you could try Timeline. Data behind this iframe is a static data used to show possible usage.

Browser code

index.ejs is similar to previous one but this time is using a Timeline graph:

<script>
        var groups = new vis.DataSet([
            {id:'Modification', value:0},
            {id:'Creation', value:1},
            {id:'Deletion', value:2}
            ]);
        var items = new vis.DataSet([]);
        var container = document.getElementById('visualization');
        var timeline = new vis.Timeline(container);

        timeline.setItems(items);
        timeline.setGroups(groups);
        timeline.setOptions({
            groupOrder: function (a, b) {
              return a.value - b.value;
            },
            editable: false,
            type: 'point'
          });

        var socket = io();
        socket.on('trsevent', function(msg){
            items.add(msg);
        });
</script>

Server code

Below is just a working part of parser:

async.whilst(
  function () { return rest != trs.nil && order > iLastOrder},
  // get ordered list or changes
  function (next) {
      var o = store.find(rest, trs.first, null)[0];
      if (typeof o !='undefined') {
        
        // o [_b:xx, trs.first, urn:rational::1-dbid-M-00000525:hostname:8443:2015-09-03:14:45:58.971:39]
        var timearr = o.object.split(':');
        if (timearr[0] == 'urn') {
          var d = timearr[6].split('-');
          var dt = new Date(d[0], d[1]-1, d[2], timearr[7], timearr[8], timearr[9]);
          
          // update order
          order = parseInt(n3.Util.getLiteralValue( store.find(o.object, trs.order, null)[0].object) );
          
          // timearr[3] |= 1-dbid-M-00000525
          //         |= 1-dbid-O-9-00000525    
          var tmp = timearr[3].split('-');
          var object = '';
          
          if (tmp[2] == "M") {
            var obj = store.find(o.object, trs.changed, null)[0].object;
            if (obj.indexOf('view') == -1) {
              object += 'Module ' + tmp[tmp.length-1];
            }
          } else if (tmp[2] == "P") {
            // Object
            object = 'Project ' + tmp[tmp.length-1];
          } else {
            // Object
            object = 'Module ' + tmp[tmp.length-1] + " object " + tmp[3];
          }
          
          if (object != '') {
            // type -> group
            var type = store.find(o.object, trs.type, null)[0].object;
            var grp = type.split('#')[1];
            
            // push
                  changes.push({start:dt, id:o.object, content:object, group:grp});
          }
        }
        
          rest = store.find(rest, trs.rest, null)[0].object;
      } else {
          rest = trs.nil;
      }
      next();
  },
  function (err) {
      last(changes);
  }
);

Same assumptions for change id.

Data passed to Timeline looks like:
{
  "start": time of event,
  "id": id of event (urn:rational...)
  "content": what to show
  "group": modification, deletion or creation
}

Conclusions

TRS gives us ordered list of changes to the resources in working set. Those are ordered from newest to the latest change. One can simply show them as a list of changes, to generate some kind of report, or a graph.

One might want to request some additional information using DXL services and show those changes in browser (see below), store them, etc. Possibilities are endless.

Showing a change made to the object with aid of DXL service and Timeline item templates:

C# integration

As many of you know DOORS has a COM interface. Some use it via VBS, other via Java.
Today I will show you how to make a very simple application with C#.

VBS example

Let's see very simple VBS example:

Set DOORSObj = CreateObject("DOORS.Application")

DOORSObj.runStr ("ack ""hello""")

It's so simple to call a DXL script from VBS!

C# Example

Well C# is little bit more complex than VBS. But once all steps are done it is very easy to use it.

CLR Assembly

First you need to create a .NET common language runtime (CLR) assembly. I used TlbImp.exe. This converts type definitions found in COM type library into equivalent definitions in CLR assembly.
You can find DOORS type library in DOORS bin folder:

C:\>"C:\Program Files (x86)\Microsoft SDKs\Windows\v7.0A\Bin\NETFX 4.0 Tools\x64\TlbImp.exe" "c:\Program Files\IBM\Rational\DOORS\9.6\bin\doors.tlb"

Above created a DOORSCOMLib.dll in c:\.

Once you have this dll you can simply add it to your C# project.

1. Select References
2. Add Referece...
3. Browse to the location where is your DOORSCOMLib.dll.
4. Once imported right-click on DOORSCOMLib and see its properties
5. Set "Embed Interop Types" to false.

Embed Interop Types set to false

Now you can use this library in your code.

DOORSCOMLib interface

DOORSCOMLib has a following interfaces:

IDoorsDXL lets you run DXL:

• void runDXL(string)
• void runFile(string) 
• string restult()

IDoorsStatus enables checking DOORS session status:

• string clientVersion
• string clientStatus (1 user logged in, 0 DOORS started but user not logged)

Usage

Usage of DOORSCOMLib is very simple; simply declare a required interface and use it.

IDoorsStatus doorsStatus = Activator.CreateInstance(typeof(DOORSClass)) as IDoorsStatus;

IDoorsDXL doorsDXL = Activator.CreateInstance(typeof(DOORSClass)) as IDoorsDXL;

Here is a simple "Hello Word" application illustrating how to use those interfaces:

using System;
using System.Collections.Generic;
using System.Linq;
using System.Text;
using DOORSCOMLib;

namespace HelloCOM
{
    class Program
    {
        static void Main(string[] args)
        {
            IDoorsStatus doorsStatus = null;
            doorsStatus = Activator.CreateInstance(typeof(DOORSClass)) as IDoorsStatus;
            if (doorsStatus == null) {
                Console.WriteLine("Null instance");
                return;
            }

            // the session is up, but user might not be logged in
            int trycount = 0;
            while (doorsStatus.clientState != 1 && trycount < 30)
            {
                Console.WriteLine("DOORS not logged in yet");
                System.Threading.Thread.Sleep(2000);
                ++trycount;
            }

            // we have a session
            Console.WriteLine("DOORS version " + doorsStatus.clientVersion);

            IDoorsDXL doors = Activator.CreateInstance(typeof(DOORSCOMLib.DOORSClass)) as IDoorsDXL;

            if (doors != null) {
                doors.runStr("ack \"hello from C# !\"");
            }
        }
    }
}

This should start a DOORS session (if not already started) and display a "hello from C# !" message.

Update

In Visual Studio 2010 or 2013 you can simply added doors.tlb to the references and you can skip creating CLR Assembly.