DWA headless authentication from Node.JS service

In my previous posts I was using dwaOauth.js to perform OAuth dance with DOORS Web Access. This was using DWA provided login page which required user to provide his DOORS credentials.

Then I showed you TRS feed reader which was using the same dwaOauth module. But it was very inconvenient to start reader and then manually authenticate with DWA. I would expect my server to connect to remote server itself and then just view the changes in the feed.

It's time to separate view from model and make them two separate applications.

Form authentication

With aid of superagent library one can perform form authentication with DWA and then use a agent to handle further TRS calls:

user1 = superagent.agent();
user1
  .post('https://your.dwa.server:port/dwa/j_acegi_security_check')
  .type('form')
  .send({ j_username: 'doors_user', j_password: 'password'})
  .end(function(err, res) {
    if (err != null) {
      console.log(err)
      user1 = null
    }
    else if (res.status != 200) {
      console.log(res.status)
    }
    else {
      console.log('user logged in to DWA')
      trs.TRSUpdate(user1, storeTRS);
    }
  })

Now user1 is authenticated and can be used to get data from DWA:

user1
  .get('https://your.dwa.server:port/dwa/rm/trs/changelog')
  .set('Accept', 'text/turtle')
  .buffer(true)
  .end(function(err, res) {
      // use res.text for plain text response from DWA
}

If you do not specify .buffer(true) res.text will be an empty string!

TRS in DOORS Next Generation

DOORS Next Generation has two TRS specifications implemented TRS and TRS 2.
With very small changes to the my previous posts you can read TRS1 feed from DNG.

Well if you remember my first post you will notice that my dwaOauth.js works just fine with DNG. So that little module gives you not only access to DWA protected resources but you can reuse it with DNG or other CLM servers.
Remember there were small changes done in order to make it working with most recent Node.js. You can read about those changes in my OSLC TRS listener.

DNG setup

You will need to prepare your DNG to provide TRS. Have a look at Register applications as TRS providers for the Lifecycle Query Engine in IBM Knowledge Center.

Usually DNG is ready to provide TRS feed, all you need is to assign a license to user who would be used to read it.

Also you will need to check permissions on /lqe/web/admin/permissions

Now yow are ready to rad TRS from DOORS Next Generation.

Changes in trs.js

There are very little changes in function getChangelogFromDWA

> var previous = '/rm/trs';

<  var previous = '/dwa/rm/trs/changelogs'

and

> if (previous == '/rm/trs') {

<  if (previous == '/dwa/rm/trs/changelog') {

So we just changes the starting point for triple feed.

You probably will notice the difference in TRS changelog id in DNG feed. It ses uuid and it looks something like:

urn:uuid:cdaf9dae-ad2d-4434-87a7-aea2488182d6

Thus we need to modify parseChangeLogs function parser:

async.whilst(
  function () { return rest != trs.nil && order > iLastOrder},
  // get ordered list or changes
  function (next) {
      var o = store.find(rest, trs.first, null)[0];
      if (typeof o !='undefined') {
        
        var timearr = o.object.split(':');
        if (timearr[0] == 'urn') {
            // update order
            order = parseInt(n3.Util.getLiteralValue( store.find(o.object, trs.order, null)[0].object) );
            idx++;
            //
            //  DO SOMETHING USEFUL HERE
            //
        }
        
          rest = store.find(rest, trs.rest, null)[0].object;
      } else {
          rest = trs.nil;
      }
      next();
  },
  function (err) {
    console.log("read " + idx + " changelog items");
      last(changes);
  });

DWA has change id has a useful format so one doesn't need further requests to gets some basic details of the change occurred but it is easy enough to write some callback functions getting changes from DNG. Of course one should not relay on format of the id as it is not standardized, defined nor promised not to change. 

Conclusion

Well TRS is TRS ;) with little changes to my inital TRS posts you can consume DWA, DNG or even Bugzilla TRS feed. In near future I will explain how to modify this parser for TRS 2.0.

Stay tuned!

OSLC Resource Selection

Last time I presented idea of OSLC DXL Services which can very powerful and useful.
One can write a DXL services against database, projects or modules, but providing DOORS resources links manually might be very inconvenient.
There is much easier way to select your resource.
DWA server is OSLC RM Provider and as such it provides resource selection delegated UI.
You can find more information on Delegated UI here.
So before I will continue with next more complex examples of OSLC DXL Services or more powerful Node.JS applications I will focus on simple resources selection.
In terms of OSLC you can select Requirement Collection (a Module) or single Requirement (Object/Requirement).

Protocols

There are two protocols which OSLC Consumer could use to select resouce:

• #oslc-core-postMessage-1.0 - Indicates that the Post Message protocol is to be used
• #oslc-core-windowName-1.0 - Indicates that the Window Name protocol is to be used

Here I will show how to use post message protocol which works with all modern browsers. OSLC specification provides UI consumer's responsibilities which should be followed:

1. Include the Delegated UI Dialog via iframe (i.e setting iframe src to the URI of the Delegated UI Dialog) or via an embedded browser. Append the fragment identifier #oslc-core-postMessage-1.0 to the URL to indicate that Post Message is the desired protocol.
2. Add a 'message' listener to the outer window to receive messages from the Delegated UI Dialog.
3. Listen for window 'message' events, ignoring events from other sources or not prefixed with "oslc-response:"
4. When message from Delegated UI Dialog indicates user completed action, free resources and handle action. 

1. Providing an iframe for Delegated UI

Let's look at very basic HTML page:

<!DOCTYPE html>
<html>
<head>
    <title>DWA OSLC Delegated UI test</title>
    <meta http-equiv="Content-Type" content="text/html; charset=utf-8">
    <script type="text/javascript" src="/scripts/delegatedui.js"></script>
</head>

<body >
    <h1>DWA OSLC Delegated UI test</h1>
    <div>
        <button id='buttonSelect' type="button" onclick="delegatedUI.selectResource()">Select module</button>
    </div>
    <div id="OSLCMainDiv"></div>
    <div id="anotherDiv"  >
    </div>
</body>

Here I have just a button and some divs. You can find full source of /scripts/delegatedui.js here.
Button "Select Module" has following callback:

function selectResource() {
        clear();
        
        // Add 'iframe' element dynamically
        var frameName = "";
        frame = createElement("iframe", frameName);
        frame.id = "OSLCContainer";
        frame.width='400px';
        frame.height='500px';

        addEvent(window, "message", onMessageReceived);
        
        var mainDiv = document.getElementById("OSLCMainDiv");
        if (mainDiv) {
            mainDiv.appendChild(frame);
        }

        frame.src = "https://server:port/dwa/rm/calm/v2/selectRequirementCollection" + "#oslc-core-postMessage-1.0";
    }

This simply dynamically adds an iframe, registers its callback and appends "#oslc-core-postMessage-1.0" as chosen protocol.
Of course one should discover OSLC Service Provider selection dialog using Service oslc:selectionDialog property (as defined specification), but I used general DWA schema for requirement collection (a module) selection.

2. Adding message listener

We need to registering 'message' event listener as called from selectResource function:

function addEvent(obj, type, func) {
        if (obj.addEventListener) {
            obj.addEventListener(type, func, false);
        }
        else if (obj.attachEvent) {
            obj.attachEvent("on" + type, func);
        }
        else {
            obj["on" + type] = func;
        }
    } 

3. Listen to events

Once message is registered we have to listen to incoming events ignoring those not prefixed with "oslc-response:" :

 function onMessageReceived(event) {
        if (frame && event.source === frame.contentWindow) {
            var prefix = "oslc-response:";
            if (event.data.substring(0, prefix.length) === prefix) {
                done(event.data.substring(prefix.length));
            }
        }
    }

With 'done' method triggered when 'oslc-response' is found in event data:

function done(data) {
        var title = JSON.parse(data, function(key, value) {
            if (key.equals('rdf:resource')) {
                clear();
                // TODO: add you logic here
                // value contains a link to the resource
                // title is a Module name
            }
        });
    }

4.  Cleanup

Whatever cleanup you require that's the place to do it. I simply remove listener and iframe element

function clear() {
        var mainDiv = document.getElementById("OSLCMainDiv");
        if (mainDiv) {
            // clear any existing child elements
            mainDiv.innerHTML = ''; 
        }

        removeEvent(window, "message", onMessageReceived);
    }

Node.js OAuth handshake with DWA

In first post I show you basic OAuth handshake between your Node.JS application and DOORS Web Access Server.

This post will be longer than other ones, here I will show you how to create application step-by-step. In next posts I will be just extending functionality of this one by adding some services or new methods.

What you'll need to build your application

If you do not have Git on your machine I can recommend Eclipse eGit, which nicely integrates Git software control manager (Git SCM) into your development environment.


Since this application is written in Node.JS you might also want to install Nodeclipse to get default formatting and language highlighting.

Any time you can also use command line tools for which you will need Node.JS and some Git client.

That's all you need to start your local server development, but we know things can quickly escalate so you might want to have a look at IBM Cloud services.

To the Clouds and beyond

If you need more services, processes, memory, storage, you might want to host your application on the Cloud. IBM Bluemix is a Next-Generation App development platform. All you need is your IBM ID and you can start developing cloud applications.

If you already have an IBM ID follow this link to begin your free trial. Otherwise you can create one using the IBM Bluemix registration page.

Once you have your IBM Bluemix account trial started a good idea would be to complete the Getting Started with IBM Bluemix tutorial.

Install dependencies

Before we can write the application we should provide Node.js dependencies. This demo application makes use of the following modules: express, oauth, request, cookie-parser and libxmljs.


These need added those to your package.json

{
  "name": "HelloDWAOAuth",
  "version": "0.0.1",
  "description": "A sample nodejs app for OAuth with DWA",
  "dependencies": {
    "express": "3.4.7",
    "jade": "^1.1.4",
                "libxmljs": "0.13.0",
                "cookie-parser":"",
                "oauth": "",
                "request":"",
                "util": "",
                "url" : ""
  },
  "engines": {
    "node": "0.10.*"
  },
  "repository": {}
}

Open command line and navigate to your project directory, and type:

> npm install

This will install all of the required dependencies based on your package.json

Note:

Sometimes you can get errors like:

The builds tools for Visual Studio 2010 (Platform Toolset = 'v100') cannot be found.install new module.
 

It might be useful to try –msvs_version=[2005, 2008, 2012] switch if default build will fail.

Writing the server application

This demo uses Express. You will need to create express instance, define routes and provide functions that might be required to execute those routes. Here is the basic code for an express server.

var express = require('express');

// setup middleware
var app = express();
app.use(app.router);
app.use(express.errorHandler());
app.use(express.static(__dirname + '/public')); //setup static public directory
app.set('view engine', 'jade');
app.set('views', __dirname + '/views'); //optional since express defaults to CWD/views

// render index page - main route
app.get('/', function(req, res){
                    res.render('index');
});

var appInfo = JSON.parse(process.env.VCAP_APPLICATION || "{}");
var services = JSON.parse(process.env.VCAP_SERVICES || "{}");
var host = (process.env.VCAP_APP_HOST || 'localhost');

// The port on the DEA for communication with the application:
var port = (process.env.VCAP_APP_PORT || 3000);

// Start server
app.listen(port, host);
console.log('App started on port ' + port);

16 lines of code is all you need for a basic HTTP server written in Node.js. Put it into a new app.js file and that will be an entry point for our application.

Certificates

If you will be working in a development environment you might need to use self-signed certificates.

Those by default will be rejected and no connection will be made. If you want to prevent self-signed certificate error add following line:

 process.env.NODE_TLS_REJECT_UNAUTHORIZED = "0";

Adding OAuth to your application

Let's add OAuth as a separate Node.js module so it won't clatter the main app.js code. Modular approach will also help you to separate OAuth module and reuse it in other applications.

So now we will have app.js and Oauth.js files.

App.js will contain main routes and application logic for our sample. OAuth 1.0 requires user to provide application key and secret used to authenticate the application with a remote server. Thus our application will have two pages, where the user will be asked to provide location of rootservices, application key and OAuth secret.

Once those are provided the user will be redirected to page providing basic access to protected resources.

In this example we are using oauth Node.js module which comes with very useful examples how to do a basic Oauth handshake.

Configuring OAuth

Let's start with gathering information required to start OAuth. Create body.jade to add a form with POST action to /set_key_and_secret:

form(name="loginform", action="/set_key_and_secret", method="post")
          | Rootservices location:
          input(type="text", name="rootservices")
          br
          | Consumer Key:
          input(type="text", name="key")
          br
          | OAuth Secret:
          input(type="password", name="secret")
          br
          input(type="submit", value="Submit")

Now we have to add POST route to our application (app.js):

app.post('/set_key_and_secret', function(req, res) {
    //if (typeof req.body.rootservices !== 'undefined') console.log(req.body.rootservices);
    if (typeof req.body !== 'undefined') {
        if (typeof req.body.rootservices !== 'undefined' && req.body.rootservices != '' &&
            typeof req.body.key !== 'undefined' && req.body.key != '' &&
            typeof req.body.secret !== 'undefined' && req.body.sercret != '')
        {
            dwaOauth.configure(req.body.rootservices, req.body.key, req.body.secret, process.env.VCAP_APP_PORT || 3000);
            console.log("got settings");
            res.redirect("/dwa_login");
        }
        else {
            res.redirect('/');
        }
    }
});

Note: request.body is not available by default in Node.js express, we need to inform our application we would like to have access to this object. In order to do this we need to add:

app.use(express.bodyParser());

This must be added before any application routes will be defined. Best place is to add it before:

app.use(app.router);

dwaOauth.js module

You could wonder what is dwaOauth.configure called in preview example:

var location=null, oauth_key=null, oauth_secret=null, port=null;
exports.configure = function(rootservices_, key_, secret_, port_) {
    location = rootservices_;
    oauth_key = key_;
    oauth_secret = secret_;
    port = port_;
}

It is a very simple function used to provide some rootservices location, application key and secret to your dwaOauth module.

dwaOauth.js module will handle OAuth 1.0 dance and will give application access to protected resources.
All we need it to get authorized OAuth access token. The following steps are required to successfully get the authorized OAuth token:

1. Login to DWA to get JSESSIONID Cookie
2. Request a OAuth token
3. Authorize the OAuth token
4. Get the access token

In '/dwa_login' route we create a OAuth object which will be used to (2) request the OAuth token and once it was correctly retrieved it will redirect user to (3) authorize it. OAuth module will handle those requests and it will also redirect user to perform (1) login to DWA in order to get JSESSIONID Cookie.

In app.js add:

//Request an OAuth Request Token, and redirects the user to authorize it
app.get('/dwa_login', dwaOauth.getOAuthURLs, dwaOauth.requestToken);

In routes/dwaOauth.js add:

exports.requestToken = function(req, res) {
    var oa = new OAuth(oauthRequestTokenUrl,
        oauthAccessTokenUrl,
        oauth_key,
        oauth_secret,
        "1.0",
        req.protocol+"://"+req.host+":"+port+ "/dwa_cb", // this should work on Bluemix and localhost
        "HMAC-SHA1");

    console.log("1. Request a OAuth Token - this will redirect to DWA login page");

    oa.getOAuthRequestToken(function(error, oauth_token, oauth_token_secret, results){
        if(error) {
            console.log('error : ' + util.inspect(error));
            res.end();
        }
        else {
            //store the tokens in the session
            req.session.oa = oa;
            req.session.oauth_token = oauth_token;
            req.session.oauth_token_secret = oauth_token_secret;

            console.log("2. Authorize the OAuth Token ");

            res.redirect(oauthUserAuthorizationUrl +"?oauth_token="+oauth_token);
        }
    });
}

Rootservices and OAuth

As you probably notice '/dwa_login' requires getOAuthURLs before it will be executed. getOAuthURLs method parses DWA rootservices so oauthRequestTokenUrl, oauthAccessTokenUrl and oauthUserAuthorizationUrl are discovered automatically. We do not need to store those in any configuration file.

// get URL from root services
exports.getOAuthURLs = function(req, res, next){
    console.log('Attemp to get OAuth URLs ' + location);
    if (location == null || oauth_key == null || oauth_secret == null) {
        return next('not configured');
    }

    request(location, function(error, response, body){
        if (!error && response.statusCode == 200) {
            var xmlDoc = libxmljs.parseXml(body);
            oauthRequestTokenUrl = xmlDoc.get('jfs:oauthRequestTokenUrl', ns.jfs).attr('resource').value();
            oauthAccessTokenUrl = xmlDoc.get('jfs:oauthAccessTokenUrl', ns.jfs).attr('resource').value();
            oauthUserAuthorizationUrl = xmlDoc.get('jfs:oauthUserAuthorizationUrl', ns.jfs).attr('resource').value();
            next();
        }
        else {
            console.log('error' + util.inspect(error));
            return next(error);
        }
    });
}

ns is defined as:

exports.ns = ns = {
  dcterms:{ 'dcterms': "http://purl.org/dc/terms/" },
  jfs: { 'jfs': 'http://jazz.net/xmlns/prod/jazz/jfs/1.0/' },
  oslc_rm:{ 'oslc_rm': 'http://open-services.net/ns/rm#' },
  oslc:   { 'oslc': 'http://open-services.net/ns/core#' },
  rdf: { 'rdf': 'http://www.w3.org/1999/02/22-rdf-syntax-ns#' },
  doors:  { 'doors': 'http://jazz.net/doors/xmlns/prod/jazz/doors/2.0/' }

So namespaces are defined and easier to use.
Above code uses request to get rootservices and libxmljs to parse it. As mentioned earlier rootservices are the only resource not protected by OAuth so we do not need to worry about setting JSESSIONID cookie or OAuth while accessing it.

Add callback route

The last step in OAuth dance is to get the access token. We will do it in callback we provided earlier while constructing OAuth object. This callback is another route defined in app.js as '/dwa_cb':

// Callback for the authorisation page
app.get('/dwa_cb', dwaOauth.callback);

and in dwaOauth.js:

exports.callback = function(req, res) {
    var oa = new OAuth(req.session.oa._requestUrl,
            req.session.oa._accessUrl,
            req.session.oa._consumerKey,
            req.session.oa._consumerSecret,
            req.session.oa._version,
            req.session.oa._authorize_callback,
            req.session.oa._signatureMethod);
    console.log("3. Get the access token");

    oa.getOAuthAccessToken(
            req.session.oauth_token,
            req.session.oauth_token_secret,
            req.param('oauth_verifier'),
            function(error, oa_access_token, oa_access_token_secret, results) {
            if(error) {
                console.log('error' + error);
            }
            else {
                req.session.oauth_access_token = oa_access_token;
                req.session.oauth_access_token_secret = oa_access_token_secret;
                //redirect to home page
                res.redirect("/");
            }
    });
}

Access protected resource (finally!)

Once we have the access token we can access protected resources. Last line in '/dwa_cb' route will automatically redirect user to a root page which as we saw earlier routes to /resource.

Let's add a resources route to app.js:

app.get('/resource', checkLogin, function(req, res) {
    res.render('resource', {"resource": ""});
});

As you can see it renders 'resource' page with resource parameter. We will use HTTP Get request to get that resource. Add following to dwaOauth.js

exports.getProtectedResource = function(req, res, resourceURL, next, onError) {
    console.log('requesting protected resource: '+resourceURL);
    var oa = new OAuth(req.session.oa._requestUrl,
          req.session.oa._accessUrl,
          req.session.oa._consumerKey,
          req.session.oa._consumerSecret,
          req.session.oa._version,
          req.session.oa._authorize_callback,
          req.session.oa._signatureMethod);

    oa._headers['OSLC-Core-Version'] = '2.0';
    oa._headers['accept'] = "application/rdf+xml";
 

    // Try to get protected data
    oa.getProtectedResource(
        resourceURL,
        "GET",
        req.session.oauth_access_token,
        req.session.oauth_access_token_secret,
        function (error, data, response) {
            if (!error && next) {
                next(req, res, data);
            }
            else if (onError) {
                onError(error);
            }
        }
    );
}

This is a general HTTP request and should work with any OAuth server (DWA, DNG, QM, etc).

Once you have this your dwaOauth.js you can use it in your app.js with following route:

function getResource(req_, res_, next) {
    if (req_.body.res_url != "") {
        dwaOauth.getProtectedResource(req_, res_, req_.body.res_url, function(req, res, data) {
            var xmlDoc = libxmljs.parseXml(data);
            var mod = xmlDoc.get('//dcterms:title', dwaOauth.ns.dcterms).text();
            next(req_, res_, "", mod);
        });    
    }
    else next(req_, res_, "", "");
}

app.post('/resource', checkLogin, function(req, res) {
    getResource(req, res, render);

});

If you provide a valid resource link, below code will attempt to read dcterms:tile of the remote resource.
Update resource.jade to include a form for Resource URL input.

form(action="/resource", method="post")
      h2 Sample 1: Get resource title
      p Please provide a URL for one of your resources (say link to module)
      p Resource URL:
       input(type="text", name="res_url")
       input(type="submit", value="Submit")
       br
       if resource == ""
        p
       else
        p Resource title: '#{resource}'
        br

You just learned know how to read data from OAuth protected server! Easy ;)

You can find listing for dwaOauth.js here.

Update:

In my original post I forgot to mention checkLogin function. Thanks to Dani for pointing that!

// main redirection method
// make sure all your routes to protected resources will go via this method 
function checkLogin(req, res, next) {
    // if session doesn't have a oauth access token redirect to login page.
    if(!req.session.oauth_access_token) {
        res.redirect("/");
        return;
    }
    next();
}

but that's not all what's needed. It's only a redirection to the default route which I need to include here as well.

//root page
app.get('/', function(req, res){    

    // check if session has oAuth token
    if(!req.session.oauth_access_token) {
            res.render("index");
        }
        else {
            res.redirect("/resource");
        }
});

index is a simple form where user provides DWA location, oAuth key and secret:

//- index.jade
body
  table
    tr
      td(style= "width:30%;" )
        img(src="/images/newapp-icon.png")
      td
        h1 Hi there!
 
        p First you will need to setup your OAuth server details:
        
        form(name="loginform", action="/set_key_and_secret", method="post")
          | Rootservices location:
          input(type="text", name="rootservices")
          br
          | Consumer Key:
          input(type="text", name="key")
          br
          | OAuth Secret:
          input(type="password", name="secret")
          br
          input(type="submit", value="Submit")

set_key_and_secret is defined earlier.