Showing posts with label layoutdxl. Show all posts
Showing posts with label layoutdxl. Show all posts

Wednesday, 2 December 2015

DOORS 9 IoT Report

Let's think about following user story:
"As a Requirement Engineer I would like to know how many errors my devices report"

In DOORS 9 module one can visualize this as:
Example report in DOORS9

So we would like to get a number of errors of given type reported by each of devices.

Device Error Reports in Sample App

First I need a device which can send some error codes. The IoT Starter Application mentioned in one of previous post publishes three kind of messages

  • touchmove - user interaction with screen
  • accel - orientation of the device and its position
  • text - a text provided by user

I extended that application to be able to send error codes.
 

Pressing "Send Error" button and selecting error code sends a MQTT message to IoTF broker.

Typical error message looks like;
{
device_id"doors9"
evt_type"error"
timestamp:
{
$date1449059192155
}
evt: 
{
errorCode"20"
}

}

For a compiled version of this application follow this link.
Now I could start updating my module.

Design for Analytics


Requirements in my DOORS module have a "ErrorCode" integer attribute which links a requirement to a error code reported by my device.

Additionally I'm using DOORS module level attributes to store values I do not need hardcoded in my DXL. Those are:
  • Authorization (string)- so I do not need to calculate Base64 on each call
  • DeviceType 
    (string) - which of my devices this module describes
  • Organization (string) - my Bluemix organisation name
With all information in place I can write some simple layout DXL (which can be convert to attribute DXL later on to improve performance)

Layout DXL 

I want a specific type of event from all devices of a given type from my organization. So I need to use a historic query for a device type:

buf = "https://internetofthings.ibmcloud.com/api/v0001/historian/" org"/" deviceType "?evt_type=" eventId

This will return a JSON string with a list of events. If you do not have a JSON parser ready you can try to parse this data with Regular Expressions. Please remember this is a very simple example and in real world one shall not attempt to parse JSON with regular expressions.

My main worker code looks like:
if (!null obj) {
  int ival = obj."ErrorCode"
  if (ival == 0) halt
 
  string val = obj."ErrorCode"
  Module m = module obj
  string auth = m."Authorization"
  string dtype = m."DeviceType"
  string org = m."Organization"
 
  if (!null m && auth != "" && org != "" && dtype != "")
  {
    Buffer b = getData(auth, org, dtype, "error")

    if (!null b)  {
      string s = stringOf b
      Regexp re = regexp2 "\"device_id\":\"([^\"]*)\"[^}]+.[^}]+{\"errorCode\":\"([^\"]*)\""
      int i = 0
      string device = "", code =""
      //  temporary skip to hold names of devices which reported
      Skip erSkp = createString
      int allErrors = 0
      int numDevices = 0
      while (!null s && re s && i<100) {  // i is just a guard, we know there is no more then a 100 results in one page
        device = s[match 1]
        code = s[match 2]
        int ireported = 0
        // if code matches attribute value
        if (code == val)
        {
          allErrors++ // icrease number of errors
         
          if (!find(erSkp, device, ireported)) {
            put(erSkp, device, 1)
            numDevices++
          }
          else {
            ireported++
            put(erSkp, device, ireported, true)
          }
        }
        s = s[end 0 +1:]
        i++
      } // while
      // clean up
      delete b
     
      // report
      if (allErrors != 0) {
        for ireported in erSkp do {
          device = (string key erSkp)
          displayRich "Device with Id {\\b "device "} reported an issue " (ireported == 1 ? "once" : ireported" times")
        }
      }
     
      delete erSkp
    } // null b
  } // module setup
} // !null obj

You can find full DXL here.

Conclusion


Above layout DXL works fine when there're not so many devices. Once there will be more of them we no longer want to see how many times each device reported given issue. Thus the DXL could be rewritten to show something like:
Example report


As you saw in this example I'm using layout DXL but I think for a better understanding and feedback, one should consider writing a utility DXL.
Maybe that utility could provide its own UI for easier navigation?

In example above there's no need to send a HttpRequest for each object... It is enough to make one call per 100 (max page) events returned by query and write a little more complex Skip management. That however would require to make one top level Skip, but I'm sure you all know how to do it.

Sunday, 29 November 2015

Moving DOORS9 towards real-time IoT sensors

Last time I shown how easy it is to use historian from your IoT enabled device data in DOORS using IoT Foundation HTTP API. I thought 'it cannot be that hard to show real-time data', I know there are no MQTT callbacks in DXL, no JSON parser (sure we can write one, but what's the point?).

Sounds challenging. So let's do it!

Let's play!

If you have spare 15 minutes please have a look at wonderful presentation by Dr John Cohn (@johncohnvt) - "The importance of play".

Let's create a very simple HTML static page you can open in any (recent!) browser and which will connect to IoT Foundation.

Very (and I mean it) simple HTML page

The page will have just two p elements, one for static info, and one used dynamically from JavaScript. It will additionally include 3 scripts; jquery, paho and MQTT connection helper class.
The entire page code is simply:
<!DOCTYPE html>
<html>
    <head>
        <meta http-equiv="Content-Type" content="text/html; charset=utf-8" />
        <script type="text/javascript" src="jquery.min.js"></script>
        <title>MQTT listener</title>
    </head>

    <body onContextMenu="return false">
        <p>Please do not close</p>
        <p id="pStatus"></p>
    </body>
        <script type="text/javascript" src="realtime.js"></script>
        <script type="text/javascript" src="mqttws31.js"></script>
        <script type="text/javascript">
          var realtime = new Realtime(org, apikey, apitoken, device_type, device_id);
        </script>
</html>

MQTT connection

As you know IoT devices communicate over MQTT, lightweight device to device protocol implemented in many languages. Among which there are many JavaScript implementations available, which you can find at http://mqtt.org/tag/javascript. Personally I'm using Paho because it is very easy to use with IBM Bluemix services.

Using Paho is straight forward, you need to create new Messaging.Client object providing host, port and clientId.
For newly created client you should provide two message callbacks:
  • onMessageArrived - function(msg)
  • onConnectionLost - function(e)
client = new Messaging.Client(hostname, 8883,clientId);
client.onMessageArrived = function(msg) {
  var topic = msg.destinationName;
  var payload = JSON.parse(msg.payloadString);
  data = payload.d;
};

client.onConnectionLost = function(e){
  console.log("Connection Lost at " + Date.now() + " : " + e.errorCode + " : " + e.errorMessage);
  this.connect(connectOptions);
}

Then you construct connection options object which will be passed to connect function of Messaging.Client. This will start connection with IoT MQTT Broker.
You have to specify:
  • timeout - number
  • useSSL - boolean
  • userName - string - as usual this will be API Key for your organization
  • password - string - and API Token  
  • onSuccess - function() - executed when connection was successful
  • onFailure - function(e) - execeuted on failure
var connectOptions = new Object();
connectOptions.keepAliveInterval = 3600;
connectOptions.useSSL=true;
connectOptions.userName=api_key;
connectOptions.password=auth_token;

connectOptions.onSuccess = function() {
  console.log("MQTT connected to host: "+client.host+" port : "+client.port+" at " + Date.now());
  $('#pStatus').text("MQTT connected to host: "+client.host+" port : "+client.port);
  self.subscribeToDevice();
}

connectOptions.onFailure = function(e) {
  console.log("MQTT connection failed at " + Date.now() + "\nerror: " + e.errorCode + " : " + e.errorMessage);
  $('#pStatus').text("MQTT  connection failed at " + Date.now() + "\nerror: " + e.errorCode + " : " + e.errorMessage);
}

console.log("about to connect to " + client.host);
$("#pStatus").text("about to connect to "+client.host);

client.connect(connectOptions);

Once you establish connection with IoT MQTT Broker you should subscribe to some topic.

var self = this;
// Subscribe to the device when the device ID is selected.
this.subscribeToDevice = function(){
  var subscribeOptions = {
    qos : 0,
    onSuccess : function() {
      console.log("subscribed to " + subscribeTopic);
    },
    onFailure : function(){
      console.log("Failed to subscribe to " + subscribeTopic);
      console.log("As messages are not available, visualization is not possible");
    }
  };
  
  if(subscribeTopic != "") {
    console.log("Unsubscribing to " + subscribeTopic);
    client.unsubscribe(subscribeTopic);
  }
  subscribeTopic = "iot-2/type/" + deviceType + "/id/" + deviceId + "/evt/+/fmt/json";
  client.subscribe(subscribeTopic,subscribeOptions);
} 

DOORS9 HTML window

Now it's time to display this very simple page in DOORS 9:
bool onB4Navigate(DBE dbe,string URL,frame,body){return true}
void onComplete(DBE dbe, string URL){}
bool onError(DBE dbe, string URL, string frame, int error){return true};
void onProgress(DBE dbe, int percentage){return true}

Module refreshModuleReff = null
void moduleRefreshCB(DBE x) {
  if (!null refreshModuleReff) {
    refresh refreshModuleReff
  }
}

string surl="c:\\iot\\realtime.html"
DB iotUIdb = create("do not close")
DBE iotUI=htmlView(iotUIdb, 100, 50, surl, onB4Navigate, onComplete, onError, onProgress)
DBE t = timer(iotUIdb, 1, moduleRefreshCB, "ping")
startTimer(t) 
realize iotUIdb 

You probably noticed I'm using timer functionality (page 48 in DXL Reference Manual), that's because layout DXL is run when the module is being refreshed, redrawn, etc. This simple code ensures it is refreshed every second.
There is a lot of space for improvement in above DXL; you can add a toggle to check if user wants refresh, you can add noError/lastError block before calling refresh refreshModuleReff. 

Now save above code as %doors_home%/lib/dxl/startup/iot.dxl and restart DOORS. This will create a top level window which can be accessed from any DXL context in your DOORS 9 session.

Layout DXL

Finally we can add some worker code. This will be using hidden perm which has following syntax:
string hidden(DBE, string)

string property = obj."accel_property"
if (property != "none") {
  noError()
  refreshModuleReff = current Module
  string strval = hidden(iotUI, "realtime." property "()")
  string err = lastError
  if (!null err) { halt }
  real val = realOf strval

  //enable blow if your Module is similar to the one from last post
  //obj."iot_data" = val
  display strval
} 

I put whole code from "MQTT Connection" chapter into realtime.js and created additional helper methods to get required
 
this.yaw = function() {
  if (data != null) return data.yaw;
  return -101;
}
this.pitch = function() {
  if (data != null) return data.pitch;
  return -101;
}
this.roll = function() {
  if (data != null) return data.roll;
  return -101;
}
var data = null;


Now the module is displaying real-time data from a device connected to IoT Foundation. It has 1 second delay, maybe you want to synchronize data object when it's retrieved from realtime.js.

Well afterwards I didn't need to write a JSON parser in DXL!

Conclusions

DOORS reading IoT real-time data is easy and possible! It was even simpler then I thought initially ;)

Remember in this blog I'm just giving examples what can be done, it's up to you to extend it with unsubscribe, change device type and device depending on current module. There are many thing you might want to add.

Tuesday, 17 November 2015

DOORS in IoT World

Last time I shown a quick example how to use HTTP request perms (DOORS functions) to get data from Cloudant database.
Today let's go a little step further and try to get more recent data.

Before you will continue with DOORS connection to IoT Foundation API please have a look at IoT Foundation Starter application. There you will learn how to create a simple IoT device, deploy a IoTStarterApp to your mobile, and start talking IoT! That's something I'm not covering in this blog (yet).

IoT connectivity

IoT devices communicate over Machine 2 Machine connectivity protocol - MQTT Message Queue Telemetry Transport. This protocol is extremely lightweight protocol much lighter than HTTP. MQTT becomes more and more popular, not only in IoT world but also in mobile data exchange, enven Facebook Messenger uses MQTT.

DOORS and IoT connectivity

MQTT protocol requires message broker and message client. From DOORS client point of view IBM IoT Foundation can be a message broker, but what with MQTT client? Well at the moment I do not see easy/best solution, one can try writing a MQTT client in DXL, or using some external library OLE interface. However MQTT messages are great for a real time data which can feed IoT Real-Time Insights or IoT Foundation.

OOTB DOORS can consume a historic data. In order to get a historic data from your device you can use IBM IoT Foundation HTTP API and you can do it using HTTP perms available in DOORS.

You can get events for:
  • every device from your organization - so all device types, all devices.
  • all devices of selected device types in your organization 
  • a single device
Each of possible historic data queries has a number of filters available.

Authorization 

IoT Foundation HTTP API uses a Basic Authentication in combination with HTTPS. You need to create  API key for your IoT service.

IoT API Key generation

When authenticating API Key is used as user name and its associated Auth Token as the password. Those should be provided in HTTP Authorization header in all requests.

Following DXL will add set authorization header:
  HttpHeader h = create
  string auth
  toBase64_(apiKey":"apiToken, auth)
  auth = auth[0:length(auth) -2]
  add(h, "Authorization", "Basic "auth)

toBase64_ is a perm which will convert a string into its Base64 representation. It it adds a newline character to the resulting string thus is it removed before use.

Historic query

Since I'm going to use my script in Layout DXL I'm not going to query for devices or their types. Let's assume I know already all these information (or it can be stored in module level attributes) so I will focus on a single query on specific device. From documentation I know I need to send a GET request (with empty body) to a URL:

https://internetofthings.ibmcloud.com/api/v0001/historian/${org}/${type}/${id}?filter

where:
${org} is organization
${type} - device type
${id} is a device ID

filter will let me narrow down response:
top - selects a number of top most events - selecting top=1 is almost equivalent to getting data in real time ;)
start/end - interval of historical data to query at
evt_type - narrows down selected events
sumarize and sumarize_typeString - allows performing aggregate functions of selected events.

My layout DXL will select an average from last 100 'accel' events over a selected attribute:
Buffer buf = create
buf = "https://internetofthings.ibmcloud.com/api/v0001/historian/" org"/" deviceType "/" deviceId "?top=100&evt_type=accel&summarize={"attr"}"

'accel' is one of the events sent by IoTStartApp if you use different IoT device, then please change that accordingly.

Proposed usage scenario

The scenario here is that you have requirement(s) for an Android device that is being measured in the real world. Those measurements may mean that error messages are passed back and ultimately you would want to report against these from your original requirements. The attribute "accel_property" in the following code is the key that identifies the thing and measurement that you are interested in (and is relevant to the requirement).

DXL IoT column in DOORS module


Whole DXL is:
Buffer getData(string apiKey, apiToken, org, deviceType, deviceId, eventId, attr)
{
  Buffer buf = create
  buf = "https://internetofthings.ibmcloud.com/api/v0001/historian/" org"/" deviceType "/" deviceId "?top=1&evt_type=" eventId "&summarize={"attr"}"
  HttpHeader h = create
  string auth
  toBase64_(apiKey":"apiToken, auth)
  auth = auth[0:length(auth) -2]
  add(h, "Authorization", "Basic "auth)
  HttpResponse resp = httpRequest(HttpGet, tempStringOf buf, null, h)
  delete h
  delete buf

  if (!null resp && resp.isOk)
  {
    HttpBody b = resp.body
    Buffer respBuf = create
    respBuf += b.value

    delete resp
    return respBuf
  }
  else {
    if (!null resp) {
      display "error getting response " resp.code""
      delete resp
    }
    else {
      display "connection error"
    }
  }

  return null
}

if (!null obj) {
  string val = obj."accel_property"
  if (val != "none") {
    Buffer b = getData("API KEY", "Auth Token", "org", "type", "id", "accel", val)

    if (!null b)  {
      string re = stringOf b
      int l,o
      if (findPlainText(re, ":", l, o, false)) {
        display val " last value " re[l+1:length(re) -3]
      }
      delete b
    }
  }
}

As you can see I'm using any fancy JSON parser, my query returns a single variable so it can be easily extracted from a string. 
I used an extra enum attribute I created for my module so each of my requirements can select different variable from query.

Conclusion

DOORS can be really easily connected to IoT Foundation and access its data. With help of htmlView DBE one can add an extra DXL window displaying IoT data in Real-Time.
There is a lot DOORS can do for you!

Bonus lines of DXL for Real-Time data

"Turn your mobile phone into an IoT device" is an interesting extension to the article on m2m I showed you in the very beginning of this post. In compare to preview post on IoT starter application it has "Step 6. Create a Bluemix app to visualize sensor data". Where you can see how to deploy IoT sample NodeJS application which will visualize real-time or historic data from your organization. Once you deploy this application to your Bluemix account run following DXL for a simple HTML viewer:

void iotDataView(string srcUrl)
{
    DB dlg = create("IoT real-time data", styleCentered)
    DBE html = htmlView(dlg, 1000, 800, srcUrl, onHTMLBeforeNavigate, onHTMLDocComplete, onHTMLError, onHTMLProgress)
    show dlg
}
iotDataView "http://(your_application_name).mybluemix.net"

This will create windows similar to the following: