Showing posts with label Internet of Things. Show all posts
Showing posts with label Internet of Things. Show all posts

Tuesday, 15 December 2015

DOORS 9 direct IoT support - build-in challenge #1

Monday morning is a good time for reflection and quick chat with coworkers before throwing yourself into work cycle.
In this way I had quick chat session with my manager yesterday. That ended up with challenge#1 (DOORS and IoT) and later on I had another quick chat my friend from DNG team, that ended up with challenge #2 (DOORS 9 itself).

Since challenge #1 is about IoT let's leave details of the other one for another post- but I like the idea so I will definitely post something in that subject.


Challenge #1

It was simply; "DXL triggers and IoT."

Well in my last post I used external tool written in NodeJS to publish DOORS 9 TRS feed to IoTF broker.

You need a full DOORS Web Access stack with some extra settings (as TRS is not enabled by default) to be able to get TRS feed which you will consume with your external application before posting that to IoT Foundation broker.

That's how complex last solution was
DWA Stack used in last post

There are no events in TRS so the last example was querying for changes every 10 seconds. That's far from real-time in IoT terms.


DOORS 9 Triggers

Have a look at DXL Reference Manual if you are not familiar with concept of triggers in DOORS 9.

Triggers are as close to real-time events as it can be in RM tool. Yeah but DOOR 9 doesn't have MQTT client.

Does it have to have it?

Well NO!

Looking at Internet of Things documentation, you can find topic on HTTP(S) Connectivity for Devices. If you read my previous post you should be familiar with HttpRequest. So your trigger DXL have to do is send a HTTP POST request to
<target server: org_id.internetofthings.ibmcloud.com>/api/v0002/device/types/{DeviceType}/devices/{DeviceID}/events/{eventID}

Looking at the URL format you can guess some event values will be set automatically:
{
"device_type": {DeviceType},
"device_id": {DeviceID},
"evt_type": {eventID},
"timestamp":
{
"$date": 1450194956425
}
All you need to add is a evt itself.

So your entire trigger DXL will look like

Module m = current Module
Buffer msg = create
msg = ""
msg += "\"id\": \"" getResourceURL m"\", "
msg += "\"content\": \"" name m "\", "
msg += "\"group\": \"Open\""
msg += "}"

string ioturl = "https://org_id.internetofthings.ibmcloud.com/api/v0002/device/types/DeviceType/devices/DeviceID/events/trigger_update"

HttpHeader h = create
string auth = ""
toBase64_("use-token-auth:YOUR_TOKEN", auth)
auth = auth[0:length(auth) -2]
add(h, "Authorization", "Basic "auth)
add(h, "Content-Type", "application/json")

HttpBody b = create
setValue(b, msg)

HttpResponse resp = HttpRequest(HttpPost, ioturl, b, h)
delete b

if (!null resp && resp.isOk)
{
// no one really needs to see it
  print "got it " (!null resp ? resp.code "" : "null") "\n"
  delete resp
}

delete h
delete msg


That's all you need to do publish events in real-time to IoTF!

Conclusion

Again HTTP Request proved to be a very powerful and useful perm. Using it the initial stack was reduced to very simple form:
Simplified stack
Final stack in DOORS to IoTF communication

DOORS 9 has a potential in it, you just need to know how to use it.

Friday, 4 December 2015

IoT data in JavaScript

Introduction

In my previous post Displaying real-time IoT data in IBM DOORS, I displayed IoT data in the DOORS Web Access hover over. This involved an investigation into how I would access IBM Foundation IoT data from JavaScript. My initial thoughts were this would be very easy, I will just use the Paho Js utility, though it didn't end up as straight forward as anticipated. Therefore the aim of this post is to make the process a little easier and highlight the gotchas I encountered.


Wednesday, 2 December 2015

DOORS 9 IoT Report

Let's think about following user story:
"As a Requirement Engineer I would like to know how many errors my devices report"

In DOORS 9 module one can visualize this as:
Example report in DOORS9

So we would like to get a number of errors of given type reported by each of devices.

Device Error Reports in Sample App

First I need a device which can send some error codes. The IoT Starter Application mentioned in one of previous post publishes three kind of messages

  • touchmove - user interaction with screen
  • accel - orientation of the device and its position
  • text - a text provided by user

I extended that application to be able to send error codes.
 

Pressing "Send Error" button and selecting error code sends a MQTT message to IoTF broker.

Typical error message looks like;
{
device_id"doors9"
evt_type"error"
timestamp:
{
$date1449059192155
}
evt: 
{
errorCode"20"
}

}

For a compiled version of this application follow this link.
Now I could start updating my module.

Design for Analytics


Requirements in my DOORS module have a "ErrorCode" integer attribute which links a requirement to a error code reported by my device.

Additionally I'm using DOORS module level attributes to store values I do not need hardcoded in my DXL. Those are:
  • Authorization (string)- so I do not need to calculate Base64 on each call
  • DeviceType 
    (string) - which of my devices this module describes
  • Organization (string) - my Bluemix organisation name
With all information in place I can write some simple layout DXL (which can be convert to attribute DXL later on to improve performance)

Layout DXL 

I want a specific type of event from all devices of a given type from my organization. So I need to use a historic query for a device type:

buf = "https://internetofthings.ibmcloud.com/api/v0001/historian/" org"/" deviceType "?evt_type=" eventId

This will return a JSON string with a list of events. If you do not have a JSON parser ready you can try to parse this data with Regular Expressions. Please remember this is a very simple example and in real world one shall not attempt to parse JSON with regular expressions.

My main worker code looks like:
if (!null obj) {
  int ival = obj."ErrorCode"
  if (ival == 0) halt
 
  string val = obj."ErrorCode"
  Module m = module obj
  string auth = m."Authorization"
  string dtype = m."DeviceType"
  string org = m."Organization"
 
  if (!null m && auth != "" && org != "" && dtype != "")
  {
    Buffer b = getData(auth, org, dtype, "error")

    if (!null b)  {
      string s = stringOf b
      Regexp re = regexp2 "\"device_id\":\"([^\"]*)\"[^}]+.[^}]+{\"errorCode\":\"([^\"]*)\""
      int i = 0
      string device = "", code =""
      //  temporary skip to hold names of devices which reported
      Skip erSkp = createString
      int allErrors = 0
      int numDevices = 0
      while (!null s && re s && i<100) {  // i is just a guard, we know there is no more then a 100 results in one page
        device = s[match 1]
        code = s[match 2]
        int ireported = 0
        // if code matches attribute value
        if (code == val)
        {
          allErrors++ // icrease number of errors
         
          if (!find(erSkp, device, ireported)) {
            put(erSkp, device, 1)
            numDevices++
          }
          else {
            ireported++
            put(erSkp, device, ireported, true)
          }
        }
        s = s[end 0 +1:]
        i++
      } // while
      // clean up
      delete b
     
      // report
      if (allErrors != 0) {
        for ireported in erSkp do {
          device = (string key erSkp)
          displayRich "Device with Id {\\b "device "} reported an issue " (ireported == 1 ? "once" : ireported" times")
        }
      }
     
      delete erSkp
    } // null b
  } // module setup
} // !null obj

You can find full DXL here.

Conclusion


Above layout DXL works fine when there're not so many devices. Once there will be more of them we no longer want to see how many times each device reported given issue. Thus the DXL could be rewritten to show something like:
Example report


As you saw in this example I'm using layout DXL but I think for a better understanding and feedback, one should consider writing a utility DXL.
Maybe that utility could provide its own UI for easier navigation?

In example above there's no need to send a HttpRequest for each object... It is enough to make one call per 100 (max page) events returned by query and write a little more complex Skip management. That however would require to make one top level Skip, but I'm sure you all know how to do it.

Monday, 30 November 2015

Displaying real-time IoT data in IBM DOORS (Doors Web Access)

There are many ways in which we could imagine surfacing IoT data in the IBM DOORS interface.

One way that I would propose is to extend the rich hover so that you could see IoT data in real-time, displayed when you hover over an IBM DOORS link-end point. Imagine being an IBM RQM user looking at a defect and being able to hover over a link to a requirement and see the history of a thing displayed.

Extending the rich hover to display IoT data in IBM DOORS Web Access


Extending the rich hover to display IBM DOORS IoT data in IBM RQM


Showing display of real-time data in hover over



As part of the technical investigation I looked into the implementation of a JavaScript client which listens for data from a device in the IoT Foundation. And then display that real-time data in the UI. The functioning graphing component is shown in the examples above. I will blog about the JavaScript / IoT implementation in a future post. 



Sunday, 29 November 2015

Moving DOORS9 towards real-time IoT sensors

Last time I shown how easy it is to use historian from your IoT enabled device data in DOORS using IoT Foundation HTTP API. I thought 'it cannot be that hard to show real-time data', I know there are no MQTT callbacks in DXL, no JSON parser (sure we can write one, but what's the point?).

Sounds challenging. So let's do it!

Let's play!

If you have spare 15 minutes please have a look at wonderful presentation by Dr John Cohn (@johncohnvt) - "The importance of play".

Let's create a very simple HTML static page you can open in any (recent!) browser and which will connect to IoT Foundation.

Very (and I mean it) simple HTML page

The page will have just two p elements, one for static info, and one used dynamically from JavaScript. It will additionally include 3 scripts; jquery, paho and MQTT connection helper class.
The entire page code is simply:
<!DOCTYPE html>
<html>
    <head>
        <meta http-equiv="Content-Type" content="text/html; charset=utf-8" />
        <script type="text/javascript" src="jquery.min.js"></script>
        <title>MQTT listener</title>
    </head>

    <body onContextMenu="return false">
        <p>Please do not close</p>
        <p id="pStatus"></p>
    </body>
        <script type="text/javascript" src="realtime.js"></script>
        <script type="text/javascript" src="mqttws31.js"></script>
        <script type="text/javascript">
          var realtime = new Realtime(org, apikey, apitoken, device_type, device_id);
        </script>
</html>

MQTT connection

As you know IoT devices communicate over MQTT, lightweight device to device protocol implemented in many languages. Among which there are many JavaScript implementations available, which you can find at http://mqtt.org/tag/javascript. Personally I'm using Paho because it is very easy to use with IBM Bluemix services.

Using Paho is straight forward, you need to create new Messaging.Client object providing host, port and clientId.
For newly created client you should provide two message callbacks:
  • onMessageArrived - function(msg)
  • onConnectionLost - function(e)
client = new Messaging.Client(hostname, 8883,clientId);
client.onMessageArrived = function(msg) {
  var topic = msg.destinationName;
  var payload = JSON.parse(msg.payloadString);
  data = payload.d;
};

client.onConnectionLost = function(e){
  console.log("Connection Lost at " + Date.now() + " : " + e.errorCode + " : " + e.errorMessage);
  this.connect(connectOptions);
}

Then you construct connection options object which will be passed to connect function of Messaging.Client. This will start connection with IoT MQTT Broker.
You have to specify:
  • timeout - number
  • useSSL - boolean
  • userName - string - as usual this will be API Key for your organization
  • password - string - and API Token  
  • onSuccess - function() - executed when connection was successful
  • onFailure - function(e) - execeuted on failure
var connectOptions = new Object();
connectOptions.keepAliveInterval = 3600;
connectOptions.useSSL=true;
connectOptions.userName=api_key;
connectOptions.password=auth_token;

connectOptions.onSuccess = function() {
  console.log("MQTT connected to host: "+client.host+" port : "+client.port+" at " + Date.now());
  $('#pStatus').text("MQTT connected to host: "+client.host+" port : "+client.port);
  self.subscribeToDevice();
}

connectOptions.onFailure = function(e) {
  console.log("MQTT connection failed at " + Date.now() + "\nerror: " + e.errorCode + " : " + e.errorMessage);
  $('#pStatus').text("MQTT  connection failed at " + Date.now() + "\nerror: " + e.errorCode + " : " + e.errorMessage);
}

console.log("about to connect to " + client.host);
$("#pStatus").text("about to connect to "+client.host);

client.connect(connectOptions);

Once you establish connection with IoT MQTT Broker you should subscribe to some topic.

var self = this;
// Subscribe to the device when the device ID is selected.
this.subscribeToDevice = function(){
  var subscribeOptions = {
    qos : 0,
    onSuccess : function() {
      console.log("subscribed to " + subscribeTopic);
    },
    onFailure : function(){
      console.log("Failed to subscribe to " + subscribeTopic);
      console.log("As messages are not available, visualization is not possible");
    }
  };
  
  if(subscribeTopic != "") {
    console.log("Unsubscribing to " + subscribeTopic);
    client.unsubscribe(subscribeTopic);
  }
  subscribeTopic = "iot-2/type/" + deviceType + "/id/" + deviceId + "/evt/+/fmt/json";
  client.subscribe(subscribeTopic,subscribeOptions);
} 

DOORS9 HTML window

Now it's time to display this very simple page in DOORS 9:
bool onB4Navigate(DBE dbe,string URL,frame,body){return true}
void onComplete(DBE dbe, string URL){}
bool onError(DBE dbe, string URL, string frame, int error){return true};
void onProgress(DBE dbe, int percentage){return true}

Module refreshModuleReff = null
void moduleRefreshCB(DBE x) {
  if (!null refreshModuleReff) {
    refresh refreshModuleReff
  }
}

string surl="c:\\iot\\realtime.html"
DB iotUIdb = create("do not close")
DBE iotUI=htmlView(iotUIdb, 100, 50, surl, onB4Navigate, onComplete, onError, onProgress)
DBE t = timer(iotUIdb, 1, moduleRefreshCB, "ping")
startTimer(t) 
realize iotUIdb 

You probably noticed I'm using timer functionality (page 48 in DXL Reference Manual), that's because layout DXL is run when the module is being refreshed, redrawn, etc. This simple code ensures it is refreshed every second.
There is a lot of space for improvement in above DXL; you can add a toggle to check if user wants refresh, you can add noError/lastError block before calling refresh refreshModuleReff. 

Now save above code as %doors_home%/lib/dxl/startup/iot.dxl and restart DOORS. This will create a top level window which can be accessed from any DXL context in your DOORS 9 session.

Layout DXL

Finally we can add some worker code. This will be using hidden perm which has following syntax:
string hidden(DBE, string)

string property = obj."accel_property"
if (property != "none") {
  noError()
  refreshModuleReff = current Module
  string strval = hidden(iotUI, "realtime." property "()")
  string err = lastError
  if (!null err) { halt }
  real val = realOf strval

  //enable blow if your Module is similar to the one from last post
  //obj."iot_data" = val
  display strval
} 

I put whole code from "MQTT Connection" chapter into realtime.js and created additional helper methods to get required
 
this.yaw = function() {
  if (data != null) return data.yaw;
  return -101;
}
this.pitch = function() {
  if (data != null) return data.pitch;
  return -101;
}
this.roll = function() {
  if (data != null) return data.roll;
  return -101;
}
var data = null;


Now the module is displaying real-time data from a device connected to IoT Foundation. It has 1 second delay, maybe you want to synchronize data object when it's retrieved from realtime.js.

Well afterwards I didn't need to write a JSON parser in DXL!

Conclusions

DOORS reading IoT real-time data is easy and possible! It was even simpler then I thought initially ;)

Remember in this blog I'm just giving examples what can be done, it's up to you to extend it with unsubscribe, change device type and device depending on current module. There are many thing you might want to add.

Thursday, 19 November 2015

DOORS IoT data example

In my last post I demonstrated how easy it is to get data from IoT Foundation. But many of you might thing, why? Well I just wanted to show how easy it is to report on IoT data using pure old DOORS 9 but there was no fun in the preview post, just boring numbers. 

Example Scenario

Let's have a hypothetical example where we measure yaw/pitch/roll and have following rules:
  1. Measurable parameters of an Android Device shall not exceed expected maximum values
  2. Each report of a value exceeding 0.77 of maximum should be indicated Orange
  3. Each report of a value exceeding 0.88 of maximum should be indicated Red

So we need something to read data from IoT device, storage for a maximum value and some nice indicators.

Module Attributes

I just extended my module with extra attributes:
  • iot_data (real) - doesn't affect change bar, doesn't affect change dates and doesn't generate history
  • range_max (real) - a regular attribute of type range

Having add those I can update my layout DXL column to render color depending on current value:

if (findPlainText(re, ":", l, o, false)) {
  string ss = re[l+1:length(re) -3]
  real r = realOf ss
  obj."iot_data" = r
  real m = obj."range_max"
  real rm = r/m
  DBE cnv = getCanvas
  realBackground(cnv, realColor_Green)
  if (rm >= 0.77 && rm < 0.88) {
    realBackground(cnv, realColor_Orange)
    realColor(cnv, realColor_White)
  }
  else if (rm >= 0.88) {
    realBackground(cnv, realColor_Red)
    realColor(cnv, realColor_White)
  }
  else {
    realColor(cnv, realColor_Black)
  }
  display ss
}

Code isn't perfect, could be faster, but that's not the point here ;)

Now let's have some fun with canvas DXL. Add new DXL column and set its DXL to:

DBE canvas = getCanvas
if (canvas==null) halt

int rh = 50 du
int rw = 100 du

setHeight rh
setWidth rw

int normalize(real  x, max) {
  real  i = 180.0+ x/max * 180.0
  return intOf i
}

real x = (obj."iot_data")
real m = (obj."range_max")

if (m == 0.0) halt

int v = normalize(x, m)
int margin = 1

realColor(canvas, realColor_Green)

if (v >= 280 && v < 320) {
  realColor(canvas, realColor_Orange)
  margin = 2
}
else if (v >= 320) {
  realColor(canvas, realColor_Red)
  margin = 5
}

ellipse(canvas, 0, 0, rw, 2*rh)
realColor(canvas, realColor_White)
ellipse(canvas, margin, margin, rw-2*margin, (2*rh)-2*margin)

realColor(canvas, realColor_Black)
polarLine(canvas, rw/2, rh-1, rh, v)
Resulting View will show us (almost) real-time events and warning from our device.

DOORS 9 View with almost real-time IoT data warnings

Remember this could be a group of devices or a single device. It all depends what you want to measure.

Conclusions

IoT Foundation and DOORS 9 are really cool and really powerful! Hope to show you some more fun stuff soon!

Tuesday, 17 November 2015

DOORS in IoT World

Last time I shown a quick example how to use HTTP request perms (DOORS functions) to get data from Cloudant database.
Today let's go a little step further and try to get more recent data.

Before you will continue with DOORS connection to IoT Foundation API please have a look at IoT Foundation Starter application. There you will learn how to create a simple IoT device, deploy a IoTStarterApp to your mobile, and start talking IoT! That's something I'm not covering in this blog (yet).

IoT connectivity

IoT devices communicate over Machine 2 Machine connectivity protocol - MQTT Message Queue Telemetry Transport. This protocol is extremely lightweight protocol much lighter than HTTP. MQTT becomes more and more popular, not only in IoT world but also in mobile data exchange, enven Facebook Messenger uses MQTT.

DOORS and IoT connectivity

MQTT protocol requires message broker and message client. From DOORS client point of view IBM IoT Foundation can be a message broker, but what with MQTT client? Well at the moment I do not see easy/best solution, one can try writing a MQTT client in DXL, or using some external library OLE interface. However MQTT messages are great for a real time data which can feed IoT Real-Time Insights or IoT Foundation.

OOTB DOORS can consume a historic data. In order to get a historic data from your device you can use IBM IoT Foundation HTTP API and you can do it using HTTP perms available in DOORS.

You can get events for:
  • every device from your organization - so all device types, all devices.
  • all devices of selected device types in your organization 
  • a single device
Each of possible historic data queries has a number of filters available.

Authorization 

IoT Foundation HTTP API uses a Basic Authentication in combination with HTTPS. You need to create  API key for your IoT service.

IoT API Key generation

When authenticating API Key is used as user name and its associated Auth Token as the password. Those should be provided in HTTP Authorization header in all requests.

Following DXL will add set authorization header:
  HttpHeader h = create
  string auth
  toBase64_(apiKey":"apiToken, auth)
  auth = auth[0:length(auth) -2]
  add(h, "Authorization", "Basic "auth)

toBase64_ is a perm which will convert a string into its Base64 representation. It it adds a newline character to the resulting string thus is it removed before use.

Historic query

Since I'm going to use my script in Layout DXL I'm not going to query for devices or their types. Let's assume I know already all these information (or it can be stored in module level attributes) so I will focus on a single query on specific device. From documentation I know I need to send a GET request (with empty body) to a URL:

https://internetofthings.ibmcloud.com/api/v0001/historian/${org}/${type}/${id}?filter

where:
${org} is organization
${type} - device type
${id} is a device ID

filter will let me narrow down response:
top - selects a number of top most events - selecting top=1 is almost equivalent to getting data in real time ;)
start/end - interval of historical data to query at
evt_type - narrows down selected events
sumarize and sumarize_typeString - allows performing aggregate functions of selected events.

My layout DXL will select an average from last 100 'accel' events over a selected attribute:
Buffer buf = create
buf = "https://internetofthings.ibmcloud.com/api/v0001/historian/" org"/" deviceType "/" deviceId "?top=100&evt_type=accel&summarize={"attr"}"

'accel' is one of the events sent by IoTStartApp if you use different IoT device, then please change that accordingly.

Proposed usage scenario

The scenario here is that you have requirement(s) for an Android device that is being measured in the real world. Those measurements may mean that error messages are passed back and ultimately you would want to report against these from your original requirements. The attribute "accel_property" in the following code is the key that identifies the thing and measurement that you are interested in (and is relevant to the requirement).

DXL IoT column in DOORS module


Whole DXL is:
Buffer getData(string apiKey, apiToken, org, deviceType, deviceId, eventId, attr)
{
  Buffer buf = create
  buf = "https://internetofthings.ibmcloud.com/api/v0001/historian/" org"/" deviceType "/" deviceId "?top=1&evt_type=" eventId "&summarize={"attr"}"
  HttpHeader h = create
  string auth
  toBase64_(apiKey":"apiToken, auth)
  auth = auth[0:length(auth) -2]
  add(h, "Authorization", "Basic "auth)
  HttpResponse resp = httpRequest(HttpGet, tempStringOf buf, null, h)
  delete h
  delete buf

  if (!null resp && resp.isOk)
  {
    HttpBody b = resp.body
    Buffer respBuf = create
    respBuf += b.value

    delete resp
    return respBuf
  }
  else {
    if (!null resp) {
      display "error getting response " resp.code""
      delete resp
    }
    else {
      display "connection error"
    }
  }

  return null
}

if (!null obj) {
  string val = obj."accel_property"
  if (val != "none") {
    Buffer b = getData("API KEY", "Auth Token", "org", "type", "id", "accel", val)

    if (!null b)  {
      string re = stringOf b
      int l,o
      if (findPlainText(re, ":", l, o, false)) {
        display val " last value " re[l+1:length(re) -3]
      }
      delete b
    }
  }
}

As you can see I'm using any fancy JSON parser, my query returns a single variable so it can be easily extracted from a string. 
I used an extra enum attribute I created for my module so each of my requirements can select different variable from query.

Conclusion

DOORS can be really easily connected to IoT Foundation and access its data. With help of htmlView DBE one can add an extra DXL window displaying IoT data in Real-Time.
There is a lot DOORS can do for you!

Bonus lines of DXL for Real-Time data

"Turn your mobile phone into an IoT device" is an interesting extension to the article on m2m I showed you in the very beginning of this post. In compare to preview post on IoT starter application it has "Step 6. Create a Bluemix app to visualize sensor data". Where you can see how to deploy IoT sample NodeJS application which will visualize real-time or historic data from your organization. Once you deploy this application to your Bluemix account run following DXL for a simple HTML viewer:

void iotDataView(string srcUrl)
{
    DB dlg = create("IoT real-time data", styleCentered)
    DBE html = htmlView(dlg, 1000, 800, srcUrl, onHTMLBeforeNavigate, onHTMLDocComplete, onHTMLError, onHTMLProgress)
    show dlg
}
iotDataView "http://(your_application_name).mybluemix.net"

This will create windows similar to the following: