HMI Cooperation with TH Ingolstadt

Since June 2014 we are working together with Prof. Grossmann from TH Ingolstadt and his student, to develop an innovative approach for an HMI. For this approach the students use typical elements from Computer games, which can then adapted in an HMI, that is used for real automation processes. At the end of the semester, the students presented their work to the MI5-Team in a big presentation. It followed, a discussion and feedback.

loading map - please wait...

ITQ GmbH: 48.252077, 11.634312
TH Ingolstadt: 48.767116, 11.431704

Below you can find an example of the report of the investigation of the different game elements. Visible is an so-called inventory view. Something similar could be used to show with what kind of actors or sensors a module is equipped. It could show a picture of a module, status information or more. Typically this is shown at some corner of the screen for an overview, or centralized for more information.

TH Ingolstadt

Inventory Elements from some games. (TH Ingolstadt)

This research was then adapted to some more practical approaches by other students. They made a list as it can be used in production to see a queue or control the process. In the picture below, you can also see a message box, that keeps the user up to date about ongoing events.

List of Events in Production with message box. Source: TH Ingolstadt

List of Events in Production with message box. (TH Ingolstadt)

With this ideas in mind, and some very good approaches, we transformed it into a more practical approach with a tool called Balsamiq. It allows us to use typical desktop elements like lists, tabs, dropdown menus in a sketchy way, so that the idea can be transported easily. Colors and fonts are not yet finalized.

Sketch for HMI. (MI5)

Sketch for HMI. (MI5)

Besides, we looked at possible use cases for the HMI. Therefore we used two different perspectives:

  1. User by skill level
  2. User by perspective

The first ist divided into the groups first-time users, that have no knowledge of the HMI, intermediate users, and expert users. With our approach we will be able to give all of them a good user experience. Next to the skill level we looked at the task of the user, e.g. maintenance operator vs production operator.

It is now July, and since our HMI shall also be portable to any modern device, responsive by design and with an intuitive touch experience, we decided to do it with web-technologies. One prototype can be seen below. It is hard-coded and runs on an Apache server.

HMI Protoype with HTML and JavaScript

HMI Protoype with HTML and JavaScript (MI5)

While developing this proof of concept, and during the following month, September 2014, we investigated technologies for further development.

Node.JS is a JavaScript Interpreter, similar to PHP5 module in Apache server, with the Google V8 Engine, that powers also the popular Google Chrome Browser. With this approach only one programming language is required. There is a big movement to use JavaScript not only for Browser-Coding but also for server-side applications. One of the main features in comparison to a classic webserver is the inherent non-blocking approach, that is perfect for an HMI and a constant connection.

Further information: creates a constant connection to a browser, or to any other REST-compatible device. It is then possible to send updates of text field directly from the server to the client. It is even possible to broadcast events, so that multiple clients (HMI screens) are updated simultaneously
Express.js handles the different requests. Due to the event-based architecture of javascript, it is asynchronous and interrupt-free.
Jade serves as a template engine. It delivers easy to read templates for the designers.
Stylus is used to generate CSS code out of JavaScript. It uses a markup language that lets you use variables, that simplify CSS development.



These technologies are then combined to one HMI module (we follow our modular paradigma). This module includes an OPC UA client, and acts as a server for the HMI-web-pages. Doing something so complex like an HMI in a web-page is in our opinion a good approach, since the web-development community is huge, and all technologies used are for free and open source. They are also bullet-proof and exist for production for a long time, just look at a website like Google or Amazon, that practically have no downtime, and run on any browser flawlessly.

Our draft for the system architecture of the HMI looks like this:

System Architecture (MI5)

System Architecture (MI5)

It is now end of September, and the JavaScript client is under heavy development. One of the most complex parts so far, is the investigation of the connection of the node-opcua module to the OPC UA Server. We investigate the compatibility of different OPC UA Servers, for example Beckhoff TwinCAT OPC UA Server or the equivalent from B&R.

As of now, we can say, that the decision for JavaScript looks very good. It looks lean and clean. Since we do not need to actually write anything else, besides JavaScript there is no cut in code development. Once all the programmers are familiar with the somewhat different way of programming, they can read all the code.

For developing we use Nodeclipse and GIT versioning.

Please let us know what you think, in the comment section below. We also attached some more screenshots and excerpts of the Students report from the TH Ingolstadt.

And don’t forget to subscribe our blog on the right.