Tutorial: Overview

MQTT Gateway (beta) Overview

This web page is about the MQTT gateway bound to Webcom, also known as the "Webcom MQTT Gateway".

Table of contents:

Requirements

In order to use the Webcom MQTT Gateway, you must fulfill the following steps:

  • If you never logged on Webcom, you must create an account
  • Once signed in, create at least one Webcom application
  • Then, you must handle authentication whose complete explanation is beyond the scope of this document (refer to the Webcom documentation)

Only registered people will be able to configure the Webcom MQTT Gateway.

Overview

The goal of Webcom MQTT Gateway is to provide a convenient way for developers using Webcom to deal with their MQTT brokers in order to:

  1. subscribe to MQTT brokers and store any update to Webcom
  2. monitor data on Webcom and publish any update to MQTT brokers

First feature is called "Subscribe to MQTT -> Put/Push to Datasync", second one is called "Listen to Datasync -> Publish to MQTT". Following chapters explain in details what should be configured to use these two features. How to configure them is explained in another page. Please, read the following chapters before the other page...

Subscribe to MQTT -> Put/Push to Datasync

As a developer using Webcom, say that you have a temperature sensor somewhere, which value is periodically published to some specific topic on your favorite MQTT broker. And each time such a value is published, you'd like it to be stored on some path of your Webcom application. All you need to do is:

  • provide the settings of your MQTT broker: URL, client id, authentication settings, topic...
  • decide on which Webcom path you want this temperature value to be stored

Webcom MQTT Gateway will then subscribe to the appropriate topic on your MQTT broker and, each time a message is published, store the value contained in the message to the appropriate Webcom path.

Additionally, you may:

  • decide if any update of your temperature value will:
    • overwrite the previous one in Webcom through a PUT operation
    • store it as an additional value to a new sub-path through a PUSH operation
  • use a template in order to:
    • extract a specific part of the MQTT message as the value
    • build the Webcom path from MQTT message content

MQTT broker settings

Webcom MQTT Gateway needs to know how to reach your MQTT broker and which topic to subscribe to. Here are the settings we need (settings in bold are MANDATORY)

  • the broker URL, for example: "wss://iot.eclipse.org:443/ws"
  • the topic, for example "my/topic"
  • your client ID, assigned to you by the broker,
  • your username and password if any,
  • the SSL certificate, needed to reach the broker, if any.

Webcom settings

Webcom MQTT Gateway needs to know how to store data in Webcom. Here are the settings we need:

  • a front path, corresponding to the path if no template is in use (see below)
  • a boolean, named overwrite, corresponding to:
    • true: a PUT operation,
    • false: a PUSH operation
  • a template, which format is detailed below,
  • a boolean, named use JSON, used to apply or not the template.

The template is made of reserved words, all starting with a "$" character, as follows:

  • $year, $month, $day, $hour, $minute, $second and $millisecond reflecting the date when data is stored in Webcom (and not when it was published on MQTT broker)
  • $payload reflecting the entire JSON message retrieved from topic.

We use $payload to navigate through MQTT JSON message, separating field names with a dot, like for example: "$payload.room.temperature". Then, we can use these $payload.xxx combinations to build a Webcom path, using "/" character as a separator. The unique combination, after the last "/" separator, is considered as the value to store in the path built from previous combinations.

Warning:

  • most of the time, there is a "payload" field in JSON messages retrieved from MQTT brokers. Pay attention not to be mistaken by its name and our $payload reserved word. Actually, the content of this "payload" field should be designed in our template by "$payload.payload".
  • do not try to build a Webcom path using a combination which reference is a JSON object or array in MQTT message. Only simple values (string, number, boolean) are correctly parsed, others are simply ignored.

Example #1:

Let us use an MQTT message with the following format:

{
    "timestamp":"2018-07-25T09:34:34 561Z",
    "payload": {
        "room": {
            "name":"kitchen",
            "data": {
                "temp":25,
                "hygro":75,
                "status":"normal"
            }
        }
    }
}

Say that you just want the 'temp' value to be overwritten each time such a message is published in your /home/kitchen/temp Webcom path. In that case, you have to define the following settings:

  • overwrite: true
  • front path: /home/kitchen/temp
  • use JSON: true
  • template: $payload.payload.room.data.temp

Example #2:

Now, say that you want the entire data regarding your kitchen in /home/kitchen Webcom path:

  • front path: /home/kitchen
  • template: $payload.payload.room.data

This will store data in Webcom as follows:

 |
 |- home
     |
     |- kitchen
           |
           |- temp: 25
           |- hygro: 75
           |- status: normal

You may obtain the same result with the following settings:

  • overwrite: true
  • front path: /home
  • use JSON: true
  • template: $payload.payload.room.name/$payload.payload.room.data

Here, "kitchen" is dynamically retrieved from the "name" field in the JSON message. The template dynamically built path from MQTT JSON field values, using the "/" character as a separator.

So, the previous example works again with the following settings:

  • overwrite: true
  • front path: nothing, defaults to /
  • use JSON: true
  • template: home/$payload.payload.room.name/$payload.payload.room.data

Remember that the unique $payload.xxx combination in the template, after the last "/" separator, is considered as the value to store in the path built from previous combinations. This is why we obtain "temp", 'hygro" and "status" sub-paths of the "/home/kitchen" path.

Example #3:

If you want to add a "data" sub-path to "/home/kitchen" path, you should use the following settings:

  • overwrite: true
  • front path: nothing, defaults to /
  • use JSON: true
  • template: home/$payload.payload.room.name/data/$payload.payload.room.data

Then, you obtain the following storage in Webcom:

 |
 |- home
     |
     |- kitchen
           |
           |- data
                |
                |- temp: 25
                |- hygro: 75
                |- status: normal

Example #4:

Let us see what happens with PUSH operations via the overwrite boolean:

  • overwrite: false
  • front path: nothing, defaults to /
  • use JSON: true
  • template: home/$payload.payload.room.name/$payload.payload.room.data

Then, you obtain the following storage in Webcom:

 |
 |- home
     |
     |- kitchen
           |
           |- -LAlgWvdSr-FAElPkBCo
           |           |
           |           |- temp: 25
           |           |- hygro: 75
           |           |- status: normal
           |
           |
           |- -LAlgkVVQZgDjAhA2pJK
                       |
                       |- temp: 24
                       |- hygro: 80
                       |- status: normal

Notice that "-LAlgWvdSr-FAElPkBCo" and "-LAlgkVVQZgDjAhA2pJK" are randomly generated sub-paths to avoid overwriting data on Webcom.

Example #5:

You may have several of these sub-paths each day, however if you intend not to update data every day, or to keep only the last values of updated data each day, you could use the following settings:

  • overwrite: true
  • front path: /home
  • use JSON: true
  • template: $year/$month/$day/$payload.payload.room.name/$payload.payload.room.data

Then, you obtain the following storage in Webcom:

 |
 |- home
     |
     |- 2018
     |    |
     |    |- 7
     |       |
     |       |- 24
     |           |
     |           |- kitchen
     |                 |
     |                 |- temp: 25
     |                 |- hygro: 75
     |                 |- status: normal
     |
     |
     |- 2018
          |
          |- 7
             |
             |- 25
                 |
                 |- kitchen
                       |
                       |- temp: 24
                       |- hygro: 80
                       |- status: normal

Of course, it is also possible to keep last value of each month or even year.

Example #6:

And finally, you could mix the two previous examples in order to obtain several values each day, plus you could group date sub-paths to reduce tree levels:

  • overwrite: false
  • front path: /home
  • use JSON: true
  • template: $year-$month-$day/$payload.payload.room.name/$payload.payload.room.data

Then, you obtain the following storage in Webcom:

 |
 |- home
     |
     |- 2018-10-24
     |     |
     |     |- kitchen
     |           |
     |           |- -LAlgWvdSr-FAElPkBCo
     |           |           |
     |           |           |- temp: 25
     |           |           |- hygro: 75
     |           |           |- status: normal
     |           |
     |           |
     |           |- -LAlgkVVQZgDjAhA2pJK
     |                       |
     |                       |- temp: 24
     |                       |- hygro: 80
     |                       |- status: normal
     |
     |
     |- 2018-10-25
           |
           |- kitchen
                 |
                 |- -LHX7Xm5lsD7XqXYlP0D
                 |           |
                 |           |- temp: 23
                 |           |- hygro: 74
                 |           |- status: normal
                 |
                 |
                 |- -LHX9UEY4Ythei8_A0rm
                             |
                             |- temp: 22
                             |- hygro: 81
                             |- status: normal

Here are a few working and non working examples of grouped reserved words used to build sub-paths (and not value):

  • $year%$month.$day_$hour$minute$second$payload.room.name
  • $payload.payload.room.name-$payload.payload.room.temp
  • $payload.payload.room.status$payload.payload.room.name
  • $year_$month_$day_$payload.payload.room.namefoobar does not work as 'namefoobar' is misinterpreted ('$payload.payload.room.name-foobar' works)
  • $year_$month_$day_$payload.payload.room.name.foobar does not work as 'payload.room.name.foobar' references nothing in MQTT message
  • $year_$month_$day_$payload does not work as '$payload' alone, without any following '.xxx' item, is ignored
  • $year_$month_$day_$payload.payload.room.data does not work as '$payload.payload.room.data' references a JSON object in MQTT message

to obtain respectively something like:

  • 2018%10.25_170742kitchen
  • kitchen-22
  • normalkitchen
  • 2018_10_25_ ('$payload.payload.room.namefoobar', that cannot be retrieved in MQTT message, is ignored)
  • 2018_10_25_ ('$payload.payload.room.name.foobar', that cannot be retrieved in MQTT message, is ignored)
  • 2018_10_25_ ('$payload' alone, without any following '.xxx' item, is ignored)
  • 2018_10_25_ ('$payload.payload.room.data', as a JSON object, is ignored)

Warning: do not group combinations or use date reserved words after the last "/" separator. The only reserved word here may be $payload alone to retrieve the entire MQTT message (not very useful) or a unique $payload.xxx combination.

Example #7:

Another solution would be to use the "timestamp" field value from MQTT JSON message, as a path information. In that case, there is no need to use randomly generated sub-paths as this timestamp changes for every received MQTT message:

  • overwrite: true
  • front path: /home
  • use JSON: true
  • template: $payload.payload.room.name/$payload.timestamp/$payload.payload.room.data

Then, you obtain the following storage in Webcom:

 |
 |- home
     |
     |- kitchen
           |
           |- 2018-07-24T01:53:56_560Z
           |           |
           |           |- temp: 25
           |           |- hygro: 75
           |           |- status: normal
           |
           |
           |- 2018-07-25T09:34:34_561Z
                       |
                       |- temp: 24
                       |- hygro: 80
                       |- status: normal

Notice that forbidden characters in Webcom paths have been replaced by an underscore character (the space, in timestamp, before milliseconds in our example)

Listen to Datasync -> Publish to MQTT

As a developer using Webcom, say that any data stored on some path of your Webcom application should be, as soon as it changes, published on some topic of your favorite MQTT broker. All you need to do is:

  • provide the settings of your MQTT broker: URL, client id, authentication settings, topic...
  • decide which Webcom path you want to monitor (or "listen to")

Webcom MQTT Gateway will then listen to the appropriate Webcom path and, each time data is updated there, publish it through an MQTT message on appropriate topic of your MQTT broker.

MQTT broker settings

Webcom MQTT Gateway needs to know how to reach your MQTT broker and which topic to publish to. Here are the settings we need (settings in bold are MANDATORY)

  • the broker URL, for example: "wss://iot.eclipse.org:443/ws"
  • the topic, for example "my/topic"
  • your client ID, assigned to you by the broker,
  • your username and password if any,
  • the SSL certificate, needed to reach the broker, if any.

Webcom settings

Webcom MQTT Gateway only needs to know which path to monitor on Webcom. Here is the setting we need:

  • listen path: the path to monitor, along with all its sub paths

On Webcom, when you listen to a path, you also listen to all its sub paths, and if something changes only for a sub path, you receive information about data change for this sub path only. In that case however, Webcom MQTT Gateway will publish the entire data stored on the path, and not only data regarding the updated sub path.

Example #1:

Let us work with the Webcom data below, and say that your listen path is /home/kitchen:

 |
 |- home
     |
     |- kitchen
           |
           |- temp: 25
           |- hygro: 75
           |- status: normal

Each time something changes on at least one of "temp", "hygro" and "status" sub paths, Webcom MQTT gateway will publish the following data to your MQTT broker (example if temperature has decreased to 24):

{
    "temp":24,
    "hygro":75,
    "status":"normal"
}

So, if you only want to publish MQTT messages about temperature, you have to use /home/kitchen/temp as the listen path and Webcom MQTT gateway will publish its only value (24 in this example) as a MQTT message.

Example #2:

Now, say that you need to format data published to your MQTT broker, for example with the famous "payload" field usually used in MQTT messages:

{
    "payload": {
        "temp":25,
        "hygro":75,
        "status":"normal"
    }
}

As there is no template in "Listen to Datasync -> Publish to MQTT" feature, you need to cheat with Webcom and add a "payload" sub path level, like the following:

 |
 |- home
     |
     |- kitchen
           |
           |- payload
                 |
                 |- temp: 25
                 |- hygro: 75
                 |- status: normal

Of course, this only works if you keep /home/kitchen as your listen path.