Tutorial: Monitors with Templated Payloads

The monitor feature of Remote Manager supports templates that customize the format used to deliver events. This can be helpful when a service is not able to easily consume the built-in XML and JSON formats and a different schema is required by the target.

For additional background on monitors see Experimenting with Monitors where we took a look at the monitor feature in Remote Manager and how it can be used to aggregate and deliver events based on a list of subscribed topics.

Supplying a Template

For a Monitor

The monSchemaType and monSchemaData fields can be used to supply a template. For example, a query monitor can be created that includes a template with a POST to /ws/Monitor with a payload such as:

<Monitor>
    <monTopic>DataPoint/*/metrics/cellular</monTopic>
    <monTransportType>polling</monTransportType>
    <monSchemaType>handlebars</monSchemaType>
    <monSchemaData>
        [
        {{#each this}}
          {{#if @index}},{{/if}}
          { "{{DataPoint.streamId}}":{{DataPoint.data}} }
        {{/each}}
        ]
    </monSchemaData>
</Monitor>

Test with the History API

A monitor may be responsible for sending data to your production system and not be a good candidate for testing or experimenting.

Supplying a template to the monitory history API is a good way to rapidly test different templates without changing a monitor. Retrieving the history of the query monitor with a GET to /ws/v1/monitors/history/<monId> shows the result of the template in the output field.

This example:

Targets monitor 1772299
Uses the start_time parameter to start with data from 5 minutes ago
Uses the size parameter to limit the maximum data items to return to 10
Uses the schema parameter to supply a template
Uses the httpIe command line tool to send the web request.

NOTE: In your own testing, you’ll have to be careful with escaping special characters (especially if you use the curl command line tool)

http remotemanager.digi.com/ws/v1/monitors/history/1772299 \
 start_time==-5m \ 
 size==10 \
 schema=='[
{{#eachFiltered this}}
  {{#if DataPoint.streamId}}
  {{#if @index}},{{/if}}
    {{json DataPoint.streamId}}
  {{/if}}
{{/eachFiltered}}]'

The result is the following:

Note:

The output field has been formatted to show proper spaces and newlines.
The device IDs have been changed to 00000000-00000000-00000000-00000000

{
    "count": 10,
    "cursor": "99084a20-b838-11ed-9b29-e2a950f87599",
    "next_uri": "/ws/v1/monitors/history/1772299?start_time=-5m&size=10&schema=%5B%0A%7B%7B%23eachFiltered+this%7D%7D%0A++%7B%7B%23if+DataPoint.streamId%7D%7D%0A++%7B%7B%23if+%40index%7D%7D%2C%7B%7B%2Fif%7D%7D%0A++++%7B%7Bjson+DataPoint.streamId%7D%7D%0A++%7B%7B%2Fif%7D%7D%0A%7B%7B%2FeachFiltered%7D%7D%5D&cursor=99084a20-b838-11ed-9b29-e2a950f87599",
    "output": "[
  "00000000-00000000-00000000-00000000/metrics/sys/mem/used",
  "00000000-00000000-00000000-00000000/metrics/sys/cpu/used",
  "00000000-00000000-00000000-00000000/metrics/sys/storage/config/used",
  "00000000-00000000-00000000-00000000/metrics/sys/storage/opt/used",
  "00000000-00000000-00000000-00000000/metrics/sys/storage/tmp/used",
  "00000000-00000000-00000000-00000000/metrics/sys/storage/var/used",
  "00000000-00000000-00000000-00000000/metrics/eth/1/rx/bytes",
  "00000000-00000000-00000000-00000000/metrics/sys/location",
  "00000000-00000000-00000000-00000000/metrics/eth/1/rx/packets",
  "00000000-00000000-00000000-00000000/metrics/eth/1/rx/overruns"
]",
    "polling_cursor": "99084a20-b838-11ed-9b29-e2a950f87599",
    "polling_uri": "/ws/v1/monitors/history/1772299?start_time=-5m&size=10&schema=%5B%0A%7B%7B%23eachFiltered+this%7D%7D%0A++%7B%7B%23if+DataPoint.streamId%7D%7D%0A++%7B%7B%23if+%40index%7D%7D%2C%7B%7B%2Fif%7D%7D%0A++++%7B%7Bjson+DataPoint.streamId%7D%7D%0A++%7B%7B%2Fif%7D%7D%0A%7B%7B%2FeachFiltered%7D%7D%5D&cursor=99084a20-b838-11ed-9b29-e2a950f87599",
    "size": 10
}

Template format

See Handlebars Template for details about the Handlebars template format.

Handlebars Playgrounds

There are playground environments online that let you quickly try out templates given a data model that you supply, such as https://handlebarsjs.com/playground.html . These playgrounds do not include the custom helpers Remote Manager provides. If you want to try out custom helpers in these environments you can paste the following into the box they have for custom helpers (captioned Preparation-Script in the above playground).

Note, these should be considered unsupported implementations of the custom helpers and are solely provided to experiment in the sandbox environments. The actual implementation of the helpers differs in Remote Manager.

Here are some experimental JavaScript Custom Helpers, this list does not include all the helpers provided by Remote Manager:

Handlebars.registerHelper('firstPathComponent', function(text) {
  return text.split('/')[0];
});
Handlebars.registerHelper('remainingPathComponents', function(text) {
  return text.split('/').splice(1).join('/');
});
Handlebars.registerHelper('formatTime', function(timestamp) {
  return new Date(timestamp).toISOString();
});
Handlebars.registerHelper("endsWith", function(text, suffix, options) {
  if (text.endsWith(suffix)) {
    return options.fn(this);
  } else {
    return options.inverse(this);
  }
});
Handlebars.registerHelper("eq", function(inputOne, inputTwo, options) {
  if (inputOne === inputTwo) {
    return options.fn(this);
  } else {
    return options.inverse(this);
  }
});

Handlebars.registerHelper("eachFiltered", function(context, options) {
  if (!options) {
    throw new Exception('Must pass iterator to #each');
  }

  let fn = options.fn,
    inverse = options.inverse,
    i = 0,
    ret = '',
    data;

  if (typeof context === 'function') {
    context = context.call(this);
  }

  if (options.data) {
    data = {};
    for (let key in options.data) {
      if (Object.prototype.hasOwnProperty.call(arguments[i], key)) {
        data[key] = arguments[i][key];
      }
    }
    data._parent = options.data;
  }

  function execIteration(field, index, last) {
    if (data) {
      data.key = field;
      data.index = index;
      data.first = index === 0;
    }

    let blockOutput = fn(context[field], {
        data: data,
        blockParams: [context[field], field]
      });

    if(blockOutput.trim()) {
      ret = ret + blockOutput;
      return true;
    } else {
      return false;
    }
  }

  if (context && typeof context === 'object') {
    if (context && typeof context === 'object'
      ? toString.call(context) === '[object Array]'
      : false) {
      let nonEmptyIndex = 0;
      for (let j = context.length; i < j; i++) {
        if (i in context) {
          if(execIteration(i, nonEmptyIndex, i === context.length - 1)) {
            nonEmptyIndex++;
          }
        }
      }
    } else if (global.Symbol && context[global.Symbol.iterator]) {
      const newContext = [];
      const iterator = context[global.Symbol.iterator]();
      for (let it = iterator.next(); !it.done; it = iterator.next()) {
        newContext.push(it.value);
      }
      context = newContext;
      let nonEmptyIndex = 0;
      for (let j = context.length; i < j; i++) {
        if(execIteration(i, nonEmptyIndex, i === context.length - 1)) {
          nonEmptyIndex++;
        }
      }
    } else {
      let priorKey;
      let nonEmptyIndex = 0;
      Object.keys(context).forEach(key => {
        // We're running the iterations one step out of sync so we can detect
        // the last iteration without have to scan the object twice and create
        // an itermediate keys array.
        if (priorKey !== undefined) {
          if(execIteration(priorKey, nonEmptyIndex)) {
            nonEmptyIndex++;
          }
        }
        priorKey = key;
        i++;
      });
      if (priorKey !== undefined) {
        if(execIteration(priorKey, nonEmptyIndex++, true)) {
          nonEmptyIndex++;
        }
      }
    }
  }

  if (i === 0) {
    ret = inverse(this);
  }

  return ret;
});
Handlebars.registerHelper('slice', function(context, sep, start, length, options) {
    if (start === 0 || length < 1) {
        throw new Exception('Bad start or length parameter');
    }
    const components = context.split(sep)
    let end = false
    if (start < 0) {
        end = true
        start = -start
    }
    start-=1
    if (!end) {
        return components.slice(start, start+length).join(sep)
    }
    else {
        return components.slice(components.length-1-start, components.length-1-start+length).join(sep)
    }
});
Handlebars.registerHelper('json', function(context, options) {
    const dft = options.hash.default ? options.hash.default : null;
    if (!context) {
        return dft
    }
    const pretty = options.hash.pretty ? 2 : null;
    const escapeHTML = !!options.hash.escapeHTML;
    let json = JSON.stringify(context, null, pretty)
    if (!escapeHTML) {
        return json;
    }
    // Better techniques for escaping HTML are available using
    // the DOM or jQuery, but this is a simple example with no
    // dependencies
    return json
        .replace(/&/g, '&amp;')
        .replace(/</g, '&lt;')
        .replace(/>/g, '&gt;')
        .replace(/"/g, '&quot;')
        .replace(/'/g, '&#039;');
});
Handlebars.registerHelper('replace', function(context) {
    var current = context;

    const options = arguments[arguments.length - 1];
    if (current) {
        let index=1;
        while (index+1 < arguments.length-1) {
            const match = arguments[index];
            const replace = arguments[index+1];
            current = current.replaceAll(match, replace);
            index += 2
        }
    }
    if (current !== context) {
        console.log(`Replaced to ${current}`);
        if (!options.data) {
            options.data = {};
        }
        options.data.replaced = current;
        return options.fn(this, options);
    } else {
        return options.inverse(this);
    }
});

Building a Template

Data Model

The data model supplied to the template engine is the list of monitor events in their JSON representation. For example, a monitor subscribed to DataPoint events would have the following available to the template:

[
   {
      "topic":"1234/DataPoint/00000000-00000000-12341234-12341234/metrics/cellular/1/sim1/tx/bytes",
      "group":"somegroup",
      "operation":"INSERTION",
      "timestamp":"2021-12-07T20:07:08.563Z",
      "context": {
        "deviceType": "Default Simulated Device Type",
        "deviceName": "deviceA",
        "vendorId": "0"
      },
      "DataPoint":{
         "id":"41a1ac50-5799-11ec-afa9-f6681641504e",
         "streamId":"00000000-00000000-12341234-12341234/metrics/cellular/1/sim1/tx/bytes",
         "streamUnits":"",
         "streamType":"UNKNOWN",
         "description":"",
         "data":"64",
         "timestamp":1638907628437,
         "serverTimestamp":1638907628437,
         "quality":0,
         "cstId":1234
      }
   },
...
   {
      "topic":"1234/DataPoint/00000000-00000000-12341234-12341234/metrics/cellular/2/sim1/tx/bytes",
      "group":"somegroup",
      "operation":"INSERTION",
      "timestamp":"2021-12-07T20:15:44.125Z",
      "context": {
      "deviceType": "Default Simulated Device Type",
      "deviceName": "deviceA",
      "vendorId": "0"
      },
         "DataPoint":{
         "id":"74f1b360-579a-11ec-afa9-f6681641504e",
         "streamId":"00000000-00000000-12341234-12341234/metrics/cellular/2/sim1/tx/bytes",
         "streamUnits":"",
         "streamType":"UNKNOWN",
         "description":"",
         "data":"72",
         "previousData":"8",
         "timestamp":1638908144022,
         "serverTimestamp":1638908144022,
         "quality":0,
         "cstId":1234
      }
   }
]

Since the template is supplied a list of events you will typically need to iterate over the content of the data model using either the each or eachFiltered helpers.

Testing Your Template

It can take a few tries to get a template that has the desired output, so there are stages you can go through that can make it easier to experiment.

Getting the Data Model

First off, start by creating a query monitor that is subscribed to the topics you are interested in. Query monitors can be queried repeatedly without having to publish new data, so tweaks to your template can be seen quickly. See above for an example of creating a query monitor with a template.

Once you have a query monitor created, wait for some events and then retrieve the raw JSON representation of the events by doing a GET to /ws/v1/monitors/history/.json?ignore_schema=true&include_context=true. The items in the list field is what will be supplied to your template.

DataPoint Context

Published DataPoint events for use in templates will include a context that has the device name, device type, and vendor ID if they are defined for the device that sent the event. Examples of the data can be retrieved using the include_context parameter as mentioned in the previous section.

Handlebars Playgrounds

If you are just starting out with a template you might want to paste the content of the list field from above into a Handlebars Playground so that you can see a live representation of what your template would output.

Once you are happy with the template, you can update the query monitor used to retrieve the data model with a PUT to /ws/Monitor/ with the updated template in the monSchemaData field, like below:

<Monitor>
  <monSchemaType>handlebars</monSchemaType>
  <monSchemaData>...updated template here</monSchemaData>
</Monitor>

After the monitor is updated you can query the output with a GET to /ws/v1/monitors/history/ to see if the format of the output field is correct.

HTTP Monitor Request Capturing and Inspection

Next, if you’re using an HTTP monitor you can create the HTTP monitor but target an endpoint that allows you to easily see what is being delivered from Remote Manager. An example site that allows capturing requests is https://requestbin.com/. You can point your HTTP monitor against a site like this and then look at the requests to see if the body of the request matches your expected template output.

Integrate Your Application

Finally, if you are planning to use an HTTP or TCP monitor you can now create the monitor with the desired monTransportType and and template and verify that it has the desired output when delivering events to your TCP monitor client or web server. The more rapid testing available with the playground and query monitor should result in a template that works quickly with the HTTP or TCP monitor, avoiding the overhead of having to wait for traffic and event delivery.

Batching

Templates do not have the ability to change how monitor events are batched, they are only allowed to determine the output that should be used for a given batch of events. If a monitor is configured to send events in batches of 1000, then a template will be supplied those 1000 events and expected to represent every event in the output.

It’s possible for a template to filter out events by choosing not to include anything for them in the output (perhaps using if or endsWith), but the monitor will still consider the event as having been processed even if a template chooses to not include anything from the event.

Common Patterns

JSON Output

A common use case is a template that generates JSON output. A starting pattern for such a template where the monitor is subscribed to DataPoint events is

[
{{#each this}}
  {{#if @index}},{{/if}}
  { "{{DataPoint.streamId}}":"{{DataPoint.data}}" }
{{/each}}
]

In this example square brackets are used to create a list in JSON and the each helper is used to iterate through each of the DataPoint events in the data model. For each event, curly braces add an object to the json output and we extract the DataPoint.streamId and DataPoint.data fields from the data model. We use the if helper along with the @index attribute from the each helper to ensure a comma delimiter is inserted between each object.

Filtered JSON Output

This use case is very similar to the JSON Output case, but there may be specific events in the data model that you want to skip. Special care needs to be taken in this case to ensure that the delimiters in your output don’t get doubled up for skipped events. An example of this is

[
{{#eachFiltered this}}
  {{#if @index}},{{/if}}
  {{#endsWith streamId 'ts1/bat'}}
  { "{{DataPoint.streamId}}":"{{DataPoint.data}}" }
  {{/endsWith}}
{{/eachFiltered}}
]

We’ve swapped the each helper for eachFiltered and added an endsWith helper to ensure only events that have a particular streamId are included. eachFiltered is necessary to ensure the @index attribute appropriately considers events we’ve filtered out…otherwise if the first event is filtered out we would still end up with a comma delimiter resulting in invalid JSON.

Tutorial: Monitors with Templated Payloads

Table Of Contents

Supplying a Template

For a Monitor

Test with the History API

Template format

Handlebars Playgrounds

Building a Template

Data Model

Testing Your Template

Getting the Data Model

DataPoint Context

Handlebars Playgrounds

Refinement through the Query Monitor

HTTP Monitor Request Capturing and Inspection

Integrate Your Application

Batching

Common Patterns

JSON Output

Filtered JSON Output