Why you shouldn’t use SYS-Topics for Monitoring


Many MQTT brokers, including HiveMQ, implement SYS-Topics. These topics are special meta topics which are used by the broker to publish information about the broker itself and its MQTT client sessions. All SYS-Topics start with $SYS and are read-only for MQTT clients, so publishing to these topics is prohibited. SYS-Topics are mentioned in the MQTT 3.1.1 specification (4.7.2) but are a non-standard feature that are available in some MQTT brokers like HiveMQ or mosquitto.

An unofficial SYS-Topic specification is available on the MQTT Wiki, it is however subject to change and applications should not rely on these information.

Brokers publish these SYS-Topics on a periodical basis, the default on most brokers is 60 seconds. All MQTT clients that subscribe to one or more SYS-Topics receive the current value on the SYS topics as soon as they subscribe. After the subscription was successful, the client will receive metrics on periodical basis. Some static SYS-Topics (e.g. broker version) are only published upon subscription.

Debugging and Development with SYS-Topics
SYS-Topics are fantastic for developing and debugging MQTT applications. Developers can quickly monitor the current state of the broker and calculate and verify metrics – like message amplification rate or network metrics. Applications like MQTT.fx even provide native support for SYS-Topics embedded in a beautiful interface.

As you can see in the following video, you can get a quick overview of the MQTT brokers metrics by subscribing to the SYS-Topics:

1. Metric resolution is not good enough
The metric publishing interval for most MQTT brokers is 60 seconds by default. Monitoring interfaces like JMX allow to poll the most current data whenever needed. So the broker does not dictate the monitoring system how the resolution should be. The monitoring system can decide how often new data should be collected, without modifying any configuration on the MQTT broker.

2. SYS-Topics provide only a subset of available metrics
While the metrics provided by SYS-Topics can be useful, they are only a small subset of all available metrics. HiveMQ exposes more than 100 different metrics (the number of exposed metrics also increase with every release). The SYS-Topic Metrics usually are more focused on MQTT session monitoring and thus don’t provide metrics that can give you an overview of the brokers health at a quick glance. The most interesting metrics for operation teams are usually product dependant, so it’s unlikely that a common SYS-Topic standard will cover these in the future.

3. SYS-Topics expose internal information to potential attackers
It’s a common best practice to conceal the actual software version used in a deployment from attackers to make it harder to use exploits that are targeted towards a specific software version. Often you even want to hide the actual software used from attackers. While this only provides Security by Obscurity (which you should never rely on as only security measure!), it’s still important to hide such deployment information. SYS-Topics expose information like Broker Software Used and Version Number to any subscriber. These may be valuable information for attackers but often don’t provide too much value to actual legitimate subscribers.

4. It’s hard to monitor broker clusters with SYS-Topics
HiveMQ SYS-Topics expose statistics only for the current cluster node. If someone would like to build a custom monitoring solution based on SYS-Topics, MQTT connections to all cluster nodes need to be established. Typically MQTT clusters are behind a load balancer, so often it’s not even possible to connect to a specific cluster node. Monitoring integrations with e.g. Graphite are built with clustering in mind and so it’s easy get a quick health overview of the whole cluster. That’s much harder to do with SYS-Topics.

5. The monitored channel should be separated from the monitoring channel
A good monitoring practice is to use a dedicated monitoring channel instead of using the same channel you are monitoring. With MQTT brokers this means that if you are monitoring the MQTT communication, you shouldn’t use MQTT (SYS-Topics) to monitor the MQTT communication. It’s the same as: Don’t use e-mail alerts for monitoring e-mail servers. If the MQTT communication is not available for any reason, you won’t get any monitoring data. The monitoring data may be most valuable especially when something bad happened. So if you’re relying on these data for alerting you may have a problem. If you’re using a dedicated monitoring channel like JMX, Graphite or Nagios, a problem with the MQTT communication should not affect your monitoring.

Leave a Reply

Your email address will not be published. Required fields are marked *