Using local MQTT broker for cloud and interprocess communication

Recently I was working on an embedded Linux IoT device that communicated with cloud using MQTT protocol. The software in the device was divided into multiple applications that also required interprocess communication. We ended up using MQTT also for the local communication, and it turned out to be a good decision.

What is MQTT?

MQTT (Message Queue Telemetry Transport) is a lightweight publish-subscribe protocol that is used on top of TCP/IP. MQTT uses a message broker that dispatches messages between senders that publish them, and receivers that are interested in these messages. Same client can both publish and subscribe messages.

Each message is published to a specific topic. The topic is the message routing information, and it is simply a string that can have slash separated hierarchy levels (e.g. “office/floor1/temperature”). Clients subscribe to these topics, and the broker delivers all messages sent with matching topics to them. It is also possible to use wildcards to easily subscribe to multiple topics.

With this communication scheme, the data producers and consumers do not need to know about each other. They only communicate using a common topic. With cloud systems, the MQTT broker is usually in the cloud, and the ‘things’ connect to it and produce data by publishing messages.

Local MQTT broker

The device in this case needed to publish measured and processed sensor information to cloud and also receive command messages such as firmware update, configuration update and reboot from the cloud. The software was divided into multiple applications that handled different aspects and therefore also published and subscribed to different topics. However, it was desirable that the device only opened one MQTT connection to the cloud instead of each application connecting individually. Individual connections would have caused problems because each physical device was registered with a single client id. The solution was to use a local broker and MQTT bridge connection.

The applications each connected locally to the broker that was running on the device. The local broker in turn connected to the MQTT broker in the cloud. The cloud connection also used encryption and authentication. The applications used unencrypted connections for simplicity because all their communication was local in the device. The connection between the local broker in the device and the remote broker in cloud is called a bridge. It uses the same publish and subscribe as normal clients but it is between two brokers.

In this case mosquitto was used as the local broker and Paho C as MQTT client library in the applications.

The broker allows to configure which topics are bridged to and from the cloud. The following example shows a simple bridge configuration for Mosquitto.

connection test-mosquitto-org
address test.mosquitto.org
topic cloud/command/# in 0
topic cloud/event/# out 0

The topics that are configured as input are forwarded from the cloud to the device (i.e. the broker on the device subscribes these topics from the cloud broker). The configuration also allows the specify which topics are published to cloud. These topic patterns can also include wildcards as shown above. All other topics are only published to local clients if there are active subscriptions.

Interprocess communication and testing

The communication scheme that uses local broker and bridge allows to decouple applications from each other while still having a single connection towards the cloud. It also allows to re-use the same client implementation for communication between local applications. From the application point-of-view the main difference between cloud and interprocess communication is just different topic names.

One additional benefit of using the local MQTT broker also for IPC is that it allows external clients to easily connect and monitor the communication (this of course should be prevented in production using e.g. iptables). Being able to see all the messages from, for instance, development PC is very useful for debugging and testing.

It is also possible to implement automated tests this way. Test system could implement the same topics and messages as the cloud or use the internal topics for integration testing. These test cases can also be automated using, for instance, Jenkin and Robot Framework which already has MQTT support.

Many cloud systems use MQTT as their communication protocol for connecting devices, so especially in these situations it makes sense to re-use the same implementation also for local communication.