FarmMaps/Documentatie
8
0
Fork 0
Code Pull Requests Releases Activity

Update page 'Message Queue Endpoint'

Browse Source
This commit is contained in:
Auke Sytsma 2020-03-01 10:35:50 +00:00
parent 65e29d9745
commit b0f72eea97
1 changed files with 63 additions and 24 deletions
Show Stats Download Patch File Download Diff File Expand all files Collapse all files

87
Message-Queue-Endpoint.md
View File

@ -4,28 +4,7 @@ The same workflow can be followed for different languages, as long as there's a
FarmMaps provides an MQTT endpoint to allow your application to send signals to FarmMaps, and listen for events. FarmMaps provides an MQTT endpoint to allow your application to send signals to FarmMaps, and listen for events.
These signals are collected in a "message queue". These signals are collected in a "message queue".
If you are not familiar with MQTT or message queues in general, please see the short explanation below. If you are not familiar with message queues or protobuff, please see the FAQ at the end.
#### What is a message queue? (skip ahead if familiar)
A message queue is commonly used to make software programs able to send messages between eachother, and thereby making it easy for data to flow from one program into another. There are many variants of message queues, some popular names are Apache Kafka, MQTT and RabbitMQ.
In message queue systems there is usually one central "hub" called the broker or **"message broker"**. This broker holds all the messages. Usually, the messages are organised in groups called **"topics"**.
Now, there are two things that external services can do.
* An external service (like a sensor) could **publish a message to a certain topic**. For example, a temperature sensor would publish the temperature at a specific time and location to the "temperatureMeasurements" topic.
* An external service can **"subscribe" to this topic** by connecting to the broker. This service will then recieve every temperature measurement.
When the subscribed service temporarily disconnects from the broker, it will not recieve any messages, but the messages will remain stored at the broker. Depending on configuration, messages will be kept longer or shorter, or be deleted after they reach the subscribers but the intent is always to ensure the messages get from the **publisher** to the **subscriber**.
Setting communication between applications up like this makes things a lot more flexible than connecting systems directly and provides a central point for management of all communication.
#### What is protobuf?
To quote the Google Documentation:
> Protocol buffers are Google's language-neutral, platform-neutral, extensible mechanism for serializing structured data – think XML, but smaller, faster, and simpler.
> You define how you want your data to be structured once, then you can use special generated source code to easily write and read your structured data to and from a variety of data streams and using a variety of languages.
More information can be found in the [protobuf documentation](https://developers.google.com/protocol-buffers/docs/overview).
### Prerequisites ### Prerequisites
To follow along with the examples, you need: To follow along with the examples, you need:
@ -47,5 +26,65 @@ py -m pip install 'protobuf==3.11.0'
``` ```
python3 -m pip install 'paho-mqtt==1.5.0' python3 -m pip install 'paho-mqtt==1.5.0'
python3 -m pip install 'protobuf==3.11.0' python3 -m pip install 'protobuf==3.11.0'
```
``` ### Workflow
The general workflow for pushing data to the FarmMaps MQTT endpoint consists of:
* Preparing your data as a protobuf message
* Connecting to the MQTT Broker
* Publishing the message and waiting for confirmation.
* Closing the connection
To be able to publish data to the broker, you need an username and password, and posibly a dedicated topic for your data.
An account can be requested through email from XXXXX. This user is only for the broker, and can not be used for the other API's.
Generally, one user is provided per application (so no individual users for different sensors within the same application).
### Settings
To connect to the MQTT Broker, we'll use the following settings:
**MQTT connection settings**
| Parameter | Default | Description |
| :---: | :------------------------------------------: | --- |
| CLIENT_ID | - | This ID is used to identify the connecting party (your software) to the broker, please use your company or brand name. |
| USER | - | Your username at the FarmMaps broker. |
| PWD | - | Your password at the FarmMaps broker. |
| HOST | `farmmaps.awtest.nl` | The address of the FarmMaps broker. |
| PORT | `1883` | The port number of the FarmMaps broker. |
| KEEPALIVE | `60` | Number of seconds to maintain the connection, even if no messages are published.
| TOPIC | `trekkerdata/sensors` | The topic to publish the messages to at the broker. |
All settings are required.
### Example: Pushing Live Tractor Data
To create messages and connect to the broker we will need the following settings:
**Message settings**
| Parameter | Default | Description |
| :---: | :------------------------------------------: | --- |
| META_DATA_URL | `https://<somedomain.com>/<somepath>` | HTTP adress where a JSON metadata file for the messages is provided (by the data source). |
| BASE_DEV_URN | `urn:dev:<company_name>:<device_serial_number>` | To identify each unique device from different parties, we add a code to each message composed of the company name and serial number. Farmmaps uses this code to link devices and data to their owners. |
All settings are required except for the `META_DATA_URL`, this parameter is optional.
### FAQ
#### What is a message queue? (skip ahead if familiar)
A message queue is commonly used to make software programs able to send messages between eachother, and thereby making it easy for data to flow from one program into another. There are many variants of message queues, some popular names are Apache Kafka, MQTT and RabbitMQ.
In message queue systems there is usually one central "hub" called the broker or **"message broker"**. This broker holds all the messages. Usually, the messages are organised in groups called **"topics"**.
Now, there are two things that external services can do.
* An external service (like a sensor) could **publish a message to a certain topic**. For example, a temperature sensor would publish the temperature at a specific time and location to the "temperatureMeasurements" topic.
* An external service can **"subscribe" to this topic** by connecting to the broker. This service will then recieve every temperature measurement.
When the subscribed service temporarily disconnects from the broker, it will not recieve any messages, but the messages will remain stored at the broker. Depending on configuration, messages will be kept longer or shorter, or be deleted after they reach the subscribers but the intent is always to ensure the messages get from the **publisher** to the **subscriber**.
Setting communication between applications up like this makes things a lot more flexible than connecting systems directly and provides a central point for management of all communication.
#### What is protobuf?
To quote the Google Documentation:
> Protocol buffers are Google's language-neutral, platform-neutral, extensible mechanism for serializing structured data – think XML, but smaller, faster, and simpler.
> You define how you want your data to be structured once, then you can use special generated source code to easily write and read your structured data to and from a variety of data streams and using a variety of languages.
More information can be found in the [protobuf documentation](https://developers.google.com/protocol-buffers/docs/overview).