Documentatie/Message-Queue-Endpoint.md

## Intro
This documentation page provides the basics on how to access the MQTT endpoint of FarmMaps, using Python code for examples.  
The same workflow can be followed for different languages, as long as there's a MQTT client and a Protobuf library for the language you are using.  
FarmMaps provides an MQTT endpoint to allow your application to send signals to FarmMaps, and listen for events.  
These signals are collected in a "message queue".  

If you are not familiar with anything mentioned, please see the [FAQ](#faq) at the end. 

### Prerequisites
To follow along with the examples, you need:
* Access to the FarmmMaps MQTT broker (user, password), these can be requested **here**
* To have Python 3.7 or higher installed.
* To come up with a unique identifier for your sensors.

Generally the URN is composed of the company or brand name and a device serial number like so:  
```python
"urn:dev:<company-name>:<device-serial-number>"
```
To run the python example code you also need to install the `paho-mqtt` and the `protobuf` modules.  

### Workflow
The general workflow for pushing data to the FarmMaps MQTT endpoint consists of:
* Preparing your data as a protobuf message
* Connecting to the MQTT Broker
* Publishing the message and waiting for confirmation.
* Closing the connection

To be able to publish data to the broker, you need an username and password, and posibly a dedicated topic for your data.
An account can be requested through email from XXXXX. This user is only for the broker, and can not be used for the other API's.
Generally, one user is provided per application (so no individual users for different sensors within the same application).

### Settings
To connect to the MQTT Broker, we'll use the following settings:

| Parameter | Default | Description |
| :---: | :------------------------------------------: | --- |
| CLIENT_ID | - | This ID is used to identify the connecting party (your software) to the broker, please use your company or brand name. |
| USER | - | Your username at the FarmMaps broker. |
| PWD | - | Your password at the FarmMaps broker. |
| HOST | `farmmaps.awtest.nl` | The address of the FarmMaps broker. |
| PORT | `1883` | The port number of the FarmMaps broker. |
| KEEPALIVE | `60` | Number of seconds to maintain the connection, even if no messages are published.
| TOPIC | `trekkerdata/sensors` | The topic to publish the messages to at the broker. | 

All settings are required.

For preparing the message, we'll use the following settings:

| Parameter | Default | Description |
| :---: | :---------------------------------------------------------------: | --- |
| META_DATA_URL | - | HTTP adress where a JSON metadata file for the messages is provided (by the data source). |
| BASE_DEV_URN | `urn:dev:<company_name>:<device_serial_number>` | To identify each unique device from different parties, we add a code to each message composed of the company name and serial number. Farmmaps uses this code to link devices and data to their owners. |

All settings are required except for the `META_DATA_URL`, this parameter is optional.


### Example: Pushing Live Tractor Data
To create messages and connect to the broker we will need the following settings: 


## [FAQ](#faq) 
#### How do I install the required Python modules?
This tutorial requires the `paho-mqtt` and `protobuf` modules.
These can be installed with the following commands:  

**Windows command prompt:**
```
py -m pip install 'paho-mqtt==1.5.0'
py -m pip install 'protobuf==3.11.0'
```
**Linux terminal:**
```
python3 -m pip install 'paho-mqtt==1.5.0
python3 -m pip install 'protobuf==3.11.0'
```

#### What is a message queue?
A message queue is commonly used to make software programs able to send messages between eachother, and thereby making it easy for data to flow from one program into another. There are many variants of message queues, some popular names are Apache Kafka, MQTT and RabbitMQ.

In message queue systems there is usually one central "hub" called the broker or **"message broker"**. This broker holds all the messages. Usually, the messages are organised in groups called **"topics"**.

Now, there are two things that external services can do. 
* An external service (like a sensor) could **publish a message to a certain topic**. For example, a temperature sensor would publish the temperature at a specific time and location to the "temperatureMeasurements" topic. 
* An external service can **"subscribe" to this topic** by connecting to the broker. This service will then recieve every temperature measurement.

When the subscribed service temporarily disconnects from the broker, it will not recieve any messages, but the messages will remain stored at the broker. Depending on configuration, messages will be kept longer or shorter, or be deleted after they reach the subscribers but the intent is always to ensure the messages get from the **publisher** to the **subscriber**.

Setting communication between applications up like this makes things a lot more flexible than connecting systems directly and provides a central point for management of all communication.

#### What is protobuf?
To quote the Google Documentation: 

> Protocol buffers are Google's language-neutral, platform-neutral, extensible mechanism for serializing structured data – think XML, but smaller, faster, and simpler. 
> You define how you want your data to be structured once, then you can use special generated source code to easily write and read your structured data to and from a variety of data streams and using a variety of languages.

More information can be found in the [protobuf documentation](https://developers.google.com/protocol-buffers/docs/overview).