FarmMaps main components

This page documents the main components of FarmMaps, to provide a general understanding of how data is stored, processed and how to access this data.

Main data structure

Farmmaps uses a few basic objects to organise the data that is put into it.
The following hierarchy (starting with the highest parent element) can be used as a reference:

Element	Description
Farm	The highest level grouping element, the data is grouped by farm.
Cropyear	All the crop's grown on the different fields of the farm are grouped in a cropyear.
Plot	All different crop's grown in the different growing seasons/or years are grouped by plot.
Cropfield	For each crop in a specific growing year/season and on a specific plot, a cropfield is created. All data related to that specific cropfield can then be grouped under this element. This can be data such as task maps, yield maps, NDVI maps or drone images.

Most agricultural data has a strong link to one or more of the elements above. A few examples:

An electrical conductivity map, which is related to the plot (electrical conductivity is a soil parameter).
A yield map, which is related to the cropfield (yields can vary per year, per crop etc.)

Data storage

On the server, FarmMaps stores data in 2 ways. Raw data is stored in files, metadata is stored in the database.

File storage (raw data).

The primary method is through files. Each dataset is saved in a file on disk and given a unique identifier. Generally, this is the raw data, and this data does not contain so called "meta data".

For example, let's say we have a .csv file like this:

timestamp,latitude,longitude,altitude,speed,direction,engine torque
24-11-2019 13:43:20,53.12694303,6.339856608,-25.649,0.099,235.3399,24
24-11-2019 13:43:19,53.12694303,6.339856608,-25.361,0.693,25.8694,24
24-11-2019 13:43:18,53.12694303,6.339856608,-23.017,0.793,38.8679,42
24-11-2019 13:43:17,53.12694303,6.339856608,-25.41,0.92,60.8391,46
24-11-2019 13:43:10,53.12694303,6.339856608,-11.667,0.029,139.1908,50
24-11-2019 13:43:09,53.12694303,6.339856608,-11.558,0.043,198.0093,45
24-11-2019 13:43:08,53.12694303,6.339856608,-11.426,0.034,183.7946,23
24-11-2019 13:43:07,53.12694303,6.339856608,-11.243,0.013,144.7128,35
24-11-2019 13:43:06,53.12694303,6.339856608,-11.248,0.022,135.7003,29

Database storage (meta-data)

For each file in the file storage, a database record exists with a reference to the file. The database record contains the metadata of the file, so files can easily be found by the system. This metadata describes how the data in the raw file should be interpreted.

While the file example above might look easy to understand, it becomes a little bit unclear what it actually means when we take a better look at it. The first line specifies the column headers:

timestamp,latitude,longitude,altitude,speed,direction,engine torque

While it gives some information, this information is hard to relate to other data because some information is missing. When we look at data in the speed column, we can se that it changes, but we do not know how it was measured. Useful metadata here would be the unit of measurement, for example km/hour or meters/second. While FarmMaps does currently not allow you to specify this by hand as a user, it is important to realise metadata might need to be provided to have the data properly processed by FarmMaps.

Endpoints

To upload and download data to and from FarmMaps, two API endpoints are provided. The first and primary API is the REST API for uploading an downloading files.
The second endpoint is the MQTT API, for pushing more stream like data. For the time being, we recommend using the REST API as your primary choice.

REST Endpoint

The REST API is the primary API for FarmMaps and allows you to:

Upload data files
Download data files
Run a processing task (i.e. convert a file into another format or similar)
Run other tasks (create a taskmap, create cropfield etc.)
Monitor the progress of the processing tasks

Several how-to's can be found in the [examples folder][/wiki/examples/] There's also swagger based reference of the REST API

MQTT Endpoint

The MQTT Endpoint provides a way to input more stream like data like a live GPS location, or live temperature measurement to FarmMaps. This endpoint is still under development. If you would like to build on top of this endpoint, please contact the FarmMaps development team. As a reference, please also see Sending data to MQTT.

Data processing infrastructure

To process the data that is provided through the endpoints, FarmMaps uses queues and workers to manage the execution of these tasks. How this works is described in the workflow article

Viewer

To visualize all data in the FarmMaps backend, FarmMaps integrates it's own viewer. This viewer is available as an open source project and can be found in the FarmMapsLibs Repository. There's also a separate documentation page which can be found here. [under development]

5.1 KiB Raw Blame History