Documentatie/Workflow.md

FarmMaps Workflow

This page explains how FarmMaps processes incoming data so that it can be used in the FarmMaps application, and data becomes easily retrievable through the interface. Depending on the type of data, the system might process the data differently, how this works is explained below.

General workflow

As mentioned earlier, the REST API is the primary API endpoint. The workflow starts with registering a new upload, and uploading the file in chunks. Once a file has been uploaded succesfully through the REST API, FarmMaps will automatically start to process the file step by step.

In the processing workflow, there are 3 common, not dependent on the type of file. After the common steps, the workflow starts tasks for the specific type of file. So, when a file has been uploaded completely, FarmMaps will:

• Start a stitching task to reasemble the chunks back into a single file.
• Start an identification task to identify the type of file, and identify what the next processing step should be.
• Start the file-type specific workflow, starting with the first task defined in the file-type specific workflow.

Each time a task is started, a separate process is created called a worker. This worker then executes the task and reports back to the main FarmMaps workflow once it has completed. The main workflow will then start the next task untill all processing steps defined for that specific type of file are completed.

Once the primary processing is completed, the data is available for viewing through the FarmMaps GUI. Also, other tasks can be started that use the pre-processed data. These are tasks such as generating a task map, or creating derived datasets from the pre-processed data.

Stitching task

Because files are uploaded in chunks in FarmMaps, these chunks need to re-assembled into the original file. This task is started automatically.

Identification task

Once a file is re-assembled, FarmMaps will try to identify the file. It does so by looking at the file extension (i.e. .csv, .shp, .tif), the binary structure, and the content structure. Once it recognizes the type of file that was uploaded, FarmMaps stores this information as metadata in the database, with a reference to the file. The system can then look up the correct file specific workflow, and related tasks.

File specific task

Once the file has been identified and the correct file-type specific workflow is known, FarmMaps will start the with the first task listed in that workflow. It uses the resulting file of the stitching task, and depending on the type of task generally creates a new file with the converted data.

A good example would be a .csv file that was uploaded, that would be converted to a .shp or shapefile.
Once the data conversion has completed, the database is updated to point to the new file instead of the original .csv. The system does however keep track of the previous version and also keeps this in store, so the same data can easily be retrieved in different formats or versions. When the next file-specific task starts, this step repeats and another "version" of the same data is created.