44 lines
3.1 KiB
Markdown
44 lines
3.1 KiB
Markdown
|
# FarmMaps Workflow
|
||
|
|
This page explains how FarmMaps processes incoming data so that it can be used in the FarmMaps application, and data becomes easily retrievable through the interface.
|
||
|
|
Depending on the type of data, the system might process the data differently, how this works is explained below.
|
||
|
|
|
||
|
|
## General workflow
|
||
|
|
As mentioned earlier, the REST API is the primary API endpoint.
|
||
|
|
The workflow starts with registering a new upload, and uploading the file in chunks.
|
||
|
|
Once a file has been uploaded succesfully through the REST API, FarmMaps will automatically start to process the file step by step.
|
||
|
|
|
||
|
|
In the processing workflow, there are 3 common, not dependent on the type of file.
|
||
|
|
After the common steps, the workflow starts tasks for the specific type of file.
|
||
|
|
So, when a file has been uploaded completely, FarmMaps will:
|
||
|
|
|
||
|
|
* Start a stitching task to reasemble the chunks back into a single file.
|
||
|
|
* Start an identification task to identify the type of file, and identify what the next processing step should be.
|
||
|
|
* Start the file-type specific workflow, starting with the first task defined in the file-type specific workflow.
|
||
|
|
|
||
|
|
Each time a task is started, a separate process is created called a worker.
|
||
|
|
This worker then executes the task and reports back to the main FarmMaps workflow once it has completed.
|
||
|
|
The main workflow will then start the next task untill all processing steps defined for that specific type of file are completed.
|
||
|
|
|
||
|
|
Once the primary processing is completed, the data is available for viewing through the FarmMaps GUI.
|
||
|
|
Also, other tasks can be started that use the pre-processed data.
|
||
|
|
These are tasks such as generating a task map, or creating derived datasets from the pre-processed data.
|
||
|
|
|
||
|
|
## Stitching task
|
||
|
|
Because files are uploaded in chunks in FarmMaps, these chunks need to re-assembled into the original file.
|
||
|
|
This task is started automatically.
|
||
|
|
|
||
|
|
## Identification task
|
||
|
|
Once a file is re-assembled, FarmMaps will try to identify the file.
|
||
|
|
It does so by looking at the file extension (i.e. `.csv`, `.shp`, `.tif`), the binary structure, and the content structure.
|
||
|
|
Once it recognizes the type of file that was uploaded, FarmMaps stores this information as metadata in the database, with a reference to the file.
|
||
|
|
The system can then look up the correct file specific workflow, and related tasks.
|
||
|
|
|
||
|
|
### File specific task
|
||
|
|
Once the file has been identified and the correct file-type specific workflow is known, FarmMaps will start the with the first task listed in that workflow.
|
||
|
|
It uses the resulting file of the stitching task, and depending on the type of task generally creates a new file with the converted data.
|
||
|
|
|
||
|
|
A good example would be a `.csv` file that was uploaded, that would be converted to a `.shp` or shapefile.
|
||
|
|
Once the data conversion has completed, the database is updated to point to the new file instead of the original `.csv`.
|
||
|
|
The system does however keep track of the previous version and also keeps this in store, so the same data can easily be retrieved in different formats or versions.
|
||
|
|
When the next file-specific task starts, this step repeats and another "version" of the same data is created.
|