updated workflow and component pages, restructure index

Home.md

@@ -2,7 +2,11 @@
 ### The documentation is still work in progress and heavily subject to change! 
 This page provides an index of the available documentation for the FarmMaps platform.  
 
-### Getting started
+## Getting started
+The main starting point for Farmmaps API access is the REST API.
+To get started, follow the steps below. The workflow and high-level architecture FarmMaps are documented on the [Farmmaps Workflow](/wiki/Workflow) and [FarmMaps Main Components](/Main-components) pages.
+
+### Environments
 
 At the moment, FarmMaps provides two environments for development.
 
@@ -11,23 +15,26 @@ At the moment, FarmMaps provides two environments for development.
 
 When developing your application, using the **acceptation environment is highly recommended**.
 
-The process for obtaining access to API's varies slightly per API.
+### Credentials
-Each API documentation page describes this for its own API.
+To get access to the API, you need an akkerweb development account  (akkerweb development and farmmaps development use the same credentials). 
+These can be created at:
 
-### FarmMaps Main Concepts
+* https://awacc.nl (for the acceptation environment)
-* The datastore
+* https://awtest.nl (for the testing environment)
-* Endpoints
+
-* Workers
+### Authentication
-* Queues
+To start uploading files, you need to authenticate to the API. The REST API uses OpenID Connect as the protocol.
-* File-chains
+
-* [Viewer (FarmMapsLibs)](/wiki/FarmMapsLibs)
+[Todo: Basic explanation, based of "Authentication.md"]
+
+## How to build a FarmMaps App
+    - Accessing the API
+    - Uploading a file
+    - Checking the file processing status
+    - Creating a crop field
+    - Pushing data to MQTT
 
-### API Documentation
-FarmMaps provides 3 main endpoints to build services upon.
 
-* A file upload/download API (REST) for file uploads in chunks.
-* An FTP endpoint to upload files through FTP
-* [A message queue endpoint (MQTT)](/wiki/Message-Queue-Endpoint) to recieve streaming data and send and recieve triggering signals.
 
 ### Use-case examples
 * [VRAPoten-API](/wiki/VRAPoten-API)

Main-components.md

@@ -0,0 +1,86 @@
+# FarmMaps main components
+This page documents the main components of FarmMaps, to provide a general understanding of how data is stored, processed and how to access this data.
+
+### Main data structure
+Farmmaps uses a few basic objects to organise the data that is put into it.  
+The following hierarchy (starting with the highest parent element) can be used as a reference:
+
+|Element|Description|
+|----|---|
+|Farm| The highest level grouping element, the data is grouped by farm.|
+|Cropyear|All the crop's grown on the different fields of the farm are grouped in a cropyear.|
+|Plot|All different crop's grown in the different growing seasons/or years are grouped by plot.|
+|Cropfield| For each crop in a specific growing year/season and on a specific plot, a cropfield is created. All data related to that specific cropfield can then be grouped under this element. This can be data such as task maps, yield maps, NDVI maps or drone images.|
+
+Most agricultural data has a strong link to one or more of the elements above. A few examples:
+- An electrical conductivity map, which is related to the plot (electrical conductivity is a soil parameter).
+- A yield map, which is related to the cropfield (yields can vary per year, per crop etc.)
+
+
+### Data storage
+On the server, FarmMaps stores data in 2 ways. Raw data is stored in files, metadata is stored in the database.
+
+#### File storage (raw data).
+The primary method is through files. Each dataset is saved in a file on disk and given a unique identifier.
+Generally, this is the raw data, and this data does not contain so called "meta data". 
+
+For example, let's say we have a `.csv` file like this:
+```
+timestamp,latitude,longitude,altitude,speed,direction,engine torque
+24-11-2019 13:43:20,53.12694303,6.339856608,-25.649,0.099,235.3399,24
+24-11-2019 13:43:19,53.12694303,6.339856608,-25.361,0.693,25.8694,24
+24-11-2019 13:43:18,53.12694303,6.339856608,-23.017,0.793,38.8679,42
+24-11-2019 13:43:17,53.12694303,6.339856608,-25.41,0.92,60.8391,46
+24-11-2019 13:43:10,53.12694303,6.339856608,-11.667,0.029,139.1908,50
+24-11-2019 13:43:09,53.12694303,6.339856608,-11.558,0.043,198.0093,45
+24-11-2019 13:43:08,53.12694303,6.339856608,-11.426,0.034,183.7946,23
+24-11-2019 13:43:07,53.12694303,6.339856608,-11.243,0.013,144.7128,35
+24-11-2019 13:43:06,53.12694303,6.339856608,-11.248,0.022,135.7003,29
+```
+
+#### Database storage (meta-data)
+For each file in the file storage, a database record exists with a reference to the file.
+The database record contains the metadata of the file, so files can easily be found by the system. 
+This metadata describes how the data in the raw file should be interpreted.
+
+While the file example above might look easy to understand, it becomes a little bit unclear what it actually means when we take a better look at it.
+The first line specifies the column headers:
+```
+timestamp,latitude,longitude,altitude,speed,direction,engine torque
+```
+While it gives some information, this information is hard to relate to other data because some information is missing.
+When we look at data in the `speed` column, we can se that it changes, but we do not know how it was measured.
+Useful metadata here would be the unit of measurement, for example `km/hour` or `meters/second`.
+While FarmMaps does currently not allow you to specify this by hand as a user, it is important to realise metadata might need to be provided to have the data properly processed by FarmMaps.
+
+### Endpoints
+To upload and download data to and from FarmMaps, two API endpoints are provided.
+The first and primary API is the REST API for uploading an downloading files.   
+The second endpoint is the MQTT API, for pushing more stream like data.
+For the time being, we recommend using the REST API as your primary choice.
+
+#### REST Endpoint
+The REST API is the primary API for FarmMaps and allows you to:
+* Upload data files
+* Download data files
+* Run a processing task (i.e. convert a file into another format or similar)
+* Run other tasks (create a taskmap, create cropfield etc.)
+* Monitor the progress of the processing tasks
+
+Several how-to's can be found in the [examples folder][/wiki/examples/] 
+There's also swagger based reference of the [REST API](https://farmmaps.awacc.nl/swagger/index.html)
+
+#### MQTT Endpoint
+The MQTT Endpoint provides a way to input more stream like data like a live GPS location, or live temperature measurement to FarmMaps.
+This endpoint is still under development. If you would like to build on top of this endpoint, please contact the FarmMaps development team.
+As a reference, please also see [Sending data to MQTT](/wiki/Message-Queue-Endpoint).
+
+### Data processing infrastructure
+To process the data that is provided through the endpoints, FarmMaps uses queues and workers to manage the execution of these tasks.
+How this works is described in [the workflow article](/wiki/Workflow)
+
+### Viewer
+To visualize all data in the FarmMaps backend, FarmMaps integrates it's own viewer.
+This viewer is available as an open source project and can be found in the [FarmMapsLibs Repository](https://git.akkerweb.nl/FarmMaps/FarmMapsLib).
+There's also a separate documentation page which can be found [here](/wiki/FarmMapsLibs). [under development]
+

Message-Queue-Endpoint.md

@@ -1,4 +1,4 @@
-## Message Queue Endpoint
+## Sending data to the MQTT endpoint
 This documentation page provides the basics on how to access the MQTT endpoint of FarmMaps, using Python code for examples.  
 The same workflow can be followed for different languages, as long as there's a MQTT client and a Protobuf library for the language you are using.  
 FarmMaps provides an MQTT endpoint to allow your application to send signals to FarmMaps, and listen for events.  

Workflow.md

@@ -0,0 +1,43 @@
+# FarmMaps Workflow
+This page explains how FarmMaps processes incoming data so that it can be used in the FarmMaps application, and data becomes easily retrievable through the interface.
+Depending on the type of data, the system might process the data differently, how this works is explained below.
+
+## General workflow
+As mentioned earlier, the REST API is the primary API endpoint.
+The workflow starts with registering a new upload, and uploading the file in chunks.
+Once a file has been uploaded succesfully through the REST API, FarmMaps will automatically start to process the file step by step.
+
+In the processing workflow, there are 3 common, not dependent on the type of file.
+After the common steps, the workflow starts tasks for the specific type of file.
+So, when a file has been uploaded completely, FarmMaps will:
+
+* Start a stitching task to reasemble the chunks back into a single file.
+* Start an identification task to identify the type of file, and identify what the next processing step should be.
+* Start the file-type specific workflow, starting with the first task defined in the file-type specific workflow.
+
+Each time a task is started, a separate process is created called a worker.
+This worker then executes the task and reports back to the main FarmMaps workflow once it has completed.
+The main workflow will then start the next task untill all processing steps defined for that specific type of file are completed.
+
+Once the primary processing is completed, the data is available for viewing through the FarmMaps GUI.
+Also, other tasks can be started that use the pre-processed data.
+These are tasks such as generating a task map, or creating derived datasets from the pre-processed data.
+
+## Stitching task
+Because files are uploaded in chunks in FarmMaps, these chunks need to re-assembled into the original file.
+This task is started automatically.
+    
+## Identification task
+Once a file is re-assembled, FarmMaps will try to identify the file.
+It does so by looking at the file extension (i.e. `.csv`, `.shp`, `.tif`), the binary structure, and the content structure.
+Once it recognizes the type of file that was uploaded, FarmMaps stores this information as metadata in the database, with a reference to the file.
+The system can then look up the correct file specific workflow, and related tasks.
+
+### File specific task
+Once the file has been identified and the correct file-type specific workflow is known, FarmMaps will start the with the first task listed in that workflow.
+It uses the resulting file of the stitching task, and depending on the type of task generally creates a new file with the converted data.  
+
+A good example would be a `.csv` file that was uploaded, that would be converted to a `.shp` or shapefile.  
+Once the data conversion has completed, the database is updated to point to the new file instead of the original `.csv`.
+The system does however keep track of the previous version and also keeps this in store, so the same data can easily be retrieved in different formats or versions.
+When the next file-specific task starts, this step repeats and another "version" of the same data is created.

examples/Home.md

@@ -0,0 +1 @@
+### Examples