From 01db9fa14f8f4aacc961ee56ddd56ecc7cb0e1b9 Mon Sep 17 00:00:00 2001 From: auke Date: Sat, 25 Apr 2020 17:13:54 +0200 Subject: [PATCH] Add Running tasks article and various corrections --- Create-a-cropfield.md | 215 ++++++++++++++++++++++++++++++++++ Home.md | 6 +- Running-tasks.md | 265 ++++++++++++++++++++++++++++++++++++++++++ Upload-a-file.md | 5 +- 4 files changed, 486 insertions(+), 5 deletions(-) create mode 100644 Create-a-cropfield.md create mode 100644 Running-tasks.md diff --git a/Create-a-cropfield.md b/Create-a-cropfield.md new file mode 100644 index 0000000..7a36d1d --- /dev/null +++ b/Create-a-cropfield.md @@ -0,0 +1,215 @@ +## Creating a cropfield +This page explains what a cropfield is, and how to create one through the API. + +Agricultural datasets generally refer to a specif crop, at a specific location at a specific time. +Therefore, we need to create a "Cropfield" item, to define this location and a timeframe our data relates to. +A cropfield provides a convienient way to group all data for a crop grown on a specific plot during a season. +Farmmaps also uses the cropfield to collect and prepare context data such as weather, satelite imagery etc. + +To create a cropfield we: +* Get the parent folder to place the cropfield under +* Create a folder under this element (optional) +* Create the cropfield + +**Prerequisites** +To create a cropfield we need: + +* an acces token +* coordinates of the cropfield (what points define the plot contour) +* a startdate and an end date (what timeframe does this data aply to) + +### Get parent element +First, we'll get the parent code needed to place the cropfield in the hierarchy. + +**Request** +Replace `` with your actual token. +```http +GET https://farmmaps.awacc.nl/api/v1/folders/my_drive? HTTP/1.1 +Host: farmmaps.awacc.nl +Accept: application/json +Authorization: Bearer +``` + +**Response** +The response will be something similar to: + +```http +HTTP/1.1 200 OK +Server: nginx/1.14.0 (Ubuntu) +Date: Tue, 21 Apr 2020 09:57:07 GMT +Content-Type: application/json; charset=utf-8 +Transfer-Encoding: chunked +Connection: keep-alive +Cache-Control: no-store,no-cache +Pragma: no-cache +Content-Encoding: br +Vary: Accept-Encoding + +{ + "url":"/api/v1/folders/f25d8765a1cd407cb235961c73c268cf:USER_FILES", + "code":"f25d8765a1cd407cb235961c73c268cf:USER_FILES", + "name":"My Drive", + "created":"2019-09-25T19:39:33.841835", + "updated":"2019-09-25T19:39:33.841835", + "itemType":"ROOT_FOLDER", + "size":0, + "state":0, + "thumbnail":false +} +``` +So the `parentcode` we need is **"f25d8765a1cd407cb235961c73c268cf:USER_FILES"** +You can also find the code of a folder by browsing the folders in your FarmMaps account through the web interface. +The item code is the last part of the URL. + +### Create folder +Depending on how you want to organise things, you might choose to make a separate folder for the cropfield. +A folder can be created by creating an item with itemType `FOLDER`. + +> **Note:** At the moment, subfolders can not be created directly under the root folder, create a folder through the web interface first. + +**Request** +```http +POST /api/v1/items HTTP/1.1 +Accept: application/json +Authorization: Bearer +Content-Type: application/json +Cache-Control: no-cache +Host: farmmaps.awacc.nl +Accept-Encoding: gzip, deflate, br +Connection: keep-alive +Content-Length: 210 + +{ + "parentCode": "f25d8765a1cd407cb235961c73c268cf:USER_FILES", + "itemType": "FOLDER", + "name": "My_new_folder", + "data": {}, + "dataDate": "2020-04-24T09:59:18.893Z", + "geometry": {}, + "tags": [ "string" ] +} +``` + +When the folder is created successfully we should recieve something like this: + +**Response** +```http +{ + "parentCode": "string", + "geometry": {}, + "data": {}, + "tags": [ + "string" + ], + "url": "string", + "code": "string", + "name": "My_new_folder", + "created": "2019-12-18T10:16:21.455Z", + "updated": "2019-12-18T10:16:21.455Z", + "dataDate": "2019-12-18T10:16:21.455Z", + "itemType": "string", + "sourceTask": "string", + "size": 0, + "state": 0, + "thumbnail": true +} +``` + +### Create the cropfield +Now we can create the cropfield. We do so by setting `"itemType": "vnd.farmmaps.itemtype.cropfield"` and adding data to the geometry parameter. +Generally, a cropfield contour would be defined as a polygon, so we add `"type":"Polygon" and all the coordinates. +We also need to specify a `dataDate` and a `dataEndDate` to indicate the the timeframe of the growing season. +The `dataDate` needs to be before `dataEndDate` and cannot be on the same day. + +**Request** + +```http +POST /api/v1/items HTTP/1.1 +Accept: application/json +Authorization: Bearer +Content-Type: application/json +Cache-Control: no-cache +Host: farmmaps.awacc.nl +Accept-Encoding: gzip, deflate, br +Connection: keep-alive +Content-Length: 916 + +{ + "parentCode": "6601a06d812b40f9830c9fe4e63b1944", + "itemType": "vnd.farmmaps.itemtype.cropfield", + "name": "cropfield for VRA", + "dataDate": "2019-1-18T10:16:21.455Z", + "dataEndDate": "2019-12-18T10:16:21.455Z", + "data": {}, + "geometry": {"type":"Polygon","coordinates":[[[6.09942873984307,53.070025028087],[6.09992507404607,53.0705617890585],[6.10036959220086,53.0710679529031],[6.10065149010421,53.0714062774307],[6.10087493644271,53.0716712354474],[6.10091082982487,53.0716936039203],[6.10165087441291,53.0712041549161],[6.10204994718318,53.0709349338005],[6.10263143118855,53.0705789370018],[6.10311578125011,53.0702657538294],[6.10331686552072,53.0701314102389],[6.103326530575,53.070119463569],[6.10309137950343,53.0699829669055],[6.10184241586523,53.0692902201371],[6.10168497998891,53.0691984306747],[6.10092987659869,53.0694894453514],[6.09942873984307,53.070025028087]]]} +} +``` + +**Response** +When the cropfield is created, it should be visible through the FarmMaps web interface (under "My Drive" in the respective parent folder). +We should have the following response: + +```http +HTTP/1.1 201 Created +Server: nginx/1.14.0 (Ubuntu) +Date: Fri, 24 Apr 2020 08:56:45 GMT +Content-Type: application/json; charset=utf-8 +Transfer-Encoding: chunked +Connection: keep-alive +Cache-Control: no-store,no-cache +Pragma: no-cache +Content-Encoding: br +Location: /api/v1/items/8ecbaa2d85d5484db7f16b281b7cd013 +Vary: Accept-Encoding + +{ + "parentCode":"6601a06d812b40f9830c9fe4e63b1944", + "geometry": { + "type":"Polygon", + "coordinates":[[ + [6.09942873984307,53.070025028087], + [6.09992507404607,53.0705617890585], + [6.10036959220086,53.0710679529031], + [6.10065149010421,53.0714062774307], + [6.10087493644271,53.0716712354474], + [6.10091082982487,53.0716936039203], + [6.10165087441291,53.0712041549161], + [6.10204994718318,53.0709349338005], + [6.10263143118855,53.0705789370018], + [6.10311578125011,53.0702657538294], + [6.10331686552072,53.0701314102389], + [6.103326530575,53.070119463569], + [6.10309137950343,53.0699829669055], + [6.10184241586523,53.0692902201371], + [6.10168497998891,53.0691984306747], + [6.10092987659869,53.0694894453514], + [6.09942873984307,53.070025028087]] + ]}, + "data":{}, + "tags":[], + "isEditable":false, + "url":"/api/v1/items/8ecbaa2d85d5484db7f16b281b7cd013", + "code":"8ecbaa2d85d5484db7f16b281b7cd013", + "name":"cropfield for VRA", + "created":"2020-04-24T08:56:45.3727807Z", + "updated":"2020-04-24T08:56:45.3727807Z", + "dataDate":"2019-01-18T10:16:21.455Z", + "itemType":"vnd.farmmaps.itemtype.cropfield", + "size":0, + "state":0, + "thumbnail":false + } +``` + +**Troubleshooting** +|Status code|Description| +|---|---| +|201|Cropfield created successfully| +|401|Error: not authenticated| +|403|Error: No write permissions in parent item| +|404|Error: Parent item not found| + +Now that the cropfield has been created, we can start the processing by running a task. +This will collext all context data for the cropfield. +We need the ItemCode of the cropfield to start the task so keep this at hand. + diff --git a/Home.md b/Home.md index 12e643b..1b2fb09 100644 --- a/Home.md +++ b/Home.md @@ -58,14 +58,14 @@ For testing purposes, [Postman](https://www.postman.com) can be used to perform We provide the following guides: * [Uploading a file](/wiki/Upload-a-file) -* Creating a cropfield -* Running a task +* [Creating a cropfield](/wiki/Create-a-cropfield) +* [Running a task](/wiki/Running-tasks) Generally, tasks can be run in the same way. However, each specific task has it's own inputs and properties. How these work can be found in the use task examples below. -### Taks examples +### Task examples * [VRAPoten-API](/wiki/VRAPoten-API) * [VRANbs-API](/wiki/VRANbs-API) * [VRAHerbicide-API](/wiki/VRAHerbicide-API) diff --git a/Running-tasks.md b/Running-tasks.md new file mode 100644 index 0000000..2b612aa --- /dev/null +++ b/Running-tasks.md @@ -0,0 +1,265 @@ +# Running a task +This page documents how to run a task from the FarmMaps API. +Once you have uploaded some data and made sure the relevant cropfield is available, you can now instruct the system to start processing the data. +Processing data is done by running tasks. There are many different types of tasks to acomplish many different things such as: + +* Identifying the type of file to start automatic processing +* Converting a CSV file to a Shapefile +* Filtering data for valid values +* Creating a taskmap of input data. + + +## Workflow task. +There is one special type of task, the `workflow` task. +When an item (i.e. file-upload, folder, cropfield etc.) is created there are certain processing tasks that can be done by "default". +As an example, for an uploaded file these tasks are: + +- Re-assembling the part-chunks back into a single file +- Identifying the type of file, and how it should be processed. + +For a cropfield there are slightly different tasks such as: + +- retrieve AHN mapping data (cut-out from public height maps) +- retrieve soil compaction data (based on a public map) +- retrieve shadow mapping data (based on satellite imagery) + +These steps are all defined in the workflow for each `itemType`. +So, once an item has been created, we can run the `workflow` task to execute the predefined tasks automatically. + +### Example: Running the workflow task for a cropfield. +We run a task by adding the `taskType` to the body of the request, with `vnd.farmmaps.task.workflow` as the type of task. +The item that the task will be executed on is specified in the url by it's itemcode. Other types of tasks can be run by changing the taskType. + +**Request** +```http +POST /api/v1/items/98a480ad8d7444a8a10ef547cd8594eb/tasks HTTP/1.1 +Authorization: Bearer +Accept: application/json +Content-Type: application/json +Cache-Control: no-cache +Host: farmmaps.awacc.nl +Accept-Encoding: gzip, deflate, br +Connection: keep-alive +Content-Length: 49 + +{ + "taskType": "vnd.farmmaps.task.workflow" +} +``` + +**Response** +```http +HTTP/1.1 201 Created +Server: nginx/1.14.0 (Ubuntu) +Date: Sat, 25 Apr 2020 13:55:28 GMT +Content-Type: application/json; charset=utf-8 +Transfer-Encoding: chunked +Connection: keep-alive +Cache-Control: no-store,no-cache +Pragma: no-cache +Content-Encoding: br +Location: /api/v1/items/98a480ad8d7444a8a10ef547cd8594eb/tasks/52173fb5bf1146efa276ecfe1fda5f60 +Vary: Accept-Encoding + +{ + "code":"52173fb5bf1146efa276ecfe1fda5f60", + "taskType":"vnd.farmmaps.task.workflow", + "attributes":{} +} +``` + +**Troubleshooting** +|Status code|Description| +|---|---| +|201|Task created| +|400|Error: Tasktype not found| +|401|Error: Not authenticated| +|403|Error: No WRITE permissions in item| +|404|Error: Item not found| + +## Polling for task status +Now that the task was created, we can retrieve its status by performing a **GET** request on the same endpoint as used for starting task. +Note that the response contains a lot of tasks, and also multiple workflow tasks, indicating that multiple workflows were triggered. + +**Request** +```http +GET /api/v1/items/98a480ad8d7444a8a10ef547cd8594eb/tasks HTTP/1.1 +Accept: application/json +Authorization: Bearer +Cache-Control: no-cache +Host: farmmaps.awacc.nl +Accept-Encoding: gzip, deflate, br +Connection: keep-alive +``` + +**Response** +```http +HTTP/1.1 200 OK +Server: nginx/1.14.0 (Ubuntu) +Date: Sat, 25 Apr 2020 14:34:01 GMT +Content-Type: application/json; charset=utf-8 +Transfer-Encoding: chunked +Connection: keep-alive +Cache-Control: no-store,no-cache +Pragma: no-cache +Content-Encoding: br +Vary: Accept-Encoding + +[ + { + "taskType":"vnd.farmmaps.task.workflow", + "code":"52173fb5bf1146efa276ecfe1fda5f60", + "state":1, + "started":"2020-04-25T13:55:28.58133", + "finished":"2020-04-25T13:55:28.750055" + }, + { + "taskType":"vnd.farmmaps.task.vandersat", + "code":"0586f0f867014d55a90bfe6adf7416d5", + "state":1, + "started":"2020-04-25T14:00:36.221527", + "finished":"2020-04-25T14:02:18.314088" + }, + { "taskType":"vnd.farmmaps.task.watbal", + "code":"f33cfaff01644313b8de73dcfd42bc20", + "state":3, + "started":"2020-04-25T14:10:46.201482" + }, + { + "taskType":"vnd.farmmaps.task.workflow", + "code":"2b76caf17f6d4f5eb497f7ba7609a08c", + "state":1, + "started":"2020-04-25T13:55:41.278677", + "finished":"2020-04-25T13:55:41.442544" + }, + { + "taskType":"vnd.farmmaps.task.crprec", + "code":"0ddd635c244c47b8a182aa5f8dddb5db", + "state":1, + "started":"2020-04-25T13:55:33.609931", + "finished":"2020-04-25T13:55:34.464812" + }, + { + "taskType":"vnd.farmmaps.task.bofek", + "code":"e95910401dd8493d8483ebde56f8814c", + "state":1, + "started":"2020-04-25T13:55:33.135703", + "finished":"2020-04-25T13:55:34.620735" + }, + { + "taskType":"vnd.farmmaps.task.tipstar", + "code":"172f42ece9004013aaf2b6a56b716e39", + "state":1, + "started":"2020-04-25T13:55:33.903717", + "finished":"2020-04-25T13:55:34.636575" + }, + { + "taskType":"vnd.farmmaps.task.soilcompaction", + "code":"1667fce90d954b1b90d9997e5e642e8d", + "state":1, + "started":"2020-04-25T13:55:33.575171", + "finished":"2020-04-25T13:55:34.767101" + }, + { + "taskType":"vnd.farmmaps.task.ahn", + "code":"23b1e66325134e8cbd778d57865590f2", + "state":1, + "started":"2020-04-25T13:55:33.465586", + "finished":"2020-04-25T13:55:35.170389" + }, + { + "taskType":"vnd.farmmaps.task.tipstar", + "code":"80bcf30893614ad288fe8409cee5f9b1", + "state":1, + "started":"2020-04-25T13:55:36.893606", + "finished":"2020-04-25T13:55:37.331862" + }, + { + "taskType":"vnd.farmmaps.task.workflow", + "code":"c4c7d23b6e534a7e9b5dfb79847ea449", + "state":1, + "started":"2020-04-25T13:55:37.237456", + "finished":"2020-04-25T13:55:37.567148" + }, + { + "taskType":"vnd.farmmaps.task.trijntje", + "code":"e29539ed7a014dfcb739ad3d87298cc2", + "state":1, + "started":"2020-04-25T13:55:44.271105", + "finished":"2020-04-25T13:55:45.64174" + }, + { + "taskType":"vnd.farmmaps.task.satellite", + "code":"b5a549be975e4196972806157ee7f2cd", + "state":1, + "started":"2020-04-25T13:55:33.778997", + "finished":"2020-04-25T13:57:13.811563" + }, + { + "taskType":"vnd.farmmaps.task.shadow", + "code":"a1daf49cf08142c289825def78986027", + "state":1, + "started":"2020-04-25T13:55:33.153348", + "finished":"2020-04-25T13:56:14.590014" + } +] +``` + +**Troubleshooting** +|Status code|Description| +|---|---| +|401|Error: Not authenticated| +|403|Error: No READ permissions in parent item| +|404|Error: Parent item not found| + +### Polling a single task +Polling for a siingle task can be done by appending g the `ItemTaskCode` to the task URL: + +**Request** +```http +GET /api/v1/items/98a480ad8d7444a8a10ef547cd8594eb/tasks/a1daf49cf08142c289825def78986027 HTTP/1.1 +Accept: application/json +Authorization: Bearer +User-Agent: PostmanRuntime/7.24.1 +Cache-Control: no-cache +Postman-Token: dafc3b2a-35be-4f68-b398-ddab1f36864f +Host: farmmaps.awacc.nl +Accept-Encoding: gzip, deflate, br +Connection: keep-alive +``` + +**Response** +```http +HTTP/1.1 200 OK +Server: nginx/1.14.0 (Ubuntu) +Date: Sat, 25 Apr 2020 15:02:05 GMT +Content-Type: application/json; charset=utf-8 +Transfer-Encoding: chunked +Connection: keep-alive +Cache-Control: no-store,no-cache +Pragma: no-cache +Content-Encoding: br +Vary: Accept-Encoding + +{ + "taskType":"vnd.farmmaps.task.shadow", + "code":"a1daf49cf08142c289825def78986027", + "state":1, + "started":"2020-04-25T13:55:33.153348", + "finished":"2020-04-25T13:56:14.590014" +} +``` +**Troubleshooting** +|Status code|Description| +|---|---| +|401|Error: Not authenticated| +|403|Error: No READ permissions in parent item| +|404|Error: Parent item not found| + +## What's next? +Now that you know how to run a task, you can start running specialized tasks to create or process your own data. +Head over to the [Task Examples](/wiki/Home#task-examples) + + + + diff --git a/Upload-a-file.md b/Upload-a-file.md index b243ef7..36ed9e7 100644 --- a/Upload-a-file.md +++ b/Upload-a-file.md @@ -1,6 +1,7 @@ # Uploading a file This page documents how to upload a file through the FarmMaps API. -The FarmMaps file upload API follows the [Google Drive API](https://developers.google.com/drive/api/v3/manage-uploads#resumable) loosely. +The FarmMaps file upload API follows the [Google Drive API](https://developers.google.com/drive/api/v3/manage-uploads#resumable) loosely. +Files smaller than 2 MB are uploaded in a single part, larger files need to be split into parts of 1 MB. The workflow for uploading a file is as follows: * Register the file for upload. @@ -13,7 +14,7 @@ The workflow for uploading a file is as follows: ### Registering the upload Before a file can be uploaded it needs to be registered. Files smaller than 2 MB can be uploaded in one chunk. Bigger files need to be uploaded in chunks of 1 MB. The uploaded files are structured in a hierarchy, so we'll specify a `parentCode` to identify its parent. -We'll first retrieve this parentcode using a GET request. +We'll first retrieve this parentcode using a GET request to the "my_drive" (root)folder. **Request** Replace `` with your actual token.