FarmMaps
/
FarmMapsApiClient
11
0
Fork 3
Code Issues Pull Requests Releases Activity

Compare commits

base: FarmMaps:master
Branches Tags
FarmMaps:master
FarmMaps:fedde-patch-1
abel:master
..
compare: abel:master
Branches Tags
abel:master
FarmMaps:master
FarmMaps:fedde-patch-1

57 Commits
master ... master

Author SHA1 Message Date
tamara 6faa5104c8 added example with greenmanure 2024-05-02 15:57:32 +02:00
tamara efb4d6833f Included relevant codelist for dataOperations operationCode and status. 2024-04-11 11:31:56 +02:00
tamara 0a9a83a5a6 updated KPIdefinitions and todolist 2024-04-10 10:29:30 +02:00
tamara fe2bd98589 added target and threshold definitions + changed productionPurposeCode to be functional 2024-03-12 11:42:16 +01:00
Pepijn van Oort ffbb45894a write A1 yield & strawYield to output 
update definitions
update TODO.txt
 2024-03-11 14:35:03 +01:00
tamara 67014e20f5 updated names of output KPI B1 and B2 + added yieldStraw to output (non functional) 2024-03-07 10:37:27 +01:00
tamara 84f27db16e DataCropfieldCharacteristics as an array containing cropyield and possibly strawyield. If no strawyield is filled in then straw remains in field. 2024-02-27 15:44:07 +01:00
tamara d756f08b4c typos 2024-02-27 13:58:09 +01:00
tamara 77532fee9f add codelist 265. Empty result? 2024-02-27 13:55:51 +01:00
tamara 5d2789a806 updated KPI definitions 2024-02-19 15:07:26 +01:00
Pepijn van Oort dea4f102f5 C1 defs 2024-02-16 15:28:15 +01:00
Pepijn van Oort 39ee3263a7 bit of code simplification 2024-02-16 14:52:48 +01:00
Pepijn van Oort 619cb5e92c small updates on KPIdefinitions.csv & TODO.txt 2024-02-16 14:36:10 +01:00
tamara 5399fd66df added detailed output for KPI C1 effective organic matter supply 2024-02-16 12:58:00 +01:00
tamara 5d02370f56 details about N & P balance written to output file 2024-02-12 15:43:00 +01:00
Pepijn van Oort e4087adc77 add miliebelastingspunten (mbp) and checks on whether KPI task is finished or not yet (if not yet then wait and call again) 2023-11-28 15:16:44 +01:00
Pepijn van Oort 4761166472 new Task<Configuration> GetConfiguration 2023-11-28 15:12:51 +01:00
Pepijn van Oort 9d691dfed1 read & write KPI "E1" = milieu belastingspunten ("mbp") 2023-11-22 16:16:59 +01:00
Pepijn van Oort d616fd3dfc in KPIApplication add bofek to cropfield, because KPI E1 (milieubelastingspunten) only calculated if soil is known 2023-11-21 14:03:14 +01:00
Pepijn van Oort 3f66676914 specify how many seconds to wait for the KPItask to be completed 2023-11-15 16:49:46 +01:00
Pepijn van Oort aa18393102 two example inputs for testing garden pea (doperwt) for whole year of part of growing period 2023-11-08 13:51:30 +01:00
Pepijn van Oort 0b0ec8f767 StartDate & EndDate of cropfield. Is relevant for N fixation by leguminose crops where currently N fixation depends on day between these two dates. If not provided a whole year is assumed which leads to high N fixation 2023-11-08 13:50:37 +01:00
Pepijn van Oort aef8c7c446 update KPIdefinitions.csv & TODO.txt 2023-11-08 13:48:01 +01:00
Pepijn van Oort f1707c432f More testing & demo's in KPIInput.json: 
* pesticide
* effect of soiltype: yes, different target & threshold for nitrogen (OK)
* effect of productionpurpose: none?
* effect of location: different nitrogen balance due to different atmospheric deposition (OK)

And update TODO.txt
 2023-11-01 13:05:49 +01:00
Pepijn van Oort cfba86cf75 Get polygon area from geometry and give to operations if operation was on whole field 
Time tracker for how much time the KPI calculations take
 2023-11-01 13:02:54 +01:00
Pepijn van Oort da808362c3 Added codelist CL251: Codering teeltdoelen (CropProductionPurposeCode) 2023-11-01 12:32:05 +01:00
Pepijn van Oort 847ff75845 download codelist CL405, soil types 2023-10-24 13:47:58 +02:00
Pepijn van Oort 6d903c3e9b error catching and console input to ask which KPIinput file to use 2023-10-24 13:06:36 +02:00
Pepijn van Oort 6eb36b2b4f remove author name from code 2023-10-23 12:48:28 +02:00
Pepijn van Oort 7f3b87ae38 and of course also add & commit this one ... 2023-10-18 17:35:34 +02:00
Pepijn van Oort 90f285a0d1 update todo list 2023-10-18 17:33:55 +02:00
Pepijn van Oort 546ce131e4 also commit this file 2023-10-18 17:33:32 +02:00
Pepijn van Oort d8c2ecb88b new project to download latest version of editeelt codelists 2023-10-18 17:32:38 +02:00
Pepijn van Oort 2d588026e2 also write totalNapplied to csv file with KPI outputs per cropfield 2023-10-16 15:34:07 +02:00
Pepijn van Oort 1a9d5af5f6 don't write duplicate KPIOutput items to output csv 2023-10-16 13:55:12 +02:00
Pepijn van Oort 8771e47b28 TODO 2023-10-13 16:38:27 +02:00
Pepijn van Oort 4102ed628e farmmapsKPI receives minimum necessary input and writes relevant KPI output 2023-10-13 16:37:50 +02:00
Pepijn van Oort 1cc36422a4 start & end date & geometry added to operations item and to cropfield characteristics item 2023-10-11 17:00:25 +02:00
Pepijn van Oort fedd363075 in KPIApplication possible to (1) add an operation, e.g. applying fertilizer and (2) add a cropfield characteristic containing cropyield. 2023-10-04 17:21:05 +02:00
Pepijn van Oort f04cc239d5 Extra note to Clone the git repository to a local drive. DON'T clone to a network drive like "My Documents". Latter may cause authentication problems. 2023-10-04 10:43:44 +02:00
Pepijn van Oort 8a770a1762 handles existing cropfielditemcode 2023-06-27 14:08:27 +02:00
Abel Hoeven, van b99191b7b1 niewste versie 2023-06-27 12:58:36 +02:00
Pepijn van Oort 9be9eb3145 update TargetFramework of all projects to netcoreapp3.1. Except FarmmapsAPi & Secrets which remain 2.1 2023-06-13 12:05:30 +02:00
Abel Hoeven, van 641e09f833 Merge branch 'master' of https://git.akkerweb.nl/abel/FarmMapsApiClient_KB34_MAST 2023-05-02 10:55:43 +02:00
Abel Hoeven, van 8c5a0aed95 de niet verplichte inputs verwijderd van de input file. Wel wordt er dus de input file gebruikt die in de C:\Users\hoeve065\Documents\hoofdmap_werk\project_KB34MAST nutrienten DB\API04.03\FarmMapsApiClient_KB34_MAST\FarmmapsKPI\bin\Debug\netcoreapp3.1\KPIInput.json staat, en niet bij het KPI programma. 2023-05-02 10:55:24 +02:00
Abel Hoeven, van 59f9ef4743 nu kan er van de gegeven input een cropfield wordern gemaakt, en hier kunnen vervolgens KPIs over berekend worden. Volgende stap is het strippen van de KPIinput om te kijken wat het minimale is dat nodig is om de kpis te berekenen. verder nu eem aantal inputs die niet nodig waren verwijdertInput aangepast, 2023-04-25 16:33:57 +02:00
Abel Hoeven, van 8bb5c6f581 Input aangepast, nu kan er van de gegeven input een cropfield wordern gemaakt, en hier kunnen vervolgens KPIs over berekend worden. Volgende stap is het strippen van de KPIinput om te kijken wat het minimale is dat nodig is om de kpis te berekenen. 2023-04-25 16:14:43 +02:00
Abel Hoeven, van 7149086ce2 KPI items opslaan werkt. alle KPI items worden nu naar een JSON file in de dowload folder geschreven 2023-04-25 12:00:08 +02:00
abel 55f00da9d0 KPI items ophalen voor een cropfield werkt nu. wel dus een bestaand cropfield ingeven bij de settings in de BIN. Alle KPIs worden opgehaald na een minuut wachten. 2023-04-07 14:33:49 +02:00
abel 3d7bf186e7 de properties voor de KPI task worden nu correct doorgegeven denk ik 2023-04-06 16:11:37 +02:00
abel 8f8f857bf0 ook in in de andere KPIinout toegevoegd 2023-04-03 13:05:00 +02:00
abel b7bedf182c aggegrate KPI = false toegevoegd aan inputs 2023-04-03 13:03:10 +02:00
abel be22d631bc geupdate runKPIitemtask, helaas nog geen KPI data gevonden als child item 2023-03-31 11:13:51 +02:00
abel 7dda21dd77 probeersel om een runKPItask te maken 2023-03-31 11:02:06 +02:00
abel c79e0d062c Merge branch 'master' of https://git.akkerweb.nl/abel/FarmMapsApiClient_KB34_MAST 2023-03-22 14:48:34 +01:00
abel 32138bef80 blah test 2023-03-22 14:48:13 +01:00
Pepijn van Oort e66820bd87 test 2023-03-22 14:41:41 +01:00
35 changed files with 3057 additions and 585 deletions
Show Stats Expand all files Collapse all files

11
FarmmapsApi/Constants.cs
View File

@ -12,20 +12,27 @@ namespace FarmmapsApiSamples
public const string SHAPE_ITEMTYPE = "vnd.farmmaps.itemtype.shape";
public const string GEOJSON_ITEMTYPE = "vnd.farmmaps.itemtype.geojson";
public const string BLIGHT_ITEMTYPE = "vnd.farmmaps.itemtype.blight";
public const string CROPREC_ITEMTYPE = "vnd.farmmaps.itemtype.crprec.operation";
public const string CROPREC_ITEMTYPE = "vnd.farmmaps.itemtype.crprec";
public const string CROPOP_ITEMTYPE = "vnd.farmmaps.itemtype.crprec.operation";
public const string CROPCHAR_ITEMTYPE = "vnd.farmmaps.itemtype.edicrop.characteristic";
public const string CROPSCHEME_ITEMTYPE = "vnd.farmmaps.itemtype.croppingscheme"; // deze toegevoegd, misschien is het type van de KPI task wel een croppingscheme
//public const string KPI_ITEM = "vnd.farmmaps.itemtype.kpi.data"; //PO20231004: originally with .data
public const string KPICONTAINER_ITEM = "vnd.farmmaps.itemtype.kpi.data.container"; //PO20231004: originally without .container
public const string VRANBS_TASK = "vnd.farmmaps.task.vranbs";
public const string VRAHERBICIDE_TASK = "vnd.farmmaps.task.vraherbicide";
public const string VRAHAULMKILLING_TASK = "vnd.farmmaps.task.vrahaulmkilling";
public const string VRAPLANTING_TASK = "vnd.farmmaps.task.vrapoten";
public const string VRAZONERING_TASK = "vnd.farmmaps.task.vrazonering";
public const string SATELLITE_TASK = "vnd.farmmaps.task.sentinelhub"; //"vnd.farmmaps.task.satellite";
public const string SATELLITE_TASK = "vnd.farmmaps.task.sentinelhub";
public const string VANDERSAT_TASK = "vnd.farmmaps.task.vandersat";
public const string TASKMAP_TASK = "vnd.farmmaps.task.taskmap";
public const string WORKFLOW_TASK = "vnd.farmmaps.task.workflow";
public const string BOFEK_TASK = "vnd.farmmaps.task.bofek";
public const string CROPREC_TASK = "vnd.farmmaps.task.crprec";
public const string SHADOW_TASK = "vnd.farmmaps.task.shadow";
public const string AHN_TASK = "vnd.farmmaps.task.ahn";
public const string WATBAL_TASK = "vnd.farmmaps.task.watbal";
public const string KPI_TASK = "vnd.farmmaps.task.kpi"; // dus nieuwe taak om de KPIs te berekenen
}
}

2
FarmmapsApi/FarmmapsApi.csproj
View File

@ -2,6 +2,8 @@
<PropertyGroup>
<TargetFramework>netstandard2.1</TargetFramework>
<PlatformTarget>x64</PlatformTarget>
<Platforms>AnyCPU;x64</Platforms>
</PropertyGroup>
<ItemGroup>

4
FarmmapsApi/Services/FarmmapsApiService.cs
View File

@ -46,6 +46,10 @@ namespace FarmmapsApi.Services
}
}
public async Task<Configuration> GetConfiguration()
{
return _configuration;
}
public async Task AuthenticateAsync()
{
if (_httpClient.DefaultRequestHeaders.Authorization != null &&

141
FarmmapsApi/Services/GeneralService.cs
View File

@ -22,14 +22,20 @@ namespace FarmmapsApi.Services
}
public async Task<Item> CreateCropfieldItemAsync(string parentItemCode, string name, int year,
string fieldGeomJson, string data = "{}") {
var currentYear = new DateTime(year, 1, 1, 0, 0, 0, DateTimeKind.Utc);
string fieldGeomJson, string data = "{}", DateTime startDate = new DateTime(), DateTime endDate = new DateTime() )
{
//If user provides no startDate or endDate, then set startDate and endDate to full year provided through 'year'
if (startDate == new DateTime() || endDate == new DateTime())
{
startDate = new DateTime(year, 1, 1);
endDate = new DateTime(year, 12, 31);
}
var cropfieldItemRequest = new ItemRequest() {
ParentCode = parentItemCode,
ItemType = CROPFIELD_ITEMTYPE,
Name = name,
DataDate = currentYear,
DataEndDate = currentYear.AddYears(1).AddDays(-1),
DataDate = startDate,
DataEndDate = endDate,
Data = JObject.Parse(data),
Geometry = JObject.Parse(fieldGeomJson)
};
@ -37,6 +43,55 @@ namespace FarmmapsApi.Services
return await _farmmapsApiService.CreateItemAsync(cropfieldItemRequest);
}
public async Task<Item> CreateOperationItemAsync(string cropRecordingItemCode, int year,
string fieldGeomJson, string data = "{}", DateTime startDate = new DateTime(), DateTime endDate = new DateTime())
{
//If user provides no startDate or endDate, then set startDate and endDate to full year provided through 'year'
if (startDate == new DateTime() || endDate == new DateTime())
{
startDate = new DateTime(year, 1, 1);
endDate = new DateTime(year, 12, 31);
}
JObject jdata = JObject.Parse(data);
string name = string.Format($"CrpRec Operation, {jdata.GetValue("name")}");
ItemRequest operationItemRequest = new ItemRequest()
{
ParentCode = cropRecordingItemCode,
ItemType = CROPOP_ITEMTYPE,
Name = name,
DataDate = startDate,
DataEndDate = endDate,
Data = jdata,
Geometry = JObject.Parse(fieldGeomJson)
};
return await _farmmapsApiService.CreateItemAsync(operationItemRequest);
}
public async Task<Item> CreateCropfieldCharacteristicItemAsync(string cropfieldItemCode, int year,
string fieldGeomJson, string data = "{}", DateTime startDate = new DateTime(), DateTime endDate = new DateTime())
{
//If user provides no startDate or endDate, then set startDate and endDate to full year provided through 'year'
if (startDate == new DateTime() || endDate == new DateTime())
{
startDate = new DateTime(year, 1, 1);
endDate = new DateTime(year, 12, 31);
}
// not sure if here we also need to specify DataDate, DataEndDate & Geometry. Do it just in case
string name = "Cropfield characteristic";
ItemRequest cropfieldCharactericsticItemRequest = new ItemRequest()
{
ParentCode = cropfieldItemCode,
ItemType = CROPCHAR_ITEMTYPE,
Name = name,
DataDate = startDate,
DataEndDate = endDate,
Data = JObject.Parse(data),
Geometry = JObject.Parse(fieldGeomJson)
};
return await _farmmapsApiService.CreateItemAsync(cropfieldCharactericsticItemRequest);
}
public async Task<Item> UploadDataAsync(UserRoot root, string itemType, string filePath, string itemName, string geoJsonString = null)
{
var startUpload = DateTime.UtcNow.AddSeconds(-3);
@ -247,6 +302,39 @@ namespace FarmmapsApi.Services
return dataItem;
}
public async Task<Item> RunCropRecordingTask(Item cropfieldItem)
{
var cropRecordingRequest = new TaskRequest { TaskType = CROPREC_TASK };
string itemTaskCode = await _farmmapsApiService.QueueTaskAsync(cropfieldItem.Code, cropRecordingRequest);
await PollTask(TimeSpan.FromSeconds(5), async (tokenSource) => {
var itemTaskStatus = await _farmmapsApiService.GetTaskStatusAsync(cropfieldItem.Code, itemTaskCode);
_logger.LogInformation($"Waiting on RunCropRecordingTask; status: {itemTaskStatus.State}");
if (itemTaskStatus.IsFinished)
tokenSource.Cancel();
});
var itemTask = await _farmmapsApiService.GetTaskStatusAsync(cropfieldItem.Code, itemTaskCode);
if (itemTask.State == ItemTaskState.Error)
{
_logger.LogError($"Something went wrong with task execution: {itemTask.Message}");
return null;
}
//the CropRecording data is a child of the cropfield
var itemName = "Crprec";
var cropRecordingItem = await FindChildItemAsync(cropfieldItem.Code,
CROPREC_ITEMTYPE, itemName);
if (cropRecordingItem == null)
{
_logger.LogError("Could not find the CropRecording data as a child item under the cropfield");
return null;
}
return cropRecordingItem;
}
public async Task<Item> RunBofekTask(Item cropfieldItem) {
var taskmapRequest = new TaskRequest { TaskType = BOFEK_TASK };
@ -277,6 +365,51 @@ namespace FarmmapsApi.Services
return bofekItem;
}
public async Task<List<Item>> GetKpiItemsForCropField(Item cropfieldItem, int waitsecs = 6)
{
TaskRequest kpiRequest = new TaskRequest { TaskType = KPI_TASK };
kpiRequest.attributes["processAggregateKpi"] = "false";
int year = cropfieldItem.DataDate.Value.Year;
kpiRequest.attributes["year"] = year.ToString();
int ms = waitsecs * 1000;
string itemTaskCode = await _farmmapsApiService.QueueTaskAsync(cropfieldItem.Code, kpiRequest);
await PollTask(TimeSpan.FromSeconds(5), async (tokenSource) => {
var itemTaskStatus = await _farmmapsApiService.GetTaskStatusAsync(cropfieldItem.Code, itemTaskCode);
_logger.LogInformation($"Waiting on retreiving KPI data; status: {itemTaskStatus.State}");
if (itemTaskStatus.IsFinished)
tokenSource.Cancel();
});
var itemTask = await _farmmapsApiService.GetTaskStatusAsync(cropfieldItem.Code, itemTaskCode);
if (itemTask.State == ItemTaskState.Error)
{
_logger.LogError($"Something went wrong with task execution: {itemTask.Message}");
return null;
}
_logger.LogInformation($"Taking {waitsecs} seconds to look up all KPI items. The longer we wait the more KPI's we get ..." +
$"");
await Task.Delay(ms); // wait 60000ms = 60 secs for task to be completed
//After the task is completed we have 1 kpiContainerItem with a code and with no data
//Because the data are in the children of this kpiContainerItem. The container will have a list of children each with an "id" aand data,
//The ids' are "a1", "b1", "b2", "c1","d1","d3","d5"
//with following meanings:
//| A1 | Opbrengst | Yield |
//| B1 | Stikstofoverschot | Nitrogen surplus |
//| B2 | Fosfaatoverschot | Phosphate surplus |
//| C1 | Effectieve Organischestof aanvoer| Effective Organic Matter supply |
//| D1 | Gebruik bestrijdingsmiddelen | Use of pesticides|
//| D3 | Gewasdiversiteit(randdichtheid) | Crop diversity(edge density) |
//| D5 | Percentage rustgewassen | Percentage of rest crops |
List <Item> kpiContainerItem = await _farmmapsApiService.GetItemChildrenAsync(cropfieldItem.Code, KPICONTAINER_ITEM);
string kpiContainerItemCode = kpiContainerItem[0].Code;
List<Item> kpiItems = await _farmmapsApiService.GetItemChildrenAsync(kpiContainerItemCode);
return kpiItems;
}
public async Task<Item> RunAhnTask(Item cropfieldItem) {
var taskmapRequest = new TaskRequest { TaskType = AHN_TASK };

70
FarmmapsApiSamples.sln
View File

@ -1,7 +1,7 @@

Microsoft Visual Studio Solution File, Format Version 12.00
# Visual Studio Version 16
VisualStudioVersion = 16.0.30320.27
# Visual Studio Version 17
VisualStudioVersion = 17.4.33122.133
MinimumVisualStudioVersion = 10.0.40219.1
Project("{9A19103F-16F7-4668-BE54-9A1E7A4F7556}") = "FarmmapsNbs", "FarmmapsNbs\FarmmapsNbs.csproj", "{E08EF7E9-F09E-42D8-825C-164E458C78F4}"
EndProject
@ -31,56 +31,122 @@ Project("{9A19103F-16F7-4668-BE54-9A1E7A4F7556}") = "Secrets", "Secrets\Secrets.
EndProject
Project("{9A19103F-16F7-4668-BE54-9A1E7A4F7556}") = "FarmmapsCleanUp", "FarmmapsCleanUp\FarmmapsCleanUp.csproj", "{5E4387F9-5953-4A9B-BCA5-DF3964EED3CB}"
EndProject
Project("{9A19103F-16F7-4668-BE54-9A1E7A4F7556}") = "FarmmapsKPI", "FarmmapsKPI\FarmmapsKPI.csproj", "{14575235-9867-4CE5-A22F-3F9FE002FF42}"
EndProject
Project("{9A19103F-16F7-4668-BE54-9A1E7A4F7556}") = "FarmmapsDownloadCL", "FarmmapsDownloadCL\FarmmapsDownloadCL.csproj", "{63E69101-D804-4DBC-B2E4-A33771CD5C5F}"
EndProject
Global
GlobalSection(SolutionConfigurationPlatforms) = preSolution
Debug|Any CPU = Debug|Any CPU
Debug|x64 = Debug|x64
Release|Any CPU = Release|Any CPU
Release|x64 = Release|x64
EndGlobalSection
GlobalSection(ProjectConfigurationPlatforms) = postSolution
{E08EF7E9-F09E-42D8-825C-164E458C78F4}.Debug|Any CPU.ActiveCfg = Debug|Any CPU
{E08EF7E9-F09E-42D8-825C-164E458C78F4}.Debug|Any CPU.Build.0 = Debug|Any CPU
{E08EF7E9-F09E-42D8-825C-164E458C78F4}.Debug|x64.ActiveCfg = Debug|Any CPU
{E08EF7E9-F09E-42D8-825C-164E458C78F4}.Debug|x64.Build.0 = Debug|Any CPU
{E08EF7E9-F09E-42D8-825C-164E458C78F4}.Release|Any CPU.ActiveCfg = Release|Any CPU
{E08EF7E9-F09E-42D8-825C-164E458C78F4}.Release|Any CPU.Build.0 = Release|Any CPU
{E08EF7E9-F09E-42D8-825C-164E458C78F4}.Release|x64.ActiveCfg = Release|Any CPU
{E08EF7E9-F09E-42D8-825C-164E458C78F4}.Release|x64.Build.0 = Release|Any CPU
{1FA9E50B-F45E-4534-953A-37C783D03C74}.Debug|Any CPU.ActiveCfg = Debug|Any CPU
{1FA9E50B-F45E-4534-953A-37C783D03C74}.Debug|Any CPU.Build.0 = Debug|Any CPU
{1FA9E50B-F45E-4534-953A-37C783D03C74}.Debug|x64.ActiveCfg = Debug|x64
{1FA9E50B-F45E-4534-953A-37C783D03C74}.Debug|x64.Build.0 = Debug|x64
{1FA9E50B-F45E-4534-953A-37C783D03C74}.Release|Any CPU.ActiveCfg = Release|Any CPU
{1FA9E50B-F45E-4534-953A-37C783D03C74}.Release|Any CPU.Build.0 = Release|Any CPU
{1FA9E50B-F45E-4534-953A-37C783D03C74}.Release|x64.ActiveCfg = Release|x64
{1FA9E50B-F45E-4534-953A-37C783D03C74}.Release|x64.Build.0 = Release|x64
{731A88CD-9DC4-4969-86F2-2315830A6998}.Debug|Any CPU.ActiveCfg = Debug|Any CPU
{731A88CD-9DC4-4969-86F2-2315830A6998}.Debug|Any CPU.Build.0 = Debug|Any CPU
{731A88CD-9DC4-4969-86F2-2315830A6998}.Debug|x64.ActiveCfg = Debug|Any CPU
{731A88CD-9DC4-4969-86F2-2315830A6998}.Debug|x64.Build.0 = Debug|Any CPU
{731A88CD-9DC4-4969-86F2-2315830A6998}.Release|Any CPU.ActiveCfg = Release|Any CPU
{731A88CD-9DC4-4969-86F2-2315830A6998}.Release|Any CPU.Build.0 = Release|Any CPU
{731A88CD-9DC4-4969-86F2-2315830A6998}.Release|x64.ActiveCfg = Release|Any CPU
{731A88CD-9DC4-4969-86F2-2315830A6998}.Release|x64.Build.0 = Release|Any CPU
{DFA89D0B-5400-4374-B824-8367B76B4B6E}.Debug|Any CPU.ActiveCfg = Debug|Any CPU
{DFA89D0B-5400-4374-B824-8367B76B4B6E}.Debug|Any CPU.Build.0 = Debug|Any CPU
{DFA89D0B-5400-4374-B824-8367B76B4B6E}.Debug|x64.ActiveCfg = Debug|Any CPU
{DFA89D0B-5400-4374-B824-8367B76B4B6E}.Debug|x64.Build.0 = Debug|Any CPU
{DFA89D0B-5400-4374-B824-8367B76B4B6E}.Release|Any CPU.ActiveCfg = Release|Any CPU
{DFA89D0B-5400-4374-B824-8367B76B4B6E}.Release|Any CPU.Build.0 = Release|Any CPU
{DFA89D0B-5400-4374-B824-8367B76B4B6E}.Release|x64.ActiveCfg = Release|Any CPU
{DFA89D0B-5400-4374-B824-8367B76B4B6E}.Release|x64.Build.0 = Release|Any CPU
{AAFAB03A-6F5C-4D91-991F-867B7898F981}.Debug|Any CPU.ActiveCfg = Debug|Any CPU
{AAFAB03A-6F5C-4D91-991F-867B7898F981}.Debug|Any CPU.Build.0 = Debug|Any CPU
{AAFAB03A-6F5C-4D91-991F-867B7898F981}.Debug|x64.ActiveCfg = Debug|Any CPU
{AAFAB03A-6F5C-4D91-991F-867B7898F981}.Debug|x64.Build.0 = Debug|Any CPU
{AAFAB03A-6F5C-4D91-991F-867B7898F981}.Release|Any CPU.ActiveCfg = Release|Any CPU
{AAFAB03A-6F5C-4D91-991F-867B7898F981}.Release|Any CPU.Build.0 = Release|Any CPU
{AAFAB03A-6F5C-4D91-991F-867B7898F981}.Release|x64.ActiveCfg = Release|Any CPU
{AAFAB03A-6F5C-4D91-991F-867B7898F981}.Release|x64.Build.0 = Release|Any CPU
{892E0932-5D11-4A37-979E-CEDB39C2E181}.Debug|Any CPU.ActiveCfg = Debug|Any CPU
{892E0932-5D11-4A37-979E-CEDB39C2E181}.Debug|Any CPU.Build.0 = Debug|Any CPU
{892E0932-5D11-4A37-979E-CEDB39C2E181}.Debug|x64.ActiveCfg = Debug|Any CPU
{892E0932-5D11-4A37-979E-CEDB39C2E181}.Debug|x64.Build.0 = Debug|Any CPU
{892E0932-5D11-4A37-979E-CEDB39C2E181}.Release|Any CPU.ActiveCfg = Release|Any CPU
{892E0932-5D11-4A37-979E-CEDB39C2E181}.Release|Any CPU.Build.0 = Release|Any CPU
{892E0932-5D11-4A37-979E-CEDB39C2E181}.Release|x64.ActiveCfg = Release|Any CPU
{892E0932-5D11-4A37-979E-CEDB39C2E181}.Release|x64.Build.0 = Release|Any CPU
{91A58C4A-4A80-4079-B43D-9B851206194F}.Debug|Any CPU.ActiveCfg = Debug|Any CPU
{91A58C4A-4A80-4079-B43D-9B851206194F}.Debug|Any CPU.Build.0 = Debug|Any CPU
{91A58C4A-4A80-4079-B43D-9B851206194F}.Debug|x64.ActiveCfg = Debug|Any CPU
{91A58C4A-4A80-4079-B43D-9B851206194F}.Debug|x64.Build.0 = Debug|Any CPU
{91A58C4A-4A80-4079-B43D-9B851206194F}.Release|Any CPU.ActiveCfg = Release|Any CPU
{91A58C4A-4A80-4079-B43D-9B851206194F}.Release|Any CPU.Build.0 = Release|Any CPU
{91A58C4A-4A80-4079-B43D-9B851206194F}.Release|x64.ActiveCfg = Release|Any CPU
{91A58C4A-4A80-4079-B43D-9B851206194F}.Release|x64.Build.0 = Release|Any CPU
{32ED9500-AAAB-4030-9C7A-F611A85DF890}.Debug|Any CPU.ActiveCfg = Debug|Any CPU
{32ED9500-AAAB-4030-9C7A-F611A85DF890}.Debug|Any CPU.Build.0 = Debug|Any CPU
{32ED9500-AAAB-4030-9C7A-F611A85DF890}.Debug|x64.ActiveCfg = Debug|Any CPU
{32ED9500-AAAB-4030-9C7A-F611A85DF890}.Debug|x64.Build.0 = Debug|Any CPU
{32ED9500-AAAB-4030-9C7A-F611A85DF890}.Release|Any CPU.ActiveCfg = Release|Any CPU
{32ED9500-AAAB-4030-9C7A-F611A85DF890}.Release|Any CPU.Build.0 = Release|Any CPU
{32ED9500-AAAB-4030-9C7A-F611A85DF890}.Release|x64.ActiveCfg = Release|Any CPU
{32ED9500-AAAB-4030-9C7A-F611A85DF890}.Release|x64.Build.0 = Release|Any CPU
{772DBDCD-9FAA-40A7-8551-2C1620C4AB67}.Debug|Any CPU.ActiveCfg = Debug|Any CPU
{772DBDCD-9FAA-40A7-8551-2C1620C4AB67}.Debug|Any CPU.Build.0 = Debug|Any CPU
{772DBDCD-9FAA-40A7-8551-2C1620C4AB67}.Debug|x64.ActiveCfg = Debug|Any CPU
{772DBDCD-9FAA-40A7-8551-2C1620C4AB67}.Debug|x64.Build.0 = Debug|Any CPU
{772DBDCD-9FAA-40A7-8551-2C1620C4AB67}.Release|Any CPU.ActiveCfg = Release|Any CPU
{772DBDCD-9FAA-40A7-8551-2C1620C4AB67}.Release|Any CPU.Build.0 = Release|Any CPU
{772DBDCD-9FAA-40A7-8551-2C1620C4AB67}.Release|x64.ActiveCfg = Release|Any CPU
{772DBDCD-9FAA-40A7-8551-2C1620C4AB67}.Release|x64.Build.0 = Release|Any CPU
{C4EE5ECA-253A-4B71-9F67-D231AC4517D6}.Debug|Any CPU.ActiveCfg = Debug|Any CPU
{C4EE5ECA-253A-4B71-9F67-D231AC4517D6}.Debug|Any CPU.Build.0 = Debug|Any CPU
{C4EE5ECA-253A-4B71-9F67-D231AC4517D6}.Debug|x64.ActiveCfg = Debug|x64
{C4EE5ECA-253A-4B71-9F67-D231AC4517D6}.Debug|x64.Build.0 = Debug|x64
{C4EE5ECA-253A-4B71-9F67-D231AC4517D6}.Release|Any CPU.ActiveCfg = Release|Any CPU
{C4EE5ECA-253A-4B71-9F67-D231AC4517D6}.Release|Any CPU.Build.0 = Release|Any CPU
{C4EE5ECA-253A-4B71-9F67-D231AC4517D6}.Release|x64.ActiveCfg = Release|x64
{C4EE5ECA-253A-4B71-9F67-D231AC4517D6}.Release|x64.Build.0 = Release|x64
{5E4387F9-5953-4A9B-BCA5-DF3964EED3CB}.Debug|Any CPU.ActiveCfg = Debug|Any CPU
{5E4387F9-5953-4A9B-BCA5-DF3964EED3CB}.Debug|Any CPU.Build.0 = Debug|Any CPU
{5E4387F9-5953-4A9B-BCA5-DF3964EED3CB}.Debug|x64.ActiveCfg = Debug|Any CPU
{5E4387F9-5953-4A9B-BCA5-DF3964EED3CB}.Debug|x64.Build.0 = Debug|Any CPU
{5E4387F9-5953-4A9B-BCA5-DF3964EED3CB}.Release|Any CPU.ActiveCfg = Release|Any CPU
{5E4387F9-5953-4A9B-BCA5-DF3964EED3CB}.Release|Any CPU.Build.0 = Release|Any CPU
{5E4387F9-5953-4A9B-BCA5-DF3964EED3CB}.Release|x64.ActiveCfg = Release|Any CPU
{5E4387F9-5953-4A9B-BCA5-DF3964EED3CB}.Release|x64.Build.0 = Release|Any CPU
{14575235-9867-4CE5-A22F-3F9FE002FF42}.Debug|Any CPU.ActiveCfg = Debug|Any CPU
{14575235-9867-4CE5-A22F-3F9FE002FF42}.Debug|Any CPU.Build.0 = Debug|Any CPU
{14575235-9867-4CE5-A22F-3F9FE002FF42}.Debug|x64.ActiveCfg = Debug|x64
{14575235-9867-4CE5-A22F-3F9FE002FF42}.Debug|x64.Build.0 = Debug|x64
{14575235-9867-4CE5-A22F-3F9FE002FF42}.Release|Any CPU.ActiveCfg = Release|Any CPU
{14575235-9867-4CE5-A22F-3F9FE002FF42}.Release|Any CPU.Build.0 = Release|Any CPU
{14575235-9867-4CE5-A22F-3F9FE002FF42}.Release|x64.ActiveCfg = Release|x64
{14575235-9867-4CE5-A22F-3F9FE002FF42}.Release|x64.Build.0 = Release|x64
{63E69101-D804-4DBC-B2E4-A33771CD5C5F}.Debug|Any CPU.ActiveCfg = Debug|Any CPU
{63E69101-D804-4DBC-B2E4-A33771CD5C5F}.Debug|Any CPU.Build.0 = Debug|Any CPU
{63E69101-D804-4DBC-B2E4-A33771CD5C5F}.Debug|x64.ActiveCfg = Debug|x64
{63E69101-D804-4DBC-B2E4-A33771CD5C5F}.Debug|x64.Build.0 = Debug|x64
{63E69101-D804-4DBC-B2E4-A33771CD5C5F}.Release|Any CPU.ActiveCfg = Release|Any CPU
{63E69101-D804-4DBC-B2E4-A33771CD5C5F}.Release|Any CPU.Build.0 = Release|Any CPU
{63E69101-D804-4DBC-B2E4-A33771CD5C5F}.Release|x64.ActiveCfg = Release|x64
{63E69101-D804-4DBC-B2E4-A33771CD5C5F}.Release|x64.Build.0 = Release|x64
EndGlobalSection
GlobalSection(SolutionProperties) = preSolution
HideSolutionNode = FALSE

279
FarmmapsBulkSatDownload/BulkSatDownloadApplication.cs
View File

@ -25,7 +25,6 @@ namespace FarmmapsBulkSatDownload
private readonly GeneralService _generalService;
public const string settingsfile = "Settings.json";
public const int firstAvailableYear = 2017;
private Settings _settings;
public BulkSatDownloadApplication(ILogger<BulkSatDownloadApplication> logger, FarmmapsApiService farmmapsApiService,
@ -52,104 +51,96 @@ namespace FarmmapsBulkSatDownload
DateTime lastdownloadedimagedate;
int cropYear;
//Use doDB to decide if reading from/writing to database (doDB = true) or
//read from file BulkSatDownloadInput.json and write satellite statistics to file(s) specified in BulkSatDownloadInput.json
//note in case of doDB == true you will need to provide a file "DBsettings.secrets.json" with login details for the database
//see empty example "DBsettings.json"
bool doDB = false;
if (doDB == true)
// Option 1: When using database need to (1) fill in database data in DBsettings.secrets.json; (2) write tailor made SELECT query for fieldinputs in following lines;
// (3) Write tailor made INSERT INTO query in Task Process() below;
// Initialize databases. Username, password etc stored in file "DBsettings.secrets.json".
// Crashes if "DBsettings.secrets.json" is absent or empty
DB dbparcels = JsonConvert.DeserializeObject<DB>(File.ReadAllText("DBsettings.secrets.json"));
string schemaname = "bigdata";
string parceltablename = "parcel_bollenrevolutie_tulips2020"; //"parcelsijbrandij" "parcel"; "parcel_flowerbulbs"; "parcel_disac"; ""parcel_bollenrevolutie_tulips2020""
string groenmonitortablename = "groenmonitor_bollenrevolutie_tulips2020"; //"groenmonitorsijbrandij" "groenmonitor" "groenmonitor_flowerbulbs" "groenmonitor_disac" "groenmonitor_bollenrevolutie_tulips2020"
// The view 'groenmonitorlatestviewname' contains per parcelid (arbid) the year in which it "exists" and the date of the latest image downloaded. It is used to prevent unneccessary downloading of image statistics already in the database
string groenmonitorlatestviewname = "groenmonitorlatest_bollenrevolutie_tulips2020"; //"groenmonitorsijbrandijlatest" "groenmonitorlatest" "groenmonitorlatest_flowerbulbs" "groenmonitorlatest_disac" "groenmonitorlatest_bollenrevolutie_tulips2020"
// Database query and connection. Geometry must be in WGS84 coordinate system, EPSG 4326
// Apparently the FarmmapsApi cannot handle MultiPolygon, so we need to convert to single Polygon
// In case database returns a MultiPolygon use ST_NumGeometries(pt.geom) to count the number of polygons
// If necessary use WHERE ST_NumGeometries(pt.geom) = 1 to select only single polygons
//
// FarmMaps get's its satellite images from www.groenmonitor.nl through the https://agrodatacube.wur.nl/.
// Many images are available at www.groenmonitor.nl, the https://agrodatacube.wur.nl/ serves only the clean images, 10-30 per year, 2019 onwards. Possibly more images will be added for earlier years
// For other images contact www.groenmonitor.nl, gerbert.roerink@wur.nl
bulkSatDownloadInputListDB = new List<BulkSatDownloadInput>();
List<string> satelliteBands = new List<string> { "wdvi", "ndvi" };
string connectionString = dbparcels.GetConnectionString();
string readSql = string.Format(
@"
SELECT pt.arbid, pt.year, gml.lastwenrdate, ST_AsGeoJSON(ST_Transform((ST_DUMP(pt.geom)).geom::geometry(Polygon),4326)) AS geojson_polygon_wgs84,
COALESCE(pt.cropfielditemcode,'') AS cropfielditemcode,
CASE WHEN pt.year >= DATE_PART('year', CURRENT_DATE) THEN '' ELSE COALESCE(pt.satellitetaskcode,'') END AS satellitetaskcode
FROM {0}.{1} pt, {0}.{2} gml
WHERE
pt.arbid = gml.arbid
AND pt.satellitetaskcode IS NULL
ORDER BY pt.arbid
LIMIT 5;", schemaname, parceltablename, groenmonitorlatestviewname); //LIMIT x for testing
using (NpgsqlConnection connection = new NpgsqlConnection(connectionString))
{
// Option 1: When using database need to (1) fill in database data in DBsettings.secrets.json; (2) write tailor made SELECT query for fieldinputs in following lines;
// (3) Write tailor made INSERT INTO query in Task Process() below;
// Initialize databases. Username, password etc stored in file "DBsettings.secrets.json".
// Crashes if "DBsettings.secrets.json" is absent or empty
DB dbparcels = JsonConvert.DeserializeObject<DB>(File.ReadAllText("DBsettings.secrets.json"));
string schemaname = "bigdata";
string parceltablename = "parcel_bollenrevolutie_tulips2020"; //"parcelsijbrandij" "parcel"; "parcel_flowerbulbs"; "parcel_disac"; ""parcel_bollenrevolutie_tulips2020""
string groenmonitortablename = "groenmonitor_bollenrevolutie_tulips2020"; //"groenmonitorsijbrandij" "groenmonitor" "groenmonitor_flowerbulbs" "groenmonitor_disac" "groenmonitor_bollenrevolutie_tulips2020"
// The view 'groenmonitorlatestviewname' contains per parcelid (arbid) the year in which it "exists" and the date of the latest image downloaded. It is used to prevent unneccessary downloading of image statistics already in the database
string groenmonitorlatestviewname = "groenmonitorlatest_bollenrevolutie_tulips2020"; //"groenmonitorsijbrandijlatest" "groenmonitorlatest" "groenmonitorlatest_flowerbulbs" "groenmonitorlatest_disac" "groenmonitorlatest_bollenrevolutie_tulips2020"
connection.Open();
// Database query and connection. Geometry must be in WGS84 coordinate system, EPSG 4326
// Apparently the FarmmapsApi cannot handle MultiPolygon, so we need to convert to single Polygon
// In case database returns a MultiPolygon use ST_NumGeometries(pt.geom) to count the number of polygons
// If necessary use WHERE ST_NumGeometries(pt.geom) = 1 to select only single polygons
//
// FarmMaps get's its satellite images from www.groenmonitor.nl through the https://agrodatacube.wur.nl/.
// Many images are available at www.groenmonitor.nl, the https://agrodatacube.wur.nl/ serves only the clean images, 10-30 per year, 2019 onwards. Possibly more images will be added for earlier years
// For other images contact www.groenmonitor.nl, gerbert.roerink@wur.nl
bulkSatDownloadInputListDB = new List<BulkSatDownloadInput>();
List<string> satelliteBands = new List<string> { "wdvi", "ndvi" };
string connectionString = dbparcels.GetConnectionString();
string readSql = string.Format(
@"
SELECT pt.arbid, pt.year, gml.lastwenrdate, ST_AsGeoJSON(ST_Transform((ST_DUMP(pt.geom)).geom::geometry(Polygon),4326)) AS geojson_polygon_wgs84,
COALESCE(pt.cropfielditemcode,'') AS cropfielditemcode,
CASE WHEN pt.year >= DATE_PART('year', CURRENT_DATE) THEN '' ELSE COALESCE(pt.satellitetaskcode,'') END AS satellitetaskcode
FROM {0}.{1} pt, {0}.{2} gml
WHERE
pt.arbid = gml.arbid
AND pt.satellitetaskcode IS NULL
ORDER BY pt.arbid
LIMIT 5;", schemaname, parceltablename, groenmonitorlatestviewname); //LIMIT x for testing
using (NpgsqlConnection connection = new NpgsqlConnection(connectionString))
// Read data (run query) = build a list of fields for which to download images
NpgsqlCommand command = connection.CreateCommand();
command.CommandText = readSql;
NpgsqlDataReader dr = command.ExecuteReader();
while (dr.Read())
{
connection.Open();
// Read data (run query) = build a list of fields for which to download images
NpgsqlCommand command = connection.CreateCommand();
command.CommandText = readSql;
NpgsqlDataReader dr = command.ExecuteReader();
while (dr.Read())
{
bulkSatDownloadInput = new BulkSatDownloadInput();
bulkSatDownloadInput.fieldID = dr.GetInt16(0);
bulkSatDownloadInput.fieldName = string.Format($"{parceltablename}_{bulkSatDownloadInput.fieldID}");
bulkSatDownloadInput.cropYear = dr.GetInt16(1); ;
bulkSatDownloadInput.lastdownloadedimagedate = dr.GetDateTime(2);
bulkSatDownloadInput.GeometryJson = JObject.Parse(dr.GetString(3));
bulkSatDownloadInput.SatelliteBands = satelliteBands;
bulkSatDownloadInput.cropfielditemcode = dr.GetString(4);
bulkSatDownloadInput.satellitetaskcode = dr.GetString(5);
bulkSatDownloadInput.database = dbparcels;
bulkSatDownloadInput.schemaname = schemaname;
bulkSatDownloadInput.cropfieldtable = parceltablename;
bulkSatDownloadInput.satelllitetable = groenmonitortablename;
bulkSatDownloadInputListDB.Add(bulkSatDownloadInput);
}
connection.Close();
bulkSatDownloadInput = new BulkSatDownloadInput();
bulkSatDownloadInput.fieldID = dr.GetInt16(0);
bulkSatDownloadInput.fieldName = string.Format($"{parceltablename}_{bulkSatDownloadInput.fieldID}");
bulkSatDownloadInput.cropYear = dr.GetInt16(1); ;
bulkSatDownloadInput.lastdownloadedimagedate = dr.GetDateTime(2);
bulkSatDownloadInput.GeometryJson = JObject.Parse(dr.GetString(3));
bulkSatDownloadInput.SatelliteBands = satelliteBands;
bulkSatDownloadInput.cropfielditemcode = dr.GetString(4);
bulkSatDownloadInput.satellitetaskcode = dr.GetString(5);
bulkSatDownloadInput.database = dbparcels;
bulkSatDownloadInput.schemaname = schemaname;
bulkSatDownloadInput.cropfieldtable = parceltablename;
bulkSatDownloadInput.satelllitetable = groenmonitortablename;
bulkSatDownloadInputListDB.Add(bulkSatDownloadInput);
}
// Now choose which list you want to use
bulkSatDownloadInputList = bulkSatDownloadInputListDB;
}
else
{
// Option 2: Example without database. Comment out this part if you want to use database
// Read cropfields "BulkSatDownloadInput.json" and write all stats to a single csv file
// Write all stats for multiple fields will be written to a single csv file
string downloadFolder;
string fileNameStats;
string headerLineStats = $"FieldName,satelliteDate,satelliteBand,max,min,mean,mode,median,stddev,minPlus,curtosis,maxMinus,skewness,variance,populationCount,variationCoefficient,confidenceIntervalLow, confidenceIntervalHigh,confidenceIntervalErrorMargin" + Environment.NewLine;
var fieldsInputJson = File.ReadAllText("BulkSatDownloadInput.json");
bulkSatDownloadInputListCsv = JsonConvert.DeserializeObject<List<BulkSatDownloadInput>>(fieldsInputJson);
for (int i = 0; i < bulkSatDownloadInputListCsv.Count; i++)
{
downloadFolder = bulkSatDownloadInputListCsv[i].downloadFolder;
fileNameStats = Path.Combine(downloadFolder, bulkSatDownloadInputListCsv[i].fileNameStats);
if (!Directory.Exists(downloadFolder))
Directory.CreateDirectory(downloadFolder);
bulkSatDownloadInputListCsv[i].fileNameStats = fileNameStats;
// Header same as in GeneralService.DownloadSatelliteStats
// Delete fileNameStats if existing. Create a new file. Add a header to csv file
File.Delete(fileNameStats);
File.AppendAllText(fileNameStats, headerLineStats);
}
// Now choose which list you want to use
bulkSatDownloadInputList = bulkSatDownloadInputListCsv;
connection.Close();
}
// Option 2: Example without database. Comment out this part if you want to use database
// Read cropfields "BulkSatDownloadInput.json" and write all stats to a single csv file
// Write all stats for multiple fields will be written to a single csv file
//string downloadFolder;
//string fileNameStats;
//string headerLineStats = $"FieldName,satelliteDate,satelliteBand,max,min,mean,mode,median,stddev,minPlus,curtosis,maxMinus,skewness,variance,populationCount,variationCoefficient,confidenceIntervalLow, confidenceIntervalHigh,confidenceIntervalErrorMargin" + Environment.NewLine;
//var fieldsInputJson = File.ReadAllText("BulkSatDownloadInput.json");
//bulkSatDownloadInputListCsv = JsonConvert.DeserializeObject<List<BulkSatDownloadInput>>(fieldsInputJson);
//for (int i = 0; i < bulkSatDownloadInputListCsv.Count; i++)
//{
// downloadFolder = bulkSatDownloadInputListCsv[i].downloadFolder;
// fileNameStats = Path.Combine(downloadFolder, bulkSatDownloadInputListCsv[i].fileNameStats);
// if (!Directory.Exists(downloadFolder))
// Directory.CreateDirectory(downloadFolder);
// bulkSatDownloadInputListCsv[i].fileNameStats = fileNameStats;
// // Header same as in GeneralService.DownloadSatelliteStats
// // Delete fileNameStats if existing. Create a new file. Add a header to csv file
// File.Delete(fileNameStats);
// File.AppendAllText(fileNameStats, headerLineStats);
//}
// Now choose which list you want to use
bulkSatDownloadInputList = bulkSatDownloadInputListDB; //bulkSatDownloadInputListDB; //bulkSatDownloadInputListCsv;
// Whichever option (database or json/csv), continue here
// Delete the settingsfile
// File.Delete(settingsfile);
// For each input download all images. Keep track to time, important when doing bulk downloads
var watch = System.Diagnostics.Stopwatch.StartNew();
TimeSpan tsSofar = new TimeSpan();
@ -200,33 +191,6 @@ namespace FarmmapsBulkSatDownload
private async Task Process(List<UserRoot> roots, BulkSatDownloadInput input)
{
//PO20220311: first time a call is made to download satellite images or statistics, an empty list is returned
//If we wait a bit longer, e.g. 10 secs, then e.g. a list of 3 images may be returned
//If we wait still longer, maybe 4 images.
//The solution implemented below is to fire calls as long as the number of images returned keeps increasing
//While in between each call, sleep for sleepSecs
//Continue this until the number no longer increases or the maximum number of calls has been reached
//If you set sleepSecs to a very low value, e.g. 5 secs, then after 1 call you might get images and after 2nd call still zero images.
//to be on the safe side, better bit higher value.
//Just accept this may take a while, have a coffee, we suggest sleepSecs = 30;
int sleepSecs = 30;
int callCntMax = 4 * 60 / sleepSecs; //4*60 = max 4 minutes
//For example we may set: "sleepSecs = 10;" and "callCntMax = 24;" and following result:
//Call no: 1. Giving FarmMaps 10 seconds to get SatelliteItems...
//Call no: 1: Received 2 images
//Call no: 2. Giving FarmMaps 10 seconds to get SatelliteItems...
//Call no: 2: Received 7 images
//Call no: 3. Giving FarmMaps 10 seconds to get SatelliteItems...
//Call no: 3: Received 7 images
//And the firing of calls would stop because the number of images returned is no longer increasing
//In the worst case, this could would lead to a total sleeping period of "sleepSecsSum = sleepSecs * callCntMax" seconds. After that we give up
//PO20230801: I also tried creating a cropfieldItem with startDate '2017-01-01' and endDate '2023-08-01'
//so that only one RunSatelliteTask would need to be executed and statistics would be returned for the e.g. 233 images in this period from start to end date
//instead of running 7 tasks (one per year) for a single field
//But code doing that returned much less images than those 233 records which I should be getting.
//So in the end, stick to downloading stats per year, even if it is for the same location in multiple years
string cropfielditemcode;
string satellitetaskcode;
Item cropfieldItem;
@ -258,10 +222,6 @@ namespace FarmmapsBulkSatDownload
DateTime lastDownloadedSatelliteDate = input.lastdownloadedimagedate;
cropfielditemcode = input.cropfielditemcode;
satellitetaskcode = input.satellitetaskcode;
int satelliteItemsCropYearCntPrev;
int satelliteItemsCropYearCnt;
int callCnt;
int sleepSecsSum;
LoadSettings(settingsfile);
@ -349,80 +309,23 @@ namespace FarmmapsBulkSatDownload
// TODO also log satellitetaskcode to settings, how?
// SaveSettings(settingsfile);
// Getting satellite items. Only for years for which available
satelliteItemsCropYearCntPrev = 0;
satelliteItemsCropYearCnt = 0;
sleepSecsSum = 0;
satelliteItemsCropYear = null;
if (cropYear >= firstAvailableYear && cropYear <= DateTime.Now.Year)
{
_logger.LogInformation(string.Format($"Running FindSatelliteItems for cropfieldItem.Code '{cropfieldItem.Code}', SatelliteTaskCode '{satellitetaskcode}'"));
//Call first time
callCnt = 1;
//if callCntMax == 0 then don't sleep
//if callCntMax = 1 then sleep first 1x
if (callCntMax > 0)
{
_logger.LogInformation($"Call no: {callCnt}. Giving FarmMaps {sleepSecs} seconds to get SatelliteItems...");
System.Threading.Thread.Sleep(1000 * sleepSecs);
sleepSecsSum = sleepSecsSum + sleepSecs;
}
satelliteItemsCropYear = await _generalService.FindSatelliteItems(cropfieldItem, satellitetaskcode);
satelliteItemsCropYearCntPrev = satelliteItemsCropYear.Count;
_logger.LogInformation($"Call no: {callCnt}. Received {satelliteItemsCropYearCntPrev} images");
callCnt++;
satelliteItemsCropYearCnt = satelliteItemsCropYearCntPrev;
//if callCntMax > 1 then sleep untill (1) no more increase in number of images received OR (2) maximum number of calls reached
if (callCntMax > 1)
{
//Call second time
_logger.LogInformation($"Call no: {callCnt}. Giving FarmMaps another {sleepSecs} seconds to get SatelliteItems...");
System.Threading.Thread.Sleep(1000 * sleepSecs);
satelliteItemsCropYear = await _generalService.FindSatelliteItems(cropfieldItem, satellitetaskcode);
satelliteItemsCropYearCnt = satelliteItemsCropYear.Count;
_logger.LogInformation($"Call no: {callCnt}. Received {satelliteItemsCropYearCnt} images");
sleepSecsSum = sleepSecsSum + sleepSecs;
//As long as there is progress, keep calling
callCnt++;
while (callCnt <= callCntMax && (satelliteItemsCropYearCnt == 0 || satelliteItemsCropYearCnt > satelliteItemsCropYearCntPrev))
{
_logger.LogInformation($"Surprise! The longer we wait, the more images we get. Sleep and call once more");
satelliteItemsCropYearCntPrev = satelliteItemsCropYearCnt;
_logger.LogInformation($"Call no: {callCnt} (max: {callCntMax}). Giving FarmMaps another {sleepSecs} seconds to get SatelliteItems...");
System.Threading.Thread.Sleep(1000 * sleepSecs);
satelliteItemsCropYear = await _generalService.FindSatelliteItems(cropfieldItem, satellitetaskcode);
satelliteItemsCropYearCnt = satelliteItemsCropYear.Count;
_logger.LogInformation($"Call no: {callCnt}. Received {satelliteItemsCropYearCnt} images");
callCnt++;
sleepSecsSum = sleepSecsSum + sleepSecs;
}
}
}
else
{
_logger.LogWarning($"// FarmmapsBulkSatDownload: crop year {cropYear} is out of bounds. No stats will be written!");
}
if (satelliteItemsCropYearCnt == 0)
{
_logger.LogWarning($"// FarmmapsBulkSatDownload: after calling one or more times and " +
$"sleeping in total {sleepSecsSum} seconds, still no images found. " +
$"Please check your settings for parameters callCntMax and sleepSecs in FarmmapsBulkSatDownload.cs or contact FarmMaps");
}
// Getting satellite items
_logger.LogInformation(string.Format($"Running FindSatelliteItems for cropfieldItem.Code '{cropfieldItem.Code}', SatelliteTaskCode '{satellitetaskcode}'"));
satelliteItemsCropYear = await _generalService.FindSatelliteItems(cropfieldItem, satellitetaskcode);
// Checking if satellite items found
satelliteItemsAvailable = true;
if (satelliteItemsCropYear == null)
{
satelliteItemsAvailable = false;
_logger.LogInformation($"// FarmmapsBulkSatDownload: No satellite tiffs found for fieldName '{fieldName}', cropYear {cropYear}, cropfielditemcode '{cropfielditemcode}'");
_logger.LogInformation($"No satellite tiffs found for fieldName '{fieldName}', cropYear {cropYear}, cropfielditemcode '{cropfielditemcode}'");
}
else
{
if (satelliteItemsCropYear.Count == 0)
{
satelliteItemsAvailable = false;
_logger.LogInformation($"// FarmmapsBulkSatDownload: No satellite tiffs found for fieldName '{fieldName}', cropYear {cropYear}, cropfielditemcode '{cropfielditemcode}'");
_logger.LogInformation($"No satellite tiffs found for fieldName '{fieldName}', cropYear {cropYear}, cropfielditemcode '{cropfielditemcode}'");
}
}
@ -432,10 +335,10 @@ namespace FarmmapsBulkSatDownload
// Download statistics to a single csv file
if (satelliteItemsAvailable && downloadFolder != null && fileNameStats != null)
{
// Write statistics for all images for all fieldName and cropYear to a single csv file, fileNameStats
// Write statistics for all images for all fieldNane and cropYear to a single csv file, fileNameStats
_logger.LogInformation($"Downloading stats for field '{fieldName}' in cropyear {cropYear} to {fileNameStats}");
string downloadedStats = await _generalService.DownloadSatelliteStats(satelliteItemsCropYear, fieldName, satelliteBands, downloadFolder);
// Add contents of this csv file to the single large csv file
// Add contents of this csv file to thee single large csv file
var retainedLines = File.ReadAllLines(downloadedStats).Skip(1);
File.AppendAllLines(fileNameStats, retainedLines);
File.Delete(downloadedStats);

323
FarmmapsBulkSatDownload/BulkSatDownloadInput.json
View File

@ -1,44 +1,46 @@
[
{
"fieldName": "MyField_1", // FarmMaps minimum needs are: fieldName, cropYear & geometryJson
"cropYear": 2023,
"fieldID": 1,
"SatelliteBands": [ "ci-red", "ndvi", "wdvi" ], // ["ndvi"] or ["wdvi"] or both: [ "wdvi", "ndvi" ]
"lastdownloadedimagedate": "2023-01-01", //downloads images from this date till end of the year
"fieldName": "5641", // FarmMaps minimum needs are: fieldName, cropYear & geometryJson
"cropYear": 1999, //For testing a year for which we know no satellite data available, program shouldn't crash
"fieldID": 5641,
"SatelliteBands": [ "wdvi", "ndvi" ], // ["ndvi"] or ["wdvi"] or both: [ "wdvi", "ndvi" ]
"lastdownloadedimagedate": "1999-01-01", //downloads images from this date till end of the year
"geometryJson": {
"type": "Polygon",
"coordinates": [
[
[ 5.563472073408009, 52.547554398144172 ],
[ 5.567425915520115, 52.547725375100377 ],
[ 5.563878143678981, 52.54048022658143 ],
[ 5.563878143678981, 52.54048022658143 ],
[ 5.563472073408009, 52.547554398144172 ]
]
]
},
"downloadFolder": "C:\\workdir\\groenmonitor\\", //if not yet existing this folder will be created
"fileNameStats": "BulkSatDownload.csv", //if file exists, probably will be overwritten. Check code in BulkSatDownloadApplication.cs
"database": null, // keep null to work with json and csv. Check code in BulkSatDownloadApplication.cs if reading/writing to/from database
"schemaname": null,
"cropfieldtable": null,
"satelllitetable": null
},
{
"fieldName": "MyField_2", // FarmMaps minimum needs are: fieldName, cropYear & geometryJson
"cropYear": 2022, // for testing same field last year (i.e. 2021)
"fieldID": 2,
"SatelliteBands": [ "ci-red", "ndvi", "wdvi" ], // ["ndvi"] or ["wdvi"] or both: [ "wdvi", "ndvi" ]
"lastdownloadedimagedate": "2022-01-01", //downloads images from this date till end of the year
"geometryJson": {
"type": "Polygon",
"coordinates": [
[
[ 5.563472073408009, 52.547554398144172 ],
[ 5.567425915520115, 52.547725375100377 ],
[ 5.563878143678981, 52.54048022658143 ],
[ 5.563878143678981, 52.54048022658143 ],
[ 5.563472073408009, 52.547554398144172 ]
[ 3.37837807779104, 51.3231095796538 ],
[ 3.38065689232502, 51.3212527499355 ],
[ 3.38022924592256, 51.3210683536359 ],
[ 3.37980548452565, 51.3208801127141 ],
[ 3.37959556105776, 51.3207540143696 ],
[ 3.3793691292654, 51.3205959677371 ],
[ 3.37822219207335, 51.3215667913007 ],
[ 3.37816999925795, 51.3216109809456 ],
[ 3.37646704574705, 51.3208025481261 ],
[ 3.37646695791282, 51.3208025061493 ],
[ 3.37608401443192, 51.3206231652693 ],
[ 3.37607169507628, 51.3206173959751 ],
[ 3.37606021048754, 51.320612017601 ],
[ 3.37582728410659, 51.3205029306946 ],
[ 3.37580409779263, 51.3206502985963 ],
[ 3.37575872019649, 51.3207993094705 ],
[ 3.37575476634361, 51.3208122883487 ],
[ 3.37571181656268, 51.3208797459348 ],
[ 3.3756624532907, 51.3209415238446 ],
[ 3.37557609963811, 51.3210110142077 ],
[ 3.37541089899821, 51.3211055871218 ],
[ 3.37477516102591, 51.3214102985009 ],
[ 3.37473173914127, 51.3214311108204 ],
[ 3.37455904622072, 51.3215138815012 ],
[ 3.37415098054777, 51.3217199232877 ],
[ 3.37313700916272, 51.3222422862785 ],
[ 3.37748824689601, 51.3242852920348 ],
[ 3.37749760805371, 51.3242713084009 ],
[ 3.37811903757028, 51.3233437635596 ],
[ 3.37818758851947, 51.3232647797363 ],
[ 3.37823803668144, 51.3232236798646 ],
[ 3.37837807779104, 51.3231095796538 ]
]
]
},
@ -50,20 +52,47 @@
"satelllitetable": null
},
{
"fieldName": "MyField_3", // FarmMaps minimum needs are: fieldName, cropYear & geometryJson
"cropYear": 2021,
"fieldID": 3,
"SatelliteBands": [ "ci-red", "ndvi", "wdvi" ], // ["ndvi"] or ["wdvi"] or both: [ "wdvi", "ndvi" ]
"fieldName": "5641", // FarmMaps minimum needs are: fieldName, cropYear & geometryJson
"cropYear": 2021, //For testing a year for which we know no satellite data available, program shouldn't crash
"fieldID": 5641,
"SatelliteBands": [ "wdvi", "ndvi" ], // ["ndvi"] or ["wdvi"] or both: [ "wdvi", "ndvi" ]
"lastdownloadedimagedate": "2021-01-01", //downloads images from this date till end of the year
"geometryJson": {
"type": "Polygon",
"coordinates": [
[
[ 5.563472073408009, 52.547554398144172 ],
[ 5.567425915520115, 52.547725375100377 ],
[ 5.563878143678981, 52.54048022658143 ],
[ 5.563878143678981, 52.54048022658143 ],
[ 5.563472073408009, 52.547554398144172 ]
[ 3.37837807779104, 51.3231095796538 ],
[ 3.38065689232502, 51.3212527499355 ],
[ 3.38022924592256, 51.3210683536359 ],
[ 3.37980548452565, 51.3208801127141 ],
[ 3.37959556105776, 51.3207540143696 ],
[ 3.3793691292654, 51.3205959677371 ],
[ 3.37822219207335, 51.3215667913007 ],
[ 3.37816999925795, 51.3216109809456 ],
[ 3.37646704574705, 51.3208025481261 ],
[ 3.37646695791282, 51.3208025061493 ],
[ 3.37608401443192, 51.3206231652693 ],
[ 3.37607169507628, 51.3206173959751 ],
[ 3.37606021048754, 51.320612017601 ],
[ 3.37582728410659, 51.3205029306946 ],
[ 3.37580409779263, 51.3206502985963 ],
[ 3.37575872019649, 51.3207993094705 ],
[ 3.37575476634361, 51.3208122883487 ],
[ 3.37571181656268, 51.3208797459348 ],
[ 3.3756624532907, 51.3209415238446 ],
[ 3.37557609963811, 51.3210110142077 ],
[ 3.37541089899821, 51.3211055871218 ],
[ 3.37477516102591, 51.3214102985009 ],
[ 3.37473173914127, 51.3214311108204 ],
[ 3.37455904622072, 51.3215138815012 ],
[ 3.37415098054777, 51.3217199232877 ],
[ 3.37313700916272, 51.3222422862785 ],
[ 3.37748824689601, 51.3242852920348 ],
[ 3.37749760805371, 51.3242713084009 ],
[ 3.37811903757028, 51.3233437635596 ],
[ 3.37818758851947, 51.3232647797363 ],
[ 3.37823803668144, 51.3232236798646 ],
[ 3.37837807779104, 51.3231095796538 ]
]
]
},
@ -75,20 +104,47 @@
"satelllitetable": null
},
{
"fieldName": "MyField_4", // FarmMaps minimum needs are: fieldName, cropYear & geometryJson
"fieldName": "5641", // FarmMaps minimum needs are: fieldName, cropYear & geometryJson
"cropYear": 2020,
"fieldID": 4,
"SatelliteBands": [ "ci-red", "ndvi", "wdvi" ], // ["ndvi"] or ["wdvi"] or both: [ "wdvi", "ndvi" ]
"fieldID": 5641,
"SatelliteBands": [ "wdvi", "ndvi" ], // ["ndvi"] or ["wdvi"] or both: [ "wdvi", "ndvi" ]
"lastdownloadedimagedate": "2020-01-01", //downloads images from this date till end of the year
"geometryJson": {
"type": "Polygon",
"coordinates": [
[
[ 5.563472073408009, 52.547554398144172 ],
[ 5.567425915520115, 52.547725375100377 ],
[ 5.563878143678981, 52.54048022658143 ],
[ 5.563878143678981, 52.54048022658143 ],
[ 5.563472073408009, 52.547554398144172 ]
[ 3.37837807779104, 51.3231095796538 ],
[ 3.38065689232502, 51.3212527499355 ],
[ 3.38022924592256, 51.3210683536359 ],
[ 3.37980548452565, 51.3208801127141 ],
[ 3.37959556105776, 51.3207540143696 ],
[ 3.3793691292654, 51.3205959677371 ],
[ 3.37822219207335, 51.3215667913007 ],
[ 3.37816999925795, 51.3216109809456 ],
[ 3.37646704574705, 51.3208025481261 ],
[ 3.37646695791282, 51.3208025061493 ],
[ 3.37608401443192, 51.3206231652693 ],
[ 3.37607169507628, 51.3206173959751 ],
[ 3.37606021048754, 51.320612017601 ],
[ 3.37582728410659, 51.3205029306946 ],
[ 3.37580409779263, 51.3206502985963 ],
[ 3.37575872019649, 51.3207993094705 ],
[ 3.37575476634361, 51.3208122883487 ],
[ 3.37571181656268, 51.3208797459348 ],
[ 3.3756624532907, 51.3209415238446 ],
[ 3.37557609963811, 51.3210110142077 ],
[ 3.37541089899821, 51.3211055871218 ],
[ 3.37477516102591, 51.3214102985009 ],
[ 3.37473173914127, 51.3214311108204 ],
[ 3.37455904622072, 51.3215138815012 ],
[ 3.37415098054777, 51.3217199232877 ],
[ 3.37313700916272, 51.3222422862785 ],
[ 3.37748824689601, 51.3242852920348 ],
[ 3.37749760805371, 51.3242713084009 ],
[ 3.37811903757028, 51.3233437635596 ],
[ 3.37818758851947, 51.3232647797363 ],
[ 3.37823803668144, 51.3232236798646 ],
[ 3.37837807779104, 51.3231095796538 ]
]
]
},
@ -100,20 +156,47 @@
"satelllitetable": null
},
{
"fieldName": "MyField_5", // FarmMaps minimum needs are: fieldName, cropYear & geometryJson
"fieldName": "5641", // FarmMaps minimum needs are: fieldName, cropYear & geometryJson
"cropYear": 2019,
"fieldID": 5,
"SatelliteBands": [ "ci-red", "ndvi", "wdvi" ], // ["ndvi"] or ["wdvi"] or both: [ "wdvi", "ndvi" ]
"fieldID": 5641,
"SatelliteBands": [ "wdvi", "ndvi" ], // ["ndvi"] or ["wdvi"] or both: [ "wdvi", "ndvi" ]
"lastdownloadedimagedate": "2019-01-01", //downloads images from this date till end of the year
"geometryJson": {
"type": "Polygon",
"coordinates": [
[
[ 5.563472073408009, 52.547554398144172 ],
[ 5.567425915520115, 52.547725375100377 ],
[ 5.563878143678981, 52.54048022658143 ],
[ 5.563878143678981, 52.54048022658143 ],
[ 5.563472073408009, 52.547554398144172 ]
[ 3.37837807779104, 51.3231095796538 ],
[ 3.38065689232502, 51.3212527499355 ],
[ 3.38022924592256, 51.3210683536359 ],
[ 3.37980548452565, 51.3208801127141 ],
[ 3.37959556105776, 51.3207540143696 ],
[ 3.3793691292654, 51.3205959677371 ],
[ 3.37822219207335, 51.3215667913007 ],
[ 3.37816999925795, 51.3216109809456 ],
[ 3.37646704574705, 51.3208025481261 ],
[ 3.37646695791282, 51.3208025061493 ],
[ 3.37608401443192, 51.3206231652693 ],
[ 3.37607169507628, 51.3206173959751 ],
[ 3.37606021048754, 51.320612017601 ],
[ 3.37582728410659, 51.3205029306946 ],
[ 3.37580409779263, 51.3206502985963 ],
[ 3.37575872019649, 51.3207993094705 ],
[ 3.37575476634361, 51.3208122883487 ],
[ 3.37571181656268, 51.3208797459348 ],
[ 3.3756624532907, 51.3209415238446 ],
[ 3.37557609963811, 51.3210110142077 ],
[ 3.37541089899821, 51.3211055871218 ],
[ 3.37477516102591, 51.3214102985009 ],
[ 3.37473173914127, 51.3214311108204 ],
[ 3.37455904622072, 51.3215138815012 ],
[ 3.37415098054777, 51.3217199232877 ],
[ 3.37313700916272, 51.3222422862785 ],
[ 3.37748824689601, 51.3242852920348 ],
[ 3.37749760805371, 51.3242713084009 ],
[ 3.37811903757028, 51.3233437635596 ],
[ 3.37818758851947, 51.3232647797363 ],
[ 3.37823803668144, 51.3232236798646 ],
[ 3.37837807779104, 51.3231095796538 ]
]
]
},
@ -125,95 +208,47 @@
"satelllitetable": null
},
{
"fieldName": "MyField_6", // FarmMaps minimum needs are: fieldName, cropYear & geometryJson
"cropYear": 2018,
"fieldID": 6,
"SatelliteBands": [ "ci-red", "ndvi", "wdvi" ], // ["ndvi"] or ["wdvi"] or both: [ "wdvi", "ndvi" ]
"fieldName": "5641", // FarmMaps minimum needs are: fieldName, cropYear & geometryJson
"cropYear": 2018, //little to no images for 2018
"fieldID": 5641,
"SatelliteBands": [ "wdvi", "ndvi" ], // ["ndvi"] or ["wdvi"] or both: [ "wdvi", "ndvi" ]
"lastdownloadedimagedate": "2018-01-01", //downloads images from this date till end of the year
"geometryJson": {
"type": "Polygon",
"coordinates": [
[
[ 5.563472073408009, 52.547554398144172 ],
[ 5.567425915520115, 52.547725375100377 ],
[ 5.563878143678981, 52.54048022658143 ],
[ 5.563878143678981, 52.54048022658143 ],
[ 5.563472073408009, 52.547554398144172 ]
]
]
},
"downloadFolder": "C:\\workdir\\groenmonitor\\",
"fileNameStats": "BulkSatDownload.csv",
"database": null,
"schemaname": null,
"cropfieldtable": null,
"satelllitetable": null
},
{
"fieldName": "MyField_6", // FarmMaps minimum needs are: fieldName, cropYear & geometryJson
"cropYear": 2017,
"fieldID": 7,
"SatelliteBands": [ "ci-red", "ndvi", "wdvi" ], // ["ndvi"] or ["wdvi"] or both: [ "wdvi", "ndvi" ]
"lastdownloadedimagedate": "2017-01-01", //downloads images from this date till end of the year
"geometryJson": {
"type": "Polygon",
"coordinates": [
[
[ 5.563472073408009, 52.547554398144172 ],
[ 5.567425915520115, 52.547725375100377 ],
[ 5.563878143678981, 52.54048022658143 ],
[ 5.563878143678981, 52.54048022658143 ],
[ 5.563472073408009, 52.547554398144172 ]
]
]
},
"downloadFolder": "C:\\workdir\\groenmonitor\\",
"fileNameStats": "BulkSatDownload.csv",
"database": null,
"schemaname": null,
"cropfieldtable": null,
"satelllitetable": null
},
{
"fieldName": "MyField_8", // FarmMaps minimum needs are: fieldName, cropYear & geometryJson. Actually no Satellite data before 2017. This here to illustrate the program does not crash
"cropYear": 2016,
"fieldID": 8,
"SatelliteBands": [ "ci-red", "ndvi", "wdvi" ], // ["ndvi"] or ["wdvi"] or both: [ "wdvi", "ndvi" ]
"lastdownloadedimagedate": "2016-01-01", //downloads images from this date till end of the year
"geometryJson": {
"type": "Polygon",
"coordinates": [
[
[ 5.563472073408009, 52.547554398144172 ],
[ 5.567425915520115, 52.547725375100377 ],
[ 5.563878143678981, 52.54048022658143 ],
[ 5.563878143678981, 52.54048022658143 ],
[ 5.563472073408009, 52.547554398144172 ]
]
]
},
"downloadFolder": "C:\\workdir\\groenmonitor\\",
"fileNameStats": "BulkSatDownload.csv",
"database": null,
"schemaname": null,
"cropfieldtable": null,
"satelllitetable": null
},
{
"fieldName": "MyField_9", // FarmMaps minimum needs are: fieldName, cropYear & geometryJson. Actually no Satellite data before 2017. This here to illustrate the program does not crash
"cropYear": 2015,
"fieldID": 9,
"SatelliteBands": [ "ci-red", "ndvi", "wdvi" ], // ["ndvi"] or ["wdvi"] or both: [ "wdvi", "ndvi" ]
"lastdownloadedimagedate": "2015-01-01", //downloads images from this date till end of the year
"geometryJson": {
"type": "Polygon",
"coordinates": [
[
[ 5.563472073408009, 52.547554398144172 ],
[ 5.567425915520115, 52.547725375100377 ],
[ 5.563878143678981, 52.54048022658143 ],
[ 5.563878143678981, 52.54048022658143 ],
[ 5.563472073408009, 52.547554398144172 ]
[ 3.37837807779104, 51.3231095796538 ],
[ 3.38065689232502, 51.3212527499355 ],
[ 3.38022924592256, 51.3210683536359 ],
[ 3.37980548452565, 51.3208801127141 ],
[ 3.37959556105776, 51.3207540143696 ],
[ 3.3793691292654, 51.3205959677371 ],
[ 3.37822219207335, 51.3215667913007 ],
[ 3.37816999925795, 51.3216109809456 ],
[ 3.37646704574705, 51.3208025481261 ],
[ 3.37646695791282, 51.3208025061493 ],
[ 3.37608401443192, 51.3206231652693 ],
[ 3.37607169507628, 51.3206173959751 ],
[ 3.37606021048754, 51.320612017601 ],
[ 3.37582728410659, 51.3205029306946 ],
[ 3.37580409779263, 51.3206502985963 ],
[ 3.37575872019649, 51.3207993094705 ],
[ 3.37575476634361, 51.3208122883487 ],
[ 3.37571181656268, 51.3208797459348 ],
[ 3.3756624532907, 51.3209415238446 ],
[ 3.37557609963811, 51.3210110142077 ],
[ 3.37541089899821, 51.3211055871218 ],
[ 3.37477516102591, 51.3214102985009 ],
[ 3.37473173914127, 51.3214311108204 ],
[ 3.37455904622072, 51.3215138815012 ],
[ 3.37415098054777, 51.3217199232877 ],
[ 3.37313700916272, 51.3222422862785 ],
[ 3.37748824689601, 51.3242852920348 ],
[ 3.37749760805371, 51.3242713084009 ],
[ 3.37811903757028, 51.3233437635596 ],
[ 3.37818758851947, 51.3232647797363 ],
[ 3.37823803668144, 51.3232236798646 ],
[ 3.37837807779104, 51.3231095796538 ]
]
]
},

178
FarmmapsBulkSatDownload/ShowGeotiff.r 100644
View File

@ -0,0 +1,178 @@
# ShowGeotiff.r
# I downloaded and calculated the stats for the polygon defined in C:\git\FarmMapsApiClient_WURtest\FarmmapsDataDownload\DataDownloadInput.json
# in which I set "SatelliteBand": "wdvi" and in which in the console I requested the image for date '2020-09-22'
# FarmmapsBulkSatDownload generates many files. Here is what I tried when inputing the same field for a number of years using BulkSatDownloadInput.json
# see list below
library(raster)
library(sf)
library(rgdal)
setwd("C:/workdir/groenmonitor/DataDownload/")
# FarmmapsDataDownload and BulkSatDownload can be used to download zip files with inside two files:
fileGeotiff <- "wenr.tif"
fileJpg <- "thumbnail.jpg"
# Here is what I tried when inputing the same field for a number of years:
# fileGeotiff <- "wheat_fld5641_20210224.tif" # 2 files for year 2021. This file is a nice example having in upperleft corner no data
# fileGeotiff <- "wheat_fld5641_20210331.tif" # 2 files for year 2021
# fileGeotiff <- "wheat_fld5641_20200321.tif" # 14 files for year 2020, earliest
# fileGeotiff <- "wheat_fld5641_20200922.tif" # 14 files for year 2020, latest
# fileGeotiff <- "wheat_fld5641_20190121.tif" # 9 files for year 2019, earliest
# fileGeotiff <- "wheat_fld5641_20191117.tif" # 9 files for year 2019, latest
# 1 file for year 2018, with error message 'End of Central Directory record could not be found' and invalid wheat_fld5641_20180630.zip
# fileGeotiff <- "wheat_fld5641_20170526.tif" # 1 file for year 2017
# Zero files for 2016
lenfilename <- nchar(fileGeotiff)
year <- substr(fileGeotiff,lenfilename-11,lenfilename-8)
imgdate <- substr(fileGeotiff,lenfilename-11,lenfilename-4)
# The thumbnail has the polygon clipped out, has 1 layer, no crs and the mean value is not the mean wdvi we are looking for
r.thumbnail <- raster(fileJpg)
plot(r.thumbnail)
crs(r.thumbnail)
#CRS arguments: NA
cellStats(r.thumbnail,'mean') #87.5128 # nonsense
stk.wenr <- stack(x=fileGeotiff)
# plot(stk.wenr) shows 5 plots (5 bands)
# I think these are:
# 1. ndvi (since it runs from 0 to 1)
# 2. wdvi (since it runs from 0 to 0.5)
# 3-5: RGB (since they run from 0 to 255)
plot(stk.wenr)
# CRS arguments:
# +proj=sterea +lat_0=52.1561605555556 +lon_0=5.38763888888889 +k=0.9999079 +x_0=155000 +y_0=463000 +ellps=bessel +units=m +no_defs
# Or use st_crs(stk.wenr) to get more info, but no EPSG code in there.
# Likely it is epsg:28992 (Amersfoort)
crs(stk.wenr)
stk.wenr <- projectRaster(stk.wenr, crs = CRS('+init=EPSG:28992'))
crs(stk.wenr)
# Looks the same but strangely, if we don't do projectRaster(stk.wenr, crs = CRS('+init=EPSG:28992')), we find below the bottom left corner of the polygon missing
r.wenr.rd.wdvi <- subset(stk.wenr,2)
dev.off()
plot(r.wenr.rd.wdvi,main=paste("wdvi",imgdate),xlab="RDX",ylab="RDY")
cellStats(r.wenr.rd.wdvi,'mean') #0.1350561
# Furthermore we can see
# shows coordinates in RD
# returns a rectangle, thus the shape of the polygon submitted is not clipped.
# The polygon was provided in WGS84. Let's draw it on top.
# First convert the raster to WGS84
r.wenr.wgs84.wdvi <- projectRaster(r.wenr.rd.wdvi, crs = CRS('+init=EPSG:4326'))
# Draw a polygon on top of the raster
# Example polygon p1
# p1 <- data.frame(id = 1, wkt = 'POLYGON((4.963 52.801, 4.966 52.801, 4.966 52.803, 4.963 52.803, 4.963 52.801))')
# p1 <- st_as_sf(p1, wkt = 'wkt', crs = targetcrs)
# plot(p1,add=TRUE, col="transparent",border="black")
# Draw the polygon on top of the raster
# Polygon p2 from C:\git\FarmMapsApiClient_WURtest\FarmmapsDataDownload\DataDownloadInput.json
p2 <- data.frame(id = 1, wkt = gsub("\n","",'POLYGON((
4.960707146896585 52.800583669708487,
4.960645975538824 52.800470217610922,
4.962140695752897 52.799177147194797,
4.967523821195745 52.801502400041208,
4.966336768950911 52.802543735879809,
4.961711880764330 52.801009996856429,
4.960707146896585 52.800583669708487))'))
# Polygon p2 from C:\git\FarmMapsApiClient_WURtest\FarmmapsBulkSatDownload\BulkSatDownloadInput.json
p2 <- data.frame(id = 1, wkt = gsub("\n","",'POLYGON((
3.37837807779104 51.3231095796538,
3.38065689232502 51.3212527499355,
3.38022924592256 51.3210683536359,
3.37980548452565 51.3208801127141,
3.37959556105776 51.3207540143696,
3.3793691292654 51.3205959677371,
3.37822219207335 51.3215667913007,
3.37816999925795 51.3216109809456,
3.37646704574705 51.3208025481261,
3.37646695791282 51.3208025061493,
3.37608401443192 51.3206231652693,
3.37607169507628 51.3206173959751,
3.37606021048754 51.320612017601,
3.37582728410659 51.3205029306946,
3.37580409779263 51.3206502985963,
3.37575872019649 51.3207993094705,
3.37575476634361 51.3208122883487,
3.37571181656268 51.3208797459348,
3.3756624532907 51.3209415238446,
3.37557609963811 51.3210110142077,
3.37541089899821 51.3211055871218,
3.37477516102591 51.3214102985009,
3.37473173914127 51.3214311108204,
3.37455904622072 51.3215138815012,
3.37415098054777 51.3217199232877,
3.37313700916272 51.3222422862785,
3.37748824689601 51.3242852920348,
3.37749760805371 51.3242713084009,
3.37811903757028 51.3233437635596,
3.37818758851947 51.3232647797363,
3.37823803668144 51.3232236798646,
3.37837807779104 51.3231095796538))'))
p2.wgs84 <- st_as_sf(p2, wkt = 'wkt', crs = CRS('+init=EPSG:4326'))
# Or other way round, in RD Amersfoort. That looks ok
p2.rd <- st_transform(p2.wgs84, "+init=epsg:28992")
# Have a look at both
# wg84
dev.off()
plot(r.wenr.wgs84.wdvi,main=paste("wdvi",imgdate),xlab="LON",ylab="LAT")
plot(p2.wgs84,add=TRUE, col="transparent",border="red")
# RD
dev.off()
plot(r.wenr.rd.wdvi,main=paste("wdvi",imgdate),xlab="RDX",ylab="RDY")
plot(p2.rd,add=TRUE, col="transparent",border="red")
#Let's clip the polygon
r.wenr.rd.wdvi.pol <- mask(r.wenr.rd.wdvi,p2.rd)
r.wenr.wgs84.wdvi.pol <- mask(r.wenr.wgs84.wdvi,p2.wgs84)
dev.off()
plot(r.wenr.wgs84.wdvi.pol,main=paste("wdvi",imgdate),xlab="LON",ylab="LAT")
plot(p2.wgs84,add=TRUE, col="transparent",border="red")
#That's what we want! Now compare the stats
cellStats(r.wenr.rd.wdvi,'mean') # [1] 0.1350561 # Stats from rectangle, RD
cellStats(r.wenr.wgs84.wdvi,'mean') # [1] 0.1351411 # Stats from rectangle, WGS84
cellStats(r.wenr.rd.wdvi.pol,'mean') # [1] 0.05723957 # Stats from raster clipped by polygon, RD
cellStats(r.wenr.wgs84.wdvi.pol,'mean') # [1] 0.05723607 # Stats from raster clipped by polygon, WGS84
# file SatelliteDataStatistics_test_satData_wdvi_2020-09-22.csv
# "mean": 0.057430520945401985 # SatelliteDataStatistics_test_satData_wdvi_2020.csv returns stats for the clipped raster (.pol). 'mean' almost the same, maybe
# cellStats cannot return median, just a few stats.
cellStats(r.wenr.wgs84.wdvi.pol,'median') # Error in .local(x, stat, ...) : invalid 'stat'. Should be sum, min, max, sd, mean, or 'countNA'
r.wenr.wgs84.wdvi.vals <- values(r.wenr.wgs84.wdvi)
median(r.wenr.wgs84.wdvi.vals) # [1] NA
median(r.wenr.wgs84.wdvi.vals,na.rm=TRUE) # [1] 0.076
r.wenr.wgs84.wdvi.pol.vals <- values(r.wenr.wgs84.wdvi.pol)
median(r.wenr.wgs84.wdvi.pol.vals) # [1] NA
median(r.wenr.wgs84.wdvi.pol.vals,na.rm=TRUE) # [1] 0.048
# "median": 0.04800000041723251 # SatelliteDataStatistics_test_satData_wdvi_2020.csv returns stats for the clipped raster (.pol).
# An image may contain NA values. Check:
cellStats(r.wenr.wgs84.wdvi,'countNA') # [1] 22956
ncell(r.wenr.wgs84.wdvi) # [1] 221696
cellStats(r.wenr.wgs84.wdvi,'countNA') / ncell(r.wenr.wgs84.wdvi) # [1] 0.1035472 # 10% no data? doesn't show in the plot?
cellStats(r.wenr.wgs84.wdvi.pol,'countNA') # [1] 147387
summary(r.wenr.wgs84.wdvi.pol.vals) # shows the same: NA's: 147387
ncell(r.wenr.wgs84.wdvi.pol) # [1] 221696
cellStats(r.wenr.wgs84.wdvi.pol,'countNA') / ncell(r.wenr.wgs84.wdvi.pol) # [1] 0.6648158 # 66% no data? doesn't show in the plot?
# The project FarmmapsNbs can generate a wenr.tif file, application.tif, uptake.tif (in rtest1.uptake.zip)and shape.shp (in rtest1.taskmap.zip)
r.application <- raster("C:/git/FarmMapsApiClient_WURtest/FarmmapsNbs/bin/Debug/netcoreapp3.1/Downloads/application.tif")
dev.off()
plot(r.application)
plot(p2.rd,add=TRUE, col="transparent",border="red")
# The application.tif file is a rectangle (polygon not yet clipped), in projection Amersfoort RD New (EPSG:28992)
r.uptake <- raster("C:/git/FarmMapsApiClient_WURtest/FarmmapsNbs/bin/Debug/netcoreapp3.1/Downloads/uptake.tif")
dev.off()
plot(r.uptake)
plot(p2.rd,add=TRUE, col="transparent",border="red")
# The uptake.tif file is a rectangle (polygon not yet clipped), in projection Amersfoort RD New (EPSG:28992)
shp.wgs84 <- readOGR(dsn="C:/git/FarmMapsApiClient_WURtest/FarmmapsNbs/bin/Debug/netcoreapp3.1/Downloads", layer="shape")
crs(shp.wgs84)
# CRS arguments: +proj=longlat +datum=WGS84 +no_defs
dev.off()
plot(r.wenr.wgs84.wdvi,main="wdvi",xlab="LON",ylab="LAT")
plot(shp.wgs84,add=TRUE, col="transparent",border="black")
plot(p2.wgs84,add=TRUE, col="transparent",border="red")
# The shape file is in WGS84

125
FarmmapsDataDownload/DataDownloadApplication.cs
View File

@ -38,7 +38,7 @@ namespace FarmmapsDataDownload
public async Task RunAsync()
{
string fieldsInputJson = File.ReadAllText("DataDownloadInput.json");
var fieldsInputJson = File.ReadAllText("DataDownloadInput.json");
List<DataDownloadInput> fieldsInputs = JsonConvert.DeserializeObject<List<DataDownloadInput>>(fieldsInputJson);
@ -61,28 +61,6 @@ namespace FarmmapsDataDownload
private async Task Process(List<UserRoot> roots, DataDownloadInput input)
{
//PO20220311: first time a call is made to download satellite images or statistics, an empty list is returned
//If we wait a bit longer, e.g. 10 secs, then e.g. a list of 3 images may be returned
//If we wait still longer, maybe 4 images.
//The solution implemented below is to fire calls as long as the number of images returned keeps increasing
//While in between each call, sleep for sleepSecs
//Continue this until the number no longer increases or the maximum number of calls has been reached
//Out of politeness, don't be too impatient. Don't set sleepSecs to 5 or 10 or 30 secs. Just accept this may take a while, have a coffee, we suggest sleepSecs = 60;
int sleepSecs = 60;
int callCntMax = 4;
//For example we may set: "sleepSecs = 10;" and "callCntMax = 24;" and following result:
//Call no: 1. Giving FarmMaps 10 seconds to get SatelliteItems...
//Call no: 1: Received 2 images
//Call no: 2. Giving FarmMaps 10 seconds to get SatelliteItems...
//Call no: 2: Received 7 images
//Call no: 3. Giving FarmMaps 10 seconds to get SatelliteItems...
//Call no: 3: Received 7 images
//And the firing of calls would stop because the number of images returned is no longer increasing
//In the worst case, this could would lead to a total sleeping period of "sleepSecsSum = sleepSecs * callCntMax" seconds. After that we give up
//This is an ugly fix. Neater would if FarmMaps would just take a bit longer and then do always deliver all satellite images on first call.
//Once this has been fixed on the side of FarmMaps we can set callCntMax = 0 and the code below will work smoothly without any sleeping
string downloadFolder = input.DownloadFolder;
if (string.IsNullOrEmpty(downloadFolder)) {
downloadFolder = "Downloads";
@ -96,8 +74,7 @@ namespace FarmmapsDataDownload
var fieldName = input.fieldName;
bool storeSatelliteStatistics = input.StoreSatelliteStatisticsSingleImage;
bool storeSatelliteStatisticsCropYear = input.StoreSatelliteStatisticsCropYear;
//List<string> SatelliteBands = new List<string>(1) { input.SatelliteBand };
List<string> satelliteBands = input.SatelliteBands;
List<string> SatelliteBands = new List<string>(1) { input.SatelliteBand };
string headerLineStats = $"FieldName,satelliteDate,satelliteBand,max,min,mean,mode,median,stddev,minPlus,curtosis,maxMinus,skewness,variance,populationCount,variationCoefficient,confidenceIntervalLow, confidenceIntervalHigh,confidenceIntervalErrorMargin" + Environment.NewLine;
@ -189,58 +166,12 @@ namespace FarmmapsDataDownload
SaveSettings(settingsfile);
}
//Call first time
int callCnt = 1;
int sleepSecsSum = 0;
//if callCntMax == 0 then don't sleep
//if callCntMax = 1 then sleep first 1x
if (callCntMax > 0)
{
_logger.LogInformation($"Call no: {callCnt}. Giving FarmMaps {sleepSecs} seconds to get SatelliteItems...");
System.Threading.Thread.Sleep(1000 * sleepSecs);
sleepSecsSum = sleepSecsSum + sleepSecs;
}
// Select all satellite items
List<Item> satelliteItemsCropYear = await _generalService.FindSatelliteItems(cropfieldItem, _settings.SatelliteTaskCode);
int satelliteItemsCropYearCntPrev = satelliteItemsCropYear.Count;
_logger.LogInformation($"Call no: {callCnt}. Received {satelliteItemsCropYearCntPrev} images");
callCnt++;
int satelliteItemsCropYearCnt = satelliteItemsCropYearCntPrev;
//if callCntMax > 1 then sleep untill (1) no more increase in number of images received OR (2) maximum number of calls reached
if (callCntMax > 1)
{
//Call second time
_logger.LogInformation($"Call no: {callCnt}. Giving FarmMaps another {sleepSecs} seconds to get SatelliteItems...");
System.Threading.Thread.Sleep(1000 * sleepSecs);
satelliteItemsCropYear = await _generalService.FindSatelliteItems(cropfieldItem, _settings.SatelliteTaskCode);
satelliteItemsCropYearCnt = satelliteItemsCropYear.Count;
_logger.LogInformation($"Call no: {callCnt}. Received {satelliteItemsCropYearCnt} images");
sleepSecsSum = sleepSecsSum + sleepSecs;
//As long as there is progress, keep calling
callCnt++;
while (callCnt <= callCntMax && (satelliteItemsCropYearCnt == 0 || satelliteItemsCropYearCnt > satelliteItemsCropYearCntPrev))
{
_logger.LogInformation($"Surprise! The longer we wait, the more images we get. Sleep and call once more");
satelliteItemsCropYearCntPrev = satelliteItemsCropYearCnt;
_logger.LogInformation($"Call no: {callCnt} (max: {callCntMax}). Giving FarmMaps another {sleepSecs} seconds to get SatelliteItems...");
System.Threading.Thread.Sleep(1000 * sleepSecs);
satelliteItemsCropYear = await _generalService.FindSatelliteItems(cropfieldItem, _settings.SatelliteTaskCode);
satelliteItemsCropYearCnt = satelliteItemsCropYear.Count;
_logger.LogInformation($"Call no: {callCnt}. Received {satelliteItemsCropYearCnt} images");
callCnt++;
sleepSecsSum = sleepSecsSum + sleepSecs;
}
}
if (satelliteItemsCropYearCnt == 0)
{
_logger.LogWarning($"DataDownloadApplication.cs: after calling one or more times and " +
$"sleeping in total {sleepSecsSum} seconds, still no images found. " +
$"Please check your settings for parameters callCntMax and sleepSecs in DataDownloadApplication.cs or contact FarmMaps");
}
satelliteItemsCropYear = satelliteItemsCropYear.OrderBy(x => x.DataDate).ToList();
if (input.StoreSatelliteStatisticsSingleImage == true && satelliteItemsCropYearCnt > 0) {
if (input.StoreSatelliteStatisticsSingleImage == true) {
_logger.LogInformation("Available satellite images:");
var count = 0;
TimeSpan.FromSeconds(0.5);
@ -257,49 +188,43 @@ namespace FarmmapsDataDownload
var SatelliteDate = selectedSatelliteItem.DataDate.Value.ToString("yyyyMMdd");
string fileName = string.Format($"satelliteGeotiff_{fieldName}_{SatelliteDate}"); // no need to add satelliteBand in the name because the tif contains all bands
string fileNameZip = Path.Combine(downloadFolder, string.Format($"{fileName}.zip"));
await _farmmapsApiService.DownloadItemAsync(selectedSatelliteItem.Code, fileNameZip);
string fileNameZip = string.Format($"{fileName}.zip");
string fileNameGeotiff = string.Format($"{fileName}.tif");
await _farmmapsApiService.DownloadItemAsync(selectedSatelliteItem.Code, Path.Combine(downloadFolder, fileNameZip));
// Download a csv file with stats
List<Item> selectedSatelliteItems = new List<Item>(1) { selectedSatelliteItem };
List<Item> selectedSatalliteItems = new List<Item>(1) { selectedSatelliteItem };
string fileNameStats = Path.Combine(downloadFolder, string.Format($"satelliteStats_{fieldName}_{SatelliteDate}.csv"));
_logger.LogInformation($"First call to get DownloadSatelliteStats for selected image...");
string downloadedStats = await _generalService.DownloadSatelliteStats(selectedSatelliteItems, fieldName, satelliteBands, downloadFolder);
string downloadedStats = await _generalService.DownloadSatelliteStats(selectedSatalliteItems, fieldName, SatelliteBands, downloadFolder);
//rename the csv file with stats
//if the targe file already exists, delete it
File.Delete(fileNameStats);
//rename
File.Move(downloadedStats, fileNameStats);
// name the tif file
string fileNameTifzipped = Path.Combine(downloadFolder, string.Format($"sentinelhub_{SatelliteDate}.tif"));
string fileNameGeotiff = Path.Combine(downloadFolder, string.Format($"sentinelhub_{fieldName}_{SatelliteDate}.tif"));
// download the geotiffs. Returns a zip file with always these two files:
// wenr.tif. Contains 5 layers: (1) ndvi, (2) wdvi, (3) Red, (4) Green and (5) Blue
// download the geotiffs. Returns a zip file with always these three files:
// data.dat.aux.xml
// thumbnail.jpg
// sentinelhub_yyyyMMdd.tif. Contains 4 layers: (1) ndvi, (2) wdvi, (3) ci-red and (4) natural. Natural has 3 layers inside: redBand, blueBand and greenBand
// wenr.tif. Contains 5 layers: (1) ndvi, (2) wdvi, (3) Red, (4) Green and (5) Blue
if (true)
{
// Extract the file fileNameTifzipped from zip, rename it to fileNameGeotiff
ZipFile.ExtractToDirectory(fileNameZip, downloadFolder, true);
//if the targe file already exists, delete it
File.Delete(fileNameGeotiff);
//rename
File.Move(fileNameTifzipped, fileNameGeotiff);
// Extract the file "wenr.tif" from zip, rename it to fileNameGeotiff
ZipFile.ExtractToDirectory(Path.Combine(downloadFolder, fileNameZip), downloadFolder, true);
File.Delete(Path.Combine(downloadFolder, fileNameGeotiff)); // Delete the fileNameGeotiff file if exists
File.Move(Path.Combine(downloadFolder, "wenr.tif"), Path.Combine(downloadFolder, fileNameGeotiff)); // Rename the oldFileName into newFileName
// Cleanup
File.Delete(fileNameZip);
File.Delete(Path.Combine(downloadFolder, "thumbnail.jpg"));
string[] filesToDelete = new string[] { fileNameZip, "wenr.tif", "thumbnail.jpg", "data.dat.aux.xml" };
foreach (string f in filesToDelete)
{
File.Delete(Path.Combine(downloadFolder, f));
}
}
//_logger.LogInformation($"Downloaded files {fileNameGeotiff} and {fileNameStats} to {downloadFolder}");
_logger.LogInformation($"Downloaded files to {downloadFolder}");
_logger.LogInformation($"Downloaded files {fileNameGeotiff} and {fileNameStats} to {downloadFolder}");
}
if (input.StoreSatelliteStatisticsCropYear == true) {
string fileNameStats = Path.Combine(downloadFolder, string.Format($"satelliteStats_{fieldName}_{cropYear}.csv"));
File.Delete(fileNameStats);
_logger.LogInformation($"First call to get DownloadSatelliteStats for whole cropYear...");
string downloadedStats = await _generalService.DownloadSatelliteStats(satelliteItemsCropYear, fieldName, satelliteBands, downloadFolder);
string downloadedStats = await _generalService.DownloadSatelliteStats(satelliteItemsCropYear, fieldName, SatelliteBands, downloadFolder);
File.Move(downloadedStats, fileNameStats);
_logger.LogInformation($"Downloaded file {fileNameStats} with stats for field '{fieldName}', cropyear {cropYear}");
}

32
FarmmapsDataDownload/DataDownloadInput.json
View File

@ -1,30 +1,34 @@
[
{
"UseCreatedCropfield": false, // if false, program will make new CropfieldItemCode and SatelliteTaskCode; if true, the program will read CropfieldItemCode and SatelliteTaskCode from a file called "..\FarmmapsDataDownload\bin\Debug\netcoreapp3.1\Settings_{fieldName}.json", which will be faster
"outputFileName": "test_BvdTFieldlabG92",
"fieldName": "test_BvdTFieldlabG92",
"UseCreatedCropfield": false,
"outputFileName": "TestData",
"fieldName": "TestField",
"DownloadFolder": "Downloads", //"C:\\workdir\\groenmonitor\\", // "Downloads", -> if you just put "Downloads" the program will download to somewhere in ..\FarmMapsApiClient_WURtest\FarmmapsDataDownload\bin\Debug\netcoreapp3.1\Downloads\
"GetCropRecordings": false,
"GetCropRecordings": true,
"CrprecItem": "...", //item code of de crop recording parrent - can be found by opening the crop recording page of a field.
"GetShadowData": false,
"GetSatelliteData": true,
"SatelliteBands": [ "ndvi", "wdvi", "ci-red" ], // ["ndvi"] or ["wdvi"] or ["ci-red"] or multiple: [ "wdvi", "ndvi" ]
"StoreSatelliteStatisticsSingleImage": true,
"StoreSatelliteStatisticsCropYear": true,
"GetSatelliteData": false,
"SatelliteBand": "wdvi", // "natural", "ndvi" or "wdvi"
"StoreSatelliteStatisticsSingleImage": false,
"StoreSatelliteStatisticsCropYear": false,
"GetVanDerSatData": false,
"StoreVanDerSatStatistics": false,
"CropYear": 2022,
"CropYear": 2020,
"geometryJson": {
"type": "Polygon",
"coordinates": [
[
[ 5.563472073408009, 52.547554398144172 ],
[ 5.567425915520115, 52.547725375100377 ],
[ 5.567917474269188, 52.540608459298582 ],
[ 5.563878143678981, 52.54048022658143 ],
[ 5.563472073408009, 52.547554398144172 ]
[ 4.960707146896585, 52.800583669708487 ],
[ 4.960645975538824, 52.800470217610922 ],
[ 4.962140695752897, 52.799177147194797 ],
[ 4.967523821195745, 52.801502400041208 ],
[ 4.966336768950911, 52.802543735879809 ],
[ 4.961711880764330, 52.801009996856429 ],
[ 4.960707146896585, 52.800583669708487 ]
]
]
}
}
]

3
FarmmapsDataDownload/Models/DataDownloadInput.cs
View File

@ -1,5 +1,4 @@
using System;
using System.Collections.Generic;
using Newtonsoft.Json.Linq;
namespace FarmmapsDataDownload.Models
@ -17,7 +16,7 @@ namespace FarmmapsDataDownload.Models
public string fieldName { get; set; }
public bool GetSatelliteData { get; set; }
public bool GetVanDerSatData { get; set; }
public List<string> SatelliteBands { get; set; }
public string SatelliteBand { get; set; }
public bool StoreSatelliteStatisticsSingleImage { get; set; }
public bool StoreSatelliteStatisticsCropYear { get; set; }
public bool StoreVanDerSatStatistics { get; set; }

114
FarmmapsDataDownload/ShowGeotiff.r
View File

@ -1,114 +0,0 @@
# ShowGeotiff.r
# Have a look at a downloaded satellite image and check if stats are correctly calculated
# I downloaded and calculated the stats for the polygon defined in C:\git\FarmMapsApiClient_WURtest\FarmmapsDataDownload\DataDownloadInput.json
# in which I set "SatelliteBand": "wdvi" and in which in the console I requested the image for date '2022-03-08'
library(raster)
library(sf)
library(rgdal)
setwd("C:/git/FarmMapsApiClient_WURtest/FarmmapsDataDownload/bin/Debug/netcoreapp3.1/Downloads")
# FarmmapsDataDownload
fileGeotiff <- "sentinelhub_test_BvdTFieldlabG92_20220308.tif"
lenfilename <- nchar(fileGeotiff)
year <- substr(fileGeotiff,lenfilename-11,lenfilename-8)
imgdate <- substr(fileGeotiff,lenfilename-11,lenfilename-4)
stk.sentinelhub <- stack(x=fileGeotiff)
# plot(stk.sentinelhub) shows 6 plots (6 bands)
# 1. ndvi
# 2. wdvi Note wdvi-red
# 3. ci-red
# 4. natural: red
# 5. natural: green
# 6. natural: blue
names(stk.sentinelhub) <- c("ndvi","wdvired","ci-red","red","green","blue")
plot(stk.sentinelhub)
crs(stk.sentinelhub)
# CRS arguments: +proj=longlat +datum=WGS84 +no_defs
stk.sentinelhub.rd <- projectRaster(stk.sentinelhub, crs = CRS('+init=EPSG:28992'))
crs(stk.sentinelhub)
r.sentinelhub.rd.wdvi <- subset(stk.sentinelhub.rd,2)
dev.off()
plot(r.sentinelhub.rd.wdvi,main=paste("wdvi",imgdate),xlab="RDX",ylab="RDY")
cellStats(r.sentinelhub.rd.wdvi,'mean') # 0.2252725
# Convert the .rd.wdvi raster to WGS84
r.sentinelhub.wgs84.wdvi <- projectRaster(r.sentinelhub.rd.wdvi, crs = CRS('+init=EPSG:4326'))
# Draw a polygon on top of the raster
# Polygon pol from C:\git\FarmMapsApiClient_WURtest\FarmmapsDataDownload\DataDownloadInput.json
pol <- data.frame(id = 1, wkt = gsub("\n","",'POLYGON((
5.563472073408009 52.547554398144172,
5.567425915520115 52.547725375100377,
5.567917474269188 52.540608459298582,
5.563878143678981 52.54048022658143,
5.563472073408009 52.547554398144172
))'))
pol.wgs84 <- st_as_sf(pol, wkt = 'wkt', crs = CRS('+init=EPSG:4326'))
pol.rd <- st_transform(pol.wgs84, "+init=epsg:28992")
#Calculate approximate middle of polygon
res <- as.data.frame(do.call("rbind", lapply(st_geometry(pol.wgs84), st_bbox)))
res$latmid <- (res$ymax+res$ymin)/2.0
res$lonmid <- (res$xmax+res$xmin)/2.0
res
# xmin ymin xmax ymax latmid lonmid
# 1 5.563472 52.54048 5.567917 52.54773 52.5441 5.565695
# Have a look at both polygons
# wg84
dev.off()
plot(r.sentinelhub.wgs84.wdvi,main=paste("wdvi",imgdate),xlab="LON",ylab="LAT")
plot(pol.wgs84,add=TRUE, col="transparent",border="red")
# RD
dev.off()
plot(r.sentinelhub.rd.wdvi,main=paste("wdvi",imgdate),xlab="RDX",ylab="RDY")
plot(pol.rd,add=TRUE, col="transparent",border="red")
# Clip the polygon from the full rectangle figure
r.sentinelhub.rd.wdvi.pol <- mask(r.sentinelhub.rd.wdvi,pol.rd)
r.sentinelhub.wgs84.wdvi.pol <- mask(r.sentinelhub.wgs84.wdvi,pol.wgs84)
dev.off()
plot(r.sentinelhub.wgs84.wdvi.pol,main=paste("wdvi",imgdate),xlab="LON",ylab="LAT")
plot(pol.wgs84,add=TRUE, col="transparent",border="red")
#That's what we want!
# Now compare the stats
cellStats(r.sentinelhub.wgs84.wdvi,'mean') # [1] 0.2250987 # Stats from rectangle, WGS84
cellStats(r.sentinelhub.rd.wdvi,'mean') # [1] 0.2252725 # Stats from rectangle, RD. Almost but not exactly same as above
cellStats(r.sentinelhub.wgs84.wdvi.pol,'mean') # [1] 0.2275067 # Stats from raster clipped by polygon, WGS84
cellStats(r.sentinelhub.rd.wdvi.pol,'mean') # [1] 0.2275073 # Stats from raster clipped by polygon, RD. Almost but not exactly same as above
# file satelliteStats_test_BvdTFieldlabG92_20220308.csv
# "wdvi" "mean": 0.22744397204465
# Mean in csv corresponds with cellStats calculated from clipped tif!
# So while the tif returned is a non-clipped image, the downloaded statistics are from the clipped image
# Exactly as we wanted.
cellStats(r.sentinelhub.wgs84.wdvi.pol,'median') # Error in .local(x, stat, ...) : invalid 'stat'. Should be sum, min, max, sd, mean, or 'countNA'
r.sentinelhub.wgs84.wdvi.vals <- values(r.sentinelhub.wgs84.wdvi)
median(r.sentinelhub.wgs84.wdvi.vals) # [1] NA
median(r.sentinelhub.wgs84.wdvi.vals,na.rm=TRUE) # [1] 0.2318
r.sentinelhub.wgs84.wdvi.pol.vals <- values(r.sentinelhub.wgs84.wdvi.pol)
median(r.sentinelhub.wgs84.wdvi.pol.vals) # [1] NA
median(r.sentinelhub.wgs84.wdvi.pol.vals,na.rm=TRUE) # [1] 0.2338
# file satelliteStats_test_BvdTFieldlabG92_20220308.csv
# "wdvi" "mean": 0.233799993991851
# Median is same as for median(r.sentinelhub.wgs84.wdvi.pol.vals,na.rm=TRUE)
# in csv corresponds with cellStats calculated from clipped tif!
# So while the tif returned is a non-clipped image, the downloaded statistics are from the clipped image
# Exactly as we wanted.
cellStats(r.sentinelhub.wgs84.wdvi,'countNA') # [1] 27896
ncell(r.sentinelhub.wgs84.wdvi) # [1] 272718
cellStats(r.sentinelhub.wgs84.wdvi,'countNA') / ncell(r.sentinelhub.wgs84.wdvi) # [1] 0.1022888 # 10% no data? doesn't show in the plot?
cellStats(r.sentinelhub.wgs84.wdvi.pol,'countNA') # [1] 57625
summary(r.sentinelhub.wgs84.wdvi.pol.vals) # shows the same: NA's: 57625
ncell(r.sentinelhub.wgs84.wdvi.pol) # [1] 272718
populationCount = ncell(r.sentinelhub.wgs84.wdvi.pol) - cellStats(r.sentinelhub.wgs84.wdvi.pol,'countNA')
populationCount # [1] 215093
# file satelliteStats_test_BvdTFieldlabG92_20220308.csv
# "wdvi" "populationCount": 214688
# similar but not same

178
FarmmapsDownloadCL/DownloadCLApplication.cs 100644
View File

@ -0,0 +1,178 @@
using System;
using System.Collections.Generic;
using System.IO;
using System.IO.Compression;
using System.Linq;
using System.Reflection;
using System.Threading.Tasks;
using FarmmapsApi;
using FarmmapsApi.Models;
using FarmmapsApi.Services;
using FarmmapsDownloadCL.Models;
using Microsoft.Extensions.Logging;
using Newtonsoft.Json;
using Newtonsoft.Json.Linq;
using static FarmmapsApiSamples.Constants;
namespace FarmmapsDownloadCL
{
public class DownloadCLApplication : IApplication
{
private readonly ILogger<DownloadCLApplication> _logger;
private readonly FarmmapsApiService _farmmapsApiService;
private readonly DownloadCLService _DownloadCLService;
private readonly GeneralService _generalService;
public readonly Dictionary<string, List<string>> _dictCl;
string _itemcode;
public DownloadCLApplication(ILogger<DownloadCLApplication> logger, FarmmapsApiService farmmapsApiService,
GeneralService generalService, DownloadCLService DownloadCLService)
{
_logger = logger;
_farmmapsApiService = farmmapsApiService;
_generalService = generalService;
_DownloadCLService = DownloadCLService;
CodelistsClasses clc = new CodelistsClasses();
_itemcode = clc.itemcode;
_dictCl = clc.dictCl;
}
public async Task RunAsync()
{
var fieldsInputJson = File.ReadAllText("DownloadCLInput.json");
DownloadCLInput clInput = JsonConvert.DeserializeObject<DownloadCLInput>(fieldsInputJson);
// !! this call is needed the first time an api is called with a fresh clientid and secret !!
await _farmmapsApiService.GetCurrentUserCodeAsync();
var roots = await _farmmapsApiService.GetCurrentUserRootsAsync();
//Where to write the output
string downloadFolder = clInput.DownloadFolder;
if (string.IsNullOrEmpty(downloadFolder))
{
downloadFolder = "Downloads";
}
if (!Directory.Exists(downloadFolder))
Directory.CreateDirectory(downloadFolder);
//Get the most recent codelists
foreach (string codelist in clInput.codelists)
{
try
{
await Process(roots, codelist, downloadFolder);
}
catch (Exception ex)
{
_logger.LogError(ex.Message);
}
}
}
private async Task Process(List<UserRoot> roots, string codelistname, string downloadFolder)
{
List<Item> codelist;
string itemtype;
string className;
string body;
string header;
string value;
string[] aboutArray = null;
string[] headerArray = null;
PropertyInfo[] props;
PropertyInfo prop;
List<string> dataList;
string clJson = Path.Combine(downloadFolder, $"{codelistname}.json");
string clCsv = Path.Combine(downloadFolder, $"{codelistname}.csv");
try
{
itemtype = _dictCl[codelistname][0];
className = _dictCl[codelistname][1];
codelist = await _farmmapsApiService.GetItemChildrenAsync(_itemcode, itemtype);
//Write full codelist in json format to clJson.
body = JsonConvert.SerializeObject(codelist);
File.WriteAllText(clJson, body);
_logger.LogInformation($"Downloaded file {clJson}");
//Write full list in csv format to clCsv.
StreamWriter sw = new StreamWriter(clCsv);
//Write metadata in top: when downloaded
sw.WriteLine($"Editeelt codelist {codelistname} downloaded on {DateTime.Now} with the FarmmapsDownloadCL application");
//Generic, for any Codelist as long as it is also defined in FarmmapsDownloadCL.Models in
string typerequest = "FarmmapsDownloadCL.Models." + className + ", FarmmapsDownloadCL, Version=1.0.0.0, Culture=neutral, PublicKeyToken=null";
Type objType = Type.GetType(typerequest);
object clitem = Activator.CreateInstance(objType);
props = clitem.GetType().GetProperties();
//Lookup about info and write to top
foreach (PropertyInfo pi in props)
{
if (pi.Name == "about")
{
aboutArray = (string[])pi.GetValue(clitem);
foreach (string about in aboutArray)
{
//Add quotes
value = "\"" + about + "\"";
//Write each about to a separate line in the csv file
sw.WriteLine(value);
}
}
}
//Write an empty line
sw.WriteLine();
//Look up the headerArray and write the header comma separated
int i = 0;
while (headerArray == null)
{
prop = props[i];
if (prop.Name == "headers")
{
headerArray = (string[])prop.GetValue(clitem);
}
i++;
}
header = string.Join(",", headerArray);
sw.WriteLine(header);
//Add quotes because a value for field 'description' could be '21% Cl, 16.5%CaO, 1330 g/l' and without the quotes that
//would show up as separate columns. So per definition we will add quotes, except if we have a fieldname in the listNotToQuote, then value for that fieldname will not be given quotes
//e.g. not code "3" but code 3.
List<string> listNotToQuote = new List<string>() { "code", "n", "p", "k", "quantity", "cropGroupCode", "cultivationGroupCode" };
//Loop through all items in the codelist
foreach (Item item in codelist)
{
dataList = new List<string> { };
clitem = JsonConvert.DeserializeObject(item.Data.ToString(), objType);
//Add values of the cLitem to the dataList in the same order as the header
foreach (string h in headerArray)
{
value = (string)clitem.GetType().GetProperty(h).GetValue(clitem, null);
if(listNotToQuote.Contains(h) == false)
value = "\"" +value + "\"";
dataList.Add(value);
}
string dataLine = string.Join(",", dataList);
sw.WriteLine(string.Join(",", dataList));
}
sw.Close();
_logger.LogInformation($"Downloaded file {clCsv}");
}
catch
{
_logger.LogWarning($"Missing lookup information on codelist {codelistname}");
}
}
}
}

5
FarmmapsDownloadCL/DownloadCLInput.json 100644
View File

@ -0,0 +1,5 @@
{
"codeLists": [ "CL022", "CL127", "CL263", "CL265", "CL405", "CL251", "CL042" ]
//"codeLists": [ "CL263", "CL265"]
//"DownloadFolder": "Downloads", //"C:\\hugoschrererdir\\kpidir\\", // "Downloads", -> if you just put "Downloads" the program will download to somewhere in ..\FarmMapsApiClient_WURtest\FarmmapsDataDownload\bin\Debug\netcoreapp3.1\Downloads\
}

27
FarmmapsDownloadCL/DownloadCLService.cs 100644
View File

@ -0,0 +1,27 @@
using System;
using System.Globalization;
using System.Threading.Tasks;
using FarmmapsApi.Models;
using FarmmapsApi.Services;
using FarmmapsDownloadCL.Models;
using Microsoft.Extensions.Logging;
using static FarmmapsApi.Extensions;
using static FarmmapsApiSamples.Constants;
namespace FarmmapsDownloadCL
{
public class DownloadCLService
{
private readonly ILogger<DownloadCLService> _logger;
private readonly FarmmapsApiService _farmmapsApiService;
private readonly GeneralService _generalService;
public DownloadCLService(ILogger<DownloadCLService> logger, FarmmapsApiService farmmapsApiService,
GeneralService generalService)
{
_logger = logger;
_farmmapsApiService = farmmapsApiService;
_generalService = generalService;
}
}
}

24
FarmmapsDownloadCL/FarmmapsDownloadCL.csproj 100644
View File

@ -0,0 +1,24 @@
<Project Sdk="Microsoft.NET.Sdk">
<PropertyGroup>
<OutputType>Exe</OutputType>
<TargetFramework>netcoreapp3.1</TargetFramework>
<PlatformTarget>x64</PlatformTarget>
<Platforms>AnyCPU;x64</Platforms>
</PropertyGroup>
<ItemGroup>
<PackageReference Include="System.Collections" Version="4.3.0" />
</ItemGroup>
<ItemGroup>
<ProjectReference Include="..\FarmmapsApi\FarmmapsApi.csproj" />
</ItemGroup>
<ItemGroup>
<None Update="DownloadCLInput.json">
<CopyToOutputDirectory>Always</CopyToOutputDirectory>
</None>
</ItemGroup>
</Project>

152
FarmmapsDownloadCL/Models/CodelistsClasses.cs 100644
View File

@ -0,0 +1,152 @@
using System;
using System.Collections.Generic;
using System.Reflection;
using System.Text;
namespace FarmmapsDownloadCL.Models
{
public class CodelistsClasses
{
public CodelistsClasses()
{
this.itemcode = "938c0aec4efc46e88783fd029f17309e%3ASYSTEM_DATA";
this.dictCl = new Dictionary<string, List<string>>();
//Add here once you have defined a new CLxxxitem below
//To add a new item, look it up in https://test.farmmaps.eu/swagger/index.html. Download the item you are interested in (like "vnd.farmmaps.itemtype.codelist.cl263")
//then below create a new CLxxxitem with fieldnames as you see them in the just downloaded new codelist
this.dictCl.Add("CL022", new List<string> { "vnd.farmmaps.itemtype.codelist.cl022", "CL022item" });
this.dictCl.Add("CL127", new List<string> { "vnd.farmmaps.itemtype.codelist.cl127", "CL127item" });
this.dictCl.Add("CL251", new List<string> { "vnd.farmmaps.itemtype.codelist.cl251", "CL251item" });
this.dictCl.Add("CL263", new List<string> { "vnd.farmmaps.itemtype.codelist.cl263", "CL263item" });
this.dictCl.Add("CL265", new List<string> { "vnd.farmmaps.itemtype.codelist.cl265", "CL265item" });
this.dictCl.Add("CL405", new List<string> { "vnd.farmmaps.itemtype.codelist.cl405", "CL405item" });
}
public string itemcode;
//Define here the codelist names, their item type and the class defining the contents of the data type (and defined in the Models.CodelistsClasses)
public Dictionary<string, List<string>> dictCl;
}
public class CL022item
{
public CL022item()
{
this.headers = new string[] { "codelist", "code", "description", "type", "n", "p", "k", "composition" };
this.codelist = "CL022";
this.about = new string[] { "EDI-Crop, coderingslijst meststoffen", "type MOR = organische meststof; MAN = anorganische meststof" };
}
public string[] headers { get; }
public string codelist { get; }
public string[] about { get; }
public string k { get; set; }
public string n { get; set; }
public string p { get; set; }
public string code { get; set; }
public string type { get; set; }
public string composition { get; set; }
public string description { get; set; }
}
public class CL127item
{
public CL127item()
{
this.headers = new string[] { "codelist", "code", "description", "description_nl", "culturalPracticeLevel1", "culturalPracticeLevel2" };
this.codelist = "CL127";
}
public string[] headers { get; }
public string codelist { get; }
public string code { get; set; }
public string description { get; set; }
public string description_nl { get; set; }
public string culturalPracticeLevel1 { get; set; }
public string culturalPracticeLevel2 { get; set; }
}
public class CL251item
{
public CL251item()
{
this.headers = new string[] { "codelist", "code", "rvo", "active", "primary", "description", "description_fr", "description_nl" };
this.codelist = "CL251";
this.about = new string[] { "Codering teeltdoelen (CropProductionPurposeCode)", "primary: 1 = primary; 2 = secondary", "rvo: 1 = yes; 0 = no" };
}
public string[] headers { get; }
public string codelist { get; }
public string[] about { get; }
public string code { get; set; }
public string rvo { get; set; }
public string active { get; set; }
public string primary { get; set; }
public string description { get; set; }
public string description_nl { get; set; }
public string description_fr { get; set; }
}
public class CL263item
{
public CL263item()
{
this.headers = new string[] { "codelist", "code", "eppoCode", "botanicName", "description", "description_fr", "description_nl", "cropGroupCode", "cropGroupName", "subCropGroupCode", "subCropGroupName", "cultivationGroupCode", "cultivationGroupName" };
this.codelist = "CL263";
this.about = new string[] { "EDI-Crop, coderingslijst gewassoorten" };
}
public string[] headers { get; }
public string codelist { get; }
public string[] about { get; }
public string code { get; set; }
public string eppoCode { get; set; }
public string botanicName { get; set; }
public string description { get; set; }
public string description_nl { get; set; }
public string description_fr { get; set; }
public string cropGroupCode { get; set; }
public string cropGroupName { get; set; }
public string subCropGroupCode { get; set; }
public string subCropGroupName { get; set; }
public string cultivationGroupCode { get; set; }
public string cultivationGroupName { get; set; }
public string composition { get; set; } //?
}
public class CL265item
{
public CL265item()
{
this.headers = new string[] { "Code", "Formaat", "Eenheid", "Waarde" };
this.codelist = "CL265";
this.about = new string[] { "EDI-Crop CL265: FarmCharacterictics, FieldCharacteristics, CropFieldCharacteristics, TreatmentCharacteristics",
"Betreft codelijst eigenschappen teelt en teelthandelingen CL265.Daar waar in kolom 'Waarde' wordt verwezen naar CL.., wordt de codelijst bedoeld die onder het betreffende nummer bekend is.",
"In het EDI-Crop bericht wordt de code gebruikt onder resp. CropFieldCharacteristic c.q.TreatmentCharacteristic middels de dataelementen PropertyVariableCode (invullen vanuit CL265) en PropertyVariableValue(voor de waarde)"};
}
public string[] headers { get; }
public string codelist { get; }
public string[] about { get; }
public string Code { get; set; }
public string Formaat { get; set; }
public string Eenheid { get; set; }
public string Waarde { get; set; }
}
public class CL405item
{
public CL405item()
{
this.headers = new string[] { "codelist", "code", "description", "description_fr", "description_nl", "hoofdgroep" };
this.codelist = "CL405";
this.about = new string[] { };
}
public string[] headers { get; }
public string codelist { get; }
public string[] about { get; }
public string code { get; set; }
public string hoofdgroep { get; set; }
public string description { get; set; }
public string description_fr { get; set; }
public string description_nl { get; set; }
}
}

12
FarmmapsDownloadCL/Models/DownloadCLInput.cs 100644
View File

@ -0,0 +1,12 @@
using System;
using Newtonsoft.Json.Linq;
namespace FarmmapsDownloadCL.Models
{
public class DownloadCLInput
{
public string[] codelists { get; set; }
public string DownloadFolder { get; set; }
}
}

21
FarmmapsDownloadCL/Program.cs 100644
View File

@ -0,0 +1,21 @@
using System.Threading.Tasks;
using FarmmapsApi;
using Microsoft.Extensions.DependencyInjection;
using Microsoft.Extensions.Logging;
namespace FarmmapsDownloadCL
{
class Program : FarmmapsProgram<DownloadCLApplication>
{
private static async Task Main(string[] args)
{
await new Program().Start(args);
}
protected override void Configure(IServiceCollection serviceCollection)
{
serviceCollection.AddLogging()
.AddTransient<DownloadCLService>();
}
}
}

2
FarmmapsHaulmkilling/FarmmapsHaulmkilling.csproj
View File

@ -2,7 +2,7 @@
<PropertyGroup>
<OutputType>Exe</OutputType>
<TargetFramework>netcoreapp3.0</TargetFramework>
<TargetFramework>netcoreapp3.1</TargetFramework>
</PropertyGroup>
<ItemGroup>

2
FarmmapsHerbicide/FarmmapsHerbicide.csproj
View File

@ -2,7 +2,7 @@
<PropertyGroup>
<OutputType>Exe</OutputType>
<TargetFramework>netcoreapp3.0</TargetFramework>
<TargetFramework>netcoreapp3.1</TargetFramework>
</PropertyGroup>
<ItemGroup>

20
FarmmapsKPI/FarmmapsKPI.csproj 100644
View File

@ -0,0 +1,20 @@
<Project Sdk="Microsoft.NET.Sdk">
<PropertyGroup>
<OutputType>Exe</OutputType>
<TargetFramework>netcoreapp3.1</TargetFramework>
<PlatformTarget>x64</PlatformTarget>
<Platforms>AnyCPU;x64</Platforms>
</PropertyGroup>
<ItemGroup>
<ProjectReference Include="..\FarmmapsApi\FarmmapsApi.csproj" />
</ItemGroup>
<ItemGroup>
<None Update="KPIInput.json">
<CopyToOutputDirectory>Always</CopyToOutputDirectory>
</None>
</ItemGroup>
</Project>

759
FarmmapsKPI/KPIApplication.cs 100644
View File

@ -0,0 +1,759 @@
using System;
using System.Collections.Generic;
using System.Diagnostics;
using System.IO;
using System.IO.Compression;
using System.Linq;
using System.Threading.Tasks;
using FarmmapsApi;
using FarmmapsApi.Models;
using FarmmapsApi.Services;
using FarmmapsKPI.Models;
using Microsoft.Extensions.Logging;
using Newtonsoft.Json;
using Newtonsoft.Json.Linq;
using static FarmmapsApiSamples.Constants;
namespace FarmmapsKPI
{
public class KPIApplication : IApplication
{
//private const string DownloadFolder = "Downloads";
private const string SettingsFile = "settings.json";
private readonly ILogger<KPIApplication> _logger;
private readonly FarmmapsApiService _farmmapsApiService;
private readonly KPIService _kpiService;
private readonly GeneralService _generalService;
private Settings _settings;
public KPIApplication(ILogger<KPIApplication> logger, FarmmapsApiService farmmapsApiService,
GeneralService generalService, KPIService kpiService)
{
_logger = logger;
_farmmapsApiService = farmmapsApiService;
_generalService = generalService;
_kpiService = kpiService;
}
public async Task RunAsync()
{
KPIInput input;
string fnKPIinput;
//Console.WriteLine("Type name of input json file. Example: KPIinput.json (in same directory as FarmmapsKPI.exe) or for example like this: C:/temp/KPIinputChemieTmp.json");
//fnKPIinput = Console.ReadLine();
//if (string.IsNullOrEmpty(fnKPIinput))
//{
fnKPIinput = "KPIinput.json";
//}
var fieldsInputJson = File.ReadAllText(fnKPIinput);
List<KPIInput> fieldsInputs = JsonConvert.DeserializeObject<List<KPIInput>>(fieldsInputJson);
// !! this call is needed the first time an api is called with a fresh clientid and secret !!
await _farmmapsApiService.GetCurrentUserCodeAsync();
var roots = await _farmmapsApiService.GetCurrentUserRootsAsync();
//Where to write the output
string downloadFolder = fieldsInputs[0].DownloadFolder;
if (string.IsNullOrEmpty(downloadFolder))
{
downloadFolder = "Downloads";
}
if (!Directory.Exists(downloadFolder))
Directory.CreateDirectory(downloadFolder);
//Write the same info to a single csv file. Note this means existing file will be overwritten!
StreamWriter sw;
string KPIItemCsv = Path.GetFileNameWithoutExtension(fnKPIinput) + "_Items.csv";
string KPIItemPathCsv = Path.Combine(downloadFolder, KPIItemCsv);
List<string> headerList = new List<string> { "parentName", "cropfieldcode", "area_ha", "cropTypeCode", "cropTypeName", "KPIid", "KPIvariable", "KPIvalue", "KPIunit", "KPItargetvalue", "KPIthresholdValue",
"mbp_productCode","mbp_productName","mbp_quantity","mbp_unitCode","mbp_date","mbp_KPIvariable","mbp_KPIvalue"};
//Create a new csv file. Means if existing then overwritten !!!
sw = new StreamWriter(KPIItemPathCsv);
Configuration cf = await _farmmapsApiService.GetConfiguration();
string endPoint = cf.Endpoint;
sw.WriteLine($"Using FarmmapsKPI application in FarmmapsApSamples.sln. Input file: '{fnKPIinput}'. Download DateTime: '{DateTime.Now}'. Calculations on: '{endPoint}'");
sw.WriteLine();
sw.WriteLine(string.Join(",", headerList));
// For each input download all KPI's. Keep track to time, important when doing bulk calculations
var watch = System.Diagnostics.Stopwatch.StartNew();
TimeSpan tsSofar = new TimeSpan();
TimeSpan tsRemaining;
TimeSpan tsTotalEstimated;
for (int i = 0; i < fieldsInputs.Count; i++)
//for (int i = 10; i < 12; i++)
{
watch.Restart();
input = fieldsInputs[i];
_logger.LogInformation(string.Format($"// FarmmapsKPI: Downloading KPI's for field {i + 1} out of {fieldsInputs.Count} to single csv file {KPIItemPathCsv}"));
try
{
await Process(roots, input, sw);
}
catch (Exception ex)
{
_logger.LogError(ex.Message);
}
watch.Stop();
tsSofar = tsSofar + watch.Elapsed;
tsTotalEstimated = tsSofar / (i + 1) * fieldsInputs.Count;
tsRemaining = tsTotalEstimated - tsSofar;
_logger.LogInformation(string.Format($"// Time (hh:mm:ss): this field: {strTime(watch.Elapsed)}. Sofar: {strTime(tsSofar)}. Estimated total: {strTime(tsTotalEstimated)}. Remaining: {strTime(tsRemaining)}"));
}
//Close the csv file, write message to screen
sw.Close();
_logger.LogInformation(string.Format($"// FarmmapsKPI:"));
_logger.LogInformation($"Done! Written all KPI for all fields in '{fnKPIinput}' to output file '{KPIItemPathCsv}'");
}
private async Task Process(List<UserRoot> roots, KPIInput input, StreamWriter sw)
{
List<Item> cropfieldChildren;
List<Item> crprecChildren;
KPIOutput kpio;
KPIOutput kpioPrevious = new KPIOutput(); //creates a new empty
//If KPI E1 is calculated, write these sub kpi's to output
string[] mbp_KPIvariables = new string[] { "aquaticLife", "groundWater", "soilLife" };
string mbp_KPIvalue;
int targetKPIitemsCount; //if we know we should be getting 8 KPI items (A1,B1,B2,C1,D1,E1,F1,F2)
int maxtries = 5; // but don't keep on trying forever; there is a maximum number of tries
int trycnt;
string downloadFolder = input.DownloadFolder;
if (string.IsNullOrEmpty(downloadFolder)) {
downloadFolder = "Downloads";
}
if (!Directory.Exists(downloadFolder))
Directory.CreateDirectory(downloadFolder);
// !!specify if you are using an already created cropfield:
bool useExistingCropfieldWithChildren = input.UseExistingCropfieldWithChildren;
int cropYear = input.CropYear;
string fieldName = input.fieldName;
string fieldGeom = input.GeometryJson.ToString(Formatting.None);
//Settings
string settingsfile = $"Settings_{fieldName}.json";
LoadSettings(settingsfile);
var uploadedRoot = roots.SingleOrDefault(r => r.Name == "USER_IN");
if (uploadedRoot == null)
{
_logger.LogError("Could not find a needed root item");
return;
}
var myDriveRoot = roots.SingleOrDefault(r => r.Name == "USER_FILES");
if (myDriveRoot == null)
{
_logger.LogError("Could not find a needed root item");
return;
}
// Use already created cropfield or create new one, added a Data input, with field specific data for the KPI calculation
Item cropfieldItem;
//1 useExistingCropfieldWithChildren = false -> create new
//2 useExistingCropfieldWithChildren = true && input.CropfieldItemCode = "" or absent -> read from settings json
//3 useExistingCropfieldWithChildren = true && input.CropfieldItemCode like "deb48a74c5b54299bb852f17288010e9" in KPIinput -> use this one
if (useExistingCropfieldWithChildren == false)
{
_logger.LogInformation($"Creating cropfield with name '{fieldName}'");
cropfieldItem = await _generalService.CreateCropfieldItemAsync(myDriveRoot.Code,
$"{fieldName}", cropYear, input.GeometryJson.ToString(Formatting.None), input.DataCropfield.ToString(Formatting.None), input.StartDate, input.EndDate);
_settings.CropfieldItemCode = cropfieldItem.Code;
SaveSettings(settingsfile);
}
else if (string.IsNullOrEmpty(input.CropfieldItemCode))
{
_logger.LogInformation("reading CropfieldItemCode from settings file");
cropfieldItem = await _farmmapsApiService.GetItemAsync(_settings.CropfieldItemCode);
}
else
{
_logger.LogInformation("reading CropfieldItemCode from KPIinput json");
cropfieldItem = await _farmmapsApiService.GetItemAsync(input.CropfieldItemCode);
SaveSettings(settingsfile);
}
// The cropfieldCharacteristicItem is used to enter crop yields and/or straw yields
Item cropfieldCharacteristicItem;
string dataCropfieldCharacteristic;
//1 useExistingCropfieldWithChildren = false -> create new
//2 useExistingCropfieldWithChildren = true && input.CropfieldCharacteristicItemCode = "" or absent -> read from settings json
//3 useExistingCropfieldWithChildren = true && input.CropfieldCharacteristicItemCode like "deb48a74c5b54299bb852f17288010e9" in KPIinput -> use this one
if (useExistingCropfieldWithChildren == false)
{
for (int i = 0; i < input.DataCropfieldCharacteristics.Length; i++)
{
dataCropfieldCharacteristic = input.DataCropfieldCharacteristics[i].ToString(Formatting.None);
dynamic data = JObject.Parse(dataCropfieldCharacteristic);
_logger.LogInformation($"CreateCropfieldCharacteristicItemAsync ... for cropfieldCharacteristic {i}: '{data.label}', value '{data.value}'");
cropfieldCharacteristicItem = await _generalService.CreateCropfieldCharacteristicItemAsync(cropfieldItem.Code, cropYear, fieldGeom, dataCropfieldCharacteristic);
_settings.CropfieldCharacteristicItemCode = cropfieldCharacteristicItem.Code;
SaveSettings(settingsfile);
}
}
else if (string.IsNullOrEmpty(input.CropfieldItemCode))
{
_logger.LogInformation("reading CropfieldCharacteristicItemCode from settings file");
cropfieldCharacteristicItem = await _farmmapsApiService.GetItemAsync(_settings.CropfieldCharacteristicItemCode);
}
else
{
_logger.LogInformation("reading CropfieldCharacteristicItemCode from KPIinput json");
cropfieldCharacteristicItem = await _farmmapsApiService.GetItemAsync(input.CropfieldCharacteristicItemCode);
SaveSettings(settingsfile);
}
// Now we can do a first KPI calculation and get the polygon area from the KPI output (where it is based on geometry)
// We need that because for operations, you need to provide the area on which the operation was applied
// And if we put that to the crop area, then we neatly get everything on a per ha basis
_logger.LogInformation($"Getting polygon area (ha))");
List<Item> KPIItems = await _generalService.GetKpiItemsForCropField(cropfieldItem, 3);
trycnt = 1;
targetKPIitemsCount = 3; // here for the area we need at least 3, but not more than that
while (KPIItems.Count < targetKPIitemsCount & trycnt < maxtries)
{
KPIItems = await _generalService.GetKpiItemsForCropField(cropfieldItem,3);
_logger.LogInformation($"Found {KPIItems.Count} KPI items");
trycnt++;
}
kpio = JsonConvert.DeserializeObject<KPIOutput>(KPIItems[0].Data.ToString());
string area_ha = kpio.data.area;
// turn the area into a JObject for later merging with operation data;
string strJarea = JsonConvert.SerializeObject(new { area = area_ha });
JObject Jarea = JObject.Parse(strJarea);
if (useExistingCropfieldWithChildren == false)
{
//Retreiving BOFEK. A cropfield has 1 soil
//Have a look at the cropfieldChildren before and after running this task, see one child (i.e. soil) has been added)
//cropfieldChildren = await _farmmapsApiService.GetItemChildrenAsync(cropfieldItem.Code);
_logger.LogInformation("Get BOFEK for field");
Item bofekItem = await _generalService.RunBofekTask(cropfieldItem);
if (bofekItem == null)
{
_logger.LogError("Something went wrong while obtaining the BOFEK data");
return;
}
//cropfieldChildren = await _farmmapsApiService.GetItemChildrenAsync(cropfieldItem.Code);
}
else
{
_logger.LogInformation("For existing cropfield we assume it already has BOFEK soil data");
}
// A cropfield has 1 crop recording and the crop recording has 0:many operations
//So first at the crop recording
Item crprecItem;
//1 useExistingCropfieldWithChildren = false -> create new
//2 useExistingCropfieldWithChildren = true && input.CropRecordingItemCode = "" or absent -> read from settings json
//3 useExistingCropfieldWithChildren = true && input.CropRecordingItemCode like "deb48a74c5b54299bb852f17288010e9" in KPIinput -> use this one
if (useExistingCropfieldWithChildren == false)
{
_logger.LogInformation("RunCropRecordingTask ...");
crprecItem = await _generalService.RunCropRecordingTask(cropfieldItem);
_settings.CropRecordingItemCode = crprecItem.Code;
SaveSettings(settingsfile);
}
else if (string.IsNullOrEmpty(input.CropfieldItemCode))
{
_logger.LogInformation("reading CropRecordingItemCode from settings file");
crprecItem = await _farmmapsApiService.GetItemAsync(_settings.CropRecordingItemCode);
}
else
{
_logger.LogInformation("reading CropRecordingItemCode from KPIinput json");
crprecItem = await _farmmapsApiService.GetItemAsync(input.CropRecordingItemCode);
SaveSettings(settingsfile);
}
// Now add the operations
List<Item> crpOperationItems = new List<Item> { };
List<string> crpOperationItemCodes = new List<string> { };
Item crpOperationItem;
string dataOperation;
string codeOperation;
//1 useExistingCropfieldWithChildren = false -> create new
//2 useExistingCropfieldWithChildren = true && input.OperationItemCode = "" or absent -> read from settings json
//3 useExistingCropfieldWithChildren = true && input.OperationItemCode like "deb48a74c5b54299bb852f17288010e9" in KPIinput -> use this one
if (useExistingCropfieldWithChildren == false)
{
for (int i = 0; i < input.DataOperations.Length; i++)
{
dataOperation = input.DataOperations[i].ToString(Formatting.None);
dynamic data = JObject.Parse(dataOperation);
_logger.LogInformation($"CreateOperationItemAsync ... for operation {i}: '{data.name}', on date '{data.from}'");
// Now check if the operation has a field called area
string? opArea = data["area"];
if (string.IsNullOrEmpty(opArea))
{
// if not having field area: add it to dataOperation ...
input.DataOperations[i].Merge(Jarea);
// ... and update the string
dataOperation = input.DataOperations[i].ToString(Formatting.None);
}
else
{
// if yes having field area: compare with polygon area. If not the same, throw warning
if (data.area != area_ha)
{
double differencePercent = 100.0*(Convert.ToDouble(area_ha) / Convert.ToDouble(data.area) - 1.0);
_logger.LogWarning($"cropfield has area {area_ha}, but in your KPIinput.json or file like that, you have an operation with area {data.area}" +
$" Difference is {area_ha} / {data.area} - 100% = {differencePercent}%." +
$" Is that correct? Example if operation was applied in part of field, e.g. in case of variable rate (VRA) application." +
$" Or did you accidentally fill in area in the KPIinput json? To use cropfield area, omit field 'area' from json" +
$" then the KPI applicataion will fill in area calculated from geometry");
}
}
//Now after optionally adding the area, add the Operation to the crop recording
crpOperationItem = await _generalService.CreateOperationItemAsync(crprecItem.Code, cropYear, fieldGeom, dataOperation);
crpOperationItems.Add(crpOperationItem);
crpOperationItemCodes.Add(crpOperationItem.Code);
}
_settings.OperationItemCodes = crpOperationItemCodes.ToArray();
SaveSettings(settingsfile);
}
else if (string.IsNullOrEmpty(input.CropfieldItemCode))
{
_logger.LogInformation("reading OperationItemCode from settings file");
for (int i = 0; i < _settings.OperationItemCodes.Length; i++)
{
codeOperation = _settings.OperationItemCodes[i];
crpOperationItem = await _farmmapsApiService.GetItemAsync(codeOperation);
crpOperationItems.Add(crpOperationItem);
crpOperationItemCodes.Add(crpOperationItem.Code);
}
}
else
{
_logger.LogInformation("reading OperationItemCodes from KPIinput json");
for (int i = 0; i < input.OperationItemCodes.Length; i++)
{
codeOperation = input.OperationItemCodes[i];
crpOperationItem = await _farmmapsApiService.GetItemAsync(codeOperation);
crpOperationItems.Add(crpOperationItem);
crpOperationItemCodes.Add(crpOperationItem.Code);
}
_settings.OperationItemCodes = crpOperationItemCodes.ToArray();
SaveSettings(settingsfile);
}
// Inspect the children and grandchildren. If all is well, cropfield will have:
// one crprec, with 0-many operations as children. And the Data of an operation will have specification of how much fertilizer / cropprotection agent was applied
// one edicrop.characteristic (with yield in the data)
cropfieldChildren = await _farmmapsApiService.GetItemChildrenAsync(cropfieldItem.Code);
crprecChildren = await _farmmapsApiService.GetItemChildrenAsync(crprecItem.Code);
//Now get the KPIs for this cropfield, mounted with operations & cropyield
//Note sometimes the KPIItems.Count is already for some crazy reason greater than or equal to targetKPIitemsCount
//But that would have strange results, since that was from above before adding the crop recordings. We want to do at least one new call -> 'while (trycnt == 0 || '
_logger.LogInformation($"GetKpiItemsForCropField('{cropfieldItem.Code}')");
//Pesticide KPI's D1 and E1 are retreived from API's of CTBG and CLM and may take a bit longer to retrieve
targetKPIitemsCount = 8; //if we know we should be getting 8 KPI items (A1,B1,B2,C1,D1,E1,F1,F2)
trycnt = 0;
bool boolAquaticLife = false;
_logger.LogInformation($"Firing calls GetKpiItemsForCropField() until we have {targetKPIitemsCount} KPIitems, but don't keep firing forever, stop after {maxtries} calls");
_logger.LogInformation($"Before we start:");
_logger.LogInformation($"* KPIItems.Count = {KPIItems.Count}");
_logger.LogInformation($"* trycnt = {trycnt}");
_logger.LogInformation($"* boolAquaticLife = {boolAquaticLife}");
//additional criterion for while loop: check if it really contains the E1 mbp elements.
//while (trycnt == 0 || ((KPIItems.Count < targetKPIitemsCount || boolAquaticLife == false) & trycnt < maxtries))
//normal while loop
targetKPIitemsCount = 8; //if we know we should be getting 8 KPI items (A1,B1,B2,C1,D1,E1,F1,F2)
while (trycnt == 0 || (KPIItems.Count < targetKPIitemsCount & trycnt < maxtries))
{
_logger.LogInformation($"Call nr {trycnt + 1}");
KPIItems = await _generalService.GetKpiItemsForCropField(cropfieldItem, 3); //number after comma is how many seconds per try
_logger.LogInformation($"Found {KPIItems.Count} KPI items");
//boolAquaticLife = GetBoolAquaticLife(KPIItems);
trycnt ++;
}
if (KPIItems.Count < targetKPIitemsCount) {
_logger.LogWarning($"Found {KPIItems.Count} KPIItems while you were aiming for {targetKPIitemsCount} KPIItems");
}
//Download KPI's into a json output file for this specific cropfield (with unique cropfieldItem.Code)
string KPIItemPathJson = Path.Combine(downloadFolder, $"KPIItems_{cropfieldItem.Code}.json");
var count = 0;
foreach (Item item in KPIItems)
{
//Console.WriteLine($"KPI item #{count}: {item.Name}");
File.AppendAllText(KPIItemPathJson, item.Data + Environment.NewLine);
count++;
}
_logger.LogInformation($"Downloaded file {KPIItemPathJson}");
_logger.LogInformation($"");
//Write to the csv file that collects all KPI's for all the crop fields
List<string> dataList;
foreach (Item item in KPIItems)
{
kpio = JsonConvert.DeserializeObject<KPIOutput>(item.Data.ToString());
if (kpio.id != null)
{
if (kpio.id != kpioPrevious.id)
{
KPIelementsOfBalance kPIelementsOfBalance = kpio.data.values;
if (kpio.id == "A1") //TtD
{
//Make a new dataList = new line to be written
//Fill the datalist with this kpi
dataList = new List<string>
{
kpio.parentName,
cropfieldItem.Code,
kpio.data.area,
kpio.data.cropTypeCode,
kpio.data.cropTypeName,
kpio.id,
kpio.quantity, // in KPI output quantity is what we call KPIvariable in headerlist of csv file
kpio.value,
kpio.unit,
kpio.targetValue,
kpio.thresholdValue
};
//Write the datalist to a line to the streamwrieter sw for the output csv file
sw.WriteLine(string.Join(",", dataList));
foreach (string elementName in kpio.A1elements)
{
// get A1element from the element called values
// TtD 20240311: note elements of A1 are in the data structure elements of kpio.data and not as below for B1, B2, etc as kpio.data.values
string elementValue = (string)kpio.data.GetType().GetProperty(elementName).GetValue(kpio.data, null);
dataList = new List<string>
{
kpio.parentName,
cropfieldItem.Code,
kpio.data.area,
kpio.data.cropTypeCode,
kpio.data.cropTypeName,
kpio.id,
elementName, // specific output variable name for A1 element
elementValue, // specific output value name for A1 element
kpio.unit,
"",
""
};
//Write the datalist to a line to the streamwrieter sw for the output csv file
sw.WriteLine(string.Join(",", dataList));
}
}
else if (kpio.id == "B1")
{
//Make a new dataList = new line to be written
//Fill the datalist with this kpi
dataList = new List<string>
{
kpio.parentName,
cropfieldItem.Code,
kpio.data.area,
kpio.data.cropTypeCode,
kpio.data.cropTypeName,
kpio.id,
kpio.quantity, // in KPI output quantity is what we call KPIvariable in headerlist of csv file
kpio.value,
kpio.unit,
kpio.targetValue,
kpio.thresholdValue
};
//Write the datalist to a line to the streamwrieter sw for the output csv file
sw.WriteLine(string.Join(",", dataList));
foreach (string elementName in kpio.B1elements)
{
// get B1element from the element called values
string elementValue = (string)kPIelementsOfBalance.GetType().GetProperty(elementName).GetValue(kPIelementsOfBalance, null);
dataList = new List<string>
{
kpio.parentName,
cropfieldItem.Code,
kpio.data.area,
kpio.data.cropTypeCode,
kpio.data.cropTypeName,
kpio.id,
elementName, // specific output variable name for B1 element
elementValue, // specific output value name for B1 element
kpio.unit,
"",
""
};
//Write the datalist to a line to the streamwrieter sw for the output csv file
sw.WriteLine(string.Join(",", dataList));
}
}
else if (kpio.id == "B2")
{
//Make a new dataList = new line to be written
dataList = new List<string>
{
//Fill the datalist with this kpi
kpio.parentName,
cropfieldItem.Code,
kpio.data.area,
kpio.data.cropTypeCode,
kpio.data.cropTypeName,
kpio.id,
kpio.quantity, // in KPI output quantity is what we call KPIvariable in headerlist of csv file
kpio.value,
kpio.unit,
kpio.targetValue,
kpio.thresholdValue
};
//Write the datalist to a line to the streamwrieter sw for the output csv file
sw.WriteLine(string.Join(",", dataList));
foreach (string elementName in kpio.B2elements)
{
// get B1element from the element called values
string elementValue = (string)kPIelementsOfBalance.GetType().GetProperty(elementName).GetValue(kPIelementsOfBalance, null);
dataList = new List<string>
{
kpio.parentName,
cropfieldItem.Code,
kpio.data.area,
kpio.data.cropTypeCode,
kpio.data.cropTypeName,
kpio.id,
elementName, // specific output variable name for B2 element
elementValue, // specific output value name for B2 element
kpio.unit,
"",
""
};
//Write the datalist to a line to the streamwrieter sw for the output csv file
sw.WriteLine(string.Join(",", dataList));
}
}
else if (kpio.id == "C1") //TtD
{
//Make a new dataList = new line to be written
dataList = new List<string>
{
kpio.parentName,
cropfieldItem.Code,
kpio.data.area,
kpio.data.cropTypeCode,
kpio.data.cropTypeName,
kpio.id,
kpio.quantity, // in KPI output quantity is what we call KPIvariable in headerlist of csv file
kpio.value,
kpio.unit,
kpio.targetValue,
kpio.thresholdValue
};
//Write the datalist to a line to the streamwrieter sw for the output csv file
sw.WriteLine(string.Join(",", dataList));
foreach (string elementName in kpio.C1elements)
{
// get C1element from the element called values
string elementValue = (string)kPIelementsOfBalance.GetType().GetProperty(elementName).GetValue(kPIelementsOfBalance, null);
dataList = new List<string>
{
kpio.parentName,
cropfieldItem.Code,
kpio.data.area,
kpio.data.cropTypeCode,
kpio.data.cropTypeName,
kpio.id,
elementName, // specific output variable name for C1 element
elementValue, // specific output value name for C1 element
kpio.unit,
"",
""
};
//Write the datalist to a line to the streamwrieter sw for the output csv file
sw.WriteLine(string.Join(",", dataList));
}
}
else if (kpio.id == "E1")
{
//for E1, environmentMeasureData is an array of elements with 1 per pesticide. Inside each element there are multiple mbp_KPIvariables to be written
foreach (KPIenvironmentMeasureData e in kpio.data.environmentMeasureData)
{
foreach (string mbp_KPIvariable in mbp_KPIvariables)
{
if (mbp_KPIvariable == "aquaticLife") { mbp_KPIvalue = e.aquaticLife; } else if (mbp_KPIvariable == "groundWater") { mbp_KPIvalue = e.groundWater; } else if (mbp_KPIvariable == "soilLife") { mbp_KPIvalue = e.soilLife; } else { mbp_KPIvalue = ""; }
//Make a new dataList = new line to be written
dataList = new List<string>
{
kpio.parentName,
cropfieldItem.Code,
kpio.data.area,
kpio.data.cropTypeCode,
kpio.data.cropTypeName,
kpio.id, //"E1"
kpio.quantity, // "mbp"
"", // not here the value
"", // not here KPIunit for this indicator
"", // not here KPItargetvalue for this indicator
"", // not here KPIthresholdValue for this indicator
e.productCode,
e.productName,
e.quantity,
e.unitCode,
e.date,
mbp_KPIvariable,
mbp_KPIvalue
};
//Write the datalist to a line to the streamwrieter sw for the output csv file
sw.WriteLine(string.Join(",", dataList));
}
}
}
else
{
//Any other KPI, example A1 or D1, with just 1 record to write
dataList = new List<string>
{
kpio.parentName,
cropfieldItem.Code,
kpio.data.area,
kpio.data.cropTypeCode,
kpio.data.cropTypeName,
kpio.id,
kpio.quantity,
kpio.value,
kpio.unit,
kpio.targetValue,
kpio.thresholdValue
};
//Write the datalist to a line to the streamwrieter sw for the output csv file
sw.WriteLine(string.Join(",", dataList));
}
}
}
kpioPrevious = kpio;
}
////Total N applied from input
//double totalNapplied = 0.0;
//double operationNapplied;
//for (int i = 0; i < crpOperationItemCodes.Count; i++)
//{
// codeOperation = crpOperationItemCodes[i];
// crpOperationItem = await _farmmapsApiService.GetItemAsync(codeOperation);
// dynamic data = JObject.Parse(crpOperationItem.Data.ToString(Formatting.None));
// if (data.n != null)
// {
// operationNapplied = crpOperationItem.Data.GetValue("n").ToObject<double>();
// totalNapplied = totalNapplied + operationNapplied;
// }
//}
////Also add totalNapplied to the csv
//dataList = new List<string> { };
//dataList.Add(kpioPrevious.parentName);
//dataList.Add(cropfieldItem.Code);
//try { dataList.Add(kpioPrevious.data.area); } catch { dataList.Add(""); };
//try { dataList.Add(kpioPrevious.data.cropTypeCode); } catch { dataList.Add(""); };
//try { dataList.Add(kpioPrevious.data.cropTypeName); } catch { dataList.Add(""); };
//dataList.Add("");
//dataList.Add("totalNapplied");
//dataList.Add(totalNapplied.ToString());
//dataList.Add("kg/ha");
//dataList.Add("");
//dataList.Add("");
//sw.WriteLine(string.Join(",", dataList));
//Clean up. Only newly created fields
//Look up instruction in Swagger / api / v1 / items /{ code}
if (useExistingCropfieldWithChildren == false && input.DeleteNewlyCreatedAfterCalc == true)
{
// Tested and found that if I delete the parent (cropfieldItem) then
// automatically deletes children (crprecItem, cropfieldCharacteristicItem, KPIcontainer)
// and automatically deletes grand children (crprecItem: operations, KPIcontainer: KPI items)
await _farmmapsApiService.DeleteItemAsync(cropfieldItem.Code);
//Do we need to wait here for the task to be completed? Or does it continue in the background?
//await Task.Delay(60000); // wait 60 secs for task to be completed
//Before the GetItem you will see cropfieldItem, crprecItem etc exist (with data).
//After the GetItem these cropfieldItem, crprecItem etc are null which means they have been succesfully deleted.
//// Check if below works, i.e. is item really deleted?
//cropfieldItem = await _farmmapsApiService.GetItemAsync(cropfieldItem.Code);
//// Check if croprecording has also been deleted. If not also delete it
//crprecItem = await _farmmapsApiService.GetItemAsync(crprecItem.Code);
//// Check if 1 operation has also been deleted. If not also delete it
//crpOperationItem = await _farmmapsApiService.GetItemAsync(crpOperationItemCodes[0]);
//// Check if 1 cropfieldCharacteristicItem has also been deleted. If not also delete it
//cropfieldCharacteristicItem = await _farmmapsApiService.GetItemAsync(cropfieldCharacteristicItem.Code);
}
}
// Functions to save previously created cropfields
private void LoadSettings(string file)
{
if (File.Exists(file))
{
var jsonText = File.ReadAllText(file);
_settings = JsonConvert.DeserializeObject<Settings>(jsonText);
}
else
{
_settings = new Settings();
}
}
private void SaveSettings(string file)
{
if (_settings == null)
return;
var json = JsonConvert.SerializeObject(_settings);
File.WriteAllText(file, json);
}
private void SaveInfo(string file)
{
if (_settings == null)
return;
var json = JsonConvert.SerializeObject(_settings);
File.WriteAllText(file, json);
}
private string strTime(TimeSpan ts)
{
return String.Format("{0:00}:{1:00}:{2:00}", ts.Hours, ts.Minutes, ts.Seconds);
}
private bool GetBoolAquaticLife(List<Item> KPIItems)
{
bool found = false;
foreach (Item item in KPIItems)
{
KPIOutput kpio = JsonConvert.DeserializeObject<KPIOutput>(item.Data.ToString());
if (kpio.id == "E1")
{
try
{
foreach (KPIenvironmentMeasureData e in kpio.data.environmentMeasureData)
{
if (e.aquaticLife != null)
{
found = true;
_logger.LogInformation($"Found e.aquaticLife = {e.aquaticLife}");
_logger.LogInformation("");
}
}
}
catch (Exception ex)
{
_logger.LogWarning($"NOT Found e.aquaticLife");
_logger.LogError(ex.Message);
}
}
}
return found;
}
}
}

883
FarmmapsKPI/KPIInput.json 100644
View File

@ -0,0 +1,883 @@
[
{
"useExistingCropfieldWithChildren": false,
"deleteNewlyCreatedAfterCalc": true,
"fieldName": "test_Potato_ZeroFertilizer_Consumption_Sand",
//Note when you run the FarmmapsKPI project, somewhere to your C:\git\FarmMapsApiClient_KB34_MAST\FarmmapsKPI\bin\Debug\netcoreapp3.1\ directory, files named like this are written:
//'Settings_aardappelveld_test_Potato_ZeroFertilizer.json', where 'aardappelveld_test_Potato_ZeroFertilizer.json' is your fieldName (see below)
//1 useExistingCropfieldWithChildren = false -> create new
//2 useExistingCropfieldWithChildren = true && CropfieldItemCode = "" or absent -> read fieldName (see below) -> read all codes from settings json file
//3 useExistingCropfieldWithChildren = true && CropfieldItemCode like "deb48a74c5b54299bb852f17288010e9" in KPIinput -> if you have in your account already an existing cropfield then use that
"CropfieldItemCode": "", // could contain for example this: "abae97f89f3c4ac08953b1b8bea9f076" if this is an exisiting CropfieldItemCode in your account.
//And then if useExistingCropfieldWithChildren = true, this would be used.
"dataCropfield": {
//"area": 4.22, //Leave empty, KPI app will calculate it from geometry
"final": true, //always true
"soilCode": "1", //From codelist CL405. Can be omitted if unknown
"soilName": "Sand", //From codelist CL405
"cropTypeCode": "1010101", //From codelist CL263
"cropTypeName": "Potato", //From codelist CL263
"productionPurposeCode": "3", //From codelist CL251. For testing, see case with consumption & starch potato
"productionPurposeName": "consumption" //From codelist CL251
},
"CropRecordingItemCode": "", // could contain for example this: "314aa1b6480b493ba28a5f55039a01d4" if this is the CropRecordingItemCode of the provided CropfieldItemCode. And then if useExistingCropfieldWithChildren = true, this would be used
"OperationItemCodes": [], // could contain for example this: ["4d1a6f28ea3f414180663baa4b08c60f"] or ["4d1a6f28ea3f414180663baa4b08c60f","21e08386f0454ad79acfebf78649d59b"] if these are the operations of the provided CropRecordingItemCode. And then if useExistingCropfieldWithChildren = true, these would be used
"dataOperations": [], //leaving the operations empty means no fertilizer is applied
"CropfieldCharacteristicItemCode": "",
"DataCropfieldCharacteristics": [
{
//PO20231029: not different yields from different parts of fields?
"code": "860619", //From codelist CL265. 860619 refers to cropyield in ton/ha
"label": "cropyield",
"value": "48.01"
}
],
//"DownloadFolder": "Downloads", //"C:\\hugoschrererdir\\kpidir\\", // "Downloads", -> if you just put "Downloads" the program will download to somewhere in ..\FarmMapsApiClient_KB34_MAST\FarmmapsKPI\bin\Debug\netcoreapp3.1\Downloads\
"CropYear": 2022,
//geometryJson = polygon of location of the cropfield, coordinates in LON,LAT (LON: decimal degrees East, LAT: decimal degrees North)
"geometryJson": {
"type": "Polygon",
"coordinates": [
[
[
5.5945257993548765,
52.57080744107003
],
[
5.598645994070678,
52.571540800206236
],
[
5.599381743127071,
52.57012773140724
],
[
5.595408698222548,
52.56968054825188
],
[
5.5945257993548765,
52.57080744107003
]
]
]
}
},
{
"useExistingCropfieldWithChildren": false,
"deleteNewlyCreatedAfterCalc": true,
"fieldName": "test_Potato_ZeroFertilizer_Consumption_Clay",
//Note when you run the FarmmapsKPI project, somewhere to your C:\git\FarmMapsApiClient_KB34_MAST\FarmmapsKPI\bin\Debug\netcoreapp3.1\ directory, files named like this are written:
//'Settings_aardappelveld_test_Potato_ZeroFertilizer.json', where 'aardappelveld_test_Potato_ZeroFertilizer.json' is your fieldName (see below)
//1 useExistingCropfieldWithChildren = false -> create new
//2 useExistingCropfieldWithChildren = true && CropfieldItemCode = "" or absent -> read fieldName (see below) -> read all codes from settings json file
//3 useExistingCropfieldWithChildren = true && CropfieldItemCode like "deb48a74c5b54299bb852f17288010e9" in KPIinput -> if you have in your account already an existing cropfield then use that
"CropfieldItemCode": "", // could contain for example this: "abae97f89f3c4ac08953b1b8bea9f076" if this is an exisiting CropfieldItemCode in your account.
//And then if useExistingCropfieldWithChildren = true, this would be used.
"dataCropfield": {
//"area": 4.22, //Leave empty, KPI app will calculate it from geometry
"final": true, //always true
"soilCode": "7", //From codelist CL405. Can be omitted if unknown
"soilName": "Clay", //From codelist CL405
"cropTypeCode": "1010101", //From codelist CL263
"cropTypeName": "Potato", //From codelist CL263
"productionPurposeCode": "3", //From codelist CL251. For testing, see case with consumption & starch potato
"productionPurposeName": "consumption" //From codelist CL251
},
"CropRecordingItemCode": "", // could contain for example this: "314aa1b6480b493ba28a5f55039a01d4" if this is the CropRecordingItemCode of the provided CropfieldItemCode. And then if useExistingCropfieldWithChildren = true, this would be used
"OperationItemCodes": [], // could contain for example this: ["4d1a6f28ea3f414180663baa4b08c60f"] or ["4d1a6f28ea3f414180663baa4b08c60f","21e08386f0454ad79acfebf78649d59b"] if these are the operations of the provided CropRecordingItemCode. And then if useExistingCropfieldWithChildren = true, these would be used
"dataOperations": [], //leaving the operations empty means no fertilizer is applied
"CropfieldCharacteristicItemCode": "",
"DataCropfieldCharacteristics": [
{
//PO20231029: not different yields from different parts of fields?
"code": "860619", //From codelist CL265. 860619 refers to cropyield in ton/ha
"label": "cropyield",
"value": "48.01"
}
],
//"DownloadFolder": "Downloads", //"C:\\hugoschrererdir\\kpidir\\", // "Downloads", -> if you just put "Downloads" the program will download to somewhere in ..\FarmMapsApiClient_KB34_MAST\FarmmapsKPI\bin\Debug\netcoreapp3.1\Downloads\
"CropYear": 2022,
//geometryJson = polygon of location of the cropfield, coordinates in LON,LAT (LON: decimal degrees East, LAT: decimal degrees North)
"geometryJson": {
"type": "Polygon",
"coordinates": [
[
[
5.5945257993548765,
52.57080744107003
],
[
5.598645994070678,
52.571540800206236
],
[
5.599381743127071,
52.57012773140724
],
[
5.595408698222548,
52.56968054825188
],
[
5.5945257993548765,
52.57080744107003
]
]
]
}
},
{
"useExistingCropfieldWithChildren": false,
"deleteNewlyCreatedAfterCalc": true,
"fieldName": "test_Potato_ZeroFertilizer_Starch_Sand",
//Note when you run the FarmmapsKPI project, somewhere to your C:\git\FarmMapsApiClient_KB34_MAST\FarmmapsKPI\bin\Debug\netcoreapp3.1\ directory, files named like this are written:
//'Settings_aardappelveld_test_Potato_ZeroFertilizer.json', where 'aardappelveld_test_Potato_ZeroFertilizer.json' is your fieldName (see below)
//1 useExistingCropfieldWithChildren = false -> create new
//2 useExistingCropfieldWithChildren = true && CropfieldItemCode = "" or absent -> read fieldName (see below) -> read all codes from settings json file
//3 useExistingCropfieldWithChildren = true && CropfieldItemCode like "deb48a74c5b54299bb852f17288010e9" in KPIinput -> if you have in your account already an existing cropfield then use that
"CropfieldItemCode": "", // could contain for example this: "abae97f89f3c4ac08953b1b8bea9f076" if this is an exisiting CropfieldItemCode in your account.
//And then if useExistingCropfieldWithChildren = true, this would be used.
"dataCropfield": {
//"area": 4.22, //Leave empty, KPI app will calculate it from geometry
"final": true, //always true
"soilCode": "1", //From codelist CL405. Can be omitted if unknown
"soilName": "Sand",
"cropTypeCode": "1010101", //From codelist CL263
"cropTypeName": "Potato", //From codelist CL263
"productionPurposeCode": "2", //From codelist CL251. For testing, see case with consumption & starch potato
"productionPurposeName": "starch" //From codelist CL251
},
"CropRecordingItemCode": "", // could contain for example this: "314aa1b6480b493ba28a5f55039a01d4" if this is the CropRecordingItemCode of the provided CropfieldItemCode. And then if useExistingCropfieldWithChildren = true, this would be used
"OperationItemCodes": [], // could contain for example this: ["4d1a6f28ea3f414180663baa4b08c60f"] or ["4d1a6f28ea3f414180663baa4b08c60f","21e08386f0454ad79acfebf78649d59b"] if these are the operations of the provided CropRecordingItemCode. And then if useExistingCropfieldWithChildren = true, these would be used
"dataOperations": [], //leaving the operations empty means no fertilizer is applied
"CropfieldCharacteristicItemCode": "",
"DataCropfieldCharacteristics": [
{
//PO20231029: not different yields from different parts of fields?
"code": "860619", //From codelist CL265. 860619 refers to cropyield in ton/ha
"label": "cropyield",
"value": "48.01"
}
],
//"DownloadFolder": "Downloads", //"C:\\hugoschrererdir\\kpidir\\", // "Downloads", -> if you just put "Downloads" the program will download to somewhere in ..\FarmMapsApiClient_KB34_MAST\FarmmapsKPI\bin\Debug\netcoreapp3.1\Downloads\
"CropYear": 2022,
//geometryJson = polygon of location of the cropfield, coordinates in LON,LAT (LON: decimal degrees East, LAT: decimal degrees North)
"geometryJson": {
"type": "Polygon",
"coordinates": [
[
[
5.5945257993548765,
52.57080744107003
],
[
5.598645994070678,
52.571540800206236
],
[
5.599381743127071,
52.57012773140724
],
[
5.595408698222548,
52.56968054825188
],
[
5.5945257993548765,
52.57080744107003
]
]
]
}
},
{
"useExistingCropfieldWithChildren": false,
"deleteNewlyCreatedAfterCalc": true,
"fieldName": "aardappelveld_test_Potato_cattlemanure144kgnha_plusUrea92kgNha",
"CropfieldItemCode": "",
"dataCropfield": {
//"area": 4.22, //Leave empty, KPI app will calculate it from geometry
"final": true, //always true
//"soilCode": "5", //From codelist CL405. Can be omitted if unknown
//"soilName": "Loam", //From codelist CL405.
"cropTypeCode": "1010101", //From codelist CL263
"cropTypeName": "Potato", //From codelist CL263
//"rootDepthMax": 45, //not needed for KPI calculation, but shown here to know this is a possible property
//"emergenceDate": "2022-05-16T00:00:00", //not needed for KPI calculation, but shown here to know this is a possible property
"productionPurposeCode": "3", //From codelist CL251. Can be omitted if unknown
"productionPurposeName": "consumption" //From codelist CL251. Can be omitted if unknown
},
"CropRecordingItemCode": "",
"OperationItemCodes": [],
"dataOperations": [
{
//"area": "0.08", //if leave empty then write code to fill in value from geometry (operation applied to whole field). Fill in different value if operation applied to part of the field (e.g. in case of variable rate application, VRA)
"contractor": false,
"designator": "Kunstmest strooien", //refers to codelist CL127 with operation methods
"from": "2022-05-23T11:34:00",
"method": "70400", //refers to codelist CL127 with operation methods
"n": "92", //refers to codelist CL022 with fertilizer types & npk contents
"name": "Kunstmest strooien", //refers to codelist CL127 with operation methods
"product": "7360", //refers to codelist CL022 with fertilizer types
"quantity": "200",
//"status": "3", // refers to codelist CL256. where 3 stand for completed. The status does not matter and is not used in the KPI calculations
"operationCode": "7", // refers to codelist CL018 consisting of main catergorie of operations. Code 7 stands for fertilization
"to": "2022-05-23T12:34:00",
"unit": "kg/ha",
"unitCode": "KGMHAR"
},
{
"k": "201.6",
"n": "144",
"p": "60.8",
"to": "2022-04-19T09:19:00",
//"area": "0.08", //if leave empty then write code to fill in value from geometry (operation applied to whole field). Fill in different value if operation applied to part of the field (e.g. in case of variable rate application, VRA)
"from": "2022-04-19T08:19:00",
"name": "Injecteren", //refers to codelist CL127 with operation methods
"unit": "kg/ha",
"method": "70700", //refers to codelist CL127 with operation methods
//"status": "3", // refers to codelist CL256. where 3 stand for completed. The status does not matter and is not used in the KPI calculations
"operationCode": "7", // refers to codelist CL018 consisting of main catergorie of operations. Code 7 stands for fertilization
"product": "2329", //refers to codelist CL022 with fertilizer types
"quantity": "32000",
"unitCode": "KGMHAR",
"contractor": false,
"designator": "Injecteren"
}
],
"CropfieldCharacteristicItemCode": "",
"DataCropfieldCharacteristics": [
{
"code": "860619", //From codelist CL265. 860619 refers to cropyield in ton/ha
"label": "cropyield",
"value": "48.01"
}
],
//"DownloadFolder": "Downloads", //"C:\\hugoschrererdir\\kpidir\\", // "Downloads", -> if you just put "Downloads" the program will download to somewhere in ..\FarmMapsApiClient_WURtest\FarmmapsDataDownload\bin\Debug\netcoreapp3.1\Downloads\
"CropYear": 2022,
"geometryJson": {
"type": "Polygon",
"coordinates": [
[
[
5.5945257993548765,
52.57080744107003
],
[
5.598645994070678,
52.571540800206236
],
[
5.599381743127071,
52.57012773140724
],
[
5.595408698222548,
52.56968054825188
],
[
5.5945257993548765,
52.57080744107003
]
]
]
}
},
{
"useExistingCropfieldWithChildren": false,
"deleteNewlyCreatedAfterCalc": true,
"fieldName": "aardappelveld_test_Potato_Urea92kgNha",
"CropfieldItemCode": "",
"dataCropfield": {
//"area": 4.22, //Leave empty, KPI app will calculate it from geometry
"final": true, //always true
//"soilCode": "5", //From codelist CL405. Can be omitted if unknown
//"soilName": "Loam", //From codelist CL405.
"cropTypeCode": "1010101", //From codelist CL263
"cropTypeName": "Potato", //From codelist CL263
//"rootDepthMax": 45, //not needed for KPI calculation, but shown here to know this is a possible property
//"emergenceDate": "2022-05-16T00:00:00", //not needed for KPI calculation, but shown here to know this is a possible property
"productionPurposeCode": "3", //From codelist CL251. Can be omitted if unknown
"productionPurposeName": "consumption" //From codelist CL251. Can be omitted if unknown
},
"CropRecordingItemCode": "",
"OperationItemCodes": [],
"dataOperations": [
{
//"area": "0.08", //if leave empty then write code to fill in value from geometry (operation applied to whole field). Fill in different value if operation applied to part of the field (e.g. in case of variable rate application, VRA)
"contractor": false,
"designator": "Kunstmest strooien", //refers to codelist CL127 with operation methods
"from": "2022-05-23T11:34:00",
"method": "70400", //refers to codelist CL127 with operation methods
"n": "92",
"name": "Kunstmest strooien", //refers to codelist CL127 with operation methods
"product": "7360", //refers to codelist CL022 with fertilizer types
"quantity": "200",
//"status": "3", // refers to codelist CL256. where 3 stand for completed. The status does not matter and is not used in the KPI calculations
"operationCode": "7", // refers to codelist CL018 consisting of main catergorie of operations. Code 7 stands for fertilization
"to": "2022-05-23T12:34:00",
"unit": "kg/ha",
"unitCode": "KGMHAR"
}
],
"CropfieldCharacteristicItemCode": "",
"DataCropfieldCharacteristics": [
{
"code": "860619", //From codelist CL265. 860619 refers to cropyield in ton/ha
"label": "cropyield",
"value": "48.01"
}
],
//"DownloadFolder": "Downloads", //"C:\\hugoschrererdir\\kpidir\\", // "Downloads", -> if you just put "Downloads" the program will download to somewhere in ..\FarmMapsApiClient_WURtest\FarmmapsDataDownload\bin\Debug\netcoreapp3.1\Downloads\
"CropYear": 2022,
"geometryJson": {
"type": "Polygon",
"coordinates": [
[
[
5.5945257993548765,
52.57080744107003
],
[
5.598645994070678,
52.571540800206236
],
[
5.599381743127071,
52.57012773140724
],
[
5.595408698222548,
52.56968054825188
],
[
5.5945257993548765,
52.57080744107003
]
]
]
}
},
{
"useExistingCropfieldWithChildren": false,
"deleteNewlyCreatedAfterCalc": true,
"fieldName": "aardappelveld_test_Potato_Urea92kgNha_ZuidLimburg",
"CropfieldItemCode": "",
"dataCropfield": {
//"area": 4.22, //Leave empty, KPI app will calculate it from geometry
"final": true, //always true
//"soilCode": "5", //From codelist CL405. Can be omitted if unknown
//"soilName": "Loam", //From codelist CL405.
"cropTypeCode": "1010101", //From codelist CL263
"cropTypeName": "Potato", //From codelist CL263
//"rootDepthMax": 45, //not needed for KPI calculation, but shown here to know this is a possible property
//"emergenceDate": "2022-05-16T00:00:00", //not needed for KPI calculation, but shown here to know this is a possible property
"productionPurposeCode": "3", //From codelist CL251. Can be omitted if unknown
"productionPurposeName": "consumption" //From codelist CL251. Can be omitted if unknown
},
"CropRecordingItemCode": "",
"OperationItemCodes": [],
"dataOperations": [
{
//"area": "0.08", //if leave empty then write code to fill in value from geometry (operation applied to whole field). Fill in different value if operation applied to part of the field (e.g. in case of variable rate application, VRA)
"contractor": false,
"designator": "Kunstmest strooien", //refers to codelist CL127 with operation methods
"from": "2022-05-23T11:34:00",
"method": "70400", //refers to codelist CL127 with operation methods
"n": "92",
"name": "Kunstmest strooien", //refers to codelist CL127 with operation methods
//"status": "3", // refers to codelist CL256. where 3 stand for completed. The status does not matter and is not used in the KPI calculations
"operationCode": "7", // refers to codelist CL018 consisting of main catergorie of operations. Code 7 stands for fertilization
"product": "7360", //refers to codelist CL022 with fertilizer types
"quantity": "200",
"to": "2022-05-23T12:34:00",
"unit": "kg/ha",
"unitCode": "KGMHAR"
}
],
"CropfieldCharacteristicItemCode": "",
"DataCropfieldCharacteristics": [
{
"code": "860619", //From codelist CL265. 860619 refers to cropyield in ton/ha
"label": "cropyield",
"value": "48.01"
}
],
//"DownloadFolder": "Downloads", //"C:\\hugoschrererdir\\kpidir\\", // "Downloads", -> if you just put "Downloads" the program will download to somewhere in ..\FarmMapsApiClient_WURtest\FarmmapsDataDownload\bin\Debug\netcoreapp3.1\Downloads\
"CropYear": 2022,
"geometryJson": {
"type": "Polygon",
"coordinates": [
[
[
5.900,
50.800
],
[
5.901,
50.800
],
[
5.901,
50.801
],
[
5.900,
50.801
],
[
5.900,
50.800
]
]
]
}
},
{
"useExistingCropfieldWithChildren": false,
"deleteNewlyCreatedAfterCalc": true,
"fieldName": "caseQuoteNnotProvidedInDataOperations",
"CropfieldItemCode": "",
"dataCropfield": {
//"area": 4.22, //Leave empty, KPI app will calculate it from geometry
"final": true, //always true
//"soilCode": "5", //From codelist CL405. Can be omitted if unknown
//"soilName": "Loam", //From codelist CL405.
"cropTypeCode": "1010101", //From codelist CL263
"cropTypeName": "Potato", //From codelist CL263
//"rootDepthMax": 45, //not needed for KPI calculation, but shown here to know this is a possible property
//"emergenceDate": "2022-05-16T00:00:00", //not needed for KPI calculation, but shown here to know this is a possible property
"productionPurposeCode": "3", //From codelist CL251. Can be omitted if unknown
"productionPurposeName": "consumption" //From codelist CL251. Can be omitted if unknown
},
"CropRecordingItemCode": "",
"OperationItemCodes": [],
"dataOperations": [
{
//"area": "0.08", //if leave empty then write code to fill in value from geometry (operation applied to whole field). Fill in different value if operation applied to part of the field (e.g. in case of variable rate application, VRA)
"contractor": false,
"designator": "Kunstmest strooien", //refers to codelist CL127 with operation methods
"from": "2022-05-23T11:34:00",
"method": "70400", //refers to codelist CL127 with operation methods
//"n": "92",
"name": "Kunstmest strooien", //refers to codelist CL127 with operation methods
//"status": "3", // refers to codelist CL256. where 3 stand for completed. The status does not matter and is not used in the KPI calculations
"operationCode": "7", // refers to codelist CL018 consisting of main catergorie of operations. Code 7 stands for fertilization
"product": "7360", //refers to codelist CL022 with fertilizer types
"quantity": "200",
"to": "2022-05-23T12:34:00",
"unit": "kg/ha",
"unitCode": "KGMHAR"
}
],
"CropfieldCharacteristicItemCode": "",
"DataCropfieldCharacteristics": [
{
"code": "860619", //From codelist CL265. 860619 refers to cropyield in ton/ha
"label": "cropyield",
"value": "48.01"
}
],
//"DownloadFolder": "Downloads", //"C:\\hugoschrererdir\\kpidir\\", // "Downloads", -> if you just put "Downloads" the program will download to somewhere in ..\FarmMapsApiClient_WURtest\FarmmapsDataDownload\bin\Debug\netcoreapp3.1\Downloads\
"CropYear": 2022,
"geometryJson": {
"type": "Polygon",
"coordinates": [
[
[
5.5945257993548765,
52.57080744107003
],
[
5.598645994070678,
52.571540800206236
],
[
5.599381743127071,
52.57012773140724
],
[
5.595408698222548,
52.56968054825188
],
[
5.5945257993548765,
52.57080744107003
]
]
]
}
},
{
"useExistingCropfieldWithChildren": false,
"deleteNewlyCreatedAfterCalc": true,
"fieldName": "examplePesticide_11767N",
"CropfieldItemCode": "", // could contain for example this: "abae97f89f3c4ac08953b1b8bea9f076" if this is an exisiting CropfieldItemCode in your account.
"dataCropfield": {
"final": true, //always true
"cropTypeCode": "1010101", //From codelist CL263
"cropTypeName": "Potato", //From codelist CL263
"productionPurposeCode": "3", //From codelist CL251. Can be omitted if unknown
"productionPurposeName": "consumption" //From codelist CL251. Can be omitted if unknown
},
"CropRecordingItemCode": "", // could contain for example this: "314aa1b6480b493ba28a5f55039a01d4" if this is the CropRecordingItemCode of the provided CropfieldItemCode. And then if useExistingCropfieldWithChildren = true, this would be used
"OperationItemCodes": [], // could contain for example this: ["4d1a6f28ea3f414180663baa4b08c60f"] or ["4d1a6f28ea3f414180663baa4b08c60f","21e08386f0454ad79acfebf78649d59b"] if these are the operations of the provided CropRecordingItemCode. And then if useExistingCropfieldWithChildren = true, these would be used
"dataOperations": [
{
"to": "2022-07-13T17:27:00",
"from": "2022-07-13T16:27:00",
"name": "Volveld spuiten (3/3)", //refers to codelist CL127 with operation methods
"method": "80200", //refers to codelist CL127 with operation methods
//"status": "3", // refers to codelist CL256. where 3 stand for completed. The status does not matter and is not used in the KPI calculations
"operationCode": "8", // refers to codelist CL018 consisting of main catergorie of operations. Code 8 stands for crop protection
"products": [
{
"product": "11767 N", //CTBG code, lookup via CTBG API. Important: with space N",
"quantity": "1.25",
"unitCode": "LTRHAR"
//"designator": "Amistar"
}
]
}
],
"CropfieldCharacteristicItemCode": "",
"DataCropfieldCharacteristics": [
{
"code": "860619",
"label": "cropyield",
"value": "48.01"
}
],
//"DownloadFolder": "Downloads", //"C:\\hugoschrererdir\\kpidir\\", // "Downloads", -> if you just put "Downloads" the program will download to somewhere in ..\FarmMapsApiClient_KB34_MAST\FarmmapsKPI\bin\Debug\netcoreapp3.1\Downloads\
"CropYear": 2022,
//geometryJson = polygon of location of the cropfield, coordinates in LON,LAT (LON: decimal degrees East, LAT: decimal degrees North)
"geometryJson": {
"type": "Polygon",
"coordinates": [
[
[
5.5945257993548765,
52.57080744107003
],
[
5.598645994070678,
52.571540800206236
],
[
5.599381743127071,
52.57012773140724
],
[
5.595408698222548,
52.56968054825188
],
[
5.5945257993548765,
52.57080744107003
]
]
]
}
},
{
"useExistingCropfieldWithChildren": false,
"deleteNewlyCreatedAfterCalc": true,
"fieldName": "doperwt_heleJaar",
"CropfieldItemCode": "",
"dataCropfield": {
"final": true, //always true
"cropTypeCode": "1030101", //From codelist CL263
"cropTypeName": "Garden pea" //From codelist CL263
},
"CropRecordingItemCode": "",
"OperationItemCodes": [],
"dataOperations": [],
"CropfieldCharacteristicItemCode": "",
"DataCropfieldCharacteristics": [
{
"code": "860619", //From codelist CL265. 860619 refers to cropyield in ton/ha
"label": "cropyield",
"value": "9.58" //CBS: https://www.cbs.nl/nl-nl/cijfers/detail/37738, 2022: 40.4 Mkg / 4216 ha = 40400 / 4216
}
],
//"DownloadFolder": "Downloads", //"C:\\hugoschrererdir\\kpidir\\", // "Downloads", -> if you just put "Downloads" the program will download to somewhere in ..\FarmMapsApiClient_WURtest\FarmmapsDataDownload\bin\Debug\netcoreapp3.1\Downloads\
"CropYear": 2022,
//"StartDate": "2022-04-15", //PO20231108: StartDate & EndDate important for legumes, where Nitrogen fixation depends on number of days from start to end. More days = more N fixation
//"EndDate": "2022-07-30", // if StartDate & EndDate left empty then whole year from cropYear
"geometryJson": {
"type": "Polygon",
"coordinates": [
[
[
5.5945257993548765,
52.57080744107003
],
[
5.598645994070678,
52.571540800206236
],
[
5.599381743127071,
52.57012773140724
],
[
5.595408698222548,
52.56968054825188
],
[
5.5945257993548765,
52.57080744107003
]
]
]
}
},
{
"useExistingCropfieldWithChildren": false,
"deleteNewlyCreatedAfterCalc": true,
"fieldName": "doperwt_groeiperiode",
"CropfieldItemCode": "",
"dataCropfield": {
"final": true, //always true
"cropTypeCode": "1030101", //From codelist CL263
"cropTypeName": "Garden pea" //From codelist CL263
},
"CropRecordingItemCode": "",
"OperationItemCodes": [],
"dataOperations": [],
"CropfieldCharacteristicItemCode": "",
"DataCropfieldCharacteristics": [
{
"code": "860619", //From codelist CL265. 860619 refers to cropyield in ton/ha
"label": "cropyield",
"value": "9.58" //CBS: https://www.cbs.nl/nl-nl/cijfers/detail/37738, 2022: 40.4 Mkg / 4216 ha = 40400 / 4216
}
],
//"DownloadFolder": "Downloads", //"C:\\hugoschrererdir\\kpidir\\", // "Downloads", -> if you just put "Downloads" the program will download to somewhere in ..\FarmMapsApiClient_WURtest\FarmmapsDataDownload\bin\Debug\netcoreapp3.1\Downloads\
"CropYear": 2022,
"StartDate": "2022-04-15", //PO20231108: StartDate & EndDate important for legumes, where Nitrogen fixation depends on number of days from start to end. More days = more N fixation
"EndDate": "2022-07-30", // if StartDate & EndDate left empty then whole year from cropYear
"geometryJson": {
"type": "Polygon",
"coordinates": [
[
[
5.5945257993548765,
52.57080744107003
],
[
5.598645994070678,
52.571540800206236
],
[
5.599381743127071,
52.57012773140724
],
[
5.595408698222548,
52.56968054825188
],
[
5.5945257993548765,
52.57080744107003
]
]
]
}
},
{
"useExistingCropfieldWithChildren": false,
"deleteNewlyCreatedAfterCalc": true,
"fieldName": "test_WinterWheat_DefaultStrawRemainsInField",
//Note when you run the FarmmapsKPI project, somewhere to your C:\git\FarmMapsApiClient_KB34_MAST\FarmmapsKPI\bin\Debug\netcoreapp3.1\ directory, files named like this are written:
//'Settings_aardappelveld_test_Potato_ZeroFertilizer.json', where 'aardappelveld_test_Potato_ZeroFertilizer.json' is your fieldName (see below)
//1 useExistingCropfieldWithChildren = false -> create new
//2 useExistingCropfieldWithChildren = true && CropfieldItemCode = "" or absent -> read fieldName (see below) -> read all codes from settings json file
//3 useExistingCropfieldWithChildren = true && CropfieldItemCode like "deb48a74c5b54299bb852f17288010e9" in KPIinput -> if you have in your account already an existing cropfield then use that
"CropfieldItemCode": "", // could contain for example this: "abae97f89f3c4ac08953b1b8bea9f076" if this is an exisiting CropfieldItemCode in your account.
//And then if useExistingCropfieldWithChildren = true, this would be used.
"dataCropfield": {
//"area": 4.22, //Leave empty, KPI app will calculate it from geometry
"final": true, //always true
"soilCode": "1", //From codelist CL405. Can be omitted if unknown
"soilName": "Sand", //From codelist CL405
"cropTypeCode": "1020101", //From codelist CL263
"cropTypeName": "Winter wheat" //From codelist CL263
},
"CropRecordingItemCode": "", // could contain for example this: "314aa1b6480b493ba28a5f55039a01d4" if this is the CropRecordingItemCode of the provided CropfieldItemCode. And then if useExistingCropfieldWithChildren = true, this would be used
"OperationItemCodes": [], // could contain for example this: ["4d1a6f28ea3f414180663baa4b08c60f"] or ["4d1a6f28ea3f414180663baa4b08c60f","21e08386f0454ad79acfebf78649d59b"] if these are the operations of the provided CropRecordingItemCode. And then if useExistingCropfieldWithChildren = true, these would be used
"dataOperations": [], //leaving the operations empty means no fertilizer is applied
"CropfieldCharacteristicItemCode": "",
"DataCropfieldCharacteristics": [
{
//PO20231029: not different yields from different parts of fields?
"code": "860619", //From codelist CL265. 860619 refers to cropyield in ton/ha
"label": "cropyield",
"value": "10.0"
}
],
//"DownloadFolder": "Downloads", //"C:\\hugoschrererdir\\kpidir\\", // "Downloads", -> if you just put "Downloads" the program will download to somewhere in ..\FarmMapsApiClient_KB34_MAST\FarmmapsKPI\bin\Debug\netcoreapp3.1\Downloads\
"CropYear": 2022,
//geometryJson = polygon of location of the cropfield, coordinates in LON,LAT (LON: decimal degrees East, LAT: decimal degrees North)
"geometryJson": {
"type": "Polygon",
"coordinates": [
[
[
5.5945257993548765,
52.57080744107003
],
[
5.598645994070678,
52.571540800206236
],
[
5.599381743127071,
52.57012773140724
],
[
5.595408698222548,
52.56968054825188
],
[
5.5945257993548765,
52.57080744107003
]
]
]
}
},
{
"useExistingCropfieldWithChildren": false,
"deleteNewlyCreatedAfterCalc": true,
"fieldName": "test_WinterWheat_StrawRemovedFromField",
//Note when you run the FarmmapsKPI project, somewhere to your C:\git\FarmMapsApiClient_KB34_MAST\FarmmapsKPI\bin\Debug\netcoreapp3.1\ directory, files named like this are written:
//'Settings_aardappelveld_test_Potato_ZeroFertilizer.json', where 'aardappelveld_test_Potato_ZeroFertilizer.json' is your fieldName (see below)
//1 useExistingCropfieldWithChildren = false -> create new
//2 useExistingCropfieldWithChildren = true && CropfieldItemCode = "" or absent -> read fieldName (see below) -> read all codes from settings json file
//3 useExistingCropfieldWithChildren = true && CropfieldItemCode like "deb48a74c5b54299bb852f17288010e9" in KPIinput -> if you have in your account already an existing cropfield then use that
"CropfieldItemCode": "", // could contain for example this: "abae97f89f3c4ac08953b1b8bea9f076" if this is an exisiting CropfieldItemCode in your account.
//And then if useExistingCropfieldWithChildren = true, this would be used.
"dataCropfield": {
//"area": 4.22, //Leave empty, KPI app will calculate it from geometry
"final": true, //always true
"soilCode": "1", //From codelist CL405. Can be omitted if unknown
"soilName": "Sand", //From codelist CL405
"cropTypeCode": "1020101", //From codelist CL263
"cropTypeName": "Winter wheat" //From codelist CL263
},
"CropRecordingItemCode": "", // could contain for example this: "314aa1b6480b493ba28a5f55039a01d4" if this is the CropRecordingItemCode of the provided CropfieldItemCode. And then if useExistingCropfieldWithChildren = true, this would be used
"OperationItemCodes": [], // could contain for example this: ["4d1a6f28ea3f414180663baa4b08c60f"] or ["4d1a6f28ea3f414180663baa4b08c60f","21e08386f0454ad79acfebf78649d59b"] if these are the operations of the provided CropRecordingItemCode. And then if useExistingCropfieldWithChildren = true, these would be used
"dataOperations": [], //leaving the operations empty means no fertilizer is applied
"CropfieldCharacteristicItemCode": "",
"DataCropfieldCharacteristics": [
{
//PO20231029: not different yields from different parts of fields?
"code": "860619", //From codelist CL265. 860619 refers to cropyield in ton/ha
"label": "cropyield",
"value": "10.0"
},
{
"code": "fm009", //TODO: which codelist?
"label": "strawyield",
"value": "6.8"
}
],
"CropYear": 2022,
//geometryJson = polygon of location of the cropfield, coordinates in LON,LAT (LON: decimal degrees East, LAT: decimal degrees North)
"geometryJson": {
"type": "Polygon",
"coordinates": [
[
[
5.5945257993548765,
52.57080744107003
],
[
5.598645994070678,
52.571540800206236
],
[
5.599381743127071,
52.57012773140724
],
[
5.595408698222548,
52.56968054825188
],
[
5.5945257993548765,
52.57080744107003
]
]
]
}
},
{
"useExistingCropfieldWithChildren": false,
"deleteNewlyCreatedAfterCalc": true,
"fieldName": "test_WinterWheat_greenmanure",
//Note when you run the FarmmapsKPI project, somewhere to your C:\git\FarmMapsApiClient_KB34_MAST\FarmmapsKPI\bin\Debug\netcoreapp3.1\ directory, files named like this are written:
//'Settings_aardappelveld_test_Potato_ZeroFertilizer.json', where 'aardappelveld_test_Potato_ZeroFertilizer.json' is your fieldName (see below)
//1 useExistingCropfieldWithChildren = false -> create new
//2 useExistingCropfieldWithChildren = true && CropfieldItemCode = "" or absent -> read fieldName (see below) -> read all codes from settings json file
//3 useExistingCropfieldWithChildren = true && CropfieldItemCode like "deb48a74c5b54299bb852f17288010e9" in KPIinput -> if you have in your account already an existing cropfield then use that
"CropfieldItemCode": "", // could contain for example this: "abae97f89f3c4ac08953b1b8bea9f076" if this is an exisiting CropfieldItemCode in your account.
//And then if useExistingCropfieldWithChildren = true, this would be used.
"dataCropfield": {
//"area": 4.22, //Leave empty, KPI app will calculate it from geometry
"final": true, //always true
"soilCode": "1", //From codelist CL405. Can be omitted if unknown
"soilName": "Sand", //From codelist CL405
"cropTypeCode": "1020101", //From codelist CL263
"cropTypeName": "Winter wheat" //From codelist CL263
},
"CropRecordingItemCode": "", // could contain for example this: "314aa1b6480b493ba28a5f55039a01d4" if this is the CropRecordingItemCode of the provided CropfieldItemCode. And then if useExistingCropfieldWithChildren = true, this would be used
"OperationItemCodes": [], // could contain for example this: ["4d1a6f28ea3f414180663baa4b08c60f"] or ["4d1a6f28ea3f414180663baa4b08c60f","21e08386f0454ad79acfebf78649d59b"] if these are the operations of the provided CropRecordingItemCode. And then if useExistingCropfieldWithChildren = true, these would be used
"dataOperations": [],
"CropfieldCharacteristicItemCode": "",
"DataCropfieldCharacteristics": [
{
"code": "860619", //From codelist CL265. 860619 refers to cropyield in ton/ha
"label": "cropyield",
"value": "10"
},
{
"code": "501505", //From codelist CL265. 501505 refers to green manure after main crop.
"label": "cropgreenmanureaftercultivation",
"value": "1040102", // Code is from CL263. You can put any CL263 crop here. However, the KPI app only works with a limited selection of greenmanure crops, see CL104.
"cropHeight": 100, //optional input. Height of greemmanure in cm. If this value is 0 then it is assumed that the crop height is unknown.
"plantingDate": "2022-08-01T12:34:00" // optional input. Plantingsdate of greenmanure. required layout "2022-07-23T12:34:00". If this value is null then it is assumed that the plantingsdate is unknown.
}
],
"CropYear": 2022,
"geometryJson": {
"type": "Polygon",
"coordinates": [
[
[
5.5945257993548765,
52.57080744107003
],
[
5.598645994070678,
52.571540800206236
],
[
5.599381743127071,
52.57012773140724
],
[
5.595408698222548,
52.56968054825188
],
[
5.5945257993548765,
52.57080744107003
]
]
]
}
}
]

27
FarmmapsKPI/KPIService.cs 100644
View File

@ -0,0 +1,27 @@
using System;
using System.Globalization;
using System.Threading.Tasks;
using FarmmapsApi.Models;
using FarmmapsApi.Services;
using FarmmapsKPI.Models;
using Microsoft.Extensions.Logging;
using static FarmmapsApi.Extensions;
using static FarmmapsApiSamples.Constants;
namespace FarmmapsKPI
{
public class KPIService
{
private readonly ILogger<KPIService> _logger;
private readonly FarmmapsApiService _farmmapsApiService;
private readonly GeneralService _generalService;
public KPIService(ILogger<KPIService> logger, FarmmapsApiService farmmapsApiService,
GeneralService generalService)
{
_logger = logger;
_farmmapsApiService = farmmapsApiService;
_generalService = generalService;
}
}
}

23
FarmmapsKPI/KPIdefinitions.csv 100644
View File

@ -0,0 +1,23 @@
KPIid,KPIvariable,Description,KPItargetvalue,KPIthresholdValue
A1,yield,observed yield (user input),target value as in benchmark value for same crop in same region. E.g. cropyield target is from Central Bureau of Statistics (CBS) for same crop in region in which the cropfield (geometry) is located.,No threshold for this KPI.
A1,strawYield,straw yield (user input; value 0 or not filled in = straw remains in field; value > 0 means straw was exported),there are no target values for individual compenets of KPI's,there are no threshold values for individual compenets of KPI's
B1,nitrogen,Nitrogen surplus = N artifical fertilizer + N manure+ N atmospheric deposition + N fixation by crop + N from seed potatoes - N removal through harvested product. Calculated KPI internal model & parameters and from user input: yield and fertilizer/manure applications.,The target value depends on the soil type. To calculate the threshold and target values data from CBS and RIVM are used. The same calculation method is also used in biodiversiteitsmonitoring akkerbouw (https://www.bo-akkerbouw.nl/kennis-en-innovatie/pps-biodiversiteitsmonitor-akkerbouw).,The threshold value depends on soil type and groundwater levels. To calculate the threshold and target values data from CBS and RIVM are used. The same calculation method is also used in biodiversiteitsmonitoring akkerbouw (https://www.bo-akkerbouw.nl/kennis-en-innovatie/pps-biodiversiteitsmonitor-akkerbouw).
B1,nHarvestedKgHa, N removal through harvested product. Calculated KPI internal model & parameters and from user input: yield.,there are no target values for individual compenets of KPI's,there are no threshold values for individual compenets of KPI's
B1,nHarvestedStrawKgHa, N removal through harvested product. Calculated KPI internal model & parameters and from user input: yieldStraw.,there are no target values for individual compenets of KPI's,there are no threshold values for individual compenets of KPI's
B1,nFixationNKgHa, N fixated by crop (legumes only). Calculated KPI internal model & parameters and from user input: planting date,there are no target values for individual compenets of KPI's,there are no threshold values for individual compenets of KPI's
B1,nDepositionNKgHa, N deposition based on RIVM maps. ,there are no target values for individual compenets of KPI's,there are no threshold values for individual compenets of KPI's
B1,nFertilizerNKgHa, N from artifical fertilizers and manure.Calculated KPI internal model & parameters and from user input: fertilizer/manure applications.,there are no target values for individual compenets of KPI's,there are no threshold values for individual compenets of KPI's
B1,sowPlantingNKgHa, N from seed potatoes (potatoes only). Calculated KPI internal model & parameters and from user input: amount of planted seed potatoes,there are no target values for individual compenets of KPI's,there are no threshold values for individual compenets of KPI's
B2,phosphate,Phosphate surplus = P fertilizer + P from seed potatoes - P removal through harvested product. Calculated KPI internal model & parameters and from user input: yield and fertilizer applications ,The phosphate target value is set at 20 kg/ha ,No threshold for this KPI.
B2,pHarvestedKgHa, P2O5 removal through harvested product. Calculated KPI internal model & parameters and from user input: yield,there are no target values for individual compenets of KPI's,there are no threshold values for individual compenets of KPI's
B2,pHarvestedStrawKgHa, P2O5 removal through harvested product. Calculated KPI internal model & parameters and from user input: yieldStraw,there are no target values for individual compenets of KPI's,there are no threshold values for individual compenets of KPI's
B2,pFertillizerPKgHa, P2O5 from artifical fertilizers and manure.Calculated KPI internal model & parameters and from user input: fertilizer/manure applications.,there are no target values for individual compenets of KPI's,there are no threshold values for individual compenets of KPI's
B2,sowPlantingPKgHa, P2O5 from seeds potatoes (potatoes only). Calculated KPI internal model & parameters and from user input: amount of planted seed potatoes,there are no target values for individual compenets of KPI's,there are no threshold values for individual compenets of KPI's
C1,effective organic matter supply,"Organic matter supply = Organic matter from manure + Crop residues to soil + organic matter from green manure. Calculated KPI internal model & parameters and from user input: yield and manure applications. ""Effective"" means netto, after correction for carbon respiration by soil microbial activity",The target value for this KPI is the amount of EOM that companies with ‘ Goede Landbouwpraktijk’ supply (2500 kg EOM/ha/year). ,The threshold value is 2000 kg EOM/ha/year. This value comes from van Doorn et al. (2022) which states that on average 2000 kg of EOM/ha degrades on Dutch agricultural land. This is the also the threshold value from biodiversiteitsmonitoring akkerbouw (https://www.bo-akkerbouw.nl/kennis-en-innovatie/pps-biodiversiteitsmonitor-akkerbouw)
C1,fertilizerEom,"Effective organic matter added through fertilizer (animal manure, compost, etc). Calculated KPI internal model & parameters and from user input: manure applications.",there are no target values for individual compenets of KPI's,there are no threshold values for individual compenets of KPI's
C1,greenManureEom,Effective organic matter added through green manure (also called cover crop). Calculated KPI internal model & parameters and from user input: cover crop.,there are no target values for individual compenets of KPI's,there are no threshold values for individual compenets of KPI's
C1,cropLeftoversEom,Effective organic matter added through crop residues of main product and (when relevant) straw. Calculated KPI internal model & parameters.,there are no target values for individual compenets of KPI's,there are no threshold values for individual compenets of KPI's
D1,pesticides,"kg/ha of active ingredient. Sum over kg a.i. of the different crop protection agents applied, with concentration (kg a.i. / kg or litre) of crop protection agents retrieved from CTBG",The target value fort his KPI is to reduce pesticide use to 50% of the amount used in the Netherlands during 2020. Data for 2020 is obtained from CBS statline ,No threshold for this KPI.
E1,mpb,"milieubelastingspunten. For each pesticide applied show 3 sub-mbp's: aquaticLife, groundWater, soilLife","0 applications with exceedances of either the target MBP or risks. MBP's consist of three categories (groundwater, soil life, and aquatic life). An value of <100 in either or these categories counts as exceedance. Each application of pesticides cannot have more then one exceedance for MBP. Risks consist of two categories (natural enemies and pollinators), a value of B or C in either category counts as exceedance. Each application of pesticides cannot have more then one exceedance for risks. Values and methodology come from Environmental Yardstick for Pesticides (https://www.milieumeetlat.nl/). The same calculation method is also used in biodiversiteitsmonitoring akkerbouw (https://www.bo-akkerbouw.nl/kennis-en-innovatie/pps-biodiversiteitsmonitor-akkerbouw).","0 applications with exceedances of either the target MBP or risks. MBP's consist of three categories (groundwater, soil life, and aquatic life). An value of <10 in either or these categories counts as exceedance. Each application of pesticides cannot have more then one exceedance for MBP. Risks consist of two categories (natural enemies and pollinators), a value of B or C in either category counts as exceedance. Each application of pesticides cannot have more then one exceedance for risks. Values and methodology come from Environmental Yardstick for Pesticides (https://www.milieumeetlat.nl/). The same calculation method is also used in biodiversiteitsmonitoring akkerbouw (https://www.bo-akkerbouw.nl/kennis-en-innovatie/pps-biodiversiteitsmonitor-akkerbouw)."
F1,greenness,time of year that the field was covered. Based on NDVI data from AgroDataCube.,Target value is 75%.,Threshold value is 25%.
F2,rotationindex,index of diversity of crops over time. Calculated in AgroDataCube,Target value is 75%.,Threshold value is 25%.
1 KPIid KPIvariable Description KPItargetvalue KPIthresholdValue
2 A1 yield observed yield (user input) target value as in benchmark value for same crop in same region. E.g. cropyield target is from Central Bureau of Statistics (CBS) for same crop in region in which the cropfield (geometry) is located. No threshold for this KPI.
3 A1 strawYield straw yield (user input; value 0 or not filled in = straw remains in field; value > 0 means straw was exported) there are no target values for individual compenets of KPI's there are no threshold values for individual compenets of KPI's
4 B1 nitrogen Nitrogen surplus = N artifical fertilizer + N manure+ N atmospheric deposition + N fixation by crop + N from seed potatoes - N removal through harvested product. Calculated KPI internal model & parameters and from user input: yield and fertilizer/manure applications. The target value depends on the soil type. To calculate the threshold and target values data from CBS and RIVM are used. The same calculation method is also used in biodiversiteitsmonitoring akkerbouw (https://www.bo-akkerbouw.nl/kennis-en-innovatie/pps-biodiversiteitsmonitor-akkerbouw). The threshold value depends on soil type and groundwater levels. To calculate the threshold and target values data from CBS and RIVM are used. The same calculation method is also used in biodiversiteitsmonitoring akkerbouw (https://www.bo-akkerbouw.nl/kennis-en-innovatie/pps-biodiversiteitsmonitor-akkerbouw).
5 B1 nHarvestedKgHa N removal through harvested product. Calculated KPI internal model & parameters and from user input: yield. there are no target values for individual compenets of KPI's there are no threshold values for individual compenets of KPI's
6 B1 nHarvestedStrawKgHa N removal through harvested product. Calculated KPI internal model & parameters and from user input: yieldStraw. there are no target values for individual compenets of KPI's there are no threshold values for individual compenets of KPI's
7 B1 nFixationNKgHa N fixated by crop (legumes only). Calculated KPI internal model & parameters and from user input: planting date there are no target values for individual compenets of KPI's there are no threshold values for individual compenets of KPI's
8 B1 nDepositionNKgHa N deposition based on RIVM maps. there are no target values for individual compenets of KPI's there are no threshold values for individual compenets of KPI's
9 B1 nFertilizerNKgHa N from artifical fertilizers and manure.Calculated KPI internal model & parameters and from user input: fertilizer/manure applications. there are no target values for individual compenets of KPI's there are no threshold values for individual compenets of KPI's
10 B1 sowPlantingNKgHa N from seed potatoes (potatoes only). Calculated KPI internal model & parameters and from user input: amount of planted seed potatoes there are no target values for individual compenets of KPI's there are no threshold values for individual compenets of KPI's
11 B2 phosphate Phosphate surplus = P fertilizer + P from seed potatoes - P removal through harvested product. Calculated KPI internal model & parameters and from user input: yield and fertilizer applications The phosphate target value is set at 20 kg/ha  No threshold for this KPI.
12 B2 pHarvestedKgHa P2O5 removal through harvested product. Calculated KPI internal model & parameters and from user input: yield there are no target values for individual compenets of KPI's there are no threshold values for individual compenets of KPI's
13 B2 pHarvestedStrawKgHa P2O5 removal through harvested product. Calculated KPI internal model & parameters and from user input: yieldStraw there are no target values for individual compenets of KPI's there are no threshold values for individual compenets of KPI's
14 B2 pFertillizerPKgHa P2O5 from artifical fertilizers and manure.Calculated KPI internal model & parameters and from user input: fertilizer/manure applications. there are no target values for individual compenets of KPI's there are no threshold values for individual compenets of KPI's
15 B2 sowPlantingPKgHa P2O5 from seeds potatoes (potatoes only). Calculated KPI internal model & parameters and from user input: amount of planted seed potatoes there are no target values for individual compenets of KPI's there are no threshold values for individual compenets of KPI's
16 C1 effective organic matter supply Organic matter supply = Organic matter from manure + Crop residues to soil + organic matter from green manure. Calculated KPI internal model & parameters and from user input: yield and manure applications. "Effective" means netto, after correction for carbon respiration by soil microbial activity The target value for this KPI is the amount of EOM that companies with ‘ Goede Landbouwpraktijk’ supply (2500 kg EOM/ha/year). The threshold value is 2000 kg EOM/ha/year. This value comes from van Doorn et al. (2022) which states that on average 2000 kg of EOM/ha degrades on Dutch agricultural land. This is the also the threshold value from biodiversiteitsmonitoring akkerbouw (https://www.bo-akkerbouw.nl/kennis-en-innovatie/pps-biodiversiteitsmonitor-akkerbouw)
17 C1 fertilizerEom Effective organic matter added through fertilizer (animal manure, compost, etc). Calculated KPI internal model & parameters and from user input: manure applications. there are no target values for individual compenets of KPI's there are no threshold values for individual compenets of KPI's
18 C1 greenManureEom Effective organic matter added through green manure (also called cover crop). Calculated KPI internal model & parameters and from user input: cover crop. there are no target values for individual compenets of KPI's there are no threshold values for individual compenets of KPI's
19 C1 cropLeftoversEom Effective organic matter added through crop residues of main product and (when relevant) straw. Calculated KPI internal model & parameters. there are no target values for individual compenets of KPI's there are no threshold values for individual compenets of KPI's
20 D1 pesticides kg/ha of active ingredient. Sum over kg a.i. of the different crop protection agents applied, with concentration (kg a.i. / kg or litre) of crop protection agents retrieved from CTBG The target value fort his KPI is to reduce pesticide use to 50% of the amount used in the Netherlands during 2020. Data for 2020 is obtained from CBS statline No threshold for this KPI.
21 E1 mpb milieubelastingspunten. For each pesticide applied show 3 sub-mbp's: aquaticLife, groundWater, soilLife 0 applications with exceedances of either the target MBP or risks. MBP's consist of three categories (groundwater, soil life, and aquatic life). An value of <100 in either or these categories counts as exceedance. Each application of pesticides cannot have more then one exceedance for MBP. Risks consist of two categories (natural enemies and pollinators), a value of B or C in either category counts as exceedance. Each application of pesticides cannot have more then one exceedance for risks. Values and methodology come from Environmental Yardstick for Pesticides (https://www.milieumeetlat.nl/). The same calculation method is also used in biodiversiteitsmonitoring akkerbouw (https://www.bo-akkerbouw.nl/kennis-en-innovatie/pps-biodiversiteitsmonitor-akkerbouw). 0 applications with exceedances of either the target MBP or risks. MBP's consist of three categories (groundwater, soil life, and aquatic life). An value of <10 in either or these categories counts as exceedance. Each application of pesticides cannot have more then one exceedance for MBP. Risks consist of two categories (natural enemies and pollinators), a value of B or C in either category counts as exceedance. Each application of pesticides cannot have more then one exceedance for risks. Values and methodology come from Environmental Yardstick for Pesticides (https://www.milieumeetlat.nl/). The same calculation method is also used in biodiversiteitsmonitoring akkerbouw (https://www.bo-akkerbouw.nl/kennis-en-innovatie/pps-biodiversiteitsmonitor-akkerbouw).
22 F1 greenness time of year that the field was covered. Based on NDVI data from AgroDataCube. Target value is 75%. Threshold value is 25%.
23 F2 rotationindex index of diversity of crops over time. Calculated in AgroDataCube Target value is 75%. Threshold value is 25%.

26
FarmmapsKPI/Models/KPIInput.cs 100644
View File

@ -0,0 +1,26 @@
using System;
using Newtonsoft.Json.Linq;
namespace FarmmapsKPI.Models
{
public class KPIInput
{
public bool UseExistingCropfieldWithChildren { get; set; }
public bool DeleteNewlyCreatedAfterCalc { get; set; }
public string fieldName { get; set; }
public string CropfieldItemCode { get; set; }
public JObject DataCropfield { get; set; }
public string CropRecordingItemCode { get; set; }
public JObject DataCropRecording { get; set; }
public string[] OperationItemCodes { get; set; }
public JObject[] DataOperations { get; set; }
public string CropfieldCharacteristicItemCode { get; set; }
public JObject[] DataCropfieldCharacteristics { get; set; }
public string DownloadFolder { get; set; }
public int CropYear { get; set; }
public DateTime StartDate { get; set; }
public DateTime EndDate { get; set; }
public JObject GeometryJson { get; set; }
}
}

73
FarmmapsKPI/Models/KPIoutput.cs 100644
View File

@ -0,0 +1,73 @@
using System;
using System.Collections.Generic;
using Newtonsoft.Json.Linq;
namespace FarmmapsKPI.Models
{
public class KPIOutput
{
public string id { get; set; }
public KPIOutputData data { get; set; }
public string type { get; set; }
public string unit { get; set; }
public string value { get; set; }
public string duration { get; set; }
public string quantity { get; set; }
public string parentName { get; set; }
public string targetValue { get; set; }
public string thresholdValue { get; set; }
public List<string> A1elements; //TtD
public List<string> B1elements;
public List<string> B2elements;
public List<string> C1elements;
public KPIOutput()
{
this.A1elements = new List<string>() { "yield", "strawYield" }; //TtD
this.B1elements = new List<string>() { "nHarvestedKgHa", "nHarvestedStrawKgHa", "nFixationNKgHa", "nDepositionNKgHa","nFertilizerNKgHa","sowPlantingNKgHa" }; //TtD
this.B2elements = new List<string>() { "pHarvestedKgHa", "pHarvestedStrawKgHa", "pFertillizerPKgHa", "sowPlantingPKgHa" }; //TtD
this.C1elements = new List<string>() { "fertilizerEom", "greenManureEom", "cropLeftoversEom" }; //TtD to do rename
}
}
public class KPIOutputData
{
public string area { get; set; }
public string cropTypeCode { get; set; }
public string cropTypeName { get; set; }
public string yield { get; set; }
public string strawYield { get; set; }
public KPIelementsOfBalance values { get; set; }
public KPIenvironmentMeasureData[] environmentMeasureData { get; set; }
}
public class KPIenvironmentMeasureData
{
public string date { get; set; }
public string productCode { get; set; }
public string productName { get; set; }
public string quantity { get; set; }
public string unitCode { get; set; }
public string soilLife { get; set; }
public string aquaticLife { get; set; }
public string groundWater { get; set; }
}
public class KPIelementsOfBalance
{
public string yield { get; set; } //TtD
public string strawYield { get; set; } //TtD
public string nHarvestedKgHa { get; set; } //TtD
public string nHarvestedStrawKgHa { get; set; } //TtD
public string nFixationNKgHa { get; set; }
public string nDepositionNKgHa { get; set; }
public string nFertilizerNKgHa { get; set; }
public string sowPlantingNKgHa { get; set; }
public string pHarvestedKgHa { get; set; } //TtD
public string pHarvestedStrawKgHa { get; set; } //TtD
public string pFertillizerPKgHa { get; set; } //TtD
public string sowPlantingPKgHa { get; set; } //TtD
public string fertilizerEom { get; set; } //TtD
public string greenManureEom { get; set; } //TtD
public string cropLeftoversEom { get; set; } //TtD tamara: rename
}
}

11
FarmmapsKPI/Models/Settings.cs 100644
View File

@ -0,0 +1,11 @@
namespace FarmmapsKPI
{
public class Settings
{
public string CropfieldItemCode { get; set; }
public string CropRecordingItemCode { get; set; }
public string[] OperationItemCodes { get; set; }
public string CropfieldCharacteristicItemCode { get; set; }
}
}

21
FarmmapsKPI/Program.cs 100644
View File

@ -0,0 +1,21 @@
using System.Threading.Tasks;
using FarmmapsApi;
using Microsoft.Extensions.DependencyInjection;
using Microsoft.Extensions.Logging;
namespace FarmmapsKPI
{
class Program : FarmmapsProgram<KPIApplication>
{
private static async Task Main(string[] args)
{
await new Program().Start(args);
}
protected override void Configure(IServiceCollection serviceCollection)
{
serviceCollection.AddLogging()
.AddTransient<KPIService>();
}
}
}

66
FarmmapsKPI/TODO.txt 100644
View File

@ -0,0 +1,66 @@
TODO.txt
###################################################################################################
# 1. DONE
###################################################################################################
Laat de farmmapsKPI zelf maar uitzoeken wat de n-p gehaltes zijn:
a. We willen als input opgeven we hebben 400 kg/ha urea gegeven, of 32000 kg/ha runderdrijfmest.
Daarvoor wel de juiste gewas code opgeven -> zie onder, FarmmapsEditeeltList 263 gewassen
Daarvoor wel de juiste meststof code opgeven -> zie onder, FarmmapsEditeeltList 022 meststoffen
b. Check of de operation data een element “n” bevat. Zoniet of als waarde 0, dan
i. Zoek onder de motorkap op: de omreken factor (bijv 200 kg/ha urea met 46% n -> 200*0.46 = 92 kg N/ha)
ii. Voeg die 92 toe aan de operation data ("n": 92)
c. Reken N surplus uit (KPI item id b1)
GEPARKEERD:
Want voor veel meststoffen is geen content bekend -> automatisch opzoeken zal in veel gevallen niet goed werken
Laat de gebruiker zelf maar de codelist CL022 raadplegen en omrekenen.
Nieuw project: FarmmapsEditeeltLists DONE
a. Why? Because
i. This is (1) so that we can at any time retrieve the most recent up-to-date list
ii. so that you can look up crops and fertilizer types and fill in KPIinput.json correctly
b. Download full list of items from vnd.farmmaps.itemtype.codelist.cl263 DONE
c. Write them to a csv file DONE
d. In any case following lists: 127 operations DONE; 022 meststoffen (fertilizer products) DONE; 263 gewassen DONE; 405 soilcode DONE
in which list the productionPurposeCode 251 DONE
Testing:
Effect of soiltype (e.g. different N concents in crop depending on soiltype? (CL405))
DONE (using KPIinput.json): same N balance, different target & threshold
Effect of location (e.g. different atmospheric N deposition).
DONE (using KPIinput.json): slightly different N balance
"DataCropfieldCharacteristic": {
"code": "860619", //PO20231004: so what does this code mean? Can we see the code list somewhere? COMPLETED code comes from codelist CL265. 860619 refers to cropyield in ton/ha
productionPurposeCode: different result depending on starch or consumption potato (different N contents in tubers)?
nContentFactor seems not to be adjusted by productionPurposeCode. Tamara check with Francisco
idem for pContentFactor
DONE (using KPIinput.json): surprisingly no effect?, format of the productionPurposeCode was wrong i.e was '002' instead of '2'. Format has been adjusted.
dataOperations: "operationCode": "7" & "8": DONE
a. which codelist? What does it mean? this refers to CL018 (bewerkinghoofdcaterogie), which contains information about the caterogie of the crop operations.
7 stands for fertilization while 8 refers to crop protection
b. using KPIinput.json: does it have an effect on KPI outcomes? YES
dataOperations: "status": "3": DONE
a. which codelist? What does it mean? this refers to CL256 (statusBewerking), which contains information about a crop operation has been planned, completed, cancelled etc. 3 refers to completed
b. using KPIinput.json: does it have an effect on KPI outcomes? NO, even without the status the crop operations is included in the KPI calculation.
As the status has no effect on the KPI calculations I have changed the lines about the status in the KPIinput.json file to comment
KPIelementsOfBalance: Tamara
a. define elements for C1 organic matter supply
b. for nitrogen B1 rename disposal to nHarvestedNKgHa COMPLETED
c. for phosporous B2 rename disposal to pHarvestedPKgHa COMPLETED
d. for phosporous B2 rename from pFertilizerNKgHa to pFertilizerPKgHa COMPLETED
e. idem sowPlantingNKgHa COMPLETED
f. add straw for N & P in output COMPLETED
g. add 2 examples in KPIinput.json: straw yes/no removed. Bijv. nStrawLeftInField EN nStrawRemoved COMPLETED
h. add amount of removed straw in A1 COMPLETED
i. update KPIdefinitions.csv COMPLETED
###################################################################################################
# 2. Pending
###################################################################################################
Testing:
KPIelementsOfBalance: Tamara
j. for C1 rename 'cropLeftoversEom' to 'cropResiduesEom' GEPARKEERD

4
FarmmapsZonering/FarmmapsZonering.csproj
View File

@ -1,8 +1,8 @@
<Project Sdk="Microsoft.NET.Sdk">
<Project Sdk="Microsoft.NET.Sdk">
<PropertyGroup>
<OutputType>Exe</OutputType>
<TargetFramework>netcoreapp3.0</TargetFramework>
<TargetFramework>netcoreapp3.1</TargetFramework>
</PropertyGroup>
<ItemGroup>

1
README.MD
View File

@ -1,6 +1,7 @@
##### NOT PRODUCTION READY CODE, JUST AN EXAMPLE
Put your clientId and clientSecret or your Farmmaps-account data (username and password) in a newly created appsettings.secrets.json file inside the root of the Secrets project.
Clone the git repository to a local drive. DON'T clone to a network drive like "My Documents". Latter may cause authentication problems.
**appsettings.secrets.json**
```

3
Secrets/Secrets.csproj
View File

@ -1,7 +1,8 @@
<Project Sdk="Microsoft.NET.Sdk">
<PropertyGroup>
<TargetFramework>netstandard1.0</TargetFramework>
<TargetFramework>netstandard2.1</TargetFramework>
<Platforms>AnyCPU;x64</Platforms>
</PropertyGroup>
<ItemGroup>

20
Secrets/appsettings.json
View File

@ -2,11 +2,11 @@
//There are three farmmaps environments, uncomment the environemnt you want to use with this sample client
//test environment
"Authority": "https://accounts.test.farmmaps.eu/",
"Endpoint": "https://test.farmmaps.eu/",
"BasePath": "api/v1",
"DiscoveryEndpointUrl": "https://accounts.test.farmmaps.eu/.well-known/openid-configuration",
"GrantClientId": "farmmapstesteu",
//"Authority": "https://accounts.test.farmmaps.eu/",
//"Endpoint": "https://test.farmmaps.eu/",
//"BasePath": "api/v1",
//"DiscoveryEndpointUrl": "https://accounts.test.farmmaps.eu/.well-known/openid-configuration",
//"GrantClientId": "farmmapstesteu",
////acceptance environment
//"Authority": "https://accounts.acc.farmmaps.eu/",
@ -16,11 +16,11 @@
//"GrantClientId": "farmmapsacceu",
////production environment
//"authority": "https://accounts.farmmaps.eu/",
//"endpoint": "https://farmmaps.eu/",
//"basepath": "api/v1",
//"discoveryendpointurl": "https://accounts.farmmaps.eu/.well-known/openid-configuration",
//"GrantClientId": "farmmaps",
"authority": "https://accounts.farmmaps.eu/",
"endpoint": "https://farmmaps.eu/",
"basepath": "api/v1",
"discoveryendpointurl": "https://accounts.farmmaps.eu/.well-known/openid-configuration",
"GrantClientId": "farmmaps",
//overige info
"RedirectUri": "http://example.nl/api",