Newer
Older
<!DOCTYPE html>
<html class="writer-html5" lang="en" >
<head>
<meta charset="utf-8" /><meta name="generator" content="Docutils 0.18.1: http://docutils.sourceforge.net/" />
<meta name="viewport" content="width=device-width, initial-scale=1.0" />
<title>Understanding the json config file — modspa-pixel 0.1 documentation</title>
<link rel="stylesheet" href="../_static/pygments.css" type="text/css" />
<link rel="stylesheet" href="../_static/css/theme.css" type="text/css" />
<link rel="stylesheet" href="../_static/graphviz.css" type="text/css" />
<link rel="stylesheet" href="../_static/custom.css" type="text/css" />
<!--[if lt IE 9]>
<script src="../_static/js/html5shiv.min.js"></script>
<![endif]-->
<script src="../_static/jquery.js?v=5d32c60e"></script>
<script src="../_static/_sphinx_javascript_frameworks_compat.js?v=2cd50e6c"></script>
<script src="../_static/documentation_options.js?v=2709fde1"></script>
<script src="../_static/doctools.js?v=888ff710"></script>
<script src="../_static/sphinx_highlight.js?v=dc90522c"></script>
<script src="../_static/js/theme.js"></script>
<link rel="index" title="Index" href="../genindex.html" />
<link rel="search" title="Search" href="../search.html" />
<link rel="next" title="Eodag configuration file" href="eodag_config.html" />
<link rel="prev" title="Installing the conda environment" href="conda_env.html" />
</head>
<body class="wy-body-for-nav">
<div class="wy-grid-for-nav">
<nav data-toggle="wy-nav-shift" class="wy-nav-side">
<div class="wy-side-scroll">
<div class="wy-side-nav-search" >
<a href="../index.html" class="icon icon-home">
modspa-pixel
</a>
<div role="search">
<form id="rtd-search-form" class="wy-form" action="../search.html" method="get">
<input type="text" name="q" placeholder="Search docs" aria-label="Search docs" />
<input type="hidden" name="check_keywords" value="yes" />
<input type="hidden" name="area" value="default" />
</form>
</div>
</div><div class="wy-menu wy-menu-vertical" data-spy="affix" role="navigation" aria-label="Navigation menu">
<p class="caption" role="heading"><span class="caption-text">Contents:</span></p>
<ul class="current">
<li class="toctree-l1 current"><a class="reference internal" href="setup.html">Setting up the correct environnment and parameters</a><ul class="current">
<li class="toctree-l2"><a class="reference internal" href="conda_env.html">Installing the conda environment</a></li>
<li class="toctree-l2 current"><a class="current reference internal" href="#">Understanding the json config file</a></li>
<li class="toctree-l2"><a class="reference internal" href="eodag_config.html">Eodag configuration file</a></li>
Jeremy Auclair
committed
<li class="toctree-l1"><a class="reference internal" href="../mode/mode.html">Choosing the run mode</a></li>
<li class="toctree-l1"><a class="reference internal" href="../preprocessing/input_preparation.html">Preparing the inputs for Modspa-Pixel</a></li>
<li class="toctree-l1"><a class="reference internal" href="../samir/run_samir.html">Run the SAMIR model</a></li>
<li class="toctree-l1"><a class="reference internal" href="../file_structure.html">Project file structure</a></li>
</ul>
</div>
</div>
</nav>
<section data-toggle="wy-nav-shift" class="wy-nav-content-wrap"><nav class="wy-nav-top" aria-label="Mobile navigation menu" >
<i data-toggle="wy-nav-top" class="fa fa-bars"></i>
<a href="../index.html">modspa-pixel</a>
</nav>
<div class="wy-nav-content">
<div class="rst-content">
<div role="navigation" aria-label="Page navigation">
<ul class="wy-breadcrumbs">
<li><a href="../index.html" class="icon icon-home" aria-label="Home"></a></li>
<li class="breadcrumb-item"><a href="setup.html">Setting up the correct environnment and parameters</a></li>
<li class="breadcrumb-item active">Understanding the json config file</li>
<li class="wy-breadcrumbs-aside">
<a href="../_sources/setup/json_config_file.rst.txt" rel="nofollow"> View page source</a>
</li>
</ul>
<hr/>
</div>
<div role="main" class="document" itemscope="itemscope" itemtype="http://schema.org/Article">
<div itemprop="articleBody">
<section id="understanding-the-json-config-file">
<span id="json-config"></span><h1>Understanding the json config file<a class="headerlink" href="#understanding-the-json-config-file" title="Link to this heading"></a></h1>
<p>All the global paramters for the modspa processing chain are located in the <code class="docutils literal notranslate"><span class="pre">json</span></code> (<strong>/modspa_pixel/config/config_modspa.json</strong>) configuration file.</p>
<p>This allows you to choose where and how to run the modspa_pixel processing chain. Different parts of the processing chain are detailed in the <strong>inputs</strong> and <strong>samir</strong> sections. You can have multiple configuration files with name extensions (<code class="docutils literal notranslate"><span class="pre">.name_ext</span></code> corresponds to the file <code class="docutils literal notranslate"><span class="pre">config_modspa.name_ext.json</span></code>), to select one configuration file for the run, use the <code class="docutils literal notranslate"><span class="pre">select_config_modspa.py</span></code> (in the <strong>modspa_pixel/config:</strong> directory) as follows:</p>
<div class="highlight-bash notranslate"><div class="highlight"><pre><span></span><span class="o">(</span>modspa_pixel<span class="o">)</span><span class="w"> </span>$<span class="w"> </span>python<span class="w"> </span>select_config_modspa.py<span class="w"> </span>name_ext
<span class="nb">source</span><span class="w"> </span>file:<span class="w"> </span>/modspa_pixel/config/config_modspa.name_ext.json
destination<span class="w"> </span>file:<span class="w"> </span>/modspa_pixel/config/config_modspa.json
/modspa_pixel/config/config_modspa.json<span class="w"> </span>updated
</pre></div>
</div>
<p>the name extension argument <code class="docutils literal notranslate"><span class="pre">name_ext</span></code> is used to select the file content you want and copy it in the regular <strong>/modspa_pixel/config/config_modspa.json</strong> file.</p>
<p>Example of a configuration file:</p>
<div class="highlight-json notranslate"><div class="highlight"><pre><span></span><span class="p">{</span>
<span class="w"> </span><span class="nt">"_comment"</span><span class="p">:</span><span class="w"> </span><span class="s2">"Sart date of the period on which the model will run"</span><span class="p">,</span>
<span class="w"> </span><span class="nt">"start_date"</span><span class="p">:</span><span class="w"> </span><span class="s2">"2019-01-01"</span><span class="p">,</span>
<span class="w"> </span><span class="nt">"_comment"</span><span class="p">:</span><span class="w"> </span><span class="s2">"End date of the period on which the model will run"</span><span class="p">,</span>
<span class="w"> </span><span class="nt">"end_date"</span><span class="p">:</span><span class="w"> </span><span class="s2">"2019-12-31"</span><span class="p">,</span>
<span class="w"> </span><span class="nt">"_comment"</span><span class="p">:</span><span class="w"> </span><span class="s2">"Choose between parcel or pixel mode for the run. You need a parcel shapefile for the parcel mode, and a boundary shapefile for the pixel mode."</span><span class="p">,</span>
<span class="w"> </span><span class="nt">"mode"</span><span class="p">:</span><span class="w"> </span><span class="s2">"pixel"</span><span class="p">,</span>
<span class="w"> </span><span class="nt">"_comment"</span><span class="p">:</span><span class="w"> </span><span class="s2">"Name of the current run, all output files will be saved under a subdirectory of Saves/ with that name, log file will also have that name"</span><span class="p">,</span>
<span class="w"> </span><span class="nt">"run_name"</span><span class="p">:</span><span class="w"> </span><span class="s2">"Aurade_test"</span><span class="p">,</span>
<span class="w"> </span><span class="nt">"_comment"</span><span class="p">:</span><span class="w"> </span><span class="s2">"Path to eodag configuration file for sentinel-2 image download"</span><span class="p">,</span>
<span class="w"> </span><span class="nt">"path_to_eodag_config_file"</span><span class="p">:</span><span class="w"> </span><span class="s2">"/home/auclairj/.config/eodag/eodag.yml"</span><span class="p">,</span>
<span class="w"> </span><span class="nt">"_comment"</span><span class="p">:</span><span class="w"> </span><span class="s2">"Prefered S2 data provider, choices = theia, copernicus"</span><span class="p">,</span>
<span class="w"> </span><span class="nt">"preferred_provider"</span><span class="p">:</span><span class="w"> </span><span class="s2">"copernicus"</span><span class="p">,</span>
<span class="w"> </span><span class="nt">"_comment"</span><span class="p">:</span><span class="w"> </span><span class="s2">"Maximum cloud cover percentage to download data, images with higher cloud cover will not be downloaded"</span><span class="p">,</span>
<span class="w"> </span><span class="nt">"cloud_cover_limit"</span><span class="p">:</span><span class="w"> </span><span class="mi">80</span><span class="p">,</span>
<span class="w"> </span><span class="nt">"_comment"</span><span class="p">:</span><span class="w"> </span><span class="s2">"Path to the directory on which the satellite image data will be downloaded"</span><span class="p">,</span>
<span class="w"> </span><span class="nt">"download_path"</span><span class="p">:</span><span class="w"> </span><span class="s2">"/mnt/e/DATA"</span><span class="p">,</span>
<span class="w"> </span><span class="nt">"_comment"</span><span class="p">:</span><span class="w"> </span><span class="s2">"Output path for output netcdf files (outpus will be stored in a subdirectory carrying the run_name)"</span><span class="p">,</span>
<span class="w"> </span><span class="nt">"output_path"</span><span class="p">:</span><span class="w"> </span><span class="s2">"/mnt/e/DATA/OUTPUT"</span><span class="p">,</span>
<span class="w"> </span><span class="nt">"_comment"</span><span class="p">:</span><span class="w"> </span><span class="s2">"Output path for netcdf era5 files (Weather)"</span><span class="p">,</span>
<span class="w"> </span><span class="nt">"era5_path"</span><span class="p">:</span><span class="w"> </span><span class="s2">"/mnt/e/DATA/WEATHER"</span><span class="p">,</span>
<span class="w"> </span><span class="nt">"_comment"</span><span class="p">:</span><span class="w"> </span><span class="s2">"Path to soil netCDF4 dataset"</span><span class="p">,</span>
<span class="w"> </span><span class="nt">"soil_path"</span><span class="p">:</span><span class="w"> </span><span class="s2">"/mnt/e/DATA/SOIL/Aurade_test/Soil_interpolated.nc"</span><span class="p">,</span>
<span class="w"> </span><span class="nt">"_comment"</span><span class="p">:</span><span class="w"> </span><span class="s2">"Path to Land Cover file (netCDF4 or GeoTiff)"</span><span class="p">,</span>
<span class="w"> </span><span class="nt">"land_cover_path"</span><span class="p">:</span><span class="w"> </span><span class="s2">"/mnt/e/DATA/LAND_COVER/Aurade_test/Aurade_test_new_LC.nc"</span><span class="p">,</span>
<span class="w"> </span><span class="nt">"_comment"</span><span class="p">:</span><span class="w"> </span><span class="s2">"Path to the shapefile to run the model on"</span><span class="p">,</span>
<span class="w"> </span><span class="nt">"shapefile_path"</span><span class="p">:</span><span class="w"> </span><span class="s2">"/home/auclairj/notebooks/Shapefiles/Aurade_tests/aurade_square.shp"</span><span class="p">,</span>
<span class="w"> </span><span class="nt">"_comment"</span><span class="p">:</span><span class="w"> </span><span class="s2">"Path to SAMIR parameter csv file"</span><span class="p">,</span>
<span class="w"> </span><span class="nt">"param_csv_file"</span><span class="p">:</span><span class="w"> </span><span class="s2">"/home/auclairj/GIT/modspa_pixel/parameters/csv_files/params_samir_Aurade_test.csv"</span><span class="p">,</span>
<span class="w"> </span><span class="nt">"_comment"</span><span class="p">:</span><span class="w"> </span><span class="s2">"Resolution in meters in which to run the processing chain. Choice is either 10 or 20."</span><span class="p">,</span>
<span class="w"> </span><span class="nt">"resolution"</span><span class="p">:</span><span class="w"> </span><span class="mi">10</span><span class="p">,</span>
<span class="w"> </span><span class="nt">"_comment"</span><span class="p">:</span><span class="w"> </span><span class="s2">"Overwrite NDVI images or not (set to true if you want the code to rewrite NDVI images, takes longer)"</span><span class="p">,</span>
<span class="w"> </span><span class="nt">"ndvi_overwrite"</span><span class="p">:</span><span class="w"> </span><span class="kc">false</span><span class="p">,</span>
<span class="w"> </span><span class="nt">"_comment"</span><span class="p">:</span><span class="w"> </span><span class="s2">"Overwrite Weather dataset or not (set to true if you want the code to rewrite weather dataset, takes longer)"</span><span class="p">,</span>
<span class="w"> </span><span class="nt">"weather_overwrite"</span><span class="p">:</span><span class="w"> </span><span class="kc">false</span><span class="p">,</span>
<span class="w"> </span><span class="nt">"_comment"</span><span class="p">:</span><span class="w"> </span><span class="s2">"Boolean to choose to automatically open the Dask Dashboard on default brower"</span><span class="p">,</span>
<span class="w"> </span><span class="nt">"open_browser"</span><span class="p">:</span><span class="w"> </span><span class="kc">false</span><span class="p">,</span>
<span class="w"> </span><span class="nt">"_comment"</span><span class="p">:</span><span class="w"> </span><span class="s2">"Max number of processor cores to use for multiprocessing calculations"</span><span class="p">,</span>
<span class="w"> </span><span class="nt">"max_cpu"</span><span class="p">:</span><span class="w"> </span><span class="mi">3</span><span class="p">,</span>
<span class="w"> </span><span class="nt">"_comment"</span><span class="p">:</span><span class="w"> </span><span class="s2">"Max amount of RAM memory (in GiB) to use for calculations"</span><span class="p">,</span>
<span class="w"> </span><span class="nt">"max_ram"</span><span class="p">:</span><span class="w"> </span><span class="mi">16</span><span class="p">,</span>
<span class="w"> </span><span class="nt">"_comment"</span><span class="p">:</span><span class="w"> </span><span class="s2">"List of variables to save as shapefiles for the parcel mode"</span><span class="p">,</span>
<span class="w"> </span><span class="nt">"parcel_shapefile_vars"</span><span class="p">:</span><span class="w"> </span><span class="p">[</span><span class="s2">"ETR"</span><span class="p">]</span>
<span class="p">}</span>
</pre></div>
</div>
<p>Parameter detail:</p>
<ol class="arabic simple">
<li><p><strong>start_date</strong>: <code class="docutils literal notranslate"><span class="pre">str</span></code>, start date of the simulation in <code class="docutils literal notranslate"><span class="pre">YYYY-MM-DD</span></code> format.</p></li>
<li><p><strong>end_date</strong>: <code class="docutils literal notranslate"><span class="pre">str</span></code>, end date of the simulation in <code class="docutils literal notranslate"><span class="pre">YYYY-MM-DD</span></code> format.</p></li>
<li><p><strong>mode</strong>: <code class="docutils literal notranslate"><span class="pre">str</span></code>, parameter to choose between the <code class="docutils literal notranslate"><span class="pre">'pixel'</span></code> or <code class="docutils literal notranslate"><span class="pre">'parcel'</span></code> mode. Both modes require a shapefile. A simple box is enough for the <code class="docutils literal notranslate"><span class="pre">'pixel'</span></code> mode (python only takes the total bounds of the shapefile). For the <code class="docutils literal notranslate"><span class="pre">'parcel'</span></code> mode, a shapefile with polygons representing each parcel and an attribute representing the land cover (for each polygon) is necessary.</p></li>
<li><p><strong>run_name</strong>: all preprocessed input files and output files will be saved in a directory with this name. Files will also carry the <em>run_name</em> in their name.</p></li>
<li><p><strong>path_to_eodag_config_file</strong>: <code class="docutils literal notranslate"><span class="pre">str</span></code>, path to the configuration file of the eodag module. This is needed to download sentinel-2 images to your machine.</p></li>
<li><p><strong>preferred_provider</strong>: <code class="docutils literal notranslate"><span class="pre">str</span></code>, choose between <strong>Copernicus</strong> or <strong>Theia</strong> sentinel-2 images (the atmospheric corrections are different for the two providers).</p></li>
<li><p><strong>cloud_cover_limit</strong>: <code class="docutils literal notranslate"><span class="pre">int</span></code>, the eodag module has an option to filter out images with more than a certain percentage of cloud cover (0 % is no clouds). These images will not be downloaded.</p></li>
<li><p><strong>download_path</strong>: <code class="docutils literal notranslate"><span class="pre">str</span></code>, path to directory where the raw input data will be downloaded. Subdirectories will be created for each input type (optical imagery, weather data, soil data, land cover data).</p></li>
</ol>
<div class="admonition warning">
<p class="admonition-title">Warning</p>
<p>For large spatial windows or over long time windows, make sure to have enough disk space for all the input (<strong>raw</strong> and <strong>preprocessed</strong>).</p>
</div>
<ol class="arabic simple" start="9">
<li><p><strong>output_path</strong>: <code class="docutils literal notranslate"><span class="pre">str</span></code>, output path for output netcdf files (outpus will be stored in a subdirectory carrying the run_name).</p></li>
<li><p><strong>era5_path</strong>: <code class="docutils literal notranslate"><span class="pre">str</span></code>, path to directory where ERA5-Land data will be downloaded and processed.</p></li>
<li><p><strong>soil_path</strong>: <code class="docutils literal notranslate"><span class="pre">str</span></code>, path to soil <code class="docutils literal notranslate"><span class="pre">netCDF4</span></code> file containing the <strong>Wilting Point</strong> and <strong>Field Capacity</strong> rasters.</p></li>
<li><p><strong>land_cover_path</strong>: <code class="docutils literal notranslate"><span class="pre">str</span></code>, path to land cover raster (<code class="docutils literal notranslate"><span class="pre">netCDF4</span></code> or <code class="docutils literal notranslate"><span class="pre">GeoTiff</span></code>). The land cover should have integer values ranging from 1 to the number of classes, each integer representing a class.</p></li>
<li><p><strong>shapefile_path</strong>: <code class="docutils literal notranslate"><span class="pre">str</span></code>, path to open the shapefile that defines your parcels (<strong>in parcel mode</strong>) or delimits the window on which you want to run the model (<strong>in pixel mode</strong>).</p></li>
<li><p><strong>param_csv_file</strong>: <code class="docutils literal notranslate"><span class="pre">str</span></code>, path to SAMIR csv parameter file.</p></li>
<li><p><strong>resolution</strong>: <code class="docutils literal notranslate"><span class="pre">int</span></code>, resolution in meters in which to run the processing chain.</p></li>
</ol>
<div class="admonition warning">
<p class="admonition-title">Warning</p>
<p>only <strong>10</strong> and <strong>20</strong> meters are currently handled.</p>
</div>
<ol class="arabic simple" start="16">
<li><p><strong>ndvi_overwrite</strong>: <code class="docutils literal notranslate"><span class="pre">bool</span></code>, choose weather to rewrite the NDVI data cube (<code class="docutils literal notranslate"><span class="pre">true</span> <span class="pre">=</span> <span class="pre">rewrite</span></code>, <code class="docutils literal notranslate"><span class="pre">false</span> <span class="pre">=</span> <span class="pre">don't</span> <span class="pre">rewrite</span></code>).</p></li>
</ol>
<div class="admonition warning">
<p class="admonition-title">Warning</p>
<p>The calculation of the NDVI data cube can take some time for large spatial windows and long time windows depending on your machine (up to an hour or more).</p>
</div>
<ol class="arabic simple" start="17">
<li><p><strong>open_browser</strong>: <code class="docutils literal notranslate"><span class="pre">bool</span></code>, boolean to choose to automatically open the Dask Dashboard on default brower when running the <code class="docutils literal notranslate"><span class="pre">'pixel'</span></code> ndvi calculation functions.</p></li>
<li><p><strong>max_cpu</strong>: <code class="docutils literal notranslate"><span class="pre">int</span></code>, maximum number of processor cores to use for calculation, the more the faster operations will be completed. Beware not to overload your machine. Set wisely when using the chain on a high performance cluster.</p></li>
<li><p><strong>max_ram</strong>: <code class="docutils literal notranslate"><span class="pre">int</span></code>, maximum amount of RAM memory (in <strong>GiB</strong>) to allocate for calculation, the more the faster operations will be completed. Large spatial windows or long time windows can make the chain very hungry for RAM, make sure to set an amount that your machine/cluster can support.</p></li>
<li><p><strong>parcel_shapefile_vars</strong>: <code class="docutils literal notranslate"><span class="pre">List[str]</span></code>; list of variables to save as shapefiles for the parcel mode. One shapefile per variable will be written in the output directory.</p></li>
</ol>
<div class="admonition note">
<p class="admonition-title">Note</p>
<p>A graphical interface to modify the <code class="docutils literal notranslate"><span class="pre">json</span></code> configuration file is planned for later versions.</p>
</div>
</section>
</div>
</div>
<footer><div class="rst-footer-buttons" role="navigation" aria-label="Footer">
<a href="conda_env.html" class="btn btn-neutral float-left" title="Installing the conda environment" accesskey="p" rel="prev"><span class="fa fa-arrow-circle-left" aria-hidden="true"></span> Previous</a>
<a href="eodag_config.html" class="btn btn-neutral float-right" title="Eodag configuration file" accesskey="n" rel="next">Next <span class="fa fa-arrow-circle-right" aria-hidden="true"></span></a>
217
218
219
220
221
222
223
224
225
226
227
228
229
230
231
232
233
234
235
236
237
238
239
240
241
242
</div>
<hr/>
<div role="contentinfo">
<p>© Copyright 2023, auclairj.</p>
</div>
Built with <a href="https://www.sphinx-doc.org/">Sphinx</a> using a
<a href="https://github.com/readthedocs/sphinx_rtd_theme">theme</a>
provided by <a href="https://readthedocs.org">Read the Docs</a>.
</footer>
</div>
</div>
</section>
</div>
<script>
jQuery(function () {
SphinxRtdTheme.Navigation.enable(true);
});
</script>
</body>
</html>