Skip to content
Snippets Groups Projects
Commit 5759e0c2 authored by Jeremy Auclair's avatar Jeremy Auclair
Browse files

Docs update and modifications.

parent e617b88e
Branches
No related tags found
No related merge requests found
Expand all Show whitespace changes
Inline Side-by-side
Showing
with 642 additions and 55 deletions
docs/build/doctrees/environment.pickle
+ 0
− 0
View file @ 5759e0c2
No preview for this file type
docs/build/doctrees/file_structure.doctree
+ 0
− 0
View file @ 5759e0c2
No preview for this file type
docs/build/doctrees/filetree.doctree
+ 0
− 0
View file @ 5759e0c2
No preview for this file type
docs/build/doctrees/index.doctree
+ 0
− 0
View file @ 5759e0c2
No preview for this file type
docs/build/doctrees/inputs/ndvi_cube.doctree
+ 0
− 0
View file @ 5759e0c2
No preview for this file type
docs/build/doctrees/inputs/weather_cube.doctree
+ 0
− 0
View file @ 5759e0c2
No preview for this file type
docs/build/doctrees/preprocessing/input_preparation.doctree 0 → 100644
+ 0
− 0
View file @ 5759e0c2
File added
docs/build/doctrees/preprocessing/land_cover.doctree 0 → 100644
+ 0
− 0
View file @ 5759e0c2
File added
docs/build/doctrees/preprocessing/ndvi_cube.doctree 0 → 100644
+ 0
− 0
View file @ 5759e0c2
File added
docs/build/doctrees/preprocessing/samir_params.doctree 0 → 100644
+ 0
− 0
View file @ 5759e0c2
File added
docs/build/doctrees/preprocessing/soil_data.doctree 0 → 100644
+ 0
− 0
View file @ 5759e0c2
File added
docs/build/doctrees/preprocessing/weather_cube.doctree 0 → 100644
+ 0
− 0
View file @ 5759e0c2
File added
docs/build/html/_modules/index.html
+ 1
− 1
View file @ 5759e0c2
......@@ -43,7 +43,7 @@
<p class="caption" role="heading"><span class="caption-text">Contents:</span></p>
<ul>
<li class="toctree-l1"><a class="reference internal" href="../setup/setup.html">Setting up the correct environnment and parameters</a></li>
<li class="toctree-l1"><a class="reference internal" href="../inputs/input_preparation.html">Preparing the inputs for Modspa-Pixel</a></li>
<li class="toctree-l1"><a class="reference internal" href="../preprocessing/input_preparation.html">Preparing the inputs for Modspa-Pixel</a></li>
<li class="toctree-l1"><a class="reference internal" href="../samir/run_samir.html">Run the SAMIR model</a></li>
<li class="toctree-l1"><a class="reference internal" href="../file_structure.html">Project file structure</a></li>
</ul>
......
docs/build/html/_modules/modspa_pixel/preprocessing/download_ERA5_weather.html
+ 29
− 6
View file @ 5759e0c2
......@@ -90,11 +90,12 @@
<span class="kn">from</span> <span class="nn">psutil</span> <span class="kn">import</span> <span class="n">cpu_count</span> <span class="c1"># to get number of physical cores available</span>
<span class="kn">import</span> <span class="nn">modspa_pixel.preprocessing.lib_era5_land_pixel</span> <span class="k">as</span> <span class="nn">era5land</span> <span class="c1"># custom built functions for ERA5-Land data download</span>
<span class="kn">from</span> <span class="nn">modspa_pixel.config.config</span> <span class="kn">import</span> <span class="n">config</span> <span class="c1"># to load modspa config file</span>
<span class="kn">from</span> <span class="nn">modspa_pixel.preprocessing.parcel_to_pixel</span> <span class="kn">import</span> <span class="n">convert_dataframe_to_xarray</span>
<div class="viewcode-block" id="request_ER5_weather">
<a class="viewcode-back" href="../../../inputs/weather_cube.html#modspa_pixel.preprocessing.download_ERA5_weather.request_ER5_weather">[docs]</a>
<span class="k">def</span> <span class="nf">request_ER5_weather</span><span class="p">(</span><span class="n">config_file</span><span class="p">:</span> <span class="nb">str</span><span class="p">,</span> <span class="n">raw_S2_image_ref</span><span class="p">:</span> <span class="nb">str</span><span class="p">,</span> <span class="n">ndvi_path</span><span class="p">:</span> <span class="nb">str</span><span class="p">)</span> <span class="o">-&gt;</span> <span class="n">Tuple</span><span class="p">[</span><span class="nb">str</span><span class="p">,</span> <span class="nb">str</span><span class="p">]:</span>
<span class="k">def</span> <span class="nf">request_ER5_weather</span><span class="p">(</span><span class="n">config_file</span><span class="p">:</span> <span class="nb">str</span><span class="p">,</span> <span class="n">ndvi_path</span><span class="p">:</span> <span class="nb">str</span><span class="p">,</span> <span class="n">raw_S2_image_ref</span><span class="p">:</span> <span class="nb">str</span> <span class="o">=</span> <span class="kc">None</span><span class="p">,</span> <span class="n">shapefile</span><span class="p">:</span> <span class="nb">str</span> <span class="o">=</span> <span class="kc">None</span><span class="p">,</span> <span class="n">mode</span><span class="p">:</span> <span class="nb">str</span> <span class="o">=</span> <span class="s1">&#39;pixel&#39;</span><span class="p">)</span> <span class="o">-&gt;</span> <span class="nb">str</span><span class="p">:</span>
<span class="w"> </span><span class="sd">&quot;&quot;&quot;</span>
<span class="sd"> Download ERA5 reanalysis daily weather files, concatenate and calculate ET0</span>
<span class="sd"> to obtain a netCDF4 dataset for precipitation and ET0 values. Weather data</span>
......@@ -105,11 +106,15 @@
<span class="sd"> </span>
<span class="sd"> 1. config_file: ``str``</span>
<span class="sd"> json configuration file</span>
<span class="sd"> 2. raw_S2_image_ref: ``str``</span>
<span class="sd"> unmodified sentinel-2 image at correct resolution for</span>
<span class="sd"> weather data reprojection</span>
<span class="sd"> 3. ndvi_path: ``str``</span>
<span class="sd"> 2. ndvi_path: ``str``</span>
<span class="sd"> path to ndvi cube, used for weather data reprojection</span>
<span class="sd"> 3. raw_S2_image_ref: ``str`` ``default = None``</span>
<span class="sd"> unmodified sentinel-2 image at correct resolution for</span>
<span class="sd"> weather data reprojection in pixel mode</span>
<span class="sd"> 4. shapefile: ``str`` ``default = None``</span>
<span class="sd"> path to shapefile for extraction in parcel mode</span>
<span class="sd"> 5. mode: ``str`` ``default = &#39;pixel&#39;``</span>
<span class="sd"> choose between ``&#39;pixel&#39;`` and ``&#39;parcel&#39;`` mode</span>
<span class="sd"> Returns</span>
<span class="sd"> =======</span>
......@@ -226,8 +231,26 @@
<span class="c1"># Aggregate monthly files</span>
<span class="n">aggregated_files</span> <span class="o">=</span> <span class="n">era5land</span><span class="o">.</span><span class="n">concat_monthly_nc_file</span><span class="p">(</span><span class="n">list_era5land_hourly_ncFiles</span><span class="p">,</span> <span class="n">variable_list</span><span class="p">,</span> <span class="n">save_dir</span><span class="p">)</span>
<span class="c1"># Generate pandas dataframe for parcel mode</span>
<span class="k">if</span> <span class="n">mode</span> <span class="o">==</span> <span class="s1">&#39;parcel&#39;</span><span class="p">:</span>
<span class="c1"># Generate daily weather datasets as Geotiffs for each variable</span>
<span class="n">weather_daily_rain</span><span class="p">,</span> <span class="n">weather_daily_ET0</span> <span class="o">=</span> <span class="n">era5land</span><span class="o">.</span><span class="n">era5Land_daily_to_yearly_parcel</span><span class="p">(</span><span class="n">aggregated_files</span><span class="p">,</span> <span class="n">weather_daily_ncFile</span><span class="p">,</span> <span class="n">h</span> <span class="o">=</span> <span class="n">wind_height</span><span class="p">)</span>
<span class="c1"># Create save path</span>
<span class="n">weather_datframe</span> <span class="o">=</span> <span class="n">weather_daily_ncFile</span> <span class="o">+</span> <span class="s1">&#39;_df.csv&#39;</span>
<span class="n">weather_dataset</span> <span class="o">=</span> <span class="n">weather_daily_ncFile</span> <span class="o">+</span> <span class="s1">&#39;_parcel.nc&#39;</span>
<span class="c1"># Generate and save weather dataframe</span>
<span class="n">era5land</span><span class="o">.</span><span class="n">extract_weather_dataframe</span><span class="p">(</span><span class="n">weather_daily_rain</span><span class="p">,</span> <span class="n">weather_daily_ET0</span><span class="p">,</span> <span class="n">shapefile</span><span class="p">,</span> <span class="n">config_file</span><span class="p">,</span> <span class="n">weather_datframe</span><span class="p">)</span>
<span class="c1"># Convert dataframe to xarray dataset</span>
<span class="n">convert_dataframe_to_xarray</span><span class="p">(</span><span class="n">weather_datframe</span><span class="p">,</span> <span class="n">weather_dataset</span><span class="p">,</span> <span class="n">variables</span> <span class="o">=</span> <span class="p">[</span><span class="s1">&#39;Rain&#39;</span><span class="p">,</span> <span class="s1">&#39;ET0&#39;</span><span class="p">],</span> <span class="n">data_types</span> <span class="o">=</span> <span class="p">[</span><span class="s1">&#39;u2&#39;</span><span class="p">,</span> <span class="s1">&#39;u2&#39;</span><span class="p">])</span>
<span class="k">return</span> <span class="n">weather_dataset</span>
<span class="c1"># Calculate ET0 over the whole time period</span>
<span class="n">weather_daily_ncFile</span> <span class="o">=</span> <span class="n">era5land</span><span class="o">.</span><span class="n">era5Land_daily_to_yearly_nc</span><span class="p">(</span><span class="n">aggregated_files</span><span class="p">,</span> <span class="n">weather_daily_ncFile</span><span class="p">,</span> <span class="n">raw_S2_image_ref</span><span class="p">,</span> <span class="n">ndvi_path</span><span class="p">,</span> <span class="n">h</span> <span class="o">=</span> <span class="n">wind_height</span><span class="p">,</span> <span class="n">max_ram</span> <span class="o">=</span> <span class="mi">16</span><span class="p">)</span>
<span class="n">weather_daily_ncFile</span> <span class="o">=</span> <span class="n">era5land</span><span class="o">.</span><span class="n">era5Land_daily_to_yearly_pixel</span><span class="p">(</span><span class="n">aggregated_files</span><span class="p">,</span> <span class="n">weather_daily_ncFile</span><span class="p">,</span> <span class="n">raw_S2_image_ref</span><span class="p">,</span> <span class="n">ndvi_path</span><span class="p">,</span> <span class="n">h</span> <span class="o">=</span> <span class="n">wind_height</span><span class="p">,</span> <span class="n">max_ram</span> <span class="o">=</span> <span class="mi">16</span><span class="p">)</span>
<span class="nb">print</span><span class="p">(</span><span class="s1">&#39;</span><span class="se">\n</span><span class="s1">&#39;</span><span class="p">,</span> <span class="n">weather_daily_ncFile</span><span class="p">)</span>
......
docs/build/html/_modules/modspa_pixel/preprocessing/download_S2.html
+ 35
− 16
View file @ 5759e0c2
......@@ -87,7 +87,7 @@
<span class="kn">from</span> <span class="nn">eodag</span> <span class="kn">import</span> <span class="n">setup_logging</span> <span class="c1"># module that downloads S2 data</span>
<span class="kn">from</span> <span class="nn">eodag</span> <span class="kn">import</span> <span class="n">EODataAccessGateway</span> <span class="c1"># module that downloads S2 data</span>
<span class="kn">import</span> <span class="nn">geopandas</span> <span class="k">as</span> <span class="nn">gpd</span> <span class="c1"># to read shapefile</span>
<span class="kn">from</span> <span class="nn">typing</span> <span class="kn">import</span> <span class="n">List</span> <span class="c1"># to declare variables</span>
<span class="kn">from</span> <span class="nn">typing</span> <span class="kn">import</span> <span class="n">List</span><span class="p">,</span> <span class="n">Union</span> <span class="c1"># to declare variables</span>
<span class="kn">import</span> <span class="nn">csv</span> <span class="c1"># for loading and saving path results in csv format</span>
<span class="kn">import</span> <span class="nn">zipfile</span> <span class="k">as</span> <span class="nn">zp</span> <span class="c1"># to open zip archives</span>
<span class="kn">from</span> <span class="nn">tqdm</span> <span class="kn">import</span> <span class="n">tqdm</span> <span class="c1"># to print progress bars during code execution</span>
......@@ -96,13 +96,13 @@
<div class="viewcode-block" id="download_S2_data">
<a class="viewcode-back" href="../../../inputs/ndvi_cube.html#modspa_pixel.preprocessing.download_S2.download_S2_data">[docs]</a>
<span class="k">def</span> <span class="nf">download_S2_data</span><span class="p">(</span><span class="n">start_date</span><span class="p">:</span> <span class="nb">str</span><span class="p">,</span> <span class="n">end_date</span><span class="p">:</span> <span class="nb">str</span><span class="p">,</span> <span class="n">preferred_provider</span><span class="p">:</span> <span class="nb">str</span><span class="p">,</span> <span class="n">save_path</span><span class="p">:</span> <span class="nb">str</span><span class="p">,</span> <span class="n">shapefile</span><span class="p">:</span> <span class="nb">str</span> <span class="o">=</span> <span class="kc">None</span><span class="p">,</span> <span class="n">cloud_cover_limit</span><span class="p">:</span> <span class="nb">int</span> <span class="o">=</span> <span class="mi">80</span><span class="p">)</span> <span class="o">-&gt;</span> <span class="n">List</span><span class="p">[</span><span class="nb">str</span><span class="p">]:</span>
<span class="k">def</span> <span class="nf">download_S2_data</span><span class="p">(</span><span class="n">start_date</span><span class="p">:</span> <span class="nb">str</span><span class="p">,</span> <span class="n">end_date</span><span class="p">:</span> <span class="nb">str</span><span class="p">,</span> <span class="n">preferred_provider</span><span class="p">:</span> <span class="nb">str</span><span class="p">,</span> <span class="n">save_path</span><span class="p">:</span> <span class="nb">str</span><span class="p">,</span> <span class="n">shapefile</span><span class="p">:</span> <span class="nb">str</span><span class="p">,</span> <span class="n">mode</span><span class="p">:</span> <span class="nb">str</span> <span class="o">=</span> <span class="s1">&#39;pixel&#39;</span><span class="p">,</span> <span class="n">cloud_cover_limit</span><span class="p">:</span> <span class="nb">int</span> <span class="o">=</span> <span class="mi">80</span><span class="p">)</span> <span class="o">-&gt;</span> <span class="n">List</span><span class="p">[</span><span class="nb">str</span><span class="p">]:</span>
<span class="w"> </span><span class="sd">&quot;&quot;&quot;</span>
<span class="sd"> download_S2_data uses the eodag module to look for all products of a given provider</span>
<span class="sd"> (copernicus or theia) during a specific time window and covering the whole shapefile</span>
<span class="sd"> enveloppe (several Sentinel-2 tiles might be needed). It then downloads that data into</span>
<span class="sd"> the download path parametered in the config file. Paths to the downloaded data are</span>
<span class="sd"> returned and saved as a ``csv`` file.</span>
<span class="sd"> enveloppe (several Sentinel-2 tiles might be needed, only one can be chosen for the</span>
<span class="sd"> pixel mode). It then downloads that data into the download path parametered in the</span>
<span class="sd"> config file. Paths to the downloaded data are returned and saved as a ``csv`` file.</span>
<span class="sd"> Arguments</span>
<span class="sd"> =========</span>
......@@ -117,7 +117,9 @@
<span class="sd"> path where a csv file containing the product paths will be saved</span>
<span class="sd"> 5. shapefile: ``str``</span>
<span class="sd"> path to the shapefile (``.shp``) for which the data is downloaded</span>
<span class="sd"> 6. cloud_cover_limit: ``int`` ``default = 80``</span>
<span class="sd"> 6. mode: ``str`` ``default = &#39;pixel&#39;``</span>
<span class="sd"> run download code in &#39;pixel&#39; or &#39;parcel&#39; mode</span>
<span class="sd"> 7. cloud_cover_limit: ``int`` ``default = 80``</span>
<span class="sd"> maximum percentage to pass the filter before download (between 0 and 100)</span>
<span class="sd"> Returns</span>
......@@ -183,7 +185,7 @@
<div class="viewcode-block" id="extract_zip_archives">
<a class="viewcode-back" href="../../../inputs/ndvi_cube.html#modspa_pixel.preprocessing.download_S2.extract_zip_archives">[docs]</a>
<span class="k">def</span> <span class="nf">extract_zip_archives</span><span class="p">(</span><span class="n">download_path</span><span class="p">:</span> <span class="nb">str</span><span class="p">,</span> <span class="n">list_paths</span><span class="p">:</span> <span class="n">List</span><span class="p">[</span><span class="nb">str</span><span class="p">],</span> <span class="n">bands_to_extract</span><span class="p">:</span> <span class="n">List</span><span class="p">[</span><span class="nb">str</span><span class="p">],</span> <span class="n">save_path</span><span class="p">:</span> <span class="nb">str</span><span class="p">,</span> <span class="n">remove_archive</span><span class="p">:</span> <span class="nb">bool</span> <span class="o">=</span> <span class="kc">False</span><span class="p">)</span> <span class="o">-&gt;</span> <span class="n">List</span><span class="p">[</span><span class="nb">str</span><span class="p">]:</span>
<span class="k">def</span> <span class="nf">extract_zip_archives</span><span class="p">(</span><span class="n">download_path</span><span class="p">:</span> <span class="nb">str</span><span class="p">,</span> <span class="n">list_paths</span><span class="p">:</span> <span class="n">Union</span><span class="p">[</span><span class="n">List</span><span class="p">[</span><span class="nb">str</span><span class="p">],</span> <span class="nb">str</span><span class="p">],</span> <span class="n">bands_to_extract</span><span class="p">:</span> <span class="n">List</span><span class="p">[</span><span class="nb">str</span><span class="p">],</span> <span class="n">save_path</span><span class="p">:</span> <span class="nb">str</span><span class="p">,</span> <span class="n">remove_archive</span><span class="p">:</span> <span class="nb">bool</span> <span class="o">=</span> <span class="kc">False</span><span class="p">)</span> <span class="o">-&gt;</span> <span class="n">List</span><span class="p">[</span><span class="nb">str</span><span class="p">]:</span>
<span class="w"> </span><span class="sd">&quot;&quot;&quot;</span>
<span class="sd"> Extract specific bands in a zip archive for a list of tar archives.</span>
......@@ -210,6 +212,14 @@
<span class="sd"> list of the paths to the extracted products</span>
<span class="sd"> &quot;&quot;&quot;</span>
<span class="c1"># Load csv file if input is a path</span>
<span class="k">if</span> <span class="nb">type</span><span class="p">(</span><span class="n">list_paths</span><span class="p">)</span> <span class="o">==</span> <span class="nb">str</span><span class="p">:</span>
<span class="k">with</span> <span class="nb">open</span><span class="p">(</span><span class="n">list_paths</span><span class="p">,</span> <span class="s1">&#39;r&#39;</span><span class="p">)</span> <span class="k">as</span> <span class="n">file</span><span class="p">:</span>
<span class="n">list_paths</span> <span class="o">=</span> <span class="p">[]</span>
<span class="n">csvreader</span> <span class="o">=</span> <span class="n">csv</span><span class="o">.</span><span class="n">reader</span><span class="p">(</span><span class="n">file</span><span class="p">,</span> <span class="n">delimiter</span><span class="o">=</span><span class="s1">&#39;</span><span class="se">\n</span><span class="s1">&#39;</span><span class="p">)</span>
<span class="k">for</span> <span class="n">row</span> <span class="ow">in</span> <span class="n">csvreader</span><span class="p">:</span>
<span class="n">list_paths</span><span class="o">.</span><span class="n">append</span><span class="p">(</span><span class="n">row</span><span class="p">[</span><span class="mi">0</span><span class="p">])</span>
<span class="c1"># Final product list</span>
<span class="n">product_list</span> <span class="o">=</span> <span class="p">[]</span>
......@@ -218,7 +228,7 @@
<span class="k">for</span> <span class="n">file_path</span> <span class="ow">in</span> <span class="n">list_paths</span><span class="p">:</span>
<span class="c1"># Change progress bar to print current file</span>
<span class="n">progress_bar</span><span class="o">.</span><span class="n">set_description_str</span><span class="p">(</span><span class="n">desc</span> <span class="o">=</span> <span class="s1">&#39;</span><span class="se">\r</span><span class="s1">Extracting &#39;</span> <span class="o">+</span> <span class="n">os</span><span class="o">.</span><span class="n">path</span><span class="o">.</span><span class="n">basename</span><span class="p">(</span><span class="n">file_path</span><span class="p">)</span> <span class="o">+</span> <span class="s1">&#39;</span><span class="se">\n</span><span class="s1">total progress&#39;</span><span class="p">)</span>
<span class="n">progress_bar</span><span class="o">.</span><span class="n">set_description_str</span><span class="p">(</span><span class="n">desc</span> <span class="o">=</span> <span class="sa">f</span><span class="s1">&#39;Extracting </span><span class="si">{</span><span class="n">os</span><span class="o">.</span><span class="n">path</span><span class="o">.</span><span class="n">basename</span><span class="p">(</span><span class="n">file_path</span><span class="p">)</span><span class="si">}</span><span class="s1">, total progress&#39;</span><span class="p">)</span>
<span class="c1"># Get path in which to extract the archive</span>
<span class="n">extract_path</span> <span class="o">=</span> <span class="n">download_path</span> <span class="o">+</span> <span class="n">os</span><span class="o">.</span><span class="n">sep</span> <span class="o">+</span> <span class="n">os</span><span class="o">.</span><span class="n">path</span><span class="o">.</span><span class="n">basename</span><span class="p">(</span><span class="n">file_path</span><span class="p">)[:</span><span class="o">-</span><span class="mi">4</span><span class="p">]</span>
......@@ -227,25 +237,34 @@
<span class="k">with</span> <span class="n">zp</span><span class="o">.</span><span class="n">ZipFile</span><span class="p">(</span><span class="n">file_path</span><span class="p">,</span> <span class="n">mode</span> <span class="o">=</span> <span class="s1">&#39;r&#39;</span><span class="p">)</span> <span class="k">as</span> <span class="n">myzip</span><span class="p">:</span>
<span class="n">file_list</span> <span class="o">=</span> <span class="p">(</span><span class="n">myzip</span><span class="o">.</span><span class="n">namelist</span><span class="p">())</span>
<span class="k">for</span> <span class="n">f</span> <span class="ow">in</span> <span class="n">file_list</span><span class="p">:</span>
<span class="k">for</span> <span class="n">band</span> <span class="ow">in</span> <span class="n">bands_to_extract</span><span class="p">:</span>
<span class="k">if</span> <span class="n">fnmatch</span><span class="p">(</span><span class="n">f</span><span class="p">,</span> <span class="n">band</span><span class="p">):</span>
<span class="c1"># Check if already extacted</span>
<span class="n">f_name</span> <span class="o">=</span> <span class="n">os</span><span class="o">.</span><span class="n">path</span><span class="o">.</span><span class="n">basename</span><span class="p">(</span><span class="n">f</span><span class="p">)</span>
<span class="k">if</span> <span class="ow">not</span> <span class="n">os</span><span class="o">.</span><span class="n">path</span><span class="o">.</span><span class="n">exists</span><span class="p">(</span><span class="n">extract_path</span> <span class="o">+</span> <span class="n">os</span><span class="o">.</span><span class="n">sep</span> <span class="o">+</span> <span class="n">f_name</span><span class="p">):</span>
<span class="c1"># Extract file</span>
<span class="n">myzip</span><span class="o">.</span><span class="n">extract</span><span class="p">(</span><span class="n">f</span><span class="p">,</span> <span class="n">path</span> <span class="o">=</span> <span class="n">extract_path</span><span class="p">)</span>
<span class="c1"># Move extracted file to the root of the directory</span>
<span class="n">f_name</span> <span class="o">=</span> <span class="n">os</span><span class="o">.</span><span class="n">path</span><span class="o">.</span><span class="n">basename</span><span class="p">(</span><span class="n">f</span><span class="p">)</span>
<span class="n">shutil</span><span class="o">.</span><span class="n">move</span><span class="p">(</span><span class="n">extract_path</span> <span class="o">+</span> <span class="n">os</span><span class="o">.</span><span class="n">sep</span> <span class="o">+</span> <span class="n">f</span><span class="p">,</span> <span class="n">extract_path</span> <span class="o">+</span> <span class="n">os</span><span class="o">.</span><span class="n">sep</span> <span class="o">+</span> <span class="n">f_name</span><span class="p">)</span>
<span class="n">product_list</span><span class="o">.</span><span class="n">append</span><span class="p">(</span><span class="n">extract_path</span> <span class="o">+</span> <span class="n">os</span><span class="o">.</span><span class="n">sep</span> <span class="o">+</span> <span class="n">f_name</span><span class="p">)</span>
<span class="n">product_list</span><span class="o">.</span><span class="n">append</span><span class="p">(</span><span class="n">extract_path</span><span class="p">)</span>
<span class="c1"># Remove unecessary empty directories</span>
<span class="k">try</span><span class="p">:</span>
<span class="n">subfolder</span> <span class="o">=</span> <span class="p">[</span> <span class="n">f</span><span class="o">.</span><span class="n">path</span> <span class="k">for</span> <span class="n">f</span> <span class="ow">in</span> <span class="n">os</span><span class="o">.</span><span class="n">scandir</span><span class="p">(</span><span class="n">extract_path</span><span class="p">)</span> <span class="k">if</span> <span class="n">f</span><span class="o">.</span><span class="n">is_dir</span><span class="p">()][</span><span class="mi">0</span><span class="p">]</span>
<span class="n">shutil</span><span class="o">.</span><span class="n">rmtree</span><span class="p">(</span><span class="n">subfolder</span><span class="p">)</span>
<span class="k">except</span><span class="p">:</span>
<span class="k">pass</span>
<span class="k">if</span> <span class="n">remove_archive</span><span class="p">:</span>
<span class="c1"># Remove zip file</span>
<span class="n">os</span><span class="o">.</span><span class="n">remove</span><span class="p">(</span><span class="n">file_path</span><span class="p">)</span>
<span class="n">progress_bar</span><span class="o">.</span><span class="n">update</span><span class="p">(</span><span class="mi">1</span><span class="p">)</span>
<span class="c1"># Close progress bar</span>
<span class="n">progress_bar</span><span class="o">.</span><span class="n">set_description_str</span><span class="p">(</span><span class="n">desc</span> <span class="o">=</span> <span class="s1">&#39;Done!&#39;</span><span class="p">)</span>
<span class="n">progress_bar</span><span class="o">.</span><span class="n">close</span><span class="p">()</span>
<span class="c1"># Save list of paths as a csv file for later use</span>
......
docs/build/html/_modules/modspa_pixel/source/modspa_samir.html
+ 2
− 2
View file @ 5759e0c2
......@@ -1161,9 +1161,9 @@
<span class="n">model_outputs</span> <span class="o">=</span> <span class="n">prepare_output_dataset</span><span class="p">(</span><span class="n">ndvi_cube</span><span class="o">.</span><span class="n">drop_vars</span><span class="p">([</span><span class="s1">&#39;NDVI&#39;</span><span class="p">]),</span> <span class="n">additional_outputs</span><span class="o">=</span><span class="n">additional_outputs</span><span class="p">)</span>
<span class="c1"># Create encoding dictionnary</span>
<span class="n">encoding_dict</span> <span class="o">=</span> <span class="p">{}</span>
<span class="k">for</span> <span class="n">variable</span> <span class="ow">in</span> <span class="nb">list</span><span class="p">(</span><span class="n">model_outputs</span><span class="o">.</span><span class="n">keys</span><span class="p">()):</span>
<span class="c1"># Write encoding dict</span>
<span class="n">encoding_dict</span> <span class="o">=</span> <span class="p">{}</span>
<span class="n">encod</span> <span class="o">=</span> <span class="p">{}</span>
<span class="n">encod</span><span class="p">[</span><span class="s1">&#39;dtype&#39;</span><span class="p">]</span> <span class="o">=</span> <span class="s1">&#39;i2&#39;</span>
<span class="n">encod</span><span class="p">[</span><span class="s1">&#39;chunksizes&#39;</span><span class="p">]</span> <span class="o">=</span> <span class="p">(</span><span class="n">x_size</span><span class="p">,</span> <span class="n">y_size</span><span class="p">,</span> <span class="n">time_slice</span><span class="p">)</span>
......
docs/build/html/_sources/filetree.rst.txt
+ 17
− 10
View file @ 5759e0c2
::
├── dl_S2.csv
├── __init__.py
├── parameters
│ ├── __init__.py
......@@ -8,24 +7,32 @@
│ │ └── params_samir_test.csv
│ └── params_samir_class.py
├── DEV_inputs_test
│ ├── dl_S2.csv
│ ├── soil_100.nc
│ ├── ndvi_100.nc
│ ├── soil_1000.nc
│ ├── raw_ndvi.csv
│ ├── ndvi.nc
│ ├── filtered_ndvi.csv
│ ├── ndvi_10.nc
│ ├── xls_NDVI_10.nc
│ ├── xls_NDVI_100.nc
│ ├── soil_10.nc
│ ├── extracted_S2.csv
│ ├── ndvi_1000.nc
│ ├── land_cover_100.nc
│ ├── xls_weather_10.nc
│ ├── pix_outputs_10.nc
│ ├── land_cover_10.nc
│ ├── xls_outputs_100.nc
│ ├── interp_ndvi.csv
│ ├── test_S2_one_tile.csv
│ ├── xls_outputs_10.nc
│ ├── xls_weather_1000.nc
│ ├── xls_weather_100.nc
│ ├── land_cover_1000.nc
│ └── xls_NDVI_1000.nc
│ ├── xls_NDVI_1000.nc
│ └── ndvi_S2.csv
├── test_samir_dask.py
├── tests.py
├── modspa_pixel_env.yml
......@@ -35,13 +42,13 @@
├── SAMIR_xls
│ ├── SAMIRv5_juin2015.xlsx
│ └── SAMIRpixel_Reference_Simonneaux2012.xls
├── test_S2_one_tile.csv
├── config
│ ├── config.py
│ └── config_modspa.json
├── dev_samir_xarray.ipynb
├── preprocessing
│ ├── download_S2.py
│ ├── extract_ndvi.py
│ ├── parcel_to_pixel.py
│ ├── lib_era5_land_pixel.py
│ ├── calculate_ndvi.py
......@@ -52,13 +59,6 @@
├── postprocessing
└── docs
├── source
│ ├── inputs
│ │ ├── samir_params.rst
│ │ ├── weather_cube.rst
│ │ ├── land_cover.rst
│ │ ├── ndvi_cube.rst
│ │ ├── input_preparation.rst
│ │ └── soil_data.rst
│ ├── conf.py
│ ├── file_structure.rst
│ ├── .templates
......@@ -67,6 +67,13 @@
│ │ ├── data_and_ram_management.rst
│ │ ├── samir_functions.rst
│ │ └── run_samir.rst
│ ├── preprocessing
│ │ ├── samir_params.rst
│ │ ├── weather_cube.rst
│ │ ├── land_cover.rst
│ │ ├── ndvi_cube.rst
│ │ ├── input_preparation.rst
│ │ └── soil_data.rst
│ ├── index.rst
│ ├── .static
│ └── setup
......
docs/build/html/_sources/index.rst.txt
+ 2
− 2
View file @ 5759e0c2
......@@ -6,14 +6,14 @@
ModSpa-Pixel
========================================
ModSpa pixel is an adaptation of the existing `modspa <https://gitlab.cesbio.omp.eu/modelisation/modspa>`_ processing chain. It uses a mix of satellite data (optical and weather data) to run models like `SAMIR or SAFY <https://www.cesbio.cnrs.fr/la-recherche/activites/modeliser-codes-et-modelisation/liste-et-descriptif-des-modeles/modspa/>`_ (*only SAMIR is currently supported*) on spatialized data. The previous modspa version used ``pandas dataframes`` to run the SAMIR model on the **parcel scale** (inputs are satellite data aggregated to the parcel scale). ModSpa pixel uses ``numpy`` to run the SAMIR model at the **pixel scale**. This allows for higher resolution modelisation at the cost of an increase in storage and calculation requirements.
ModSpa pixel is an adaptation of the existing `modspa <https://gitlab.cesbio.omp.eu/modelisation/modspa>`_ processing chain. It uses a mix of satellite data (optical and weather data) to run models like `SAMIR or SAFY <https://www.cesbio.cnrs.fr/la-recherche/activites/modeliser-codes-et-modelisation/liste-et-descriptif-des-modeles/modspa/>`_ (*only SAMIR is currently supported*) on spatialized data. The previous modspa version used ``pandas dataframes`` to run the SAMIR model on the **parcel scale** (inputs are satellite data aggregated to the parcel scale). ModSpa pixel uses ``numpy`` to run the SAMIR model at the **pixel scale**. This allows for higher resolution modelisation at the cost of an increase in storage and calculation requirements. However, this ModSpa version can still run at the parcel scale, the user can **choose between parcel and pixel mode**.
.. toctree::
:maxdepth: 2
:caption: Contents:
setup/setup
inputs/input_preparation
preprocessing/input_preparation
samir/run_samir
file_structure
......
0% or .
You are about to add 0 people to the discussion. Proceed with caution.
Please to comment