Skip to content
Snippets Groups Projects
Commit 23c3fc9f authored by TGermain's avatar TGermain
Browse files

rtd docstrings

parent cecd084f
No related branches found
No related tags found
No related merge requests found
Hide whitespace changes
Inline Side-by-side
Showing
with 1228 additions and 164 deletions
.readthedocs.yaml 0 → 100644
+ 36
− 0
View file @ 23c3fc9f
# .readthedocs.yaml
# Read the Docs configuration file
# See https://docs.readthedocs.io/en/stable/config-file/v2.html for details
# Required
version: 2
# Set the version of Python and other tools you might need
build:
os: ubuntu-22.04
tools:
python: "3.7"
apt_packages:
- libgdal-dev
- gdal-bin
- python3-gdal
# You can also specify other tool versions:
# nodejs: "19"
# rust: "1.64"
# golang: "1.19"
# Build documentation in the docs/ directory with Sphinx
sphinx:
configuration: docs/source/conf.py
# If using Sphinx, optionally build your docs in additional formats such as PDF
# formats:
# - pdf
# Optionally declare the Python requirements required to build your docs
python:
install:
- requirements: docs/requirements.txt
- requirements: requirements.txt
- method: pip
path: .
docs/Makefile
+ 5
− 5
View file @ 23c3fc9f
# Minimal makefile for Sphinx documentation
#
# You can set these variables from the command line.
SPHINXOPTS =
SPHINXBUILD = sphinx-build
SPHINXPROJ = sen2chain
# You can set these variables from the command line, and also
# from the environment for the first two.
SPHINXOPTS ?=
SPHINXBUILD ?= sphinx-build
SOURCEDIR = source
BUILDDIR = build
......@@ -17,4 +17,4 @@ help:
# Catch-all target: route all unknown targets to Sphinx using the new
# "make mode" option. $(O) is meant as a shortcut for $(SPHINXOPTS).
%: Makefile
@$(SPHINXBUILD) -M $@ "$(SOURCEDIR)" "$(BUILDDIR)" $(SPHINXOPTS) $(O)
\ No newline at end of file
@$(SPHINXBUILD) -M $@ "$(SOURCEDIR)" "$(BUILDDIR)" $(SPHINXOPTS) $(O)
docs/make.bat
+ 5
− 6
View file @ 23c3fc9f
......@@ -9,9 +9,6 @@ if "%SPHINXBUILD%" == "" (
)
set SOURCEDIR=source
set BUILDDIR=build
set SPHINXPROJ=sen2chain
if "%1" == "" goto help
%SPHINXBUILD% >NUL 2>NUL
if errorlevel 9009 (
......@@ -22,15 +19,17 @@ if errorlevel 9009 (
echo.may add the Sphinx directory to PATH.
echo.
echo.If you don't have Sphinx installed, grab it from
echo.http://sphinx-doc.org/
echo.https://www.sphinx-doc.org/
exit /b 1
)
%SPHINXBUILD% -M %1 %SOURCEDIR% %BUILDDIR% %SPHINXOPTS%
if "%1" == "" goto help
%SPHINXBUILD% -M %1 %SOURCEDIR% %BUILDDIR% %SPHINXOPTS% %O%
goto end
:help
%SPHINXBUILD% -M help %SOURCEDIR% %BUILDDIR% %SPHINXOPTS%
%SPHINXBUILD% -M help %SOURCEDIR% %BUILDDIR% %SPHINXOPTS% %O%
:end
popd
docs/requirements.txt 0 → 100644
+ 7
− 0
View file @ 23c3fc9f
#requirements for rtd building
sphinx==4.5.0
sphinx_rtd_theme==1.1.1
readthedocs-sphinx-search==0.1.1
sphinx-math-dollar==1.2.1
myst_parser==0.18.1
sphinx-book-theme==0.3.3
docs/source/conf.py
+ 34
− 153
View file @ 23c3fc9f
# -*- coding: utf-8 -*-
#
# Configuration file for the Sphinx documentation builder.
#
# This file does only contain a selection of the most common options. For a
# full list see the documentation:
# http://www.sphinx-doc.org/en/master/config
# For the full list of built-in configuration values, see the documentation:
# https://www.sphinx-doc.org/en/master/usage/configuration.html
# -- Path setup --------------------------------------------------------------
# -- Project information -----------------------------------------------------
# https://www.sphinx-doc.org/en/master/usage/configuration.html#project-information
project = 'Sen2chain'
copyright = '2023, ESPACE-DEV'
author = 'ESPACE-DEV'
release = '1.0'
# If extensions (or modules to document with autodoc) are in another directory,
# add these directories to sys.path here. If the directory is relative to the
# documentation root, use os.path.abspath to make it absolute, like shown here.
#
import os
import sys
sys.path.insert(0, os.path.abspath('../..'))
# -- Project information -----------------------------------------------------
project = 'sen2chain'
copyright = '2018, Jérémy Commins'
author = 'Jérémy Commins'
doc_title = 'sen2chain documentation'
# The short X.Y version
version = ''
# The full version, including alpha/beta/rc tags
release = '0.0.1'
# math
from sphinx_math_dollar import NODE_BLACKLIST
from docutils.nodes import header,FixedTextElement, literal,math
from docutils.nodes import comment, doctest_block, image, literal_block, math_block, paragraph, pending, raw, rubric, substitution_definition, target
math_dollar_node_blacklist = (literal,math,doctest_block, image, literal_block, math_block, pending, raw,rubric, substitution_definition,target)
# -- General configuration ---------------------------------------------------
# https://www.sphinx-doc.org/en/master/usage/configuration.html#general-configuration
# If your documentation needs a minimal Sphinx version, state it here.
#
# needs_sphinx = '1.0'
extensions = ["myst_parser",'sphinx.ext.autodoc', 'sphinx.ext.coverage', 'sphinx.ext.napoleon','sphinx.ext.autosummary','sphinx_math_dollar','sphinx.ext.mathjax']
# Add any Sphinx extension module names here, as strings. They can be
# extensions coming with Sphinx (named 'sphinx.ext.*') or your custom
# ones.
#extensions = [
# 'sphinx.ext.autodoc',
# 'sphinx.ext.intersphinx',
# 'sphinx.ext.ifconfig',
# 'sphinx_autodoc_annotation',
#]
autodoc_mock_imports = ["osgeo"]
extensions = [
'sphinx.ext.autodoc',
'sphinx_autodoc_typehints',
'sphinx.ext.intersphinx',
]
# Add any paths that contain templates here, relative to this directory.
templates_path = ['_templates']
# The suffix(es) of source filenames.
# You can specify multiple suffix as a list of string:
#
# source_suffix = ['.rst', '.md']
source_suffix = '.rst'
# The master toctree document.
master_doc = 'index'
# The language for content autogenerated by Sphinx. Refer to documentation
# for a list of supported languages.
#
# This is also used if you do content translation via gettext catalogs.
# Usually you set "language" from the command line for these cases.
language = None
# List of patterns, relative to source directory, that match files and
# directories to ignore when looking for source files.
# This pattern also affects html_static_path and html_extra_path .
exclude_patterns = []
# The name of the Pygments (syntax highlighting) style to use.
pygments_style = 'sphinx'
autodoc_member_order = 'bysource'
# -- Options for HTML output -------------------------------------------------
# The theme to use for HTML and HTML Help pages. See the documentation for
# a list of builtin themes.
#
# html_theme = 'alabaster'
html_theme = "sphinx_rtd_theme"
# Theme options are theme-specific and customize the look and feel of a theme
# further. For a list of options available for each theme, see the
# documentation.
#
# html_theme_options = {}
# Add any paths that contain custom static files (such as style sheets) here,
# relative to this directory. They are copied after the builtin static files,
# so a file named "default.css" will overwrite the builtin "default.css".
html_static_path = ['_static']
# Custom sidebar templates, must be a dictionary that maps document names
# to template names.
#
# The default sidebars (for documents that don't match any pattern) are
# defined by theme itself. Builtin themes are using these templates by
# default: ``['localtoc.html', 'relations.html', 'sourcelink.html',
# 'searchbox.html']``.
#
# html_sidebars = {}
# -- Options for HTMLHelp output ---------------------------------------------
# Output file base name for HTML help builder.
htmlhelp_basename = 'sen2chaindoc'
# -- Options for LaTeX output ------------------------------------------------
latex_elements = {
# The paper size ('letterpaper' or 'a4paper').
#
# 'papersize': 'letterpaper',
# The font size ('10pt', '11pt' or '12pt').
#
# 'pointsize': '10pt',
# Additional stuff for the LaTeX preamble.
#
# 'preamble': '',
# Latex figure (float) alignment
#
# 'figure_align': 'htbp',
mathjax_config = {
'tex2jax': {
'inlineMath': [ ["\\(","\\)"] ],
'displayMath': [["\\[","\\]"] ],
},
}
# Grouping the document tree into LaTeX files. List of tuples
# (source start file, target name, title,
# author, documentclass [howto, manual, or own class]).
latex_documents = [
(master_doc, 'sen2chain.tex', 'sen2chain Documentation',
'Jérémy Commins', 'manual'),
]
# -- Options for manual page output ------------------------------------------
# One entry per manual page. List of tuples
# (source start file, name, description, authors, manual section).
man_pages = [
(master_doc, 'sen2chain', 'sen2chain Documentation',
[author], 1)
]
# -- Options for Texinfo output ----------------------------------------------
# Grouping the document tree into Texinfo files. List of tuples
# (source start file, target name, title, author,
# dir menu entry, description, category)
texinfo_documents = [
(master_doc, 'sen2chain', 'sen2chain Documentation',
author, 'sen2chain', 'Download and process Sentinel-2 images.',
'Miscellaneous'),
]
# -- Extension configuration -------------------------------------------------
mathjax3_config = {
"tex": {
"inlineMath": [['\\(', '\\)']],
"displayMath": [["\\[", "\\]"]],
}
}
# -- Options for intersphinx extension ---------------------------------------
# -- Options for HTML output -------------------------------------------------
# https://www.sphinx-doc.org/en/master/usage/configuration.html#options-for-html-output
# Example configuration for intersphinx: refer to the Python standard library.
intersphinx_mapping = {'https://docs.python.org/': None}
html_theme = 'sphinx_book_theme'
html_permalinks_icon = '<span>#</span>'
html_static_path = ['_static']
docs/source/configure_sen2cor/index.rst 0 → 100644
+ 35
− 0
View file @ 23c3fc9f
.. _configure_sen2cor:
===================================
Configure previous Sen2Cor versions
===================================
To retrieve information about the Sen2Cor version and process L1C products accordingly, Sen2chain uses Sen2Cor install paths listed in ``~/sen2chain_data/config/config.cfg``.
Therefore, please keep your Sen2Cor install directory names unchanged (ie Sen2Cor-02.0\*.0\*-Linux64) so that Sen2Chain finds this pattern.
If you have multiple Sen2Cor versions installed, your Sen2chain configuration file should look like so :
.. code-block:: bash
[SEN2COR PATH]
sen2cor_bashrc_paths = /home/thomas/Sen2Cor-02.10.00-Linux64/L2A_Bashrc
/home/thomas/Sen2Cor-02.10.01-Linux64/L2A_Bashrc
/home/thomas/Sen2Cor-02.09.00-Linux64/L2A_Bashrc
Sen2Chain will not override your Sen2Cor configuration file.
In Sen2Cor configuration file ``~/sen2cor/2.11/cfg/L2A_GIPP.xml``, you have to directly edit DEM_Directory and DEM_Reference paths in order to auto-download and use DEM correction.
If a DEM is used the land cover data should be used to improve Sen2Cor classification algorithms.
At ESPACE-DEV in La Reunion we still don't use the DEM correction as it seems not relevant in high shadow / steep slope environments (as in Reunion Island).
.. toctree::
:maxdepth: 2
:caption: Versions:
sen2cor_2_9
sen2cor_2_10
docs/source/configure_sen2cor/sen2cor_2_10.rst 0 → 100644
+ 45
− 0
View file @ 23c3fc9f
Sen2Cor 2.10
============
.. role:: bash(code)
:language: bash
* Download `Sen2Cor v2.10.01 <https://step.esa.int/thirdparties/sen2cor/2.10.0/Sen2Cor-02.10.01-Linux64.run>`_ and put it to its final install directory.
* Make it executable :bash:`chmod +x Sen2Cor-02.10.01-Linux64.run`
* Install Sen2Cor :bash:`./Sen2Cor-02.10.01-Linux64.run`
* Update your configuration:
1. Move or symlink `ESACCI-LC package for SenCor module <https://earth.esa.int/eogateway/ftp/Sentinel-2/ESACCI-LC-L4-ALL-FOR-SEN2COR-2.10.tar.gz>`_ content to Sen2Cor directory:
.. code-block:: bash
ln -s /path/to/ESACCI-LC-L4-ALL-FOR-SEN2COR-2.10/ESACCI-LC-L4-LCCS-Map-300m-P1Y-2015-v2.0.7.tif /Path/to/Sen2Cor-02.10.01-Linux64/lib/python2.7/site-packages/sen2cor/aux_data
ln -s /path/to/ESACCI-LC-L4-WB-Map-150m-P13Y-2000-v4.0.tif Sen2Cor-02.10.01-Linux64/lib/python2.7/site-packages/sen2cor/aux_data/
ln -s /path/to/ESACCI-LC-L4-Snow-Cond-500m-MONTHLY-2000-2012-v2.4 Sen2Cor-02.10.01-Linux64/lib/python2.7/site-packages/sen2cor/aux_data/
2. Create or symlink DEM directory
.. code-block:: bash
ln -s /path/to/dem ~/sen2cor/2.10/dem
3. Update Sen2Cor config file :bash:`nano ~/sen2cor/2.10/cfg/L2A_GIPP.xml`
.. code-block:: html
<DEM_Directory>dem/srtm</DEM_Directory>
<DEM_Reference>http://srtm.csi.cgiar.org/wp-content/uploads/files/srtm_5x5/TIFF/</DEM_Reference>
<DEM_Terrain_Correction>FALSE</DEM_Terrain_Correction> # If you don't want DEM terrain correction
4. Update Sen2Chain configuration file :bash:`nano ~/sen2chain_data/config/config.cfg`
.. code-block:: bash
sen2cor_bashrc_path = /path/to/Sen2Cor-02.10.01-Linux64/L2A_Bashrc
* Of course you have to update Sen2Chain
docs/source/configure_sen2cor/sen2cor_2_9.rst 0 → 100644
+ 45
− 0
View file @ 23c3fc9f
Sen2Cor 2.9
============
.. role:: bash(code)
:language: bash
* Download `Sen2Cor v2.09.00 <http://step.esa.int/main/snap-supported-plugins/sen2cor/sen2cor-v2-9/>`_ and put it to its final install directory.
* Make it executable :bash:`chmod +x Sen2Cor-02.09.00-Linux64.run`
* Install Sen2Cor :bash:`./Sen2Cor-02.09.00-Linux64.run`
* Update your configuration:
1. Move or symlink `ESACCI-LC package for SenCor module <https://earth.esa.int/eogateway/ftp/Sentinel-2/ESACCI-LC-L4-ALL-FOR-SEN2COR-2.10.tar.gz>`_ content to Sen2Cor directory:
.. code-block:: bash
ln -s /path/to/ESACCI-LC-L4-LCCS-Map-300m-P1Y-2015-v2.0.7.tif Sen2Cor-02.09.00-Linux64/lib/python2.7/site-packages/sen2cor/aux_data/
ln -s /path/to/ESACCI-LC-L4-WB-Map-150m-P13Y-2000-v4.0.tif Sen2Cor-02.09.00-Linux64/lib/python2.7/site-packages/sen2cor/aux_data/
ln -s /path/to/ESACCI-LC-L4-Snow-Cond-500m-P13Y7D-2000-2012-v2.0 Sen2Cor-02.09.00-Linux64/lib/python2.7/site-packages/sen2cor/aux_data/
2. Create or symlink DEM directory
.. code-block:: bash
ln -s /path/to/dem ~/sen2cor/2.9/dem
3. Update Sen2Cor config file :bash:`nano ~/sen2cor/2.9/cfg/L2A_GIPP.xml`
.. code-block:: html
<DEM_Directory>dem/srtm</DEM_Directory>
<DEM_Reference>http://data_public:GDdci@data.cgiar-csi.org/srtm/tiles/GeoTIFF/</DEM_Reference>
<DEM_Terrain_Correction>FALSE</DEM_Terrain_Correction> # If you don't want DEM terrain correction
4. Update Sen2Chain configuration file :bash:`nano ~/sen2chain_data/config/config.cfg`
.. code-block:: bash
sen2cor_bashrc_path = /path/to/Sen2Cor-02.09.00-Linux64/L2A_Bashrc
* Of course you have to update Sen2Chain
docs/source/contribute.rst 0 → 100644
+ 5
− 0
View file @ 23c3fc9f
==========
Contribute
==========
You can submit your own developments and branches to our 'framagit <https://framagit.org/espace-dev/sen2chain>'
docs/source/esdev.png.aux.xml 0 → 100644
+ 32
− 0
View file @ 23c3fc9f
<PAMDataset>
<Metadata domain="IMAGE_STRUCTURE">
<MDI key="INTERLEAVE">PIXEL</MDI>
</Metadata>
<PAMRasterBand band="1">
<Metadata>
<MDI key="STATISTICS_MAXIMUM">255</MDI>
<MDI key="STATISTICS_MEAN">224.90926240332</MDI>
<MDI key="STATISTICS_MINIMUM">0</MDI>
<MDI key="STATISTICS_STDDEV">67.116161408218</MDI>
<MDI key="STATISTICS_VALID_PERCENT">100</MDI>
</Metadata>
</PAMRasterBand>
<PAMRasterBand band="2">
<Metadata>
<MDI key="STATISTICS_MAXIMUM">255</MDI>
<MDI key="STATISTICS_MEAN">221.1103565365</MDI>
<MDI key="STATISTICS_MINIMUM">5</MDI>
<MDI key="STATISTICS_STDDEV">69.493557576905</MDI>
<MDI key="STATISTICS_VALID_PERCENT">100</MDI>
</Metadata>
</PAMRasterBand>
<PAMRasterBand band="3">
<Metadata>
<MDI key="STATISTICS_MAXIMUM">255</MDI>
<MDI key="STATISTICS_MEAN">227.7741396502</MDI>
<MDI key="STATISTICS_MINIMUM">18</MDI>
<MDI key="STATISTICS_STDDEV">59.174002889391</MDI>
<MDI key="STATISTICS_VALID_PERCENT">100</MDI>
</Metadata>
</PAMRasterBand>
</PAMDataset>
docs/source/getting_started/configuration.rst 0 → 100644
+ 77
− 0
View file @ 23c3fc9f
Sen2Chain
==========
Installation
--------------
Download Sen2Chain and install the python dependencies
.. code-block:: bash
git clone https://framagit.org/espace-dev/sen2chain.git
cd sen2chain
pip3 install --user -e .
Or install directly from Git :
.. code-block:: bash
python3 -m pip install git+https://framagit.org/espace-dev/sen2chain.git
Once installed, import Sen2Chain in a python console to initialize the package :
.. code-block:: python
from sen2chain import Tile
exit()
Update Sen2Chain
-----------------
| Updates are regularly made on the processing chain, and can be easily retrieved with Git.
| Go to the install directory, pull the last updates from Framagit, and if necessary reinstall Sen2Chain (to get new requirement libraries).
.. code-block:: bash
cd sen2chain
git pull
pip3 install --user -e .
Sen2Chain Configuration
-----------------------
After the first launch of Sen2Chain, default folders are created in ``~/sen2chain_data/data``
You can change the settings by editing the configuration file: ``nano ~/sen2chain_data/config/config.cfg``
.. code-block:: bash
# Directory paths that will receive the different products from the processing steps of Sen2Chain.
[DATA PATHS]
temp_path = /default/path/sen2chain_data/data/TEMP
l1c_path = /default/path/sen2chain_data/data/L1C
l1c_archive_path = /default/path/sen2chain_data/data/L1C_ARCHIVE
l2a_path = /default/path/sen2chain_data/data/L2A
l2a_archive_path = /default/path/sen2chain_data/data/L2A_ARCHIVE
indices_path = /default/path/sen2chain_data/data/INDICES
time_series_path = /default/path/sen2chain_data/data/TIME_SERIES
temporal_summaries_path = /default/path/sen2chain_data/data/TEMPORAL_SUMMARIES
cloudmasks_path = /default/path/sen2chain_data/data/CLOUDMASKS
extraction_path = /default/path/sen2chain_data/data/EXTRACTION
# Path to L2A_Bashrc file of your Sen2Cor installation
[SEN2COR PATH]
sen2cor_bashrc_path = /default/path/Sen2Cor-02.10.01-Linux64/L2A_Bashrc
sen2cor_alternative_bashrc_path = /default/path/older_sen2cor/L2A_Bashrc
# Path for Jobs routine logs
[LOG PATH]
log_path = /default/path/sen2chain_data/logs
**Sen2Chain is now ready to run!**
docs/source/getting_started/eodag.rst 0 → 100644
+ 43
− 0
View file @ 23c3fc9f
EODAG
======
| `EODAG <https://eodag.readthedocs.io/en/stable/>`_ is a command-line tool and a Python package for searching and downloading remotely sensed images while offering a unified API for data access regardless of the data provider.
| As of June 2023, Sen2chain only supports EODAG v2.8.0. A specific plugin was released for the Scihub provider. Both packages can be installed with pip.
.. code-block:: bash
python -m pip install eodag==2.8.0
python3 -m pip install eodag-sentinelsat
Then, import eodag in a python console to initialize the package :
.. code-block:: python
from eodag import EODataAccessGateway
exit()
Hubs logins & EODAG configuration file
=======================================
Register to Scihub, PEPS and any provider you deem relevant for your work, and get the required credentials (username/password).
| Follow the guidelines provided in `EODAG's documentation <https://eodag.readthedocs.io/en/stable/getting_started_guide/configure.html>`_ to configure the plugin.
| You can create a dedicated configuration file or edit the default one:
.. code-block:: bash
~/.config/eodag/eodag.yml
.. code-block:: bash
scihub:
priority: 1 # Highest number for your preferred provider, see more in eodag documentation
api:
credentials:
username: "PLEASE_CHANGE_ME" # Your own username
password: "PLEASE_CHANGE_ME" # Your own password
docs/source/getting_started/index.rst 0 → 100644
+ 16
− 0
View file @ 23c3fc9f
===========================
Getting started
===========================
Why sen2chain, how to install, configure it, go!
.. toctree::
:maxdepth: 2
:caption: Contents:
my_install
eodag
sen2cor
configuration
docs/source/getting_started/my_install.rst 0 → 100644
+ 53
− 0
View file @ 23c3fc9f
Prerequisites
===============
Sen2Chain requires pip, GDAL, Git and Sen2Cor
pip
-----
.. code-block:: bash
sudo apt install python3-pip
GDAL
-----
.. code-block:: bash
sudo apt install gdal-bin libgdal-dev python3-gdal
Git
----
.. code-block:: bash
sudo apt-get install git
Sen2cor 2.11
----------------
1. Download Sen2Cor v2.11.00
.. code-block:: bash
wget https://step.esa.int/thirdparties/sen2cor/2.10.0/Sen2Cor-02.11.00-Linux64.run
2. Make it executable
.. code-block:: bash
chmod +x Sen2Cor-02.11.00-Linux64.run
3. Install Sen2Cor
.. code-block:: bash
./Sen2Cor-02.11.00-Linux64.run
.. note::
Installation guide and requirements for previous versions of Sen2Chain and Sen2Cor are available :ref:`here <configure_sen2cor>`. Previous versions are needed to process imagery before 2018.
| You can add extra data input to Sen2Cor in order to improve some features of Sen2Chain.
| These functions are not directly mobilized by Sen2Chain but for optimal operation you can add ESA landcover data and the SRTM. Please find the process at this :ref:`page <sen2cor>`.
docs/source/getting_started/sen2cor.rst 0 → 100644
+ 45
− 0
View file @ 23c3fc9f
.. _sen2cor:
Sen2Cor Configuration
----------------------------------
.. note::
If you want to process products from 2017 and before, you need to install previous versions of Sen2Cor and follow the guidelines :ref:`here <configure_sen2cor>`.
This section focuses on Sen2Cor 2.11 configuration.
1. Move or create symlink to `ESACCI-LC package for SC module <https://earth.esa.int/eogateway/ftp/Sentinel-2/ESACCI-LC-L4-ALL-FOR-SEN2COR-2.10.tar.gz>`_ content to Sen2Cor directory:
.. code-block:: bash
ln -s /path/to/ESACCI-LC-L4-ALL-FOR-SEN2COR-2.10/ESACCI-LC-L4-LCCS-Map-300m-P1Y-2015-v2.0.7.tif /path/to/Sen2Cor-02.10.01-Linux64/lib/python2.7/site-packages/sen2cor/aux_data
ln -s /path/to/ESACCI-LC-L4-WB-Map-150m-P13Y-2000-v4.0.tif /path/to/Sen2Cor-02.10.01-Linux64/lib/python2.7/site-packages/sen2cor/aux_data/
ln -s /path/to/ESACCI-LC-L4-Snow-Cond-500m-MONTHLY-2000-2012-v2.4 /path/to/Sen2Cor-02.10.01-Linux64/lib/python2.7/site-packages/sen2cor/aux_data/
2. Create or symlink DEM directory
.. code-block:: bash
ln -s /path/to/dem ~/sen2cor/2.11/dem
3. Update Sen2Cor configuration file ``nano ~/sen2cor/2.11/cfg/L2A_GIPP.xml``
.. code-block:: html
<DEM_Directory>dem/srtm</DEM_Directory>
<DEM_Reference>http://srtm.csi.cgiar.org/wp-content/uploads/files/srtm_5x5/TIFF/</DEM_Reference>
<DEM_Terrain_Correction>FALSE</DEM_Terrain_Correction> # If you don't want DEM terrain correction
4. Update Sen2Chain config file ``nano ~/sen2chain_data/config/config.cfg``
.. code-block:: bash
sen2cor_bashrc_paths = /path/to/Sen2Cor-02.11.01-Linux64/L2A_Bashrc
5. Then update Sen2Chain by using any method :
.. code-block:: python
from sen2chain import Tile
Tile('40KCB').info
docs/source/going_further/Jobs.md 0 → 100644
+ 41
− 0
View file @ 23c3fc9f
# Setting up recurrent routines : Jobs
Sen2Chain uses jobs to execute whole processing operations (downloading L1C, computing L2A with Sen2Cor, masking clouds, and producing indices).
Jobs config files are stored in your \~/sen2chain_data/config/jobs/.
There is two kind of files for each job :
- job_jid.cfg
- job_jid.py
With _jid_ the job identifier, can be any character chain.
```python
>>> from sen2chain import Jobs, Job
```
## Jobs
The _Jobs_ class is used to list all jobs present for your user.
```python
>>> Jobs()
job_id config_file python_script logging timing cron_status cron_timing
0 0123456789 True True False 0 5 * * * absent None
1 335 True True False 10 10 * * * absent None
2 012 True False False * * * * * absent None
3 013 True False False * * * * * absent None
```
Jobs can be removed using the _remove_ function and their _job_id_ identifier
```python
>>> Jobs().remove("335")
10094:2022-03-17 17:03:35:INFO:sen2chain.jobs:Removing Python script...
10094:2022-03-17 17:03:35:INFO:sen2chain.jobs:Removing config file...
```
## Job
to be filled
docs/source/going_further/Jobs.rst 0 → 100644
+ 163
− 0
View file @ 23c3fc9f
====================================
Setting up recurrent routines : Jobs
====================================
Sen2Chain uses jobs to execute whole processing operations (downloading L1C, computing L2A with Sen2Cor, masking clouds, and producing indices) on any tile.
All of Sen2Chain's functions parameters can be specified, allowing for the production of multiple products in one go.
Jobs can be launched once or at scheduled hours using crontab.
Jobs config files are stored in your ``~/sen2chain_data/config/jobs/``.
Each job is made of 2 files :
- job_jid.cfg to configure the job
- job_jid.py automatically created once job is configured
With *jid* the job identifier.
Job listing
***********
The *Jobs* class is used to list all jobs created in your Sen2Chain install.
.. code-block::
>>> from sen2chain import Jobs
>>> Jobs()
job_id config_file python_script logging timing cron_status cron_timing
0 0123456789 True True False 0 5 * * * absent None
1 335 True True False 10 10 * * * absent None
2 012 True False False * * * * * absent None
3 tes True False False * * * * * absent None
```
Jobs can be removed using the *remove* function and their *jid* identifier
.. code-block::
>>> Jobs().remove("335")
10094:2022-03-17 17:03:35:INFO:sen2chain.jobs:Removing Python script...
10094:2022-03-17 17:03:35:INFO:sen2chain.jobs:Removing config file...
Job
***
Create a new Job
-----------------
To create a new Job or select an existing one use the command line ``Job(jid="jid")``
.. code-block:: python
from sen2chain import Job
j=Job(jid="test")
This command creates a configuration file in ``~/sen2chain_data/config/jobs/`` :
.. code-block:: bash
logs = True
timing = 0 20 * * *
provider = peps
tries = 2
sleep = 4
nb_proc = 18
copy_L2A_sideproducts = False
clean_before = True
clean_after = True
tile;date_min;date_max;max_clouds;l1c;l2a;cloudmasks;indices;remove;comments
40KCB;;today;80;True;True;CM004-CSH1-CMP1-CHP1-TCI1-ITER1;NDVI/NDWIGAO/MNDWI/NDRE/IRECI/BIGR/BIRNIR/BIBG/EVI/NBR;l1c/l2a;
The configuration file first section is a list of global parameters for the job execution :
- **logs**: True | False
- **timing**: Recurrence of the job when added to cron, in cron format
- **provider**: Provider to download L1C products from, Default peps, values: peps | scihub
- **tries**: Number of tries the download should loop before stopping, to download OFFLINE products
- **sleep**: Time in min to wait between download tries
- **nb_proc**: Number of cpu cores to use for this job, Default 8
- **copy_L2A_side_products**: Duplicate *msk_cldprb_20m* and *scl_20m* from L2A folder to Cloudmask folder after L2A production. Interesting if you plan to remove L2A to save disk space, but want to keep these 2 files for cloudmask generation and better extraction. Possible values: True | False
- **clean_before / clean_after**: Clean corrupted files in Tile folder, before and/or after job execution : True | False
The second section of the configuration file is a list of tasks that will be processed on a single Tile when the Job is executed :
- **tile**: tile identifier, format --XXX, comment line using ! before tile name
- **date_min**: Start date for this task, possible values: empty (2015-01-01 will be used) | any date | today-xx (xx nb of days before today to consider)
- **date_max**: Last date for this task, possible values: empty (9999-12-31 will be used) | any date | today
- **max_clouds**: Max cloud cover to consider for downloading images
- **l1c**: Download l1c: True|False
- **l2a**: Compute L2A with sen2chain: True | False
- **cloudmasks**: Cloudmask(s) to compute and mask indice(s). Possible values range from none (False) to multiple cloudmasks: False | CM001/CM002/CM003-PRB1-ITER5/CM004-CSH1-CMP1-CHP1-TCI1-ITER0/etc.
- **indices**: False | All | NDVI/NDWIGAO/etc.
- **remove**: Remove downloaded L1C and/or produced L2A when task is done, possible values: False | l1c | l2a | l1c/l2a
- **comments**: free user comments, ie tile name, etc.
Configure Job
---------------
To configure a job with a large number of tasks on different tiles, we recommend manually editing the configuration file ``nano ~/sen2chain_data/config/jobs/job_jid.cfg``
Make sure to keep the same table structure. A job can also be configured with command lines.
Add a task to a job config file with the *task_add()* method :
.. code-block:: python
>>>from sen2chain import Job
>>>j=Job(jid="jid")
INFO:sen2chain.jobs:Reading existing config...
>>>j.task_add()
Edit a task with *task_edit(task_id, **kwargs)* :
.. code-block:: python
>>> j.task_edit(task_id=0,tile='40KEC',remove='l1C')
Remove a task with *task_remove(task_id)*
.. code-block:: python
>>> j.task_remove(task_id=0)
Save and Launch a Job
---------------------
Save the config file to your local database. If the job you created is not saved, you will not be able to load it next time.
.. code-block:: python
j.save()
To launch a job directly from the command line :
.. code-block:: python
j.run()
Job in cron
-----------
You can add, deactivate and delete a job in cron with *cron_enable()*, *cron_disable()* and *cron_remove()*.
Cron will run the job at the frequency specified by the cron parameter (minute - hour - day - month - weekday) in ``job_jid.cfg``.
.. code-block:: python
j.save()
j.cron_enable()
j.cron_disable()
j.cron_remove()
docs/source/going_further/Library.md 0 → 100644
+ 100
− 0
View file @ 23c3fc9f
# Working on multiple tiles : Library
The _Library_ class offers the possibility to get information and perform some actions on all or a group of tiles.
Initialization:
```python
>>> from sen2chain import Library
>>> l = Library()
```
## Database information and management
### Information
- This function returns tiles in the L1C library folder
```python
>>> l.l1c
```
- This function returns tiles in the L2A library folder
```python
>>> l.l2a
```
- This function returns tiles in the indices library folder
```python
>>> l.indices
```
### Cleaning
The _clean_ function performs the _clean_lib_ function (See Tile Section/link) for each tile present in L1C database or in the provided list. Use the _remove = True_ parameter to effectively remove products (default value False)
```python
>>> l.clean() # All L1C database, nothing removed
>>> l.clean(clean_list = ["40KCB", "38KND"]) # 2 tiles analysed, nothing removed
>>> l.clean(clean_list = ["38KQE"], remove = True) # 1 tile analysed, error SAFE folders removed
```
## Computing products
### Computing L2A
You can compute L2A products for multiple provided tiles using multiprocessing
```python
>>> # Compute all missing L2A products for the 2 provided tiles,
>>> # after the specified date,
>>> # and using 6 CPU cores
>>> l.compute_l2a(tile_list = ["40KCB", "38KND"],
date_min = "2020-01-01",
nb_proc = 6)
```
### Computing cloudmasks
You can compute cloudmasks products for multiple provided tiles using multiprocessing, with specific parameters (see Tile for description).
```python
>>> l.compute_cloudmasks(tile_list = ["40KCB", "38KND"],
cm_version = "cm001",
probability = 1,
iterations = 5,
reprocess = False,
date_min = None,
date_max = None,
nb_proc = 4)
```
### Computing indices
You can compute index products for multiple provided tiles using multiprocessing, with specific parameters (see Tile for description).
```python
>>> l.compute_indices(tile_list = ["40KCB"],
indice_list = ["NDVI", "NDWIGAO"],
reprocess = False,
nodata_clouds = True,
cm_version = "cm001",
probability = 1,
iterations = 5,
date_min = None,
date_max = "2021-12-31",
nb_proc = 4)
```
### Computing L1C and L2A quicklooks
You can compute quicklooks for multiple provided tiles for L1C and/or L2A products. If no tile is provided, whole L1C + L2A product database is used. You can set specific output QL resolution (default 750m/px) and specific output format (JPEG by default or TIFF).
```python
>>> l.compute_ql(tile_list = ["40KCB"],
product_list = ["l1c", "l2a"],
resolution = 750,
jpg = True)
```
docs/source/going_further/Tile.md 0 → 100644
+ 214
− 0
View file @ 23c3fc9f
# Working a single tile : Tile
The use of the Tile object is a good way to access tile database, process l2a, cloudmasks and indices.
First define a Tile object using its identifier (2 digits + 3 letters format)
```python
>>> from sen2chain import Tile
>>> t = Tile("40KCB")
```
## Database information and management
### Information
You can get informations on the number of files in database, downloaded l1c, and produced l2a, cloudmasks and indices using the _info_ function.
```python
>>> t.info
43780:2021-10-06 12:53:34:INFO:sen2chain.tiles:l1c: 344
43780:2021-10-06 12:53:34:INFO:sen2chain.tiles:l2a: 345
43780:2021-10-06 12:53:34:INFO:sen2chain.tiles:cloud_masks: 625
43780:2021-10-06 12:53:34:INFO:sen2chain.tiles:ndwigao (raw / masked): 345 / 345
43780:2021-10-06 12:53:34:INFO:sen2chain.tiles:bigr (raw / masked): 345 / 349
43780:2021-10-06 12:53:34:INFO:sen2chain.tiles:birnir (raw / masked): 215 / 211
43780:2021-10-06 12:53:34:INFO:sen2chain.tiles:ireci (raw / masked): 2 / 2
43780:2021-10-06 12:53:34:INFO:sen2chain.tiles:ndwimcf (raw / masked): 345 / 345
43780:2021-10-06 12:53:34:INFO:sen2chain.tiles:bibg (raw / masked): 211 / 211
43780:2021-10-06 12:53:34:INFO:sen2chain.tiles:ndvi (raw / masked): 345 / 352
43780:2021-10-06 12:53:34:INFO:sen2chain.tiles:ndre (raw / masked): 11 / 4
43780:2021-10-06 12:53:34:INFO:sen2chain.tiles:mndwi (raw / masked): 54 / 57
```
### Size
The _size_ function retrieves the used disk space for each kind of product. Execution can take time, depending on database size, disk or network speed. Local and archived disk spaces are analysed.
```python
>>> t.size
43780:2021-10-06 12:53:41:INFO:sen2chain.tiles:l1c: 198GB (local: 44GB / archived: 154GB)
43780:2021-10-06 12:53:45:INFO:sen2chain.tiles:l2a: 258GB (local: 65GB / archived: 192GB)
43780:2021-10-06 12:53:45:INFO:sen2chain.tiles:ndwigao: 25GB
43780:2021-10-06 12:53:45:INFO:sen2chain.tiles:bigr: 24GB
43780:2021-10-06 12:53:45:INFO:sen2chain.tiles:birnir: 15GB
43780:2021-10-06 12:53:45:INFO:sen2chain.tiles:ireci: 324MB
43780:2021-10-06 12:53:45:INFO:sen2chain.tiles:ndwimcf: 25GB
43780:2021-10-06 12:53:45:INFO:sen2chain.tiles:bibg: 15GB
43780:2021-10-06 12:53:45:INFO:sen2chain.tiles:ndvi: 25GB
43780:2021-10-06 12:53:45:INFO:sen2chain.tiles:ndre: 791MB
43780:2021-10-06 12:53:45:INFO:sen2chain.tiles:mndwi: 4GB
```
## Listing products
Different functions are called to retrieve a list all database products :
```python
>>> t.l1c # list downloaded l1c indice products
>>> t.l2a # list l2a indice products
>>> t.cloudmasks # list cloudmask products
>>> t.ndvi # list ndvi indice products
>>> t.ndwigao # list ndwi Gao indice products
>>> t.mndwi # list mndwi indice products
```
As a ProductList class this retrieved list of products can be reduced using multiples functions :
```python
>>> t.l2a.products # list of product names
>>> t.l2a.dates # list of product dates
>>> t.l2a.first # the first (oldest) product in database
>>> t.l2a.last # the last (newest) product in database
>>> t.l2a.filter_dates(date_min = "yyyy-mm-dd", date_max = "yyyy-mm-dd") # list of products between date_min and date_max
>>> t.l2a.filter_dates(date_min = "yyyy-mm-dd") # list of products after date_min
>>> t.l2a.filter_dates(date_max = "yyyy-mm-dd") # list of products before date_max
>>> t.l2a.filter_clouds(cover_min = 0, cover_max = 100) # list of products with a cloud cover between cover_min and cover_max
>>> t.l2a.filter_clouds(cover_min = 20) # list of products with more than 20% cloud cover
>>> t.l2a.filter_clouds(cover_max = 50) # list of products with less than 50% cloud cover
```
Cloudmask and indice lists as CloudMaskList class objects can be reduced using 4 different functions, one for each type of cloud mask :
```python
>>> t.cloudmasks.cm001 # list cloudmask products of type CM001
>>> t.cloudmasks.cm002 # list cloudmask products of type CM002
>>> t.cloudmasks.cm003 # list cloudmask products of type CM003
>>> t.cloudmasks.cm004 # list cloudmask products of type CM004
```
For cloudmasks of types CM003 and CM004 the _params_ function is used to select cloudmasks with specific parameters. See specific wiki page for cloudmasks implementation within Sen2Chain (**_to do_**). Default values are :
- **probability**: _1_, the max pixel cloud cover % probability value (0-100) above which the pixel should be considered as cloud. Used only for CM003.
- **iterations**: _5_, the number of dilatation cycles to apply on thresholded cloudmasks. Used for CM003 & CM004
- **cld_shad**: _True_, used with CM004 only. Boolean value for masking cloud shadows from 20m Sen2Chain Scene Classification.
- **cld_med_prob**: _True_, used with CM004. Boolean value for masking clouds with medium probability from 20m Sen2Chain Scene Classification.
- **cld_hi_prob**: _True_, used with CM004. Boolean value for masking clouds with high probability from 20m Sen2Chain Scene Classification.
- **thin_cir**: _True_, used with CM004. Boolean value for masking thin cirrus clouds from 20m Sen2Chain Scene Classification.
```python
>>> t.cloudmasks.cm003.params() # list cloudmask products of type CM003 with default values
>>> t.cloudmasks.cm003.params(probability = 25) # list cloudmask products of type CM003 with pixel cloud probability of 25%
>>> t.cloudmasks.cm004.params() # list cloudmask products of type CM004 with default values
>>> t.cloudmasks.cm004.params(cld_shad = False) # list cloudmask products of type CM004, without considering cloud shadows
```
Indice lists can be filtered using one of the 3 defined functions: _raws_, _masks_, and _quicklooks_
```python
>>> t.ndvi.raws #list only non masked ndvi indices
>>> t.ndwigao.masks #list only masked ndwi Gao indices (by any mask)
>>> t.ndwigao.masks.cm001 #list only ndwi Gao indices masked by CM001 cloudmasks
>>> t.mndwi.quicklooks #list only mndwi indice quicklooks
```
## Cleaning corrupted products
Data in image database can sometimes be corrupted during their production, while downloading and unzipping l1c products, computing l2a with Sen2Cor (frequent corruptions with old 2.5.5 Sen2Cor version), cloudmasks or indices. The _clean_lib_ function has been introduced to check for data integrity, with the possibility to remove corrupted products (whole SAFE folder). Removed products have to be downloaded / computed again.
```python
>>> t.clean_lib() # Check corrupted products (remove nothing)
{'problems': [], 'identified_problems': 0, 'removed_problems': 0}
>>> t.clean_lib(remove = True) # Check and remove corrupted products
```
## Archiving products
Archive functions offer the possibility to move L1C, L2A or both products (SAFE folders) to archive paths, specified in the configuration file (<span dir="">\~</span>/sen2chain_data/config/config.cfg). This function can be useful in case of different kind of available hardware disks (speed vs size).\
The _size_only_ option offers the possibility to get the info, whithout moving anything (default value _False_).
- L1C products are moved from `l1c_path` to `l1c_archive_path`
- L2A products are moved from `l2a_path` to `l2a_archive_path`
Every moved SAFE folder is replaced by its symlink in the original product folder. L2A SAFE products can be moved only if there is no file error (_clean_lib_ function automatically performed first).
```python
>>> t.archive_l1c() # Archive L1C products
>>> t.archive_l2a(size_only = True) # Checking L2A products ready to archive (nothing moved)
>>> t.archive_all() # Chaining L1C and L2A archive functions
```
## Removing products
Two functions offers the possibility to remove the entire L1C or L2A products from database. For L2A a provided list of identifiers offers the possibility to remove specific L2A products, if nothing provided whole L2A DB is removed.
**!!! To use with care !!!**
```python
>>> t.remove_l1c()
>>> t.remove_l2a(identifier_list = list)
```
## Computing products
Three functions are used to compute the different products from downloaded raw L1C: L2A with Sen2Cor, cloudmasks and indices. For all 3, the process is the same: entire database is scanned to identify a list of missing products for which the computing will be done. In case of no missing products, nothing is done. Each function has its own set of specific parameters (see details bellow), with these in common:
- _date_min_ and _date_max_: the date after, date before, or time range between which the missing products will be computed. Default values _None_ results in the entire database being scanned.
- _reprocess (boolean)_: possibility to reprocess already computed products. Default value _False_.
- _nb_proc_: All products in the process list are computed using multiprocessing to reduce processing time. Default value is 4, to increase according to your hardware. Limited by the number of cores of your processor.
### Computing L2A
This function is used to compute missing L2A for L1C products present in the local database. Possibility to specify dates or reprocessing, see parameters above.
```python
>>> Tile("40KCB").compute_l2a()
>>> Tile("40KCB").compute_l2a(date_min="2019-01-01", nb_proc=12)
43780:2021-10-06 13:05:17:INFO:sen2chain.tiles:All l2a products already computed
```
### Computing cloudmasks
This function computes missing cloudmasks for L2A products present in local database. This stage is mandatory if you want to mask indices later. In addition to those already mentioned above, many parameters are available, depending on the cloudmask version to compute:
- **cm_version**: _"cm001"_, _"cm002"_, _"cm003"_, or _"cm004"_. _Default value = "cm001"_. See specific wiki page for cloudmasks implementation within Sen2Chain (_to do_).
- **probability**: _DV = 1_, the max pixel cloud cover % probability value (0-100) above which the pixel should be considered as cloud. Used only for CM003.
- **iterations**: _DV = 5_, the number of dilatation cycles to apply on thresholded cloudmasks. Used for CM003 & CM004
- **cld_shad**: _DV =_ _True_, used with CM004 only. Boolean value for masking cloud shadows from 20m Sen2Chain Scene Classification.
- **cld_med_prob**: _DV = True_, used with CM004. Boolean value for masking clouds with medium probability from 20m Sen2Chain Scene Classification.
- **cld_hi_prob**: _DV = True_, used with CM004. Boolean value for masking clouds with high probability from 20m Sen2Chain Scene Classification.
- **thin_cir**: _DV =_ _True_, used with CM004. Boolean value for masking thin cirrus clouds from 20m Sen2Chain Scene Classification.
The output cloudmask is named according to cloudmask version and the set of parameters, and produced only if not already present.
```python
>>> Tile("40KCB").compute_cloudmasks()
>>> Tile("40KCB").compute_cloudmasks(cm_version="cm004",
iterations=2,
cld_shad=False,
thin_cir=False,
nb_proc=8)
```
### Computing indices
This function computes all missing indices for l2a products. Indices are given as a list. If indices are not provided, it will compute missing dates of already existing indices for this tile (no new indice computed). Indices won't be masked if no cloudmasks are present, you have to compute cloudmasks first. Specific parameters:
- **nodata_clouds**: Default value _True_, to produce a second set of cloudmasked index images (cloudmasks have to be be present). If not specified CM001 will be used. For CM003 & CM004, other CM specific parameters can be adjusted.
```python
>>> Tile("40KCB").compute_indices(["NDVI"], date_max="2020-12-31" )
>>> Tile("40KCB").compute_indices(["NDVI"], cm_version="cm002", nb_proc=2)
```
### Computing quicklooks
You can compute quicklooks for multiple provided tiles for L1C and/or L2A products. If no tile provided, whole L1C + L2A product database is used. Possibility to set specific output QL resolution (default 750m/px). Possibility to set output format JPEG (default) or TIFF.
```python
>>> Tile("40KCB").compute_ql(product_list = ["l2a"],
resolution = 100,
jpg = False)
```
docs/source/going_further/Tile_test.rst 0 → 100644
+ 227
− 0
View file @ 23c3fc9f
=============================
Working a single tile : Tile
=============================
The use of the Tile object is a good way to access tile database, process l2a, cloudmasks and indices.
First define a Tile object using its identifier (2 digits + 3 letters format)
.. code-block:: python
from sen2chain import Tile
t = Tile("40KCB")
From Downloading L1C to computing radiometric indices
======================================================
Downloading L1C Product
-----------------------
Computing products
------------------
Three functions are used to compute the different products from downloaded raw L1C: L2A with Sen2Cor, cloudmasks and indices.
For all 3, the process is the same: entire database is scanned to identify a list of missing products for which the computing will be done.
In case of no missing products, nothing is done. Each function has its own set of specific parameters (see details bellow), with these in common:
- _date_min_ and _date_max_: the date after, date before, or time range between which the missing products will be computed. Default values _None_ results in the entire database being scanned.
- _reprocess (boolean)_: possibility to reprocess already computed products. Default value _False_.
- _nb_proc_: All products in the process list are computed using multiprocessing to reduce processing time. Default value is 4, to increase according to your hardware. Limited by the number of cores of your processor.
### Computing L2A
This function is used to compute missing L2A for L1C products present in the local database. Possibility to specify dates or reprocessing, see parameters above.
```python
>>> Tile("40KCB").compute_l2a()
>>> Tile("40KCB").compute_l2a(date_min="2019-01-01", nb_proc=12)
43780:2021-10-06 13:05:17:INFO:sen2chain.tiles:All l2a products already computed
```
### Computing cloudmasks
This function computes missing cloudmasks for L2A products present in local database. This stage is mandatory if you want to mask indices later. In addition to those already mentioned above, many parameters are available, depending on the cloudmask version to compute:
- **cm_version**: _"cm001"_, _"cm002"_, _"cm003"_, or _"cm004"_. _Default value = "cm001"_. See specific wiki page for cloudmasks implementation within Sen2Chain (_to do_).
- **probability**: _DV = 1_, the max pixel cloud cover % probability value (0-100) above which the pixel should be considered as cloud. Used only for CM003.
- **iterations**: _DV = 5_, the number of dilatation cycles to apply on thresholded cloudmasks. Used for CM003 & CM004
- **cld_shad**: _DV =_ _True_, used with CM004 only. Boolean value for masking cloud shadows from 20m Sen2Chain Scene Classification.
- **cld_med_prob**: _DV = True_, used with CM004. Boolean value for masking clouds with medium probability from 20m Sen2Chain Scene Classification.
- **cld_hi_prob**: _DV = True_, used with CM004. Boolean value for masking clouds with high probability from 20m Sen2Chain Scene Classification.
- **thin_cir**: _DV =_ _True_, used with CM004. Boolean value for masking thin cirrus clouds from 20m Sen2Chain Scene Classification.
The output cloudmask is named according to cloudmask version and the set of parameters, and produced only if not already present.
```python
>>> Tile("40KCB").compute_cloudmasks()
>>> Tile("40KCB").compute_cloudmasks(cm_version="cm004",
iterations=2,
cld_shad=False,
thin_cir=False,
nb_proc=8)
```
### Computing indices
This function computes all missing indices for l2a products. Indices are given as a list. If indices are not provided, it will compute missing dates of already existing indices for this tile (no new indice computed). Indices won't be masked if no cloudmasks are present, you have to compute cloudmasks first. Specific parameters:
- **nodata_clouds**: Default value _True_, to produce a second set of cloudmasked index images (cloudmasks have to be be present). If not specified CM001 will be used. For CM003 & CM004, other CM specific parameters can be adjusted.
```python
>>> Tile("40KCB").compute_indices(["NDVI"], date_max="2020-12-31" )
>>> Tile("40KCB").compute_indices(["NDVI"], cm_version="cm002", nb_proc=2)
```
### Computing quicklooks
You can compute quicklooks for multiple provided tiles for L1C and/or L2A products. If no tile provided, whole L1C + L2A product database is used. Possibility to set specific output QL resolution (default 750m/px). Possibility to set output format JPEG (default) or TIFF.
```python
>>> Tile("40KCB").compute_ql(product_list = ["l2a"],
resolution = 100,
jpg = False)
```
## Database information and management
### Information
You can get informations on the number of files in database, downloaded l1c, and produced l2a, cloudmasks and indices using the _info_ function.
```python
>>> t.info
43780:2021-10-06 12:53:34:INFO:sen2chain.tiles:l1c: 344
43780:2021-10-06 12:53:34:INFO:sen2chain.tiles:l2a: 345
43780:2021-10-06 12:53:34:INFO:sen2chain.tiles:cloud_masks: 625
43780:2021-10-06 12:53:34:INFO:sen2chain.tiles:ndwigao (raw / masked): 345 / 345
43780:2021-10-06 12:53:34:INFO:sen2chain.tiles:bigr (raw / masked): 345 / 349
43780:2021-10-06 12:53:34:INFO:sen2chain.tiles:birnir (raw / masked): 215 / 211
43780:2021-10-06 12:53:34:INFO:sen2chain.tiles:ireci (raw / masked): 2 / 2
43780:2021-10-06 12:53:34:INFO:sen2chain.tiles:ndwimcf (raw / masked): 345 / 345
43780:2021-10-06 12:53:34:INFO:sen2chain.tiles:bibg (raw / masked): 211 / 211
43780:2021-10-06 12:53:34:INFO:sen2chain.tiles:ndvi (raw / masked): 345 / 352
43780:2021-10-06 12:53:34:INFO:sen2chain.tiles:ndre (raw / masked): 11 / 4
43780:2021-10-06 12:53:34:INFO:sen2chain.tiles:mndwi (raw / masked): 54 / 57
```
### Size
The _size_ function retrieves the used disk space for each kind of product. Execution can take time, depending on database size, disk or network speed. Local and archived disk spaces are analysed.
```python
>>> t.size
43780:2021-10-06 12:53:41:INFO:sen2chain.tiles:l1c: 198GB (local: 44GB / archived: 154GB)
43780:2021-10-06 12:53:45:INFO:sen2chain.tiles:l2a: 258GB (local: 65GB / archived: 192GB)
43780:2021-10-06 12:53:45:INFO:sen2chain.tiles:ndwigao: 25GB
43780:2021-10-06 12:53:45:INFO:sen2chain.tiles:bigr: 24GB
43780:2021-10-06 12:53:45:INFO:sen2chain.tiles:birnir: 15GB
43780:2021-10-06 12:53:45:INFO:sen2chain.tiles:ireci: 324MB
43780:2021-10-06 12:53:45:INFO:sen2chain.tiles:ndwimcf: 25GB
43780:2021-10-06 12:53:45:INFO:sen2chain.tiles:bibg: 15GB
43780:2021-10-06 12:53:45:INFO:sen2chain.tiles:ndvi: 25GB
43780:2021-10-06 12:53:45:INFO:sen2chain.tiles:ndre: 791MB
43780:2021-10-06 12:53:45:INFO:sen2chain.tiles:mndwi: 4GB
```
## Listing products
Different functions are called to retrieve a list all database products :
```python
>>> t.l1c # list downloaded l1c indice products
>>> t.l2a # list l2a indice products
>>> t.cloudmasks # list cloudmask products
>>> t.ndvi # list ndvi indice products
>>> t.ndwigao # list ndwi Gao indice products
>>> t.mndwi # list mndwi indice products
```
As a ProductList class this retrieved list of products can be reduced using multiples functions :
```python
>>> t.l2a.products # list of product names
>>> t.l2a.dates # list of product dates
>>> t.l2a.first # the first (oldest) product in database
>>> t.l2a.last # the last (newest) product in database
>>> t.l2a.filter_dates(date_min = "yyyy-mm-dd", date_max = "yyyy-mm-dd") # list of products between date_min and date_max
>>> t.l2a.filter_dates(date_min = "yyyy-mm-dd") # list of products after date_min
>>> t.l2a.filter_dates(date_max = "yyyy-mm-dd") # list of products before date_max
>>> t.l2a.filter_clouds(cover_min = 0, cover_max = 100) # list of products with a cloud cover between cover_min and cover_max
>>> t.l2a.filter_clouds(cover_min = 20) # list of products with more than 20% cloud cover
>>> t.l2a.filter_clouds(cover_max = 50) # list of products with less than 50% cloud cover
```
Cloudmask and indice lists as CloudMaskList class objects can be reduced using 4 different functions, one for each type of cloud mask :
```python
>>> t.cloudmasks.cm001 # list cloudmask products of type CM001
>>> t.cloudmasks.cm002 # list cloudmask products of type CM002
>>> t.cloudmasks.cm003 # list cloudmask products of type CM003
>>> t.cloudmasks.cm004 # list cloudmask products of type CM004
```
For cloudmasks of types CM003 and CM004 the _params_ function is used to select cloudmasks with specific parameters. See specific wiki page for cloudmasks implementation within Sen2Chain (**_to do_**). Default values are :
- **probability**: _1_, the max pixel cloud cover % probability value (0-100) above which the pixel should be considered as cloud. Used only for CM003.
- **iterations**: _5_, the number of dilatation cycles to apply on thresholded cloudmasks. Used for CM003 & CM004
- **cld_shad**: _True_, used with CM004 only. Boolean value for masking cloud shadows from 20m Sen2Chain Scene Classification.
- **cld_med_prob**: _True_, used with CM004. Boolean value for masking clouds with medium probability from 20m Sen2Chain Scene Classification.
- **cld_hi_prob**: _True_, used with CM004. Boolean value for masking clouds with high probability from 20m Sen2Chain Scene Classification.
- **thin_cir**: _True_, used with CM004. Boolean value for masking thin cirrus clouds from 20m Sen2Chain Scene Classification.
```python
>>> t.cloudmasks.cm003.params() # list cloudmask products of type CM003 with default values
>>> t.cloudmasks.cm003.params(probability = 25) # list cloudmask products of type CM003 with pixel cloud probability of 25%
>>> t.cloudmasks.cm004.params() # list cloudmask products of type CM004 with default values
>>> t.cloudmasks.cm004.params(cld_shad = False) # list cloudmask products of type CM004, without considering cloud shadows
```
Indice lists can be filtered using one of the 3 defined functions: _raws_, _masks_, and _quicklooks_
```python
>>> t.ndvi.raws #list only non masked ndvi indices
>>> t.ndwigao.masks #list only masked ndwi Gao indices (by any mask)
>>> t.ndwigao.masks.cm001 #list only ndwi Gao indices masked by CM001 cloudmasks
>>> t.mndwi.quicklooks #list only mndwi indice quicklooks
```
## Cleaning corrupted products
Data in image database can sometimes be corrupted during their production, while downloading and unzipping l1c products, computing l2a with Sen2Cor (frequent corruptions with old 2.5.5 Sen2Cor version), cloudmasks or indices. The _clean_lib_ function has been introduced to check for data integrity, with the possibility to remove corrupted products (whole SAFE folder). Removed products have to be downloaded / computed again.
```python
>>> t.clean_lib() # Check corrupted products (remove nothing)
{'problems': [], 'identified_problems': 0, 'removed_problems': 0}
>>> t.clean_lib(remove = True) # Check and remove corrupted products
```
## Archiving products
Archive functions offer the possibility to move L1C, L2A or both products (SAFE folders) to archive paths, specified in the configuration file (<span dir="">\~</span>/sen2chain_data/config/config.cfg). This function can be useful in case of different kind of available hardware disks (speed vs size).\
The _size_only_ option offers the possibility to get the info, whithout moving anything (default value _False_).
- L1C products are moved from `l1c_path` to `l1c_archive_path`
- L2A products are moved from `l2a_path` to `l2a_archive_path`
Every moved SAFE folder is replaced by its symlink in the original product folder. L2A SAFE products can be moved only if there is no file error (_clean_lib_ function automatically performed first).
```python
>>> t.archive_l1c() # Archive L1C products
>>> t.archive_l2a(size_only = True) # Checking L2A products ready to archive (nothing moved)
>>> t.archive_all() # Chaining L1C and L2A archive functions
```
## Removing products
Two functions offers the possibility to remove the entire L1C or L2A products from database. For L2A a provided list of identifiers offers the possibility to remove specific L2A products, if nothing provided whole L2A DB is removed.
**!!! To use with care !!!**
```python
>>> t.remove_l1c()
>>> t.remove_l2a(identifier_list = list)
```
0% or .
You are about to add 0 people to the discussion. Proceed with caution.
Finish editing this message first!
Please register or to comment