Skip to content
Snippets Groups Projects
Commit d3720624 authored by TGermain's avatar TGermain
Browse files

quick update on rtd build

parent 453a44b4
No related branches found
No related tags found
No related merge requests found
Hide whitespace changes
Inline Side-by-side
Showing
with 136 additions and 293 deletions
rtd/source/first_steps/download_data.md deleted 100644 → 0
+ 0
− 41
View file @ 453a44b4
# Downloading data
### By tile's number
To use this script you need to fill in the file [download_tiles.csv](https://framagit.org/espace-dev/sen2chain/tree/master/examples/scripts/download_tiles.csv). **_dire aussi que ce script se trouve quelques part dans l'installation (examples/script)_**
In this .csv, the only required parameter is the tile number. The other parameters are optional. If there is no start date the download will start on the earliest date, if there is no end date the download will end at the most recent acquisition. You can also filter the download with the maxclouds parameter of the Sentinel-2 images. The last two parameters are simply some informations on the localisation of the tile.
**_INdiquer ici que les données sont mises par défuat là et sinon à l'endroit indiqué dans le fichier de configuration - est-ce qu'on laisse l1c archives etc.?_**
For example if you write in the .csv file:
```
tile;start_time;end_time;max_clouds;site;nom
40KCB;2019-01-01;;;France;Reunion
```
And run download_tiles.py, you will download all the images on the Reunion Island since the 1st of January 2019.
### By tile's identifier
To use this script you need to fill in the file [download_tiles_by_identifier.csv](https://framagit.org/espace-dev/sen2chain/tree/master/examples/scripts/download_tiles_by_identifier.csv).
If you know exactly the data you need to download you can fill the .csv with the official identifier name of the images and then run the download_tiles_by_identifier.py. For example if you write in the .csv :
```
pre1_pre2_starttime_pre3_pre4_tile_pre5_endtime_maxclouds_site_nom
# add below your identifier list
S2A_MSIL1C_20190221T063511_N0207_R134_T40KCB_20190221T075913
```
And then run :
```
python3 download_tiles.py
```
You will download the image of the 21th february of 2019 on the Reunion Island.
You could list in the .csv all the identifiers of the images you need to download.
rtd/source/first_steps/index.rst deleted 100644 → 0
+ 0
− 22
View file @ 453a44b4
===========================
First Steps with sen2chain
===========================
Sen2Chain is a processing chain that allows you to download, process and generate indice time series from Sentinel-2 satellite data. Below are some scripts developed to facilitate the use of Sen2Chain for beginners.
These scripts work like the following :
* a config file (.csv or .txt) allows the user to configure his query/processing
* a python script (ex: download.py) run by the user in the python command prompt.
With this first chapter you'll learn how to use the pre-built scripts in example folder
.. toctree::
:maxdepth: 2
:caption: Contents:
download_data
process_data
rtd/source/first_steps/process_data.md deleted 100644 → 0
+ 0
− 81
View file @ 453a44b4
# Processing sentinel products
### With process_l2a_and_products.py
This script processes the L1C files that have have been downloaded but have not yet been processed. In the script you can change different parameters to define the tiles number you want to process and the indices you want.
_**Indiquer peut-être où on trouve le script et ce qu'on peut faire varier.**_
The available indices are :
* NDVI
* NDWIGAO
* NDWIMCF
* MNDWI
* BIGR
* BIRNIR
* BIBG
_(A FAIRE : You can see the formulas of these indices on the product description page /link/ et page à créer)_
For example, if you want to process the L2A products and the MNDWI on the Reunion Island you must edit the python script like so:
In \[29\]:
```
l2a.process_indices(["MNDWI"], True, True)
```
In \[35\]:
```
tiles = ["40KCB"]
```
Run the script in a console:
```
python3 process_l2a_and_products.py
```
### With multiprocessing
#### By tile's number
The script allows to process L1C to L2A and to generate several products. This script starts parallel processing on several cores. By default, 4 images are processed in L2A in parallel and it's the same for product calculation.
The configuration file of this script is [process_l2a_and_indices_multiprocessing.csv](https://framagit.org/espace-dev/sen2chain/tree/master/examples/scripts/process_l2a_and_indices_multiprocessing.csv). You must fill in the number of tiles you want to download and the products you want to generate. You could also fill in the start and end date.
For example to download and process products on all the images on the Reunion Island since the beginning of 2019:
```
tile;start_time;end_time;max_clouds;site;nom;indices
# max_clouds parameter not implemented yet
40KCB;2019-01-01;;;France;Reunion;NDVI/NDWIGAO/NDWIMCF/MNDWI/BIGR/BIRNIR/BIBG
```
And then launch into a terminal:
```
python3 process_l2a_and_indices_multiprocessing.py
```
#### By tile's identifier
To use this script you need to fill in the [process_l2a_and_indices_by_identifier_multiprocessing.csv](https://framagit.org/espace-dev/sen2chain/-/blob/master/examples/scripts/process_l2a_and_indices_by_identifier_multiprocessing.csv)
If you know exactly the data you need, you can fill the .csv with the official identifier name of the images and then run the download_tiles_by_identifier.py. For example if you write in the .csv :
```
pre1_pre2_starttime_pre3_pre4_tile_pre5_endtime_maxclouds_site_nom
\# add below your identifier list
S2A_MSIL1C_20190221T063511_N0207_R134_T40KCB_20190221T075913
```
And then run :
```
python3 process_l2a_and_indices_by_identifier_multiprocessing.py
```
You will download the image of the 21st february of 2019 on the Reunion Island.
rtd/source/getting_started/eodag.rst
+ 3
− 2
View file @ d3720624
......@@ -3,11 +3,12 @@ EODAG
| `EODAG <https://eodag.readthedocs.io/en/stable/>`_ is a command-line tool and a Python package for searching and downloading remotely sensed images while offering a unified API for data access regardless of the data provider.
| A specific plugin was released for the Scihub provider. Both can be installed with pip.
| As of June 2023, Sen2chain only supports EODAG v2.8.0. A specific plugin was released for the Scihub provider. Both packages can be installed with pip.
.. code-block:: bash
pip install eodag==2.8.0
python3 -m pip install eodag-sentinelsat
......@@ -24,7 +25,7 @@ Register to Scihub, PEPS and any provider you deem relevant for your work, and g
~/.config/eodag/eodag.yml
.. code-block::
.. code-block:: bash
scihub:
priority: 1 # Highest number for your preferred provider, see more in eodag documentation
......
rtd/source/index.rst
+ 11
− 14
View file @ d3720624
......@@ -9,35 +9,33 @@ Welcome to sen2chain's documentation!
Currently under development. -thomasG
|
Sen2Chain is a simple utility written in Python 3 to download and process Sentinel-2 images.
It uses the sentinelsat and peps_download packages to find and download data, and ESA's Sen2Cor processor to perform atmospheric, terrain and cirrus correction.
It uses EODAG package to find and download data, and ESA's Sen2Cor processor to perform atmospheric, terrain and cirrus correction.
Main features:
- downloading of L1C (top of atmosphere) products from Scihub and PEPS mirrors;
- processing of L1C to L2A (bottom of atmosphere) products;
- computing of radiometric indices;
- time series extraction of thoses indices;
- full automatization of the above features.
- Downloading of L1C (top of atmosphere) products from EODAG available providers;
- Processing of L1C to L2A (bottom of atmosphere) products;
- Computing of radiometric indices;
- Time series extraction of thoses indices;
- Full automatization of the above features.
All these features can be launched in multiprocessing.
Contributing
************
Scientific projects :
TOSCA S2-Malaria project, funded by CNES (TOSCA 2017-2020);
INTERREG Renovrisk-impact project (2018-2020).
Development and improvment :
Development and improvments :
Jérémy Commins ;
Pascal Mouquet ;
Charlotte Wolff.
Jérémy Commins, Pascal Mouquet, Charlotte Wolff.
Conceptualization and Coordination :
Vincent Herbreteau ;
Christophe Révillion.
Vincent Herbreteau, Christophe Révillion.
|
|
Content
--------
......@@ -47,7 +45,6 @@ Content
getting_started/index
product_user_guide/index
first_steps/index
going_further/index
package_content
......
rtd/source/my_install.md deleted 100644 → 0
+ 0
− 131
View file @ 453a44b4
# Installation
## Prerequisites
This package requires pip3, Python 3, GDAL and Sen2Cor 2.10.01.
Note I: The current version of sen2chain is not available on Windows (yet)
Note II: Installation guide and requirements for previous versions of Sen2Chain and Sen2Cor are available here (_**link to another page of wiki**_). This may be needed to process imagery from 2021 and before.
**<span dir="">GDAL</span>**
```
sudo apt install gdal-bin libgdal-dev python3-gdal
```
**<span dir="">Sen2Cor 2.10.01</span>**
<span dir="">Download Sen2Cor v2.10.01</span>
```
wget https://step.esa.int/thirdparties/sen2cor/2.10.0/Sen2Cor-02.10.01-Linux64.run
```
Make it executable
```
chmod +x Sen2Cor-02.10.01-Linux64.run
```
Install Sen2Cor
```
./Sen2Cor-02.10.01-Linux64.run
```
It's possible to add some data to improve some features of sen2chain. These functions are not directly mobilized by sen2chain but for optimal operation you can add ESA landcover data and the SRTM. Please find the process at this page (_**page à faire avec les différentes fonctions de sen2chain**_)
**EODAG**
EODDAG is a command-line tool and a Python package for searching and downloading remotely sensed images while offering a unified API for data access regardless of the data provider. A specific plugin was released for the Scihub provider. Both can be installed with pip.
Install pip first
```
sudo apt install python3-pip
```
Then EODAG
```
python3 -m pip install eodag-sentinelsat
```
## <span dir="">Install sen2chain from sources</span>
Install Git
```
sudo apt-get install git
```
Download Sen2Chain and install the python dependencies
```
git clone https://framagit.org/espace-dev/sen2chain.git
cd sen2chain
pip3 install --user -e .
```
**Or** install directly from Git:
```
python3 -m pip install git+https://framagit.org/espace-dev/sen2chain.git
```
# <span dir="">Updating sen2chain</span>
<span dir="">Updates are regularly made on the processing chain, it can be easily retrieved with Git. Go to the install directory, pull the last updates from Framagit, and if necessary reinstall Sen2Chain (to get new requirement libraries).</span>
```
cd sen2chain
git pull
pip3 install --user -e .
```
# <span dir="">Configuration</span>
<span dir="">The different configuration files are located in ~/sen2chain_data. This files are only created after the first sen2chain import in python. To initialize the config files you can lauch:</span>
```
python3
from sen2chain import Tile
exit()
```
### <span dir="">Configure Sentinel-2 images library</span>
<span dir="">After the first launch of sen2chain, default folders are created in:
```
~/sen2chain_data/data
```
<span dir="">You can change the settings by editing the configuration file:</span>
```
~/sen2chain_data/data/config.cfg
```
Here you can configure all the paths that will receive the different data from the processing steps of sen2chain. In this file indicate the full path to the bashrc of your sen2cor installation to process L1C to L2A _**(ajouter ici la possibilité de mettre un autre sen2cor pour les données pré 2018)**_:
```
/path/to/Sen2Cor-02.10.01-Linux64/L2A_Bashrc
```
### <span dir="">Hubs logins</span> & EODAG configuration file
Register to Scihub, PEPS and any provider you deem relevant for your work, and get the required credentials (username/password).
Follow the guidelines provided in [EODAG's documentation](https://eodag.readthedocs.io/en/latest/getting_started_guide/configure.html) to configure the plugin. You can create a dedicated configuration file or edit the default one:
```
~/.config/eodag/eodag.yml
```
Then
```
scihub:
priority: 1 # Highest number for your preferred provider, see more in eodag documentation
api:
credentials:
username: "PLEASE_CHANGE_ME" # Your own username
password: "PLEASE_CHANGE_ME" # Your own password
```
**<span dir="">Sen2Chain is now ready to run!</span>**
\ No newline at end of file
rtd/source/product_user_guide/Cloudmasks.rst 0 → 100644
+ 58
− 0
View file @ d3720624
======================
Cloud Masks
======================
| Sen2chain offers multiple algorythms for cloud masking :
CloudMask 1
***********
Sen2chain historical cloud mask method is mask CM001. This product is directly extracted from the cloud probability mask output of Sen2cor, at 20m resolution.
This layer is a Quality indicator that calculates the probability(0-100%) that the earth surface is obstructed by clouds or optically thick aerosols like ice or snow.
Then, a binary image is generated by applying a threshold of cloud porbbaility over 25%.
Then, two mathematical morphology operations are successively performed on the data: a binary erosion of a 3x3 pixel square kernel is followed by a binary dilatation of a 11x11 pixel rounded shape kernel.
This double operation, called opening, removes objects smaller than the structuring element, and can be used together with closing for noise removal.
CloudMask 2
***********
Sen2chain second mask is CM002, an evolution of CM001 that allows in some specific cases to reduce false cloud detection linked to unusually high snow probabilities in the CLDPRB layer of Sen2Cor.
These high probability values can be found on high reflectance rivers or in beach areas (specular surface reflections, high turbidity, white sand beaches...)
To get rid of these artefacts the SWIR spectral band, centered around 1600nm and relatively insensitive to these anomalies, is used.
SWIR Band B11 is transformed in a mask in 2 steps : First a binary image is generated with a reflectance threshold of 15%, then a binary dilatation filter of a 11x11 pixel round kernel is applied.
Then, the newly computed Cloud Mask CM002 is applied on the CM001, followed by a dilatation on a round kernel of 13x13 pixels.
The cloudmask CM002 is less sensitive on the areas mentionned before, but also slightly more dilated than CM001, allowing for a better interpretation of clouds with ambiguous boundaries.
CloudMask 3
***********
Sen2chain's third mask, CM003, is based on the same data source as CM001 (cloud probability layer MSK_CLDPRB at 20m spatial resolution), but adds 2 optional parameters :
- The cloud probability value for thresholding (0-100%)
- The number of iterations of the binary dilatation kernel (A 5x5 pixel rounded shape, without prior erosion like in CM001 case)
CloudMask 4
***********
| Sen2chain 4th and latest cloud mask uses the Scene Classification (SC) with 20m spatial resolution band of Sen2cor.
| The Sen2Cor algorithm used to produce the SC notably detects clouds and their shadows and generates a classification map, which consists of 3 different classes for clouds (including cirrus), as well
as six different classifications for shadows, cloud shadows, vegetation, soils/deserts, water and snow.
| A total of 6 parameters can be specificed by the user, allowing for a better precision of the mask and specific use cases :
Like CM003:
- The cloud probability value for thresholding (0-100%)
- The number of iterations of the binary dilatation kernel (A 5x5 pixel rounded shape, without prior erosion)
4 Classes from the SC layer used to generate the final mask:
- CLOUD_SHADOWS
- CLOUD_MEDIUM_PROBABILITY
- CLOUD_HIGH_PROBABILITY
- THIN_CIRRUS
To compute CloudMasks from any L2A product, call for compute_cloudmasks() Tile's method
.. code-block:: python
Tile('40KCB').compute_cloudmasks(cm_version="CM004", iterations=5, ...)
rtd/source/product_user_guide/Indices.rst 0 → 100644
+ 40
− 0
View file @ d3720624
==============
Indices
==============
Here are the radiometric indices available ofr computing in Sen2chain. Indices can me masked by any of the 4 cloudmasks.
.. list-table::
:widths: 50 50
:header-rows: 1
* - Name
- Formula
* - NDVI
-
* - NDWIMCF
-
* - NDWIGAO
-
* - MNDWI
-
* - NDRE
-
* - IRECI
-
* - BIGR
-
* - BIRNIR
-
* - BIBG
-
* - NBR
-
* - EVI
-
To compute, call for compute_indices() Tile's method. Specify the indices and the Cloud mask version you want to mask them with, including the optional parameters of the cloudmask version (only 1 at a time)
.. code-block:: python
Tile('40KCB').compute_indices(indices=['NDVI','BIBG','MNDWI'],cm_version='CM001', ...)
rtd/source/product_user_guide/L2A.rst 0 → 100644
+ 23
− 0
View file @ d3720624
=====================================
Products
=====================================
L1C Product
-----------
Quick description of L1C
L2A Products
------------
| Sen2chain uses ESA's Sen2cor processing chain to process Top-Of-Atmosphere Level 1C input data into L2A Surface Reflectance products.
| Sen2Cor creates Bottom-Of-Atmosphere, optionally terrain- and cirrus corrected reflectance images; and additionally, Aerosol Optical Thickness-, Water Vapor-, Scene Classification Map and Quality Indicators for cloud and snow probabilities.
| Its output product format is equivalent to the Level 1C User Product: JPEG 2000 images, with three different resolutions, 60, 20 and 10 m.
To process L1C products into L2A, call for compute_l2a() Tile's method
.. code-block:: python
Tile('40KCB').compute_l2a(...)
rtd/source/product_user_guide/index.rst
+ 1
− 2
View file @ d3720624
......@@ -3,8 +3,7 @@ Product User Guide
========================
.. toctree::
L1C
L2A
Cloudmasks
Indices
Time Series
TimeSeries
0% or .
You are about to add 0 people to the discussion. Proceed with caution.
Finish editing this message first!
Please register or to comment