Skip to content
Snippets Groups Projects
Commit 17147d1c authored by Christophe R.'s avatar Christophe R.
Browse files

Merge branch 'maj_doc8' into 'master'

Some documentation changes and syntax corrections

See merge request espace-dev/sen2chain!71
parents 95023e16 d8f0d43a
No related branches found
No related tags found
No related merge requests found
Pipeline #39617 passed
......@@ -3,20 +3,20 @@ EODAG
| `EODAG <https://eodag.readthedocs.io/en/stable/>`_ is a command-line tool and a Python package for searching and downloading remotely sensed images while offering a unified API for data access regardless of the data provider.
| As of June 2023, Sen2chain only supports EODAG v2.10.0. A specific plugin was released for the Scihub provider. Both packages can be installed with pip.
| The EODAG package can be installed with pip.
.. code-block:: bash
python -m pip install eodag
~$ python -m pip install eodag
Then, import eodag in a python console to initialize the package :
.. code-block:: python
from eodag import EODataAccessGateway
dag=EODataAccessGateway()
exit()
>>> from eodag import EODataAccessGateway
>>> dag=EODataAccessGateway()
>>> exit()
Hubs logins & EODAG configuration file
......@@ -29,12 +29,12 @@ Hubs logins & EODAG configuration file
.. code-block:: bash
~/.config/eodag/eodag.yml
~$ ~/.config/eodag/eodag.yml
.. code-block:: bash
PEPS :
~$ PEPS :
priority: 1 # Highest number for your preferred provider, see more in eodag documentation
api:
credentials:
......
......@@ -12,6 +12,7 @@ Why sen2chain, how to install, configure it, go!
my_install
venv_python
eodag
sen2cor
configuration
sen2cor
......@@ -7,42 +7,42 @@ pip
-----
.. code-block:: bash
sudo apt install python3-pip
~$ sudo apt install python3-pip
GDAL
-----
.. code-block:: bash
sudo apt install gdal-bin libgdal-dev python3-gdal
~$ sudo apt install gdal-bin libgdal-dev python3-gdal
Git
----
.. code-block:: bash
sudo apt-get install git
~$ sudo apt-get install git
Sen2cor 2.11
Sen2cor 2.12.03
----------------
1. Download Sen2Cor v2.12.03
.. code-block:: bash
wget https://step.esa.int/thirdparties/sen2cor/2.12.03/Sen2Cor-02.12.03-Linux64.run
~$ wget https://step.esa.int/thirdparties/sen2cor/2.12.0/Sen2Cor-02.12.03-Linux64.run
2. Make it executable
.. code-block:: bash
chmod +x Sen2Cor-02.12.03-Linux64.run
~$ chmod +x Sen2Cor-02.12.03-Linux64.run
3. Install Sen2Cor
.. code-block:: bash
./Sen2Cor-02.12.03-Linux64.run
~$ ./Sen2Cor-02.12.03-Linux64.run
.. note::
......
......@@ -14,15 +14,15 @@ This section focuses on Sen2Cor 2.12 configuration.
.. code-block:: bash
ln -s /path/to/ESACCI-LC-L4-ALL-FOR-SEN2COR-2.10/ESACCI-LC-L4-LCCS-Map-300m-P1Y-2015-v2.0.7.tif /path/to/Sen2Cor-02.12.03-Linux64/lib/python2.7/site-packages/sen2cor/aux_data
ln -s /path/to/ESACCI-LC-L4-ALL-FOR-SEN2COR-2.10/ESACCI-LC-L4-WB-Map-150m-P13Y-2000-v4.0.tif /path/to/Sen2Cor-02.12.03-Linux64/lib/python2.7/site-packages/sen2cor/aux_data/
ln -s /path/to/ESACCI-LC-L4-ALL-FOR-SEN2COR-2.10/ESACCI-LC-L4-Snow-Cond-500m-MONTHLY-2000-2012-v2.4 /path/to/Sen2Cor-02.12.03-Linux64/lib/python2.7/site-packages/sen2cor/aux_data/
~$ ln -s /path/to/ESACCI-LC-L4-ALL-FOR-SEN2COR-2.10/ESACCI-LC-L4-LCCS-Map-300m-P1Y-2015-v2.0.7.tif /path/to/Sen2Cor-02.12.03-Linux64/lib/python2.7/site-packages/sen2cor/aux_data/
~$ ln -s /path/to/ESACCI-LC-L4-ALL-FOR-SEN2COR-2.10/ESACCI-LC-L4-WB-Map-150m-P13Y-2000-v4.0.tif /path/to/Sen2Cor-02.12.03-Linux64/lib/python2.7/site-packages/sen2cor/aux_data/
~$ ln -s /path/to/ESACCI-LC-L4-ALL-FOR-SEN2COR-2.10/ESACCI-LC-L4-Snow-Cond-500m-MONTHLY-2000-2012-v2.4 /path/to/Sen2Cor-02.12.03-Linux64/lib/python2.7/site-packages/sen2cor/aux_data/
2. Create or symlink DEM directory
.. code-block:: bash
ln -s /path/to/dem ~/sen2cor/2.12/dem
~$ ln -s /path/to/dem ~/sen2cor/2.12/dem
3. Update Sen2Cor configuration file ``nano ~/sen2cor/2.11/cfg/L2A_GIPP.xml``
......@@ -36,12 +36,12 @@ This section focuses on Sen2Cor 2.12 configuration.
.. code-block:: bash
sen2cor_bashrc_paths = /path/to/Sen2Cor-02.12.03-Linux64/L2A_Bashrc
~$ sen2cor_bashrc_paths = /path/to/Sen2Cor-02.12.03-Linux64/L2A_Bashrc
5. Then update Sen2Chain by using any method :
.. code-block:: python
from sen2chain import Tile
Tile('40KCB').info
>>> from sen2chain import Tile
>>> Tile('40KCB').info
Setting up the virtual environement
======
| It's best to use virtualization to install Sen2Chain properly. There are different virtual environments for Python. Here we present the procedure for venv.
| Since the latest versions of Ubuntu (> 23.04), it is highly recommended to use a virtual environment for Python to avoid conflicts and prevent breaking your system. So, it's best to use virtualization to install Sen2Chain properly. There are different virtual environments for Python. Here we present the procedure for venv.
| First you need to install venv Python :
.. code-block:: bash
sudo apt install python-3-venv
~$ sudo apt install python3-venv
| Then, create your venv workspace where Sen2Chain will be installed :
| Then create a venv workspace (called here venv_sen2chain) where Sen2Chain will be installed :
.. code-block:: bash
python3 -m venv --system-site-package ~/.venv_sen2chain
~$ python3 -m venv --system-site-package ~/.venv_sen2chain
| The --system-site-package is important because it allows you to retrieve glbally installed packages such as GDAL. The command will create a folder containing the Python packages for this virtual machine.
| The --system-site-package is important because it allows you to retrieve globally installed packages such as GDAL. The command will create a folder containing the Python packages for this virtual environment.
| Next, activate the new environment (the basic name of the start prompt will change):
| Next, activate the new environment (the base name of command prompt will change):
.. code-block:: bash
source ~/.venv_sen2chain/bin/activate
~$ source ~/.venv_sen2chain/bin/activate
| This virtual environment can then be deactivated as follows
.. code-block:: bash
deactivate
~$ deactivate
| Once the environment has been activated, the following python packages can be installed.
......@@ -16,7 +16,7 @@ Each job is made of 2 files :
- job_jid.cfg to configure the job
- job_jid.py automatically created once job is configured
With **jid** the job identifier.
With **jid** the job identifier who could be any combination of number or character string.
Job config file contains global job parameters and tasks.
......@@ -59,8 +59,9 @@ To create a new Job or select an existing one use the command line ``Job(jid="ji
.. code-block:: python
from sen2chain import Job
j=Job(jid="test")
>>> from sen2chain import Job
>>> j=Job(jid="test")
>>> j.save()
This command creates a configuration file in ``~/sen2chain_data/config/jobs/`` :
......@@ -82,30 +83,35 @@ This command creates a configuration file in ``~/sen2chain_data/config/jobs/`` :
The configuration file first section is a list of global parameters for the job execution :
- **logs**: True | False
- **timing**: Recurrence of the job when added to cron, in cron format
- **provider**: Provider to download L1C products from, Default peps, values: peps | scihub
- **tries**: Number of tries the download should loop before stopping, to download OFFLINE products
- **sleep**: Time in min to wait between download tries
- **nb_proc**: Number of cpu cores to use for this job, Default 8
- **copy_L2A_side_products**: Duplicate *msk_cldprb_20m* and *scl_20m* from L2A folder to Cloudmask folder after L2A production. Interesting if you plan to remove L2A to save disk space, but want to keep these 2 files for cloudmask generation and better extraction. Possible values: True | False
- **clean_before / clean_after**: Clean corrupted files in Tile folder, before and/or after job execution : True | False
- # logs: True | False
- # timing: in cron format
- # tries: the number of times the download should loop before stopping, to download OFFLINE products
- # sleep: the time in min to wait between loops
- # nb_proc: the number of cpu cores to use for this job, default 8
- # copy_l2a_side_products: to duplicate msk_cldprb_20m and scl_20m from l2a folder to cloudmask folder after l2a production.
- # Interesting if you plan to remove l2a to save disk space, but want to keep these 2 files for cloudmask generation and better extraction
- # Possible values: True | False
- # clean_before: set to False or True to call a Libray.clean(remove=True) on selected tiles
- # before starting the job, default True
- # clean_before: set to False or True call a Libray.clean(remove=True) on selected tiles
- # after finishing the job, default True
- # process_by_line: split job execution into n lines for processing of large datasets
The second section of the configuration file is a list of tasks that will be processed on a single Tile when the Job is executed :
- **tile**: tile identifier, format --XXX, comment line using ! before tile name
- **date_min**: Start date for this task, possible values: empty (2015-01-01 will be used) | any date | today-xx (xx nb of days before today to consider)
- **date_max**: Last date for this task, possible values: empty (9999-12-31 will be used) | any date | today
- **max_clouds**: Max cloud cover to consider for downloading images
- **provider: the provider where to get the data (l1c/l2a), possible values: empty (peps will be used) | peps | cop_dataspace
- **pb_min the minimum processing baseline for this task, possible values: empty (0 will be used) | any positive float number
- **pb_max the maximum processing baseline for this task, possible values: empty (98 will be used) | any positive float number
- **download: product type to download: l1c|l2a|False
- **compute_l2a: computing l2a using sen2chain / sen2cor: True | False
- **cloudmasks: the cloudmask(s) to compute and use to mask indice(s). Possible values range from none (False) to multiple cloudmasks: empty or False | CM001/CM002/CM003-PRB1-ITER5/CM004-CSH1-CMP1-CHP1-TCI1-ITER0/etc.
- **indices: empty or False | All | NDVI/NDWIGAO/etc.
- **remove: used to remove L1C and L2A products, considering only new products (-new) (dowloaded or produced) or the whole time serie (-all) and with possible filtering to remove products above specified cloud couver (-ccXX), example of possible values: empty or False | l1c-new-cc80 | l1c-all | l2a-new | l1c-all/l2a-new-cc25
- **comments: free user comments, ie tile name, etc.
- # tile: tile identifier, format ##XXX, comment line using ! before tile name
- # date_min the start date for this task, possible values: empty (2015-01-01 will be used) | any date | today-xx (xx nb of days before today to consider)
- # date_max the last date for this task, possible values: empty (9999-12-31 will be used) | any date | today
- # max_clouds: max cloud cover to consider for downloading images, computing l2a, cloudmask and indice products
- # provider: the provider where to get the data (l1c/l2a), possible values: empty (peps will be used) | peps | cop_dataspace
- # pb_min the minimum processing baseline for this task, possible values: empty (0 will be used) | any positive float number
- # pb_max the maximum processing baseline for this task, possible values: empty (98 will be used) | any positive float number
- # download: product type to download: l1c|l2a|False
- # compute_l2a: computing l2a using sen2chain / sen2cor: True | False
- # cloudmasks: the cloudmask(s) to compute and use to mask indice(s). Possible values range from none (False) to multiple cloudmasks: empty or False | CM001/CM002/CM003-PRB1-ITER5/CM004-CSH1-CMP1-CHP1-TCI1-ITER0/etc.
- # indices: empty or False | All | NDVI/NDWIGAO/etc.
- # remove: used to remove L1C and L2A products, considering only new products (-new) (dowloaded or produced) or the whole time serie (-all) and with possible filtering to remove products above specified cloud couver (-ccXX) example of possible values: empty or False | l1c-new-cc80 | l1c-all | l2a-new | l1c-all/l2a-new-cc25
- # comments: free user comments, ie tile name, etc.
Configure Job
---------------
......@@ -120,10 +126,10 @@ Add a task to a job config file with the *task_add()* method :
.. code-block:: python
>>>from sen2chain import Job
>>>j=Job(jid="jid")
>>> from sen2chain import Job
>>> j=Job(jid="jid")
INFO:sen2chain.jobs:Reading existing config...
>>>j.task_add()
>>> j.task_add()
Edit a task with *task_edit(task_id, **kwargs)* :
......@@ -144,13 +150,13 @@ Save the config file to your local database. If the job you created is not saved
.. code-block:: python
j.save()
>>> j.save()
To launch a job directly from the command line :
.. code-block:: python
j.run()
>>> j.run()
Job in cron
......@@ -162,10 +168,10 @@ Cron will run the job at the frequency specified by the cron parameter (minute -
.. code-block:: python
j.save()
j.cron_enable()
j.cron_disable()
j.cron_remove()
>>> j.save()
>>> j.cron_enable()
>>> j.cron_disable()
>>> j.cron_remove()
# Working on multiple tiles : Library
Working on multiple tiles : Library
====================================
The _Library_ class offers the possibility to get information and perform some actions on all or a group of tiles.
Initialization:
```python
>>> from sen2chain import Library
>>> l = Library()
```
.. code-block:: python
>>> from sen2chain import Library
>>> l = Library()
## Database information and management
......@@ -15,31 +16,31 @@ Initialization:
- This function returns tiles in the L1C library folder
```python
>>> l.l1c
```
.. code-block:: python
>> l.tiles_l1c
- This function returns tiles in the L2A library folder
```python
>>> l.l2a
```
.. code-block:: python
>>> l.tiles_l2a
- This function returns tiles in the indices library folder
```python
>>> l.indices
```
.. code-block:: python
>>> l.indices
### Cleaning
The _clean_ function performs the _clean_lib_ function (See Tile Section/link) for each tile present in L1C database or in the provided list. Use the _remove = True_ parameter to effectively remove products (default value False)
```python
>>> l.clean() # All L1C database, nothing removed
>>> l.clean(clean_list = ["40KCB", "38KND"]) # 2 tiles analysed, nothing removed
>>> l.clean(clean_list = ["38KQE"], remove = True) # 1 tile analysed, error SAFE folders removed
```
.. code-block:: python
>>> l.clean() # All L1C database, nothing removed
>>> l.clean(clean_list = ["40KCB", "38KND"]) # 2 tiles analysed, nothing removed
>>> l.clean(clean_list = ["38KQE"], remove = True) # 1 tile analysed, error SAFE folders removed
## Computing products
......@@ -47,21 +48,22 @@ The _clean_ function performs the _clean_lib_ function (See Tile Section/link) f
You can compute L2A products for multiple provided tiles using multiprocessing
```python
>>> # Compute all missing L2A products for the 2 provided tiles,
>>> # after the specified date,
>>> # and using 6 CPU cores
>>> l.compute_l2a(tile_list = ["40KCB", "38KND"],
.. code-block:: python
>>> # Compute all missing L2A products for the 2 provided tiles,
>>> # after the specified date,
>>> # and using 6 CPU cores
>>> l.compute_l2a(tile_list = ["40KCB", "38KND"],
date_min = "2020-01-01",
nb_proc = 6)
```
### Computing cloudmasks
You can compute cloudmasks products for multiple provided tiles using multiprocessing, with specific parameters (see Tile for description).
```python
>>> l.compute_cloudmasks(tile_list = ["40KCB", "38KND"],
.. code-block:: python
>>> l.compute_cloudmasks(tile_list = ["40KCB", "38KND"],
cm_version = "cm001",
probability = 1,
iterations = 5,
......@@ -69,14 +71,14 @@ You can compute cloudmasks products for multiple provided tiles using multiproce
date_min = None,
date_max = None,
nb_proc = 4)
```
### Computing indices
You can compute index products for multiple provided tiles using multiprocessing, with specific parameters (see Tile for description).
```python
>>> l.compute_indices(tile_list = ["40KCB"],
.. code-block:: python
>>> l.compute_indices(tile_list = ["40KCB"],
indice_list = ["NDVI", "NDWIGAO"],
reprocess = False,
nodata_clouds = True,
......@@ -86,15 +88,14 @@ You can compute index products for multiple provided tiles using multiprocessing
date_min = None,
date_max = "2021-12-31",
nb_proc = 4)
```
### Computing L1C and L2A quicklooks
You can compute quicklooks for multiple provided tiles for L1C and/or L2A products. If no tile is provided, whole L1C + L2A product database is used. You can set specific output QL resolution (default 750m/px) and specific output format (JPEG by default or TIFF).
```python
>>> l.compute_ql(tile_list = ["40KCB"],
.. code-block:: python
>>> l.compute_ql(tile_list = ["40KCB"],
product_list = ["l1c", "l2a"],
resolution = 750,
jpg = True)
```
......@@ -8,8 +8,8 @@ First define a Tile object using its identifier (2 digits + 3 letters format)
.. code-block:: python
from sen2chain import Tile
t = Tile("40KCB")
>>> from sen2chain import Tile
>>> t = Tile("40KCB")
From Downloading L1C to computing radiometric indices
======================================================
......@@ -31,7 +31,7 @@ In the following case, the line of code will launch the download of the 40KCB ti
.. code-block:: python
Tile("40KCB").get_l1c(provider='peps', start='2022-01-01', max_cloudcover=80)
>>> Tile("40KCB").get_l1c(provider='peps', start='2022-01-01', max_cloudcover=80)
Computing products
......@@ -50,8 +50,8 @@ This function is used to compute missing L2A for L1C products present in the loc
.. code-block:: python
Tile("40KCB").compute_l2a()
Tile("40KCB").compute_l2a(date_min="2019-01-01", nb_proc=12)
>>> Tile("40KCB").compute_l2a()
>>> Tile("40KCB").compute_l2a(date_min="2019-01-01", nb_proc=12)
### Computing cloudmasks
......@@ -69,8 +69,8 @@ The output cloudmask is named according to cloudmask version and the set of para
.. code-block:: python
Tile("40KCB").compute_cloudmasks()
Tile("40KCB").compute_cloudmasks(cm_version="cm004", iterations=2, cld_shad=False, thin_cir=False, nb_proc=8)
>>> Tile("40KCB").compute_cloudmasks()
>>> Tile("40KCB").compute_cloudmasks(cm_version="cm004", iterations=2, cld_shad=False, thin_cir=False, nb_proc=8)
### Computing indices
......@@ -80,8 +80,8 @@ This function computes all missing indices for l2a products. Indices are given a
.. code-block:: python
Tile("40KCB").compute_indices(["NDVI"], date_max="2020-12-31" )
Tile("40KCB").compute_indices(["NDVI"], cm_version="cm002", nb_proc=2)
>>> Tile("40KCB").compute_indices(["NDVI"], date_max="2020-12-31" )
>>> Tile("40KCB").compute_indices(["NDVI"], cm_version="cm002", nb_proc=2)
### Computing quicklooks
......@@ -89,7 +89,7 @@ You can compute quicklooks for multiple provided tiles for L1C and/or L2A produc
.. code-block:: python
Tile("40KCB").compute_ql(product_list = ["l2a"],
>>> Tile("40KCB").compute_ql(product_list = ["l2a"],
resolution = 100,
jpg = False)
......@@ -101,7 +101,7 @@ You can get informations on the number of files in database, downloaded l1c, and
.. code-block:: python
t.info
>>> t.info
43780:2021-10-06 12:53:34:INFO:sen2chain.tiles:l1c: 344
43780:2021-10-06 12:53:34:INFO:sen2chain.tiles:l2a: 345
43780:2021-10-06 12:53:34:INFO:sen2chain.tiles:cloud_masks: 625
......@@ -121,7 +121,7 @@ The _size_ function retrieves the used disk space for each kind of product. Exec
.. code-block:: python
t.size
>>> t.size
43780:2021-10-06 12:53:41:INFO:sen2chain.tiles:l1c: 198GB (local: 44GB / archived: 154GB)
43780:2021-10-06 12:53:45:INFO:sen2chain.tiles:l2a: 258GB (local: 65GB / archived: 192GB)
43780:2021-10-06 12:53:45:INFO:sen2chain.tiles:ndwigao: 25GB
......@@ -172,10 +172,10 @@ Cloudmask and indice lists as CloudMaskList class objects can be reduced using 4
.. code-block:: python
t.cloudmasks.cm001 # list cloudmask products of type CM001
t.cloudmasks.cm002 # list cloudmask products of type CM002
t.cloudmasks.cm003 # list cloudmask products of type CM003
t.cloudmasks.cm004 # list cloudmask products of type CM004
t.cloudmasks.CM001 # list cloudmask products of type CM001
t.cloudmasks.CM002 # list cloudmask products of type CM002
t.cloudmasks.CM003 # list cloudmask products of type CM003
t.cloudmasks.CM004 # list cloudmask products of type CM004
For cloudmasks of types CM003 and CM004 the _params_ function is used to select cloudmasks with specific parameters. See specific wiki page for cloudmasks implementation within Sen2Chain (**_to do_**). Default values are :
......
......@@ -11,8 +11,10 @@ Multiprocessing is available to speed up calculations. Finally, output data is e
.. code-block:: python
from sen2chain import TimeSeries
ts=TimeSeries(vectors_layers='/path/to/shapefile.shp',date_min=date_min,date_max=date_max,indices=['NDVI','NBR','BIGR'],cm_version='CM004',iterations =1,nb_proc=10)
ts=TimeSeries(vectors_file='/path/to/shapefile.shp',date_min=date_min,date_max=date_max,indices=['NDVI','NBR','BIGR'],cm_version='CM004',iterations =1)
....
....
ts.to_csv(out_path='/path/out_path')
With this command, the data is not reprojected. Vector data must therefore be in WGS 84 (EPSG: 4326) for extraction to work properly.
......@@ -7,14 +7,14 @@ Welcome to Sen2Chain documentation!
=====================================
| Sen2Chain is a simple utility written in Python 3 to download and process Sentinel-2 images. The tool can also be used to automate these processes.
| It uses EODAG package to find and download data, and ESA's Sen2Cor processor to perform atmospheric, terrain and cirrus correction.
| It uses `EODAG <https://eodag.readthedocs.io/en/stable/>`_ package to find and download data, and `ESA's Sen2Cor <https://step.esa.int/main/snap-supported-plugins/sen2cor/>`_ processor to perform atmospheric, terrain and cirrus correction.
| Main features:
- Downloading of L1C (top of atmosphere) products from EODAG available providers;
- Downloading of L1C (top of atmosphere) products from available providers;
- Processing of L1C to L2A (bottom of atmosphere) products;
- Computing of radiometric indices;
- Time series extraction of thoses indices;
- Computing of cloud masks and radiometric indices;
- Time series extraction of thoses indices in regions of interest;
- Full automatization of the above features.
All these features can be launched in multiprocessing.
......@@ -28,7 +28,7 @@ INTERREG Renovrisk-impact project (2018-2020).
**Development and improvements :**
Pascal Mouquet, Thomas Germain, Jérémy Commins, Charlotte Wolff
Pascal Mouquet, Thomas Germain, Jérémy Commins, Charlotte Wolff, Juliette Miranville, George Ge
**Conceptualization and Coordination :**
......
0% Loading or .
You are about to add 0 people to the discussion. Proceed with caution.
Finish editing this message first!
Please register or to comment