Newer
Older
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
=============================
Working a single tile : Tile
=============================
The use of the Tile object is a good way to access tile database, process l2a, cloudmasks and indices.
First define a Tile object using its identifier (2 digits + 3 letters format)
.. code-block:: python
from sen2chain import Tile
t = Tile("40KCB")
From Downloading L1C to computing radiometric indices
======================================================
Downloading L1C Product
-----------------------
Computing products
------------------
Three functions are used to compute the different products from downloaded raw L1C: L2A with Sen2Cor, cloudmasks and indices.
For all 3, the process is the same: entire database is scanned to identify a list of missing products for which the computing will be done.
In case of no missing products, nothing is done. Each function has its own set of specific parameters (see details bellow), with these in common:
- _date_min_ and _date_max_: the date after, date before, or time range between which the missing products will be computed. Default values _None_ results in the entire database being scanned.
- _reprocess (boolean)_: possibility to reprocess already computed products. Default value _False_.
- _nb_proc_: All products in the process list are computed using multiprocessing to reduce processing time. Default value is 4, to increase according to your hardware. Limited by the number of cores of your processor.
### Computing L2A
This function is used to compute missing L2A for L1C products present in the local database. Possibility to specify dates or reprocessing, see parameters above.
```python
>>> Tile("40KCB").compute_l2a()
>>> Tile("40KCB").compute_l2a(date_min="2019-01-01", nb_proc=12)
43780:2021-10-06 13:05:17:INFO:sen2chain.tiles:All l2a products already computed
```
### Computing cloudmasks
This function computes missing cloudmasks for L2A products present in local database. This stage is mandatory if you want to mask indices later. In addition to those already mentioned above, many parameters are available, depending on the cloudmask version to compute:
- **cm_version**: _"cm001"_, _"cm002"_, _"cm003"_, or _"cm004"_. _Default value = "cm001"_. See specific wiki page for cloudmasks implementation within Sen2Chain.
- **probability**: _DV =_ _1_, the max pixel cloud cover % probability value (0-100) above which the pixel should be considered as cloud. Used only for CM003.
- **iterations**: _DV =_ _5_, the number of dilatation cycles to apply on thresholded cloudmasks. Used for CM003 & CM004
- **cld_shad**: _DV =_ _True_, used with CM004 only. Boolean value for masking cloud shadows from 20m Sen2Chain Scene Classification.
- **cld_med_prob**: _DV =_ _True_, used with CM004. Boolean value for masking clouds with medium probability from 20m Sen2Chain Scene Classification.
- **cld_hi_prob**: _DV =_ _True_, used with CM004. Boolean value for masking clouds with high probability from 20m Sen2Chain Scene Classification.
- **thin_cir**: _DV =_ _True_, used with CM004. Boolean value for masking thin cirrus clouds from 20m Sen2Chain Scene Classification.
The output cloudmask is named according to cloudmask version and the set of parameters, and produced only if not already present.
```python
>>> Tile("40KCB").compute_cloudmasks()
>>> Tile("40KCB").compute_cloudmasks(cm_version="cm004", iterations=2, cld_shad=False, thin_cir=False, nb_proc=8)
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
150
151
152
153
154
155
156
157
158
159
160
161
162
163
164
165
166
167
168
169
170
171
172
173
174
175
176
177
178
179
180
181
182
183
184
185
186
187
188
189
190
191
192
193
194
195
196
197
198
199
200
201
202
203
204
205
206
207
208
209
210
211
212
213
214
215
216
217
218
219
220
221
222
223
```
### Computing indices
This function computes all missing indices for l2a products. Indices are given as a list. If indices are not provided, it will compute missing dates of already existing indices for this tile (no new indice computed). Indices won't be masked if no cloudmasks are present, you have to compute cloudmasks first. Specific parameters:
- **nodata_clouds**: Default value _True_, to produce a second set of cloudmasked index images (cloudmasks have to be be present). If not specified CM001 will be used. For CM003 & CM004, other CM specific parameters can be adjusted.
```python
>>> Tile("40KCB").compute_indices(["NDVI"], date_max="2020-12-31" )
>>> Tile("40KCB").compute_indices(["NDVI"], cm_version="cm002", nb_proc=2)
```
### Computing quicklooks
You can compute quicklooks for multiple provided tiles for L1C and/or L2A products. If no tile provided, whole L1C + L2A product database is used. Possibility to set specific output QL resolution (default 750m/px). Possibility to set output format JPEG (default) or TIFF.
```python
>>> Tile("40KCB").compute_ql(product_list = ["l2a"],
resolution = 100,
jpg = False)
```
## Database information and management
### Information
You can get informations on the number of files in database, downloaded l1c, and produced l2a, cloudmasks and indices using the _info_ function.
```python
>>> t.info
43780:2021-10-06 12:53:34:INFO:sen2chain.tiles:l1c: 344
43780:2021-10-06 12:53:34:INFO:sen2chain.tiles:l2a: 345
43780:2021-10-06 12:53:34:INFO:sen2chain.tiles:cloud_masks: 625
43780:2021-10-06 12:53:34:INFO:sen2chain.tiles:ndwigao (raw / masked): 345 / 345
43780:2021-10-06 12:53:34:INFO:sen2chain.tiles:bigr (raw / masked): 345 / 349
43780:2021-10-06 12:53:34:INFO:sen2chain.tiles:birnir (raw / masked): 215 / 211
43780:2021-10-06 12:53:34:INFO:sen2chain.tiles:ireci (raw / masked): 2 / 2
43780:2021-10-06 12:53:34:INFO:sen2chain.tiles:ndwimcf (raw / masked): 345 / 345
43780:2021-10-06 12:53:34:INFO:sen2chain.tiles:bibg (raw / masked): 211 / 211
43780:2021-10-06 12:53:34:INFO:sen2chain.tiles:ndvi (raw / masked): 345 / 352
43780:2021-10-06 12:53:34:INFO:sen2chain.tiles:ndre (raw / masked): 11 / 4
43780:2021-10-06 12:53:34:INFO:sen2chain.tiles:mndwi (raw / masked): 54 / 57
```
### Size
The _size_ function retrieves the used disk space for each kind of product. Execution can take time, depending on database size, disk or network speed. Local and archived disk spaces are analysed.
```python
>>> t.size
43780:2021-10-06 12:53:41:INFO:sen2chain.tiles:l1c: 198GB (local: 44GB / archived: 154GB)
43780:2021-10-06 12:53:45:INFO:sen2chain.tiles:l2a: 258GB (local: 65GB / archived: 192GB)
43780:2021-10-06 12:53:45:INFO:sen2chain.tiles:ndwigao: 25GB
43780:2021-10-06 12:53:45:INFO:sen2chain.tiles:bigr: 24GB
43780:2021-10-06 12:53:45:INFO:sen2chain.tiles:birnir: 15GB
43780:2021-10-06 12:53:45:INFO:sen2chain.tiles:ireci: 324MB
43780:2021-10-06 12:53:45:INFO:sen2chain.tiles:ndwimcf: 25GB
43780:2021-10-06 12:53:45:INFO:sen2chain.tiles:bibg: 15GB
43780:2021-10-06 12:53:45:INFO:sen2chain.tiles:ndvi: 25GB
43780:2021-10-06 12:53:45:INFO:sen2chain.tiles:ndre: 791MB
43780:2021-10-06 12:53:45:INFO:sen2chain.tiles:mndwi: 4GB
```
## Listing products
Different functions are called to retrieve a list all database products :
```python
>>> t.l1c # list downloaded l1c indice products
>>> t.l2a # list l2a indice products
>>> t.cloudmasks # list cloudmask products
>>> t.ndvi # list ndvi indice products
>>> t.ndwigao # list ndwi Gao indice products
>>> t.mndwi # list mndwi indice products
```
As a ProductList class this retrieved list of products can be reduced using multiples functions :
```python
>>> t.l2a.products # list of product names
>>> t.l2a.dates # list of product dates
>>> t.l2a.first # the first (oldest) product in database
>>> t.l2a.last # the last (newest) product in database
>>> t.l2a.filter_dates(date_min = "yyyy-mm-dd", date_max = "yyyy-mm-dd") # list of products between date_min and date_max
>>> t.l2a.filter_dates(date_min = "yyyy-mm-dd") # list of products after date_min
>>> t.l2a.filter_dates(date_max = "yyyy-mm-dd") # list of products before date_max
>>> t.l2a.filter_clouds(cover_min = 0, cover_max = 100) # list of products with a cloud cover between cover_min and cover_max
>>> t.l2a.filter_clouds(cover_min = 20) # list of products with more than 20% cloud cover
>>> t.l2a.filter_clouds(cover_max = 50) # list of products with less than 50% cloud cover
```
Cloudmask and indice lists as CloudMaskList class objects can be reduced using 4 different functions, one for each type of cloud mask :
```python
>>> t.cloudmasks.cm001 # list cloudmask products of type CM001
>>> t.cloudmasks.cm002 # list cloudmask products of type CM002
>>> t.cloudmasks.cm003 # list cloudmask products of type CM003
>>> t.cloudmasks.cm004 # list cloudmask products of type CM004
```
For cloudmasks of types CM003 and CM004 the _params_ function is used to select cloudmasks with specific parameters. See specific wiki page for cloudmasks implementation within Sen2Chain (**_to do_**). Default values are :
- **probability**: _1_, the max pixel cloud cover % probability value (0-100) above which the pixel should be considered as cloud. Used only for CM003.
- **iterations**: _5_, the number of dilatation cycles to apply on thresholded cloudmasks. Used for CM003 & CM004
- **cld_shad**: _True_, used with CM004 only. Boolean value for masking cloud shadows from 20m Sen2Chain Scene Classification.
- **cld_med_prob**: _True_, used with CM004. Boolean value for masking clouds with medium probability from 20m Sen2Chain Scene Classification.
- **cld_hi_prob**: _True_, used with CM004. Boolean value for masking clouds with high probability from 20m Sen2Chain Scene Classification.
- **thin_cir**: _True_, used with CM004. Boolean value for masking thin cirrus clouds from 20m Sen2Chain Scene Classification.
```python
>>> t.cloudmasks.cm003.params() # list cloudmask products of type CM003 with default values
>>> t.cloudmasks.cm003.params(probability = 25) # list cloudmask products of type CM003 with pixel cloud probability of 25%
>>> t.cloudmasks.cm004.params() # list cloudmask products of type CM004 with default values
>>> t.cloudmasks.cm004.params(cld_shad = False) # list cloudmask products of type CM004, without considering cloud shadows
```
Indice lists can be filtered using one of the 3 defined functions: _raws_, _masks_, and _quicklooks_
```python
>>> t.ndvi.raws #list only non masked ndvi indices
>>> t.ndwigao.masks #list only masked ndwi Gao indices (by any mask)
>>> t.ndwigao.masks.cm001 #list only ndwi Gao indices masked by CM001 cloudmasks
>>> t.mndwi.quicklooks #list only mndwi indice quicklooks
```
## Cleaning corrupted products
Data in image database can sometimes be corrupted during their production, while downloading and unzipping l1c products, computing l2a with Sen2Cor (frequent corruptions with old 2.5.5 Sen2Cor version), cloudmasks or indices. The _clean_lib_ function has been introduced to check for data integrity, with the possibility to remove corrupted products (whole SAFE folder). Removed products have to be downloaded / computed again.
```python
>>> t.clean_lib() # Check corrupted products (remove nothing)
{'problems': [], 'identified_problems': 0, 'removed_problems': 0}
>>> t.clean_lib(remove = True) # Check and remove corrupted products
```
## Archiving products
Archive functions offer the possibility to move L1C, L2A or both products (SAFE folders) to archive paths, specified in the configuration file (<span dir="">\~</span>/sen2chain_data/config/config.cfg). This function can be useful in case of different kind of available hardware disks (speed vs size).\
The _size_only_ option offers the possibility to get the info, whithout moving anything (default value _False_).
- L1C products are moved from `l1c_path` to `l1c_archive_path`
- L2A products are moved from `l2a_path` to `l2a_archive_path`
Every moved SAFE folder is replaced by its symlink in the original product folder. L2A SAFE products can be moved only if there is no file error (_clean_lib_ function automatically performed first).
```python
>>> t.archive_l1c() # Archive L1C products
>>> t.archive_l2a(size_only = True) # Checking L2A products ready to archive (nothing moved)
>>> t.archive_all() # Chaining L1C and L2A archive functions
```
## Removing products
Two functions offers the possibility to remove the entire L1C or L2A products from database. For L2A a provided list of identifiers offers the possibility to remove specific L2A products, if nothing provided whole L2A DB is removed.
**!!! To use with care !!!**
```python
>>> t.remove_l1c()
>>> t.remove_l2a(identifier_list = list)
```