Add flash age compositor for li instruments

[ClementLaplace]

This Merge request is there to create the flash age compositor related to the li instrument.

• Add the flash_age compositor feature
• Document the code related to the feature
• Test the flash_age compositor

[codecov]

Codecov Report

All modified and coverable lines are covered by tests ✅

Project coverage is 96.11%. Comparing base (3765a5e) to head (01237e2).
Report is 73 commits behind head on main.

Additional details and impacted files

@@           Coverage Diff            @@
##             main    #2895    +/-   ##
========================================
  Coverage   96.10%   96.11%            
========================================
  Files         381      383     +2     
  Lines       55544    55673   +129     
========================================
+ Hits        53383    53511   +128     
- Misses       2161     2162     +1     

Flag	Coverage Δ
behaviourtests	3.89% <0.00%> (-0.01%)	⬇️
unittests	96.21% <100.00%> (+<0.01%)	⬆️

Flags with carried forward coverage won't be shown. Click here to find out more.

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

[coveralls]

Pull Request Test Coverage Report for Build 10700852657

Details

• 18 of 34 (52.94%) changed or added relevant lines in 1 file are covered.
• No unchanged relevant lines lost coverage.
• Overall first build on add_flash_age_compositor at 96.124%

---

Changes Missing Coverage	Covered Lines	Changed/Added Lines	%
satpy/composites/lightning.py	18	34	52.94%

Totals
Change from base Build 10654375136:	96.1%
Covered Lines:	52426
Relevant Lines:	54540

---

💛 - Coveralls

[ameraner]

Thanks for this first draft! I left a comment regarding the reference time.

[djhoese]

I'm really surprised that our pre-commit/linters are not catching the spacing inconsistencies. I've made some comments on small style things, but also on some bigger things that should be changed. Thanks for getting this going.

On a higher-level discussion, shouldn't this kind of functionality be something implemented in the reader?

[mraspaud]

@ameraner @ClementLaplace any comments about the last comment by @djhoese ?

[ameraner]

I would keep this "time-based normalisation" as a compositor, at least for now, since its only use really is inside an image. Since it's also a "made-up" dataset, i.e. it's not present inside the files at all, I'm not sure if it fits the satpy philosophy to put it in a reader (?).

[ClementLaplace]

Some issue happened after the merging with the main branch we susperce it is beacuse of PR #2985
the issue that we get is

raceback (most recent call last):
  File "/tcenas/home/claplace/project/satpy_script_test/test_flash_age.py", line 17, in <module>
    scn.load(['flash_age'])
  File "/tcenas/home/claplace/project/satpy/satpy/scene.py", line 1458, in load
    self.generate_possible_composites(unload)
  File "/tcenas/home/claplace/project/satpy/satpy/scene.py", line 1521, in generate_possible_composites
    keepables = self._generate_composites_from_loaded_datasets()
                ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/tcenas/home/claplace/project/satpy/satpy/scene.py", line 1540, in _generate_composites_from_loaded_datasets
    return self._generate_composites_nodes_from_loaded_datasets(needed_comp_nodes)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/tcenas/home/claplace/project/satpy/satpy/scene.py", line 1546, in _generate_composites_nodes_from_loaded_datasets
    self._generate_composite(node, keepables)
  File "/tcenas/home/claplace/project/satpy/satpy/scene.py", line 1604, in _generate_composite
    composite = compositor(prereq_datasets,
                ^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/tcenas/home/claplace/project/satpy/satpy/composites/lightning.py", line 106, in __call__
    return self._normalize_time(data, new_attrs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/tcenas/home/claplace/project/satpy/satpy/composites/lightning.py", line 66, in _normalize_time
    data = data.where(data >= begin_time, drop=True)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/tcenas/proj/optcalimg/claplace/virtual_env/satpy-dev/lib/python3.11/site-packages/xarray/core/common.py", line 1225, in where
    self = self.isel(**indexers)
           ^^^^^^^^^^^^^^^^^^^^^
  File "/tcenas/proj/optcalimg/claplace/virtual_env/satpy-dev/lib/python3.11/site-packages/xarray/core/dataarray.py", line 1485, in isel
    ds = self._to_temp_dataset()._isel_fancy(
         ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/tcenas/proj/optcalimg/claplace/virtual_env/satpy-dev/lib/python3.11/site-packages/xarray/core/dataset.py", line 3000, in _isel_fancy
    new_var = var.isel(indexers=var_indexers)
              ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/tcenas/proj/optcalimg/claplace/virtual_env/satpy-dev/lib/python3.11/site-packages/xarray/core/variable.py", line 1018, in isel
    return self[key]
           ~~~~^^^^^
  File "/tcenas/proj/optcalimg/claplace/virtual_env/satpy-dev/lib/python3.11/site-packages/xarray/core/variable.py", line 782, in __getitem__
    dims, indexer, new_order = self._broadcast_indexes(key)
                               ^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/tcenas/proj/optcalimg/claplace/virtual_env/satpy-dev/lib/python3.11/site-packages/xarray/core/variable.py", line 622, in _broadcast_indexes
    self._validate_indexers(key)
  File "/tcenas/proj/optcalimg/claplace/virtual_env/satpy-dev/lib/python3.11/site-packages/xarray/core/variable.py", line 674, in _validate_indexers
    raise KeyError(
KeyError: 'Indexing with a boolean dask array is not allowed. This will result in a dask array of unknown shape. Such arrays are unsupported by Xarray.Please compute the indexer first using .compute()'

[mraspaud]

Interresting! why is this not showing in the tests?

[ClementLaplace]

@mraspaud The test were not working because I did not take in account the dask wrapper into my test, with the new version of my test (see bellow )

def test_flash_age_compositor():
    """Test the flash_age compsitor by comparing two xarrays object."""
    comp = LightningTimeCompositor("flash_age",prerequisites=["flash_time"],
                                   standard_name="ligtning_time",
                                   time_range=60,
                                   reference_time="end_time")
    attrs_flash_age = {"variable_name": "flash_time","name": "flash_time",
                       "start_time": datetime.datetime(2024, 8, 1, 10, 50, 0),
                       "end_time": datetime.datetime(2024, 8, 1, 11, 0, 0),"reader": "li_l2_nc"}
    flash_age_value = da.array(["2024-08-01T09:00:00",
            "2024-08-01T10:00:00", "2024-08-01T10:30:00","2024-08-01T11:00:00"], dtype="datetime64[ns]")
    flash_age = xr.DataArray(
    flash_age_value,
    dims=["y"],
    coords={
        "crs": "8B +proj=longlat +ellps=WGS84 +type=crs"
    },attrs = attrs_flash_age,name="flash_time")
    res = comp([flash_age])
    print(res)
    expected_attrs = {"variable_name": "flash_time","name": "lightning_time",
                       "start_time": datetime.datetime(2024, 8, 1, 10, 50, 0),
                       "end_time": datetime.datetime(2024, 8, 1, 11, 0, 0),"reader": "li_l2_nc",
                       "standard_name": "ligtning_time"
                       }
    expected_array = xr.DataArray(
    da.array([0.0,0.5,1.0]),
    dims=["y"],
    coords={
        "crs": "8B +proj=longlat +ellps=WGS84 +type=crs"
    },attrs = expected_attrs,name="flash_time")
    xr.testing.assert_equal(res,expected_array)

I'm able to reproduce this issue

[mraspaud]

LGTM