TimeSeries Desc Sort gets skipped with Lucene 10 upgrade

[expani]

Description

We override IndexSearcher#search(List<LeafReaderContext> leaves, Weight weight, Collector collector) and it contains the logic to apply time series desc optimisation for scanning segments in reverse for descending sort use-cases.

Existing Lucene 9.12.1 and OS 2.19 call flow

Lucene 10.1.0 and OS 3.0 call flow

With Lucene 10 changes, the new replacement method

search(LeafReaderContextPartition[] partitions, Weight weight, Collector collector)

only gets invoked via IndexSearcher#search(Query, CollecterManager) which is not used by OpenSearch in QueryPhase#searchWithCollector.

So, it was never getting called causing the time series desc optimization to be skipped.

My changes ensure it will be called in the same way that IndexSearcher does for CollectorManager variant.

Related Issues

Resolves #16934

[expani]

@msfroh @reta Please help in reviewing the same.

I am trying to see if there is an easy to catch this with existing UTs as well.

[reta]

I am trying to see if there is an easy to catch this with existing UTs as well.

This is explains the xxx_desc regressions in OSB, thanks @expani !

[expani]

@reta With this change, OS 2.19 and OS 3.0 are at same for following query types in big5

asc_sort_timestamp
asc_sort_timestamp_can_match_shortcut
asc_sort_timestamp_no_can_match_shortcut

Without it, the latency for OS 3.0 was 10ms more than OS 2.19

[github-actions]

❌ Gradle check result for 21d5903: FAILURE

Please examine the workflow log, locate, and copy-paste the failure(s) below, then iterate to green. Is the failure a flaky test unrelated to your change?

[github-actions]

❌ Gradle check result for 368383f: FAILURE

Please examine the workflow log, locate, and copy-paste the failure(s) below, then iterate to green. Is the failure a flaky test unrelated to your change?

[reta]

@msfroh this change make sense to me to restore searchContext.shouldUseTimeSeriesDescSortOptimization() optimization, wdyt?

[github-actions]

❌ Gradle check result for 21a924d: FAILURE

Please examine the workflow log, locate, and copy-paste the failure(s) below, then iterate to green. Is the failure a flaky test unrelated to your change?

[reta]

@expani I think we should get all partitions across all slices and call search only once:

LeafReaderContextPartition[] partitions = Arrays.stream(getSlices()).flatMap(LeafSlice::partitions).toArray();
search(partitions, weight, collector)

[msfroh]

If we can override the logic in getSlices, we could even handle the desc sort use-case more efficiently by returning slices that contain leafcontextpartitions with the higher doc ids and front-load those.

[expani]

Thanks for the suggestions @reta @msfroh

I am still facing regression with desc_sort_timestamp and it's variants let me try fixing using these and will update in sometime.