[BUG] HTTP REST API hangs with Accept-Encoding zstd

[mmguero]

Describe the bug

As of v2.19.0 on Linux, when using cURL or libcurl against the OpenSearch REST API and zstd is in the Accept-Encoding headers, the response from OpenSearch hangs.

Related component

Other

To Reproduce

Broken:

$ curl 'http://localhost:9200/_cat/health?format=json' --verbose -H "Accept-Encoding: deflate, gzip, br, zstd" -o /dev/null
* Host localhost:9200 was resolved.
* IPv6: ::1
* IPv4: 127.0.0.1
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
  0     0    0     0    0     0      0      0 --:--:-- --:--:-- --:--:--     0*   Trying [::1]:9200...
* Connected to localhost (::1) port 9200
> GET /_cat/health?format=json HTTP/1.1
> Host: localhost:9200
> User-Agent: curl/8.5.0
> Accept: */*
> Accept-Encoding: deflate, gzip, br, zstd
> 
< HTTP/1.1 200 OK
< content-type: application/json; charset=UTF-8
< content-length: 300
< content-encoding: zstd
< 

The API call never returns, until the client times out or the process is killed.

Expected behavior

Here's what happens if you remove the zstd from the Accept-Encoding header.

$ curl 'http://localhost:9200/_cat/health?format=json' --verbose -H "Accept-Encoding: deflate, gzip, br" -o /dev/null
* Host localhost:9200 was resolved.
* IPv6: ::1
* IPv4: 127.0.0.1
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
  0     0    0     0    0     0      0      0 --:--:-- --:--:-- --:--:--     0*   Trying [::1]:9200...
* Connected to localhost (::1) port 9200
> GET /_cat/health?format=json HTTP/1.1
> Host: localhost:9200
> User-Agent: curl/8.5.0
> Accept: */*
> Accept-Encoding: deflate, gzip, br
> 
< HTTP/1.1 200 OK
< content-type: application/json; charset=UTF-8
< content-encoding: gzip
< content-length: 214
< 
{ [214 bytes data]
100   214  100   214    0     0   161k      0 --:--:-- --:--:-- --:--:--  208k
* Connection #0 to host localhost left intact

Also, this is the behavior in v2.18.0, even with the zstd included in Accept-Encoding:

$ curl 'http://localhost:9200/_cat/health?format=json' --verbose -H "Accept-Encoding: deflate, gzip, br, zstd" -o /dev/null
* Host localhost:9200 was resolved.
* IPv6: ::1
* IPv4: 127.0.0.1
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
  0     0    0     0    0     0      0      0 --:--:-- --:--:-- --:--:--     0*   Trying [::1]:9200...
* Connected to localhost (::1) port 9200
> GET /_cat/health?format=json HTTP/1.1
> Host: localhost:9200
> User-Agent: curl/8.5.0
> Accept: */*
> Accept-Encoding: deflate, gzip, br, zstd
> 
< HTTP/1.1 200 OK
< content-type: application/json; charset=UTF-8
< content-encoding: gzip
< content-length: 214
< 
{ [214 bytes data]
100   214  100   214    0     0   101k      0 --:--:-- --:--:-- --:--:--  104k
* Connection #0 to host localhost left intact

Note that in v2.18.0 content-encoding: gzip but in v2.19.0 content-encoding: zstd with it hanging.

Additional Details

Plugins

$ opensearch-plugin list
opensearch-alerting
opensearch-anomaly-detection
opensearch-asynchronous-search
opensearch-cross-cluster-replication
opensearch-custom-codecs
opensearch-flow-framework
opensearch-geospatial
opensearch-index-management
opensearch-job-scheduler
opensearch-knn
opensearch-ltr
opensearch-ml
opensearch-neural-search
opensearch-notifications
opensearch-notifications-core
opensearch-observability
opensearch-reports-scheduler
opensearch-security-analytics
opensearch-skills
opensearch-sql
opensearch-system-templates
query-insights

Note that I have removed the opensearch-security and opensearch-performance-analyzer but I think that's probably not relevant (?).

Host/Environment (please complete the following information):

• OS: Linux, x86_64 in Docker (based on official opensearchproject/opensearch:2.19.0 image)
• Version: v2.19.0 (regression since v2.18.0)

**Workaround

Setting http.compression to false causes the problem to go away.

[reta]

I seems like OpenSearch never supported ZSTD compression (for HTTP), but 4.1.116 has a bunch of changes that may trigger this behavior (since OpenSearch has zstd-jni on the classpath).

[1] netty/netty#14466
[2] netty/netty#14519