Update Forward Protocol Specification for zstd compression support

[daipom]

Is your feature request related to a problem? Please describe.

The following PR supports zstd compression.

• compressable:Zstd comp support #4657

So, we need to update Forward Protocol Specification - CompressedPackedForward Mode.

Describe the solution you'd like

Update the following description and add zstd value to compressed option.

• https://github.com/fluent/fluentd/wiki/Forward-Protocol-Specification-v1.5#compressedpackedforward-mode
• https://github.com/fluent/fluentd/wiki/Forward-Protocol-Specification-v1.5#option

Would that be Forward Protocol Specification v1.6?

Describe alternatives you've considered

Having no idea.

Additional context

No response

[daipom]

I propose changes to the following two sections.

CompressedPackedForward Mode

https://github.com/fluent/fluentd/wiki/Forward-Protocol-Specification-v1.5#compressedpackedforward-mode

 ### CompressedPackedForward Mode
 
-It carries a series of events as a msgpack binary, compressed by gzip, on a single request. The supported compression algorithm is only gzip.
+It carries a series of events as a msgpack binary, compressed by gzip or zstd, on a single request.
 
-- `entries` is a gzipped binary chunk of `MessagePackEventStream`, which MAY be a concatenated binary of multiple gzip binary strings.
-- Client MUST send an option with `compressed` key with the value `gzip`.
-- Client MUST send a gzipped chunk as msgpack `bin` format.
+- `entries` is a gzip/zstd binary chunk of `MessagePackEventStream`, which MAY be a concatenated binary of multiple gzip/zstd binary strings.
+- Client MUST send an option with `compressed` key with the value `gzip` or `zstd`.
+- Client MUST send a gzip/zstd chunk as msgpack `bin` format.
 - Server MUST accept `bin` format.
 - Server MAY decompress and decode individual events on demand but MAY NOT do right after request arrival. It means it MAY costs less, compared to `Forward` mode, when decoding is not needed by any plugins.

Option

https://github.com/fluent/fluentd/wiki/Forward-Protocol-Specification-v1.5#option

 - Server MAY just ignore any options given.
 - `size`: Clients MAY send the `size` option to show the number of event records in an entries by an integer as a value. Server can know the number of events without unpacking entries (especially for PackedForward and CompressedPackedForward mode).
 - `chunk`: Clients MAY send the `chunk` option to confirm the server receives event records. The value is a string of Base64 representation of 128 bits `unique_id` which is an ID of a set of events.
-- `compressed`: Clients MUST send the `compressed` option with value `gzip` to tell servers that entries is `CompressedPackedForward`. Other values will be ignored.
+- `compressed`: Clients MUST send the `compressed` option with value `gzip` or `zstd` to tell servers that entries is `CompressedPackedForward`. Other values will be ignored.
 
 ```json
 {"chunk": "p8n9gmxTQVC8/nh2wlKKeQ==", "size": 4097}

[daipom]

Does this revision require a voting process?

[Athishpranav2003]

The changes suggested here looks good to me.

[daipom]

If there are no objections, I would like to set it to v1.6 with the agreement of both Fluentd and Fluent Bit maintainers.

@cosmo0920
Could you please check #4758 (comment) ?

[ashie]

LGTM

[cosmo0920]

This would be good:

-- entries is a gzipped binary chunk of MessagePackEventStream, which MAY be a concatenated binary of multiple gzip binary strings.
-- Client MUST send an option with compressed key with the value gzip.
-- Client MUST send a gzipped chunk as msgpack bin format.
+- entries is a gzip/zstd binary chunk of MessagePackEventStream, which MAY be a concatenated binary of multiple gzip/zstd binary strings.
+- Client MUST send an option with compressed key with the value gzip or zstd.
+- Client MUST send a gzip/zstd chunk as msgpack bin format.

However, I have a question for these sentences. Currently, we're able to decompress compressed chunks one-by-one. So, in paper, we are also able to take turns to decompress gzip compressed chunks and zstd compressed chunks.
Is this specification needed to point out explicitly in the revised specification?

[Athishpranav2003]

@cosmo0920 are u pointing to the case where different sources send different type of compressed chunks?

Incase of forward protocol the message itself has the compression type so the decompression happens wrt the metadata of the compressed chunk.

[cosmo0920]

@cosmo0920 are u pointing to the case where different sources send different type of compressed chunks?

Incase of forward protocol the message itself has the compression type so the decompression happens wrt the metadata of the compressed chunk.

Yes, different sources use the same in_forward case. This is surely occurred because Fluentd is able to be used as an aggregator which will collect the different Fluentd instances with pointing the same forward endpoint. So, probably we need to clarify on it even if the implementation already does.

[Athishpranav2003]

@cosmo0920
https://github.com/fluent/fluentd/pull/4657/files#diff-71dd20388a1f90eaec72fb257002fc6d44a497d457821d0c105acc0510844be1R314
In this line the compression format is also used as an argument during decompression. I guess this should answer your question.

[cosmo0920]

@cosmo0920
https://github.com/fluent/fluentd/pull/4657/files#diff-71dd20388a1f90eaec72fb257002fc6d44a497d457821d0c105acc0510844be1R314
In this line the compression format is also used as an argument during decompression. I guess this should answer your question.

It’s not answered my question. My question is: Should we write down the current decompression behavior explicitly in the specification document of forward protocol? It's not covered for the actual implementation. This is because Fluent Protocol is shared between Fluentd and Fluent Bit. So, we need to describe the specification in the document instead of the implementations.

[Athishpranav2003]

got it @cosmo0920 .

Previously there was a possibility that multiple sources could send either gzip compressed chunk or uncompressed one. If we consider uncompressed data also as a compression type then essentially the previous specification was good enough to express that 2 types were supported. Now we are just adding one more number to it so like idk if we really need to be explicit here