Fix: Avoid unnecessary producer epoch bumps #4972
Open
+5
−10
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Fixes: #4953
When the producer is idempotent and
max.in.flight.requests.per.connection
is set to a value between 2 and 5, it's normal to receiveOUT_OF_ORDER_SEQUENCE_NUMBER
produce responses for requests R2 through R5 when the R1 failed for any other reason.Bumping the producer epoch in this scenario violates the "exactly-once" guarantees. Therefore, we believe that it's unnecessary to bump the producer's epoch; re-enqueuing the messages is sufficient.
The same "retry, but don't bump producer epoch" behavior is implemented in the Java client: https://github.com/apache/kafka/blob/a6a588fbed9982598377060c63f94ee6184b4295/clients/src/main/java/org/apache/kafka/clients/producer/internals/TransactionManager.java#L1015-L1016