Skip to content

docs: explain how a record that is never acked blocks Kafka spout commits - #9280

Open
linzch3 wants to merge 1 commit into
apache:masterfrom
linzch3:docs-kafka-undecodable-record
Open

linzch3 wants to merge 1 commit into
apache:masterfrom
linzch3:docs-kafka-undecodable-record

Conversation

@linzch3

@linzch3 linzch3 commented Nov 17, 2026

Copy link
Copy Markdown
Contributor

Addresses the docs part of #9078. The design questions there (signalling the drop back to the spout, dead-lettering) are left alone.

What is the purpose of the change

docs/storm-kafka-client.md says nothing about what happens when a tuple from the Kafka spout is never acked, for example because it always fails to deserialize on the receiving worker. In at-least-once mode that quietly stops commits for the partition and lag keeps climbing, and with the default retry settings it never recovers.

I added a short "Records that are never acked" section that explains:

  • why commits stop and when polling of the partition stops (max.uncommitted.offsets)
  • that the default retry service retries forever (DEFAULT_MAX_RETRIES is Integer.MAX_VALUE)
  • the current mitigation: a KafkaSpoutRetryExponentialBackoff with a finite maxRetries, plus KafkaTupleListener.onMaxRetryReached to log or keep track of the record before the spout acks it
  • the trade-offs: each retry waits for the message timeout, and a finite maxRetries also gives up on tuples that fail for temporary reasons

Docs only; no code changes.

How was the change tested

I checked the description against KafkaSpout.fail, KafkaSpoutConfig (defaults and builder methods) and KafkaTupleListener on master, and that the example uses the actual constructor and method names. I also read the rendered Markdown.

Happy to adjust the wording or move it somewhere else in the doc if you'd prefer.

An AI assistant helped draft this change; I reviewed it and take responsibility for it.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant