[sbalabanov]

TL;DR: Recover dead-lettered logs so public request status no longer stays permanently stale.

Summary

Requests could finish while their public status stayed stuck: logs in log_dlq had no reader, so unlimited retries could not help. Add a gateway reader that saves those logs through the normal path and keeps trying when saving fails. Started readers are cleaned up after a failed startup and during shutdown.

Repro

1. After a request finishes, keep gateway log writes failing until its final status log is sent to log_dlq.
2. Restore gateway storage and publish no more logs.
3. Before: nobody reads the retained log, so the public status stays stuck. After: the gateway reader retries it and eventually updates status and history.

Test Plan

Verified with existing tests:

• The normal-log integration test saves a log and exposes the updated request status.
• Invalid queue metadata is logged and rejected; unreadable log payloads with valid metadata are retried rather than marked saved.
• Failed startup cleans up started readers; shutdown attempts both readers and reports cleanup failures.

Revert Plan

Revert this change and leave the log_dlq backlog intact.

Issues

[sbalabanov]

[independent-review] Should fix — shutdown failures can be hidden (95% confidence).

In service/submitqueue/gateway/server/main.go:381, joining context.Canceled with a consumer-stop error keeps errors.Is(err, context.Canceled) true. Repro: SIGTERM wins the server select, then an in-flight controller exceeds the consumer shutdown timeout. main takes its ordinary signal branch instead of reporting the cleanup failure (the previous code replaced expected cancellation when cleanup failed).

Smallest correction: return the stop error instead of expected cancellation when cleanup fails; still join genuine listener/server failures and keep stopping both consumers. Extend the existing cleanup table with cancellation + stop error; assert both consumers stop, the error contains stopErr, and it does not contain context.Canceled. No new test suite is needed.

All four affected package tests, including fresh Docker integration, passed independently. The lifecycle/policy refactoring already removes duplicated branching; no additional safe complexity reduction was established.

Generated with the repository's .agents/skills/autoreview workflow; finding rechecked against PR #1037 at 46fd1258d94373f55a845035a7c453a2367a68e7.

[sbalabanov]

[agent-response SQC-010] Addressed #1037 (comment) in commit 2669cfb. A consumer-stop failure now replaces expected signal cancellation; genuine listener/server errors still join cleanup failures, and both consumers are stopped. Extended TestServeGateway_ConsumerCleanup with cancellation plus a stop failure, verifying stopErr is retained and context.Canceled is absent. The focused Bazel package test and formatting/lint/tidy/Gazelle checks passed locally. The PR remains a draft; skipped code-CI checks are not counted as passes.

[cla-assistant]

Thank you for your submission! We really appreciate it. Like many open source projects, we ask that you sign our Contributor License Agreement before we can accept your contribution.
_{You have signed the CLA already but the status is still pending? Let us recheck it.}

[cla-assistant]

Thank you for your submission! We really appreciate it. Like many open source projects, we ask that you sign our Contributor License Agreement before we can accept your contribution.
_{You have signed the CLA already but the status is still pending? Let us recheck it.}