Repository navigation
fix: surface server detail and cause on connection failures - #587
vuanhphung wants to merge 4 commits into
Conversation
Thrift THTTPException messages now append the server's error detail (Thrift/Databricks error headers, else a JSON or plain-text body, capped at 500 characters), keeping the existing message prefix. Kernel errors decoded from the napi binding keep the binding error as a non-enumerable `cause`; the Reyden fallback appends the Thrift KP001 rejection at the end of that chain instead of dropping it. Fixes #467 Co-authored-by: Isaac <no-reply@databricks.com> Signed-off-by: Vu Anh Phung <vu.phung@databricks.com>
0b9e9e3 to
c625ed7
Compare
There was a problem hiding this comment.
Verdict: 1 Low
Looks good, with 1 low concern: the error body is read in full even though only the first 4 KiB is inspected. The header-first detail extraction, the 200-path invariance, the kernel cause descriptor and the Reyden cause-chain append are all correct, and each has targeted unit and e2e tests. Nit: putting the Thrift KP001 at the tail of the chain makes it the cause of the napi binding error, which reads a little oddly, but it keeps the existing cause intact as intended.
| } | ||
| } | ||
|
|
||
| let body: Buffer; |
There was a problem hiding this comment.
🔵 Low — On the header-less error path, response.buffer() reads the whole error body into memory, but only the first 4 KiB (MAX_ERROR_BODY_SCAN) is ever scanned for detail. Before this change, non-200 bodies were never read. Now a large error page from a proxy or gateway, or a slow body stream, is fully buffered (and awaited) before the caller gets the exception. That is new work on a path that used to fail fast.
The buffered copy is kept so that err.response stays readable, which is a deliberate trade-off. Consider reading at most a bounded prefix, for example by streaming response.body until MAX_ERROR_BODY_SCAN bytes. That would mean the exposed response holds only a truncated body, so it is a design choice for the author. At minimum, note the unbounded read in the code comment.
|
@bot Could we bound this diagnostic body read by both bytes and time, falling back to the status-only error when either budget is exhausted? MAX_ERROR_BODY_SCAN only limits parsing; response.buffer() still consumes the complete body and delays reporting an already-known HTTP failure. |
Signed-off-by: peco-engineer-bot[bot] <peco-engineer-bot[bot]@users.noreply.github.com>
|
Pushed Started from c625ed7. Replaced the full response.buffer() read in ThriftHttpConnection.ts. The body is now read for at most 64 KiB and 1 s. A Content-Length over 64 KiB skips the read. When a budget runs out, the error carries the status only, and error.response still returns the full body: the bytes already read, then the rest of the stream. Added 3 tests: chunked body over the byte limit, oversized Content-Length, and a body that never ends. Updated CHANGELOG. npm test (1555 passing), type-check and prettier pass. npm run lint did not run: ESLint can't find the airbnb-base config here. E2E tests not run. |
|
Thanks for your contribution! To satisfy the DCO policy in our contributing guide every commit message must include a sign-off message. One or more of your commits is missing this message. You can reword previous commit messages with an interactive rebase ( |
There was a problem hiding this comment.
Verdict: 1 Low
Looks good, with one low concern: a JSON error body longer than 4 KiB loses its message field and the error shows a raw JSON prefix instead. The other changes behave as described and the tests cover them: the bounded body read keeps response readable, the kernel cause uses a non-enumerable descriptor, and the Reyden chain appends at the end. Follow-up, not caused by this PR: the 10-level depth cap in the Reyden cause walk still drops the Thrift error when a cause chain is cyclic or deeper than 10 levels. Before this PR, any existing cause made the code drop it, so this is no worse.
|
|
||
| function getErrorDetailFromBody(body: Buffer): string | undefined { | ||
| const text = body.toString('utf8', 0, MAX_ERROR_BODY_SCAN).trim(); | ||
| // Skip HTML pages and binary (e.g. Thrift-encoded) payloads |
There was a problem hiding this comment.
🔵 Low — getErrorDetailFromBody only decodes the first MAX_ERROR_BODY_SCAN (4096) bytes before calling JSON.parse, even though the whole body (up to 64 KiB) is already in prefix. When a JSON error body is longer than 4 KiB (for example, one with a long details array or a stack trace), the truncated text doesn't parse, so the code falls through to normalizeErrorDetail(text). The message then gets the raw JSON prefix ({"error_code":...,"message":...) instead of the message field. The PR description says the detail is the "JSON message, else plain text", and that doesn't hold for these bodies. Running JSON.parse on the full buffered body and applying the 4096 cap only in the plain-text fallback would keep that contract, at no extra cost since the body is already in memory.
|
@bot |
Signed-off-by: peco-engineer-bot[bot] <peco-engineer-bot[bot]@users.noreply.github.com>
|
Pushed Started from d9c01e4. |
|
Thanks for your contribution! To satisfy the DCO policy in our contributing guide every commit message must include a sign-off message. One or more of your commits is missing this message. You can reword previous commit messages with an interactive rebase ( |
When the server rejects a session open, the Thrift backend threw
THTTPExceptionwith onlyReceived a response with a bad HTTP status code: 403, dropping the detail the server sent (x-databricks-reason-phrase: Invalid access token.plus a JSON body). The kernel backend'sAuthenticationErroralready carried the detail but had nocause, so the underlying binding error was lost.THTTPExceptionnow appends the server detail after the unchanged prefix, e.g.Received a response with a bad HTTP status code: 403: Invalid access token.. The detail comes fromx-thriftserver-error-message,x-databricks-error-or-redirect-messageorx-databricks-reason-phrase. When none of those headers is present, it falls back to the response body (JSONmessage, else plain text; HTML and binary bodies are skipped), capped at 500 characters. When the body is read,err.responseis replaced with an equivalent buffered response, so it stays readable. The 200 path is unchanged. Kernel errors decoded from the napi binding now keep the binding error as a non-enumerablecause(same descriptor as native{ cause }, internal__databricks_error__:prefix stripped). The Reyden KP001 fallback now appends the Thrift rejection to the end of that cause chain instead of dropping it. Error classes,statusCode,sqlStateandkernelMetadataare unchanged.Verification: new unit tests drive
ThriftHttpConnectionagainst a local HTTP server (header precedence, JSON/text body fallback, HTML/binary skipping, length cap, body still readable) and cover the kernelcausefor every mapped class plus the Reyden chain.npm test, lint, type-check and build pass. Live against a warehouse, using an invalid token and a nonexistent warehouse, both backends now surface the server detail (Thrift 404:ENDPOINT_NOT_FOUND: SQL warehouse … does not exist), and the kernel errors carrycause. The new e2e specs (tests/e2e/connection_failure.test.ts,tests/e2e/kernel/connection-failure-e2e.test.ts) pass on this branch and fail onmain. No perf impact: the success path is byte-for-byte unchanged and the new work only runs after the server has already returned an error.Not addressed here: the coverage test also expects a
causeon the Thrift error. On Thrift,THTTPExceptionis itself the transport error (it carriesstatusCodeandresponse), so there is no underlying error to preserve, and adding a synthetic one would only duplicate the message.Fixes #467
This pull request and its description were written by Isaac.