Skip to content

fix: keep DECIMAL parameter scale on the Thrift backend - #585

Merged
vuanhphung merged 2 commits into
mainfrom
fix/issue-474-decimal-param-scale
Nov 12, 2026
Merged

vuanhphung merged 2 commits into
mainfrom
fix/issue-474-decimal-param-scale

Conversation

@vuanhphung

Copy link
Copy Markdown
Collaborator

On the Thrift backend, a DBSQLParameter typed DECIMAL was sent with the bare type DECIMAL. The server binds that as a scale-0 decimal, so "123.45" came back as 123 (decimal(3,0)), "007.50" was rounded to 8, and values with more than 10 integer digits failed with a precision error. The kernel backend was already lossless because KernelPositionalParams derived DECIMAL(p,s) itself.

toSparkParameter now does that derivation for both backends, and the kernel's copy is removed. An explicitly typed bare DECIMAL (any case) becomes DECIMAL(p,s), where precision is significant integer digits plus scale. Plain numerals are sent unchanged. Exponent forms such as 1e-7, which Number#toString produces for small and large numbers, are rewritten as plain numerals. The kernel used to reject these, and Thrift silently truncated them. Non-numeric values and values needing more than 38 digits throw a ParameterError at bind time. On Thrift, those used to fail on the server or truncate. Inferred types and explicitly qualified types such as DECIMAL(10,2) are unchanged.

Verification: new unit tests cover strings, numbers, bigint/Int64, negatives, leading zeros, exponent forms, the 38-digit limit and invalid input. They fail on main and pass here. Against a live warehouse, SELECT :v AS v, typeof(:v) now returns the same type on Thrift and the kernel for 22 cases, including "123.45" → decimal(5,2) / 123.45 on Thrift. With preserveBigNumericPrecision, 38-digit values round-trip exactly. New e2e tests in query_parameters.test.ts (Thrift) and kernel/execution-e2e.test.ts (kernel) pass live, and the Thrift one fails on main. No extra RPCs. In a toSparkParameter micro-benchmark (medians of 3 fresh-process runs), the cost for non-DECIMAL parameters is unchanged (17.4 vs 17.1 ns per call). An explicit DECIMAL parameter costs about 150 ns for the derivation.

Fixes #474

This pull request and its description were written by Isaac.

@peco-review-bot peco-review-bot Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

✅ No issues identified by the review bot.

@vuanhphung
vuanhphung force-pushed the fix/issue-474-decimal-param-scale branch from a5086c2 to c53d06b Compare November 12, 2026 07:57

@peco-review-bot peco-review-bot Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

✅ No issues identified by the review bot.

@vuanhphung

Copy link
Copy Markdown
Collaborator Author

@bot
Finding 2 (lib/DBSQLParameter.ts:189):
node-int64's toString() returns "Infinity" for values ≥ 2^53, so new Int64(0x200000, 1) typed DECIMAL now throws "Infinity" is not a decimal number. Large values are the main reason to use Int64 — can we stringify it exactly (buffer → BigInt), or at least add a test and a clearer error?

Finding 3 (lib/DBSQLParameter.ts:144):
NUMERIC / DEC (reachable from JS or via a cast) still go out bare and get truncated to scale 0, the same bug as #474. Consider matching /^(DECIMAL|DEC|NUMERIC)$/i on the trimmed type.

Finding 5 (lib/DBSQLParameter.ts:67):
nit: the Node driver uses the kernel's raw-param path, which forwards params verbatim. A bare DECIMAL isn't rejected there; SEA binds it as DECIMAL(10,0) and rounds, same as hrift.

@peco-engineer-bot

Copy link
Copy Markdown
Contributor

Pushed 2cad597 to fix/issue-474-decimal-param-scale addressing your request.

Started from c53d06b. Fixed all three. F2: Int64 values are now converted to strings exactly via toBuffer().readBigInt64BE, so 2^53+1 gives "9007199254740993", not Infinity. This also fixes BIGINT, which sent "Infinity" before. F3: bare DECIMAL/DEC/NUMERIC types are matched case-insensitively after trimming; qualified ones like NUMERIC(10,2) are left as given. F5: the doc now says the kernel forwards the type unchanged and SEA binds DECIMAL(10,0) and rounds. Added unit tests and a CHANGELOG note. Unit tests, type-check and prettier pass. Lint couldn't run: ESLint picked up a parent-directory .eslintrc that needs airbnb-base, which isn't installed. The e2e tests weren't run (no warehouse here).

peco-engineer-bot Bot added a commit that referenced this pull request Nov 12, 2026
Signed-off-by: peco-engineer-bot[bot] <peco-engineer-bot[bot]@users.noreply.github.com>
vuanhphung and others added 2 commits November 12, 2026 08:17
A DBSQLParameter typed DECIMAL was sent to the Thrift server as the bare
type "DECIMAL", which the server binds as a scale-0 decimal, so "123.45"
came back as 123. toSparkParameter now qualifies it as DECIMAL(p,s)
derived from the value, so both backends share the derivation the kernel
path already did. Exponent forms such as 1e-7 are rewritten as plain
numerals, and non-numeric or >38-digit values throw a ParameterError.

Fixes #474

Co-authored-by: Isaac <no-reply@databricks.com>
Signed-off-by: Vu Anh Phung <vu.phung@databricks.com>
Signed-off-by: peco-engineer-bot[bot] <peco-engineer-bot[bot]@users.noreply.github.com>
Signed-off-by: Vu Anh Phung <vu.phung@databricks.com>
@vuanhphung
vuanhphung force-pushed the fix/issue-474-decimal-param-scale branch from 2cad597 to 6815355 Compare November 12, 2026 08:17

@peco-review-bot peco-review-bot Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

✅ No issues identified by the review bot.

This branch was successfully deployed

1 active deployment
azure-prod — 68153551 Deployed Nov 12, 2026 by vuanhphung via e2e-test (22.23.4, 22_23_4) #1937
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

[coverage] Conformance findings: PARAMQUERY-022,PARAMQUERY-023

1 participant