Repository navigation
fix: keep DECIMAL parameter scale on the Thrift backend - #585
Conversation
a5086c2 to
c53d06b
Compare
|
@bot Finding 3 (lib/DBSQLParameter.ts:144): Finding 5 (lib/DBSQLParameter.ts:67): |
|
Pushed Started from c53d06b. Fixed all three. F2: Int64 values are now converted to strings exactly via toBuffer().readBigInt64BE, so 2^53+1 gives "9007199254740993", not Infinity. This also fixes BIGINT, which sent "Infinity" before. F3: bare DECIMAL/DEC/NUMERIC types are matched case-insensitively after trimming; qualified ones like NUMERIC(10,2) are left as given. F5: the doc now says the kernel forwards the type unchanged and SEA binds DECIMAL(10,0) and rounds. Added unit tests and a CHANGELOG note. Unit tests, type-check and prettier pass. Lint couldn't run: ESLint picked up a parent-directory .eslintrc that needs airbnb-base, which isn't installed. The e2e tests weren't run (no warehouse here). |
Signed-off-by: peco-engineer-bot[bot] <peco-engineer-bot[bot]@users.noreply.github.com>
A DBSQLParameter typed DECIMAL was sent to the Thrift server as the bare type "DECIMAL", which the server binds as a scale-0 decimal, so "123.45" came back as 123. toSparkParameter now qualifies it as DECIMAL(p,s) derived from the value, so both backends share the derivation the kernel path already did. Exponent forms such as 1e-7 are rewritten as plain numerals, and non-numeric or >38-digit values throw a ParameterError. Fixes #474 Co-authored-by: Isaac <no-reply@databricks.com> Signed-off-by: Vu Anh Phung <vu.phung@databricks.com>
Signed-off-by: peco-engineer-bot[bot] <peco-engineer-bot[bot]@users.noreply.github.com> Signed-off-by: Vu Anh Phung <vu.phung@databricks.com>
2cad597 to
6815355
Compare
On the Thrift backend, a
DBSQLParametertypedDECIMALwas sent with the bare typeDECIMAL. The server binds that as a scale-0 decimal, so"123.45"came back as123(decimal(3,0)),"007.50"was rounded to8, and values with more than 10 integer digits failed with a precision error. The kernel backend was already lossless becauseKernelPositionalParamsderivedDECIMAL(p,s)itself.toSparkParameternow does that derivation for both backends, and the kernel's copy is removed. An explicitly typed bareDECIMAL(any case) becomesDECIMAL(p,s), where precision is significant integer digits plus scale. Plain numerals are sent unchanged. Exponent forms such as1e-7, whichNumber#toStringproduces for small and large numbers, are rewritten as plain numerals. The kernel used to reject these, and Thrift silently truncated them. Non-numeric values and values needing more than 38 digits throw aParameterErrorat bind time. On Thrift, those used to fail on the server or truncate. Inferred types and explicitly qualified types such asDECIMAL(10,2)are unchanged.Verification: new unit tests cover strings, numbers, bigint/Int64, negatives, leading zeros, exponent forms, the 38-digit limit and invalid input. They fail on
mainand pass here. Against a live warehouse,SELECT :v AS v, typeof(:v)now returns the same type on Thrift and the kernel for 22 cases, including"123.45"→decimal(5,2)/123.45on Thrift. WithpreserveBigNumericPrecision, 38-digit values round-trip exactly. New e2e tests inquery_parameters.test.ts(Thrift) andkernel/execution-e2e.test.ts(kernel) pass live, and the Thrift one fails onmain. No extra RPCs. In atoSparkParametermicro-benchmark (medians of 3 fresh-process runs), the cost for non-DECIMAL parameters is unchanged (17.4 vs 17.1 ns per call). An explicit DECIMAL parameter costs about 150 ns for the derivation.Fixes #474
This pull request and its description were written by Isaac.