Skip to content

fix(appkit): return numbers and booleans from analytics queries - #709

Merged
fjakobs merged 5 commits into
mainfrom
fjakobs/la-2230-numeric-types
Nov 12, 2026
Merged

fjakobs merged 5 commits into
mainfrom
fjakobs/la-2230-numeric-types

Conversation

@fjakobs

@fjakobs fjakobs commented Nov 6, 2026 •

Copy link
Copy Markdown
Collaborator

Why

  • The type generator types numeric columns as number and BOOLEAN columns as boolean in analytics.d.ts.
  • The /query route returns them as strings. The warehouse's JSON_ARRAY format sends every cell as a string, and the connector passed them through unchanged.
  • So useAnalyticsQuery rows had total_trips: "21932" while the type said number. Apps (and agents writing apps) kept adding their own Number(...) workarounds. The dev-playground did the same in its KPI cards.

Reproduced on e2-dogfood (Serverless Starter Warehouse). This is a query run through the real type generator and the connector's JSON_ARRAY path:

column generated type value before
BIGINT number "42"
DECIMAL(10,2) number "1.50"
INT / SMALLINT / TINYINT number "7" / "3" / "1"
DOUBLE / FLOAT number "2.5" / "1.25"
BOOLEAN boolean "true"

Change

  • The SQL warehouse connector attaches the manifest column types (columnTypes) next to the columnNames it already carries. It does this on JSON rows and on inline Arrow results. Cell values are unchanged at this layer.
  • deliverJsonResult takes a coerceScalars option. With it set, coerceScalarCells converts cells by manifest typeName:
    • BYTE, SHORT, INT, LONG, FLOAT, DOUBLE and DECIMAL become numbers.
    • BOOLEAN becomes a boolean.
    • This happens on both the native JSON_ARRAY path and the ARROW_STREAM fallback, so the two paths still match.
  • Only the /query route sets coerceScalars. The /metric route keeps its documented string | null wire values, so metric-view types and docs don't change.
  • NULL stays null. DATE and TIMESTAMP stay ISO strings. STRUCT, ARRAY and MAP values stay as delivered.
  • DOUBLE/FLOAT NaN, Infinity and -Infinity stay the strings "NaN", "Infinity" and "-Infinity". JSON has no number for them, and converting them would put null on the wire, the same as SQL NULL.
  • Dev-playground: remove toNum and the other string-coercion guards in kpi-cards.tsx.
  • Docs: add a "Result value types" note to type-generation.md.

Trade-offs

  • A BIGINT beyond Number.MAX_SAFE_INTEGER and a DECIMAL beyond about 15 significant digits lose precision as JS numbers. DECIMAL also loses trailing zeros ("1.50" becomes 1.5). The docs say to CAST(... AS STRING) for exact values. The generated type then becomes string, so it stays truthful.
  • This changes runtime values on /query: they become what the generated types already claimed. Code that relied on getting strings at runtime will see numbers.
  • Not in scope: nested types (STRUCT, ARRAY, MAP) still arrive as JSON strings although they are typed as unknown[] or Record<string, unknown>.

Testing

  • Unit tests:
    • connector: columnNames and columnTypes are attached.
    • deliverJsonResult: native path, Reyden fallback using the captured fixture, and no conversion without the option.
    • /query route: converts values.
    • /metric route: keeps strings.
  • pnpm build, pnpm docs:build, pnpm check:fix, pnpm -r typecheck and pnpm test all pass (5934 tests).
  • End to end, dev-playground against e2-dogfood:
    • POST /api/analytics/query/dashboard_kpis now returns {"total_trips":21932,"avg_fare":12.35,…}.
    • Sparkline rows have numeric values and trip_date stays "2015-12-31".
    • POST /api/analytics/metric/revenue still returns {"arr":"1.604847408E9"}.
    • The smart-dashboard KPI cards, sparklines and trends render correctly without toNum.
  • After review: a query with every scalar type, typed NULLs and DOUBLE NaN/±Infinity through /query, on both a standard serverless warehouse and a Reyden (REALTIME) warehouse. Every value matches its generated type except the three non-finite strings, and NULLs are null. Also ran the ARROW_STREAM fallback with live Reyden Arrow bytes (the JSON_ARRAY rejection was simulated, since the warehouse now serves JSON_ARRAY): the same values as native JSON_ARRAY.

This pull request and its description were written by Isaac.

@fjakobs
fjakobs requested a review from a team as a code owner November 6, 2026 13:57
@github-actions

github-actions Bot commented Nov 6, 2026 •

Copy link
Copy Markdown
Contributor

📦 Bundle size report

Compared against bundle-size-baseline.json (main).

@databricks/appkit

npm tarball (packed): 1.3 MB (+2.3 KB) — gzipped download (dist + bin; excludes release-only docs/NOTICE).

dist raw gzip
JS (runtime) 1.3 MB (+2.6 KB) 479 KB (+938 B)
Type declarations 470 KB 169 KB
Source maps 2.6 MB (+4.4 KB) 889 KB (+1.4 KB)
Other 11 KB 3.7 KB
Total 4.4 MB (+6.9 KB) 1.5 MB (+2.3 KB)
Per-entry composition (own code — deps external (as shipped))
Entry Initial (gz) Lazy (gz) Total (gz) node_modules (min) Own code (min)
. 108 KB (+257 B) 2.5 KB 110 KB (+257 B) external 348 KB (+645 B)
./beta 113 KB (+32 B) 485 B 114 KB (+32 B) external 339 KB (+154 B)
./testing 46 KB 33 KB 79 KB external 229 KB
./tsdown 520 B 0 B 520 B external 813 B
./type-generator 26 KB 0 B 26 KB external 74 KB

Chunks:

Entry Chunk Load Size (gz)
. index.js initial 103 KB
. utils.js initial 4.6 KB
. remote-tunnel-manager.js lazy 2.5 KB
./beta beta.js initial 93 KB
./beta modular.js initial 6.0 KB
./beta stream-manager.js initial 5.9 KB
./beta service-context.js initial 4.3 KB
./beta databricks.js initial 3.3 KB
./beta client.js initial 649 B
./beta index.js initial 20 B
./beta supervisor-api.js lazy 193 B
./beta databricks.js lazy 178 B
./beta index.js lazy 114 B
./testing manifest.js initial 33 KB
./testing index.js initial 10 KB
./testing wide-event-emitter.js initial 2.9 KB
./testing index.js lazy 29 KB
./testing remote-tunnel-manager.js lazy 2.5 KB
./testing utils.js lazy 1.8 KB
./tsdown index.js initial 520 B
./type-generator index.js initial 26 KB

@databricks/appkit-ui

npm tarball (packed): 406 KB (-5 B) — gzipped download (dist + bin; excludes release-only docs/NOTICE).

dist raw gzip
JS (runtime) 436 KB 150 KB
Type declarations 271 KB 100 KB (-1 B)
Source maps 862 KB 291 KB
CSS 16 KB 3.2 KB
Total 1.5 MB 544 KB (-1 B)
Per-entry composition (consumer bundle — deps bundled, peerDeps external)
Entry Initial (gz) Lazy (gz) Total (gz) node_modules (min) Own code (min)
./js 5.3 KB 49 KB 55 KB 208 KB 14 KB
./js/beta 2.1 KB 0 B 2.1 KB 0 B 5.0 KB
./react 440 KB 49 KB 489 KB 1.4 MB 178 KB
./react/beta 14 KB 0 B 14 KB 25 KB 16 KB

Chunks:

Entry Chunk Load Size (gz)
./js index.js initial 5.2 KB
./js chunk initial 120 B
./js apache-arrow lazy 49 KB
./js/beta beta.js initial 2.1 KB
./react index.js initial 438 KB
./react tslib initial 2.1 KB
./react apache-arrow lazy 49 KB
./react/beta beta.js initial 14 KB

@github-actions

github-actions Bot commented Nov 6, 2026 •

Copy link
Copy Markdown
Contributor

🤖 AppKit PR bot

🔬 Run evals

Start an eval for this PR from the evals-monitor app: Go to Evals Monitor →

📦 Try this PR's app template

Scaffolds a new app from this PR's SDK build. Run it in any folder (requires the GitHub CLI — gh auth login — and the Databricks CLI):

gh run download 43204330191 -R databricks/appkit -n appkit-template-0.94.0-pr.b55196f-fjakobs-la-2230-numeric-types-709 -D appkit-pr-709 \
  && unzip -o "appkit-pr-709/appkit-template-0.94.0-pr.b55196f-fjakobs-la-2230-numeric-types-709.zip" -d "appkit-pr-709" \
  && databricks apps init --template "appkit-pr-709"

The template pins @databricks/appkit and @databricks/appkit-ui to tarballs built from this branch, so the scaffolded app runs against this PR's code.

@fjakobs
fjakobs requested a review from pffigueiredo November 6, 2026 15:47
@fjakobs

fjakobs commented Nov 6, 2026

Copy link
Copy Markdown
Collaborator Author

@pffigueiredo thanks for the review. I replied in your thread, which is resolved and so collapsed: #709 (comment)

Short version: I kept plain Number() / === "true" instead of zod. .parse throws on the "NaN" / "Infinity" / "-Infinity" strings the warehouse really returns for DOUBLE/FLOAT, and that would fail the whole query. Your .finite() did point at a real bug, though: those values were reaching the client as null, the same as SQL NULL. 4f6d390 keeps them as the warehouse strings and adds tests and a docs note. Could you take another look?

@fjakobs

fjakobs commented Nov 9, 2026

Copy link
Copy Markdown
Collaborator Author

@pffigueiredo friendly ping: I replied to your zod suggestion in the (resolved) thread and pushed a follow-up commit for the NaN/Infinity case it uncovered. Could you take another look and update your review?

fjakobs and others added 2 commits November 9, 2026 15:22
JSON_ARRAY returns every cell as a string, so analytics query rows carried
"42" for BIGINT, "1.50" for DECIMAL and "true" for BOOLEAN while the
generated query types promise number and boolean.

The SQL warehouse connector now attaches the manifest column types next to
the column names it already carries, and the /query route converts numeric
and BOOLEAN cells on both the native JSON_ARRAY path and the ARROW_STREAM
fallback. NULL, date/time and nested values are unchanged, and the /metric
route keeps its documented string | null wire values.

Remove the dev-playground's number-coercion workarounds in the KPI cards.

Co-authored-by: Isaac <no-reply@databricks.com>
Signed-off-by: Fabian Jakobs <fabian.jakobs@databricks.com>
Under JSON_ARRAY the warehouse returns DOUBLE and FLOAT NaN and ±Infinity
as "NaN", "Infinity" and "-Infinity". The /query coercion turned them into
NaN/Infinity numbers, which the SSE JSON then sent as null, the same value
as SQL NULL. Numeric cells that don't parse to a finite number now keep the
warehouse string, since JSON has no number for them.

Co-authored-by: Isaac <no-reply@databricks.com>
Signed-off-by: Fabian Jakobs <fabian.jakobs@databricks.com>
@fjakobs
fjakobs force-pushed the fjakobs/la-2230-numeric-types branch from 14d8afc to 623aa14 Compare November 9, 2026 15:25
@fjakobs

fjakobs commented Nov 9, 2026

Copy link
Copy Markdown
Collaborator Author

@MarioCadenas could you review this when you get a chance? It needs a code-owner approval; pffigueiredo has approved, CI is green and it's up to date with main.

@fjakobs
fjakobs merged commit bc36740 into main Nov 12, 2026
12 checks passed
@fjakobs
fjakobs deleted the fjakobs/la-2230-numeric-types branch November 12, 2026 13:19
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

4 participants