[vuanhphung]

What type of PR is this?

• Other (performance)

Description

fetchall() and large fetchmany() calls convert the Arrow result with one to_pandas() call and one DataFrame.to_numpy(dtype=object) over the whole table, so a DataFrame, a full object array and the final Row list are all alive at the peak. On a 5M-row mixed-type result (INT/BIGINT with NULLs, DOUBLE, STRING, DECIMAL, DATE, TIMESTAMP, BOOLEAN, ARRAY), conversion peaked at about +5.4 GB RSS, while the returned rows alone hold +3.85 GB. The process also kept about 0.9 GB more after fetchall() returned.

This converts 100,000 rows per to_pandas() call. That is the default arraysize, so fetchone() and default-size fetchmany() take the same single-call path as before. Columns whose pandas conversion depends on the column's other values are still converted once over the whole table. pyarrow picks the element dtype of a list or map from the whole column (one NULL element turns every ARRAY<BIGINT> value into a float64 array) and of a struct per Arrow chunk. Their values are written into each slice's object array over null placeholders. The placeholders keep the slice's DataFrame block layout the same as before, which matters because pandas < 2.2 returns durations differently from a single-block DataFrame. Returned values and Python types are unchanged.

Related open PRs: #982 and #1007 call _convert_arrow_table unchanged and merge cleanly with this (#982 conflicts only in CHANGELOG, as it already does with main). #979 (NaN vs NULL) and #960 (nested conversion) edit the same function body. Whichever lands second needs a small rebase: #979's NaN restore moves into to_object_array, since float columns are converted per slice, and #960's Arrow-native nested conversion replaces the whole-column to_object_array(table_renamed.select(whole)) call. This PR does not change either behavior.

How is this tested?

• Unit tests
• E2E Tests
• Manually

New unit tests check that sliced conversion matches whole-table conversion value by value and type by type (nested, struct, duration, tz timestamps, decimals, duplicate column names). They also check that slices are bounded and that small or nested-only tables are not sliced. The test file passes on pandas 1.5.3/pyarrow 14, 2.2.3/18, 3.1.1/23 and 3.1.1/26. A separate script compared main and this branch on about 2,900 table layouts and slice sizes under the same four version pairs, and on live Thrift inline, CloudFetch and kernel results. Every value and type was identical. Slicing the nested columns as well is caught as a difference.

Performance was measured on the mixed-type query above (FROM range(N), 9 columns), with a fresh process per run, main and this branch interleaved, median of 5. "Local" times _convert_arrow_table alone on the same table read from an Arrow file. "Live" is execute() + fetchall() against a SQL warehouse, with peak RSS measured above the pre-query baseline. p95 values are within 2% of the medians.

	main	this PR
local 5M rows: peak RSS, wall	+5.44 GB, 24.7 s	+3.90 GB, 24.0 s
local 20M rows: peak RSS, wall	+17.9 GB, 100.1 s	+15.2 GB, 97.6 s
live Thrift CloudFetch 5M: peak RSS, fetchall()	+5.89 GB, 25.2 s	+4.42 GB, 25.0 s
live Thrift inline 5M	+5.63 GB, 29.8 s	+4.27 GB, 29.8 s
live kernel 5M	+5.00 GB, 25.1 s	+4.44 GB, 23.6 s

At 1M rows the peak drops 4–7% on every backend, with no change in time. _disable_pandas=True is unaffected. Slice sizes from 50k to 1M rows gave the same wall time. With this change the peak is within 1% of what the returned rows hold, so what remains is the Row list itself.

This pull request and its description were written by Isaac.

[peco-review-bot]

✅ No issues identified by the review bot.