You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
{{ message }}
Repository navigation
Data files written by the pyarrow writer never report nan_value_counts, so NaN-aware scan pruning is silently unavailable #4396
data_file_statistics_from_parquet_metadata returns nan_value_counts={} unconditionally — the dict is initialized and returned but never populated. Data files written by the pyarrow writer (and files imported through parquet_file_to_data_file) therefore never report NaN counts, while the DataFile schema and the scan-planning metrics evaluator both fully support them.
Probe on a table containing 2 NaNs per column (written with pq.write_table, metadata read back and passed through the function):
Consequence — conservative, not wrong results: _InclusiveMetricsEvaluator uses nan_counts for pruning (visit_is_nan: nan_counts.get(field_id) == 0 → ROWS_CANNOT_MATCH; _contains_nans_only for not_nan/less_than/greater_than). With empty nan counts every check degrades to ROWS_MIGHT_MATCH, so:
is_nan filters can never prune files, even files with zero NaNs in that column are always scanned,
not_nan/range filters never prune all-NaN or NaN-heavy files (the sensor-data and ML-feature use case),
files written by parquet-mr/Java do carry nan_value_counts and get that pruning — the gap only affects pyiceberg-written tables.
This looks adjacent to #4313 (drop bounds when a row group has no min/max) and #4385 (omit incomplete null counts): the same statistics-completeness family, except NaN counts are not partially accumulated — they are never accumulated at all.
Fix options
pyarrow 26's ColumnStatistics exposes no nan_count/has_nan_count, so a row-group-level accumulation mirroring #4385 is not possible with the public API today:
A. Compute at write time — the writer sees the arrow table before close(); NaN counts per float column are one vectorized op (pa.compute.is_nan(col) sums to the non-null NaN count, which matches the spec's NaN-count semantics) and can be threaded into data_file_statistics_from_parquet_metadata (or merged into its returned DataFileStatistics).
B. Upstream — expose nan_count from the parquet statistics in pyarrow (the thrift Statistics message carries it since parquet-format 2.5), then mirror the #4385 accumulation including a completeness guard.
Happy to send A as a PR if that design is preferred.
AI disclosure: this report was compiled with AI assistance; every claim above is reproducible from the probe snippet and the cited code paths (pyiceberg/io/pyarrow.pydata_file_statistics_from_parquet_metadata, pyiceberg/expressions/visitors.py_InclusiveMetricsEvaluator).
Activity