Skip to content

Data files written by the pyarrow writer never report nan_value_counts, so NaN-aware scan pruning is silently unavailable #4396

Open

Description

@AP24110011695

PyIceberg version

main @ fce8185 (2026-11-15)

Describe the bug

data_file_statistics_from_parquet_metadata returns nan_value_counts={} unconditionally — the dict is initialized and returned but never populated. Data files written by the pyarrow writer (and files imported through parquet_file_to_data_file) therefore never report NaN counts, while the DataFile schema and the scan-planning metrics evaluator both fully support them.

Probe on a table containing 2 NaNs per column (written with pq.write_table, metadata read back and passed through the function):

null_value_counts: {1: 0, 2: 1}      # correct
nan_value_counts: {}                 # always empty
# true non-null NaNs: field 1 = 2, field 2 = 2

Consequence — conservative, not wrong results: _InclusiveMetricsEvaluator uses nan_counts for pruning (visit_is_nan: nan_counts.get(field_id) == 0 → ROWS_CANNOT_MATCH; _contains_nans_only for not_nan/less_than/greater_than). With empty nan counts every check degrades to ROWS_MIGHT_MATCH, so:

  • is_nan filters can never prune files, even files with zero NaNs in that column are always scanned,
  • not_nan/range filters never prune all-NaN or NaN-heavy files (the sensor-data and ML-feature use case),
  • files written by parquet-mr/Java do carry nan_value_counts and get that pruning — the gap only affects pyiceberg-written tables.

This looks adjacent to #4313 (drop bounds when a row group has no min/max) and #4385 (omit incomplete null counts): the same statistics-completeness family, except NaN counts are not partially accumulated — they are never accumulated at all.

Fix options

pyarrow 26's ColumnStatistics exposes no nan_count/has_nan_count, so a row-group-level accumulation mirroring #4385 is not possible with the public API today:

A. Compute at write time — the writer sees the arrow table before close(); NaN counts per float column are one vectorized op (pa.compute.is_nan(col) sums to the non-null NaN count, which matches the spec's NaN-count semantics) and can be threaded into data_file_statistics_from_parquet_metadata (or merged into its returned DataFileStatistics).

B. Upstream — expose nan_count from the parquet statistics in pyarrow (the thrift Statistics message carries it since parquet-format 2.5), then mirror the #4385 accumulation including a completeness guard.

Happy to send A as a PR if that design is preferred.

AI disclosure: this report was compiled with AI assistance; every claim above is reproducible from the probe snippet and the cited code paths (pyiceberg/io/pyarrow.py data_file_statistics_from_parquet_metadata, pyiceberg/expressions/visitors.py _InclusiveMetricsEvaluator).

Activity

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions