Skip to content

fix(pyarrow): omit incomplete parquet null counts - #4385

Merged
Fokko merged 1 commit into
apache:mainfrom
iceberg-kitten:fix/pyarrow-incomplete-parquet-null-counts-20261115
Nov 16, 2026
Merged

Fokko merged 1 commit into
apache:mainfrom
iceberg-kitten:fix/pyarrow-incomplete-parquet-null-counts-20261115

Conversation

@iceberg-kitten

Copy link
Copy Markdown
Contributor

What

  • Omit file-level null_value_counts when any Parquet row group has no null count.

Why

  • Adding only known row-group counts can undercount nulls and let an IS NULL scan skip a matching file.
  • The Iceberg spec defines file-level null counts. Java keeps the total unknown if any chunk count is missing.

Implementation

  • Track fields with incomplete counts and omit their partial totals after scanning all row groups.

@Fokko Fokko left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM! Thanks @iceberg-kitten 🚀 Verified locally: the new test fails on main (reports a partial {1: 0} count) and passes with the fix. Matches the Java behavior in ParquetMetrics where an unknown chunk count propagates through addNullCount and the entry is omitted. All 20 tests in test_pyarrow_stats.py pass, ruff is clean.

@Fokko
Fokko added this pull request to the merge queue Nov 16, 2026
Merged via the queue into apache:main with commit 33b4ce5 Nov 16, 2026
20 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants