Repository navigation
Conversation
manuzhang
reviewed
Nov 12, 2026
|
|
||
| Result<bool> In(const std::shared_ptr<Bound>& expr, | ||
| const BoundSetPredicate::LiteralSet& values) override { | ||
| return AnyMatch(expr, [&](const Literal& item) { return values.contains(item); }); |
Member
There was a problem hiding this comment.
This will return false for NaN.
Collaborator
Author
There was a problem hiding this comment.
Literal uses a custom comparison, so contains can match the same NaN literal in the set, I'm not sure if this is your concern.
manuzhang
reviewed
Nov 12, 2026
| Result<bool> StartsWith(const std::shared_ptr<Bound>& expr, | ||
| const Literal& value) override { | ||
| return AnyMatch(expr, [&](const Literal& item) { | ||
| return std::get<std::string>(item.value()) |
Member
There was a problem hiding this comment.
This can thrown uncaught if the variant is not a string.
Collaborator
Author
There was a problem hiding this comment.
Binder already checks that the term is a string and converts the prefix literal to the same type. ParquetMetricsRowGroupFilter also relies on this and uses std::getstd::string for StartsWith and NotStartsWith.
Collaborator
Author
There was a problem hiding this comment.
After a second check, it seems users can still manually create an invalid BoundPredicate and bypass Binder's checks. But this is rare, we can address this for both filters in a follow-up if needed.
Use column dictionaries to prune row groups after footer-statistics filtering. Support comparison, set, prefix and NaN predicates, and cache decoded dictionaries within each row group. Keep row groups when encoding metadata cannot rule out plain fallback or field IDs and types cannot be resolved safely. Preserve file-relative row positions and row IDs after pruning. Share physical-value conversion with Parquet metrics and add tests for predicate handling, type promotion, mixed encoding, splits and invalid dictionary pages.
Read the byte range between dictionary_page_offset and data_page_offset instead of loading the full column chunk during planning. Validate the range and keep using Arrow for page decompression. Skip dictionary pruning for encrypted columns so the normal reader handles them. Treat unreadable optional statistics as possible nulls instead of failing the scan. Keep errors for missing or truncated dictionary pages, and explain why rewritten null branches remain inclusive. Add regression tests for planning read ranges with multi-megabyte data pages, invalid offsets, unreadable statistics, signed zeros, and compressed V1/V2 pages. The complete Parquet suite passes all 213 tests, and pre-commit passes.
Use metadata-only projection to keep output-column prefetch out of the dictionary I/O assertion. Add float/double IN and NOT IN coverage with NaN literals.
zhjwpku
force-pushed
the
feat/parquet-dictionary-row-group-filter
branch
from
November 12, 2026 10:47
d1de99e to
a81101f
Compare
This branch has not been deployed
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Use column dictionaries to prune row groups after footer-statistics filtering. Support comparison, set, prefix and NaN predicates, and cache decoded dictionaries within each row group.
Keep row groups when encoding metadata cannot rule out plain fallback or field IDs and types cannot be resolved safely. Preserve file-relative row positions and row IDs after pruning.
Share physical-value conversion with Parquet metrics and add tests for predicate handling, type promotion, mixed encoding, splits and invalid dictionary pages.