Repository navigation
Read a list or map column referenced by a null filter - #4400
Open
gedaliakliger wants to merge 1 commit into
Open
gedaliakliger wants to merge 1 commit into
gedaliakliger wants to merge 1 commit into
Conversation
A scan filter that tests a list or map column for null added the column's own field ID to the projected IDs, and prune_columns rejects a list or map selected by its own ID. Project the column's nested fields instead.
This branch has not been deployed
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Closes #4399
Rationale for this change
A scan whose
row_filtertests a list or map column for null (tags is null,props is not null) fails into_arrow()withValueError: Cannot explicitly project List or Map types.ArrowScan._projected_field_idsleaves list and map IDs out of the projected schema's IDs, becauseprune_columnsrebuilds those columns from their nested fields, but it adds the filter's field IDs as they are. A null test on a list or map column adds the column's own ID, andprune_columnsrejects it. This change projects the nested fields of such a column instead, the same way a selected list or map column is read.Are these changes tested?
Yes.
test_projection_null_filter_on_list_and_mapintests/io/test_pyarrow.pyrunsIsNullandNotNullon a list, a map and a list of structs, with the column selected and with onlyidselected. It fails onmainwith theValueErrorabove and passes with this change.Ran locally:
pytest tests/io/test_pyarrow.py -m unit, and theruff,ruff-formatandmypyhooks on the two changed files. The rest of the suite and the integration tests are left to CI.Are there any user-facing changes?
IS NULLandIS NOT NULLfilters on list and map columns return the matching rows instead of raising.