Skip to content

Read a list or map column referenced by a null filter - #4400

Open
gedaliakliger wants to merge 1 commit into
apache:mainfrom
gedaliakliger:fix/4399-null-filter-list-map
Open

gedaliakliger wants to merge 1 commit into
apache:mainfrom
gedaliakliger:fix/4399-null-filter-list-map

Conversation

@gedaliakliger

Copy link
Copy Markdown

Closes #4399

Rationale for this change

A scan whose row_filter tests a list or map column for null (tags is null, props is not null) fails in to_arrow() with ValueError: Cannot explicitly project List or Map types.

ArrowScan._projected_field_ids leaves list and map IDs out of the projected schema's IDs, because prune_columns rebuilds those columns from their nested fields, but it adds the filter's field IDs as they are. A null test on a list or map column adds the column's own ID, and prune_columns rejects it. This change projects the nested fields of such a column instead, the same way a selected list or map column is read.

Are these changes tested?

Yes. test_projection_null_filter_on_list_and_map in tests/io/test_pyarrow.py runs IsNull and NotNull on a list, a map and a list of structs, with the column selected and with only id selected. It fails on main with the ValueError above and passes with this change.

Ran locally: pytest tests/io/test_pyarrow.py -m unit, and the ruff, ruff-format and mypy hooks on the two changed files. The rest of the suite and the integration tests are left to CI.

Are there any user-facing changes?

IS NULL and IS NOT NULL filters on list and map columns return the matching rows instead of raising.

A scan filter that tests a list or map column for null added the column's own
field ID to the projected IDs, and prune_columns rejects a list or map selected
by its own ID. Project the column's nested fields instead.

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

row_filter with IS NULL / IS NOT NULL on a list or map column raises "Cannot explicitly project List or Map types"

1 participant