Skip to content

Reject colliding names in case-insensitive Schema lookups - #4389

Open
rahulsmahadev wants to merge 1 commit into
apache:mainfrom
rahulsmahadev:rahul/case-insensitive-name-collision
Open

rahulsmahadev wants to merge 1 commit into
apache:mainfrom
rahulsmahadev:rahul/case-insensitive-name-collision

Conversation

@rahulsmahadev

Copy link
Copy Markdown
Contributor

Rationale for this change

A case-insensitive lookup on a schema with two columns whose names differ only by case uses the last one. Schema._lazy_name_to_id_lower builds the lower-case index with a dict comprehension, so id overwrites ID, and Schema.find_field and Schema.select use that index for case-insensitive name lookups:

# columns ID (field 1) and id (field 2); rows (ID, id): (5, 7) and (6, 5)
tbl.scan(row_filter=EqualTo("ID", 5), case_sensitive=False).to_arrow()  # returns (6, 5), the row whose id is 5
tbl.delete("ID = 5", case_sensitive=False)  # removes (6, 5); (5, 7) is kept

The same wrong column is used by sort order updates and by schema updates (delete_column("ID") drops id, rename_column("ID", "key") renames id).

Java does not pick a column here. TypeUtil.indexByLowerCaseName throws Cannot build lower case index: ID and id collide, and Schema.caseInsensitiveFindField and SchemaUpdate use that index, so the lookup fails.

This change makes the index raise ValueError: Cannot build lower case index: ID and id collide (the same message as Java) when two different field IDs share a lower-case name. As in Java's Schema lookup, building the lower-case index fails for any requested name when the schema contains a collision, including when the requested name is unambiguous. Names that map to the same field ID, such as the short name of a list element struct field, are not collisions. Case-sensitive lookups do not change. StructType.field_by_name uses a separate lookup and is outside this change; it can still return the first matching field (it serves union_by_name).

Callers that treated any ValueError from a lookup as "not found" (UpdateSchema.add_column, the move methods, the identifier field check on commit, and the metadata column check in scan projection) now use a private lookup that returns None for a missing name and still raises for an ambiguous one. Without updating the add-column check, an index-only fix would let table.update_schema(case_sensitive=False).add_column("Id", LongType()).commit() add a third case variant.

Are these changes tested?

Yes. test_case_insensitive_operations_raise_on_case_variant_columns in tests/table/test_init.py runs a delete, a scan, a sort order update, and schema updates with case_sensitive=False on a table with ID and id, for the memory, sql and sql_without_rowcount catalogs. Each must raise, and the table must be unchanged afterwards. It fails on main, where the delete does not raise. test_case_sensitive_and_unambiguous_operations_are_unchanged checks exact-case lookups on the same table and a case-insensitive delete on a table without a collision; it passes on main and with this change. In tests/test_schema.py, test_schema_case_insensitive_lookup_of_case_variant_names_raises and test_schema_case_insensitive_lookup_of_nested_case_variant_names_raises fail on main with DID NOT RAISE, and test_schema_case_insensitive_lookup_of_list_element_struct_field covers the same-ID short names and passes on main and with this change. test_schema_case_insensitive_lookup_rejects_collision_for_any_name, test_metadata_column_projection_propagates_case_collision and test_identifier_field_lookup_propagates_case_collision cover unambiguous and missing names, the metadata column projection and the identifier field check; they fail on main.

Are there any user-facing changes?

Yes: case-insensitive name lookups through Schema.find_field and Schema.select now raise ValueError when the schema has names that differ only by case, instead of silently using the last column. Case-sensitive lookups are unchanged.

This pull request and its description were written by Isaac.

The lower-case name index kept only the last of two columns whose
names differ only by case (ID and id), so case-insensitive lookups
silently used it. Raise ValueError when two field IDs share a
lower-case name, as Java does, and let the schema update and metadata
column checks propagate that error instead of reading it as a missing
name. Exact-case lookups are unchanged.

Co-authored-by: Isaac <no-reply@databricks.com>
@rahulsmahadev

Copy link
Copy Markdown
Contributor Author

@rambleraptor, this updates the metadata-column lookup you added in #4375 so schema name collisions propagate. @kevinjqliu, you recently worked on Schema behavior in #3958. Could you take a look at the Java-compatible lookup behavior and the caller error handling when you have a chance?

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant