Repository navigation
Reject colliding names in case-insensitive Schema lookups - #4389
Open
rahulsmahadev wants to merge 1 commit into
Open
rahulsmahadev wants to merge 1 commit into
rahulsmahadev wants to merge 1 commit into
Conversation
The lower-case name index kept only the last of two columns whose names differ only by case (ID and id), so case-insensitive lookups silently used it. Raise ValueError when two field IDs share a lower-case name, as Java does, and let the schema update and metadata column checks propagate that error instead of reading it as a missing name. Exact-case lookups are unchanged. Co-authored-by: Isaac <no-reply@databricks.com>
Contributor
Author
|
@rambleraptor, this updates the metadata-column lookup you added in #4375 so schema name collisions propagate. @kevinjqliu, you recently worked on Schema behavior in #3958. Could you take a look at the Java-compatible lookup behavior and the caller error handling when you have a chance? |
This branch has not been deployed
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Rationale for this change
A case-insensitive lookup on a schema with two columns whose names differ only by case uses the last one.
Schema._lazy_name_to_id_lowerbuilds the lower-case index with a dict comprehension, soidoverwritesID, andSchema.find_fieldandSchema.selectuse that index for case-insensitive name lookups:The same wrong column is used by sort order updates and by schema updates (
delete_column("ID")dropsid,rename_column("ID", "key")renamesid).Java does not pick a column here.
TypeUtil.indexByLowerCaseNamethrowsCannot build lower case index: ID and id collide, andSchema.caseInsensitiveFindFieldandSchemaUpdateuse that index, so the lookup fails.This change makes the index raise
ValueError: Cannot build lower case index: ID and id collide(the same message as Java) when two different field IDs share a lower-case name. As in Java'sSchemalookup, building the lower-case index fails for any requested name when the schema contains a collision, including when the requested name is unambiguous. Names that map to the same field ID, such as the short name of a list element struct field, are not collisions. Case-sensitive lookups do not change.StructType.field_by_nameuses a separate lookup and is outside this change; it can still return the first matching field (it servesunion_by_name).Callers that treated any
ValueErrorfrom a lookup as "not found" (UpdateSchema.add_column, the move methods, the identifier field check on commit, and the metadata column check in scan projection) now use a private lookup that returnsNonefor a missing name and still raises for an ambiguous one. Without updating the add-column check, an index-only fix would lettable.update_schema(case_sensitive=False).add_column("Id", LongType()).commit()add a third case variant.Are these changes tested?
Yes.
test_case_insensitive_operations_raise_on_case_variant_columnsintests/table/test_init.pyruns a delete, a scan, a sort order update, and schema updates withcase_sensitive=Falseon a table withIDandid, for the memory, sql and sql_without_rowcount catalogs. Each must raise, and the table must be unchanged afterwards. It fails on main, where the delete does not raise.test_case_sensitive_and_unambiguous_operations_are_unchangedchecks exact-case lookups on the same table and a case-insensitive delete on a table without a collision; it passes on main and with this change. Intests/test_schema.py,test_schema_case_insensitive_lookup_of_case_variant_names_raisesandtest_schema_case_insensitive_lookup_of_nested_case_variant_names_raisesfail on main withDID NOT RAISE, andtest_schema_case_insensitive_lookup_of_list_element_struct_fieldcovers the same-ID short names and passes on main and with this change.test_schema_case_insensitive_lookup_rejects_collision_for_any_name,test_metadata_column_projection_propagates_case_collisionandtest_identifier_field_lookup_propagates_case_collisioncover unambiguous and missing names, the metadata column projection and the identifier field check; they fail on main.Are there any user-facing changes?
Yes: case-insensitive name lookups through
Schema.find_fieldandSchema.selectnow raiseValueErrorwhen the schema has names that differ only by case, instead of silently using the last column. Case-sensitive lookups are unchanged.This pull request and its description were written by Isaac.