Repository navigation
Apply rename_field onto the name of a void partition field - #4391
Open
rahulsmahadev wants to merge 1 commit into
Open
rahulsmahadev wants to merge 1 commit into
rahulsmahadev wants to merge 1 commit into
Conversation
rename_field recursed on the live field when the new name belonged to a void partition field, renaming the live field to "<name>_<field id>" and returning, so the requested rename was dropped. Rename the void field to "<new name>_<field id>" and then record the requested rename, as Java's BaseUpdatePartitionSpec.renameField does. Co-authored-by: Isaac <no-reply@databricks.com>
Contributor
Author
|
@sohailshk could you take a look when you have a chance? You changed |
This branch has not been deployed
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Rationale for this change
rename_fieldgives the wrong result when the new name belongs to a void partition field. A void field keeps its name afterremove_fieldon a v1 table. On a v2 table, an explicit void field (add_fieldwithVoidTransform) keeps its name too. Renaming a live field onto that name should give the live field the requested name and move the void field out of the way, as Java does. Instead, the live field is renamed to<old name>_<void field id>and the requested name is dropped:The cause is the recursive call in
UpdateSpec.rename_field. For a void target it calledrename_field(name, name + "_" + field_id). That renames the live field and returns, so the requested rename is never recorded. The fix renames the void field to<new name>_<field id>and then records the requested rename. This is the same sequence asBaseUpdatePartitionSpec.renameFieldin Java. Related: #4307 handles removing and adding a field with the same name in one update; this change handles renaming onto an existing void field's name.add_fieldalso callsrename_field, so nested void-name collisions use the corrected sequence. Field ids, source ids and transforms do not change, only the names do.Are these changes tested?
Yes. Five tests in
tests/table/test_init.py:test_update_spec_rename_onto_removed_partition_field_name(runs for eachcatalogfixture variant): a v1remove_field, then a rename onto the removed name. It reloads the table from the catalog and checks names, field ids, source ids and transforms.test_update_spec_rename_onto_explicit_void_partition_field_name(v2 table): an explicit void field, then a rename onto its name.test_update_spec_rename_to_unused_partition_field_name: a control that renames onto an unused name, which must keep working as before.test_update_spec_rename_onto_void_rejects_removal_in_same_update: checks both operation orders on v1 and v2.test_update_spec_cascades_void_name_collisions: checks nested void-name collisions through bothrename_fieldandadd_field.Without this change, the two rename tests fail with the live field named
x_1001, the removal test fails because nothing is raised, and the cascade test fails on the resulting names. All five tests pass with this change, and the unused-name control passes on both.Are there any user-facing changes?
Yes: renaming onto the name of a void partition field now gives that name to the renamed field, and the void field becomes
<name>_<field id>. Removing the existing void field and then renaming onto its name in the same update now raisesCannot delete and rename partition field <name>, and the reverse order raisesCannot rename and delete field <name>. Both checks match Java; before, both sequences committed the wrong name for the live field.This pull request and its description were written by Isaac.