Skip to content

Apply rename_field onto the name of a void partition field - #4391

Open
rahulsmahadev wants to merge 1 commit into
apache:mainfrom
rahulsmahadev:rahul/rename-partition-field-void-name
Open

rahulsmahadev wants to merge 1 commit into
apache:mainfrom
rahulsmahadev:rahul/rename-partition-field-void-name

Conversation

@rahulsmahadev

Copy link
Copy Markdown
Contributor

Rationale for this change

rename_field gives the wrong result when the new name belongs to a void partition field. A void field keeps its name after remove_field on a v1 table. On a v2 table, an explicit void field (add_field with VoidTransform) keeps its name too. Renaming a live field onto that name should give the live field the requested name and move the void field out of the way, as Java does. Instead, the live field is renamed to <old name>_<void field id> and the requested name is dropped:

# v1 table: identity(x) (field id 1000), bucket[16](y) named y_shard (field id 1001)
table.update_spec().remove_field("y_shard").commit()
table.update_spec().rename_field("x", "y_shard").commit()
# before: [(1000, "x_1001", identity), (1001, "y_shard", void)]
# after:  [(1000, "y_shard", identity), (1001, "y_shard_1001", void)]

The cause is the recursive call in UpdateSpec.rename_field. For a void target it called rename_field(name, name + "_" + field_id). That renames the live field and returns, so the requested rename is never recorded. The fix renames the void field to <new name>_<field id> and then records the requested rename. This is the same sequence as BaseUpdatePartitionSpec.renameField in Java. Related: #4307 handles removing and adding a field with the same name in one update; this change handles renaming onto an existing void field's name. add_field also calls rename_field, so nested void-name collisions use the corrected sequence. Field ids, source ids and transforms do not change, only the names do.

Are these changes tested?

Yes. Five tests in tests/table/test_init.py:

  • test_update_spec_rename_onto_removed_partition_field_name (runs for each catalog fixture variant): a v1 remove_field, then a rename onto the removed name. It reloads the table from the catalog and checks names, field ids, source ids and transforms.
  • test_update_spec_rename_onto_explicit_void_partition_field_name (v2 table): an explicit void field, then a rename onto its name.
  • test_update_spec_rename_to_unused_partition_field_name: a control that renames onto an unused name, which must keep working as before.
  • test_update_spec_rename_onto_void_rejects_removal_in_same_update: checks both operation orders on v1 and v2.
  • test_update_spec_cascades_void_name_collisions: checks nested void-name collisions through both rename_field and add_field.

Without this change, the two rename tests fail with the live field named x_1001, the removal test fails because nothing is raised, and the cascade test fails on the resulting names. All five tests pass with this change, and the unused-name control passes on both.

Are there any user-facing changes?

Yes: renaming onto the name of a void partition field now gives that name to the renamed field, and the void field becomes <name>_<field id>. Removing the existing void field and then renaming onto its name in the same update now raises Cannot delete and rename partition field <name>, and the reverse order raises Cannot rename and delete field <name>. Both checks match Java; before, both sequences committed the wrong name for the live field.

This pull request and its description were written by Isaac.

rename_field recursed on the live field when the new name belonged to a
void partition field, renaming the live field to "<name>_<field id>" and
returning, so the requested rename was dropped. Rename the void field to
"<new name>_<field id>" and then record the requested rename, as Java's
BaseUpdatePartitionSpec.renameField does.

Co-authored-by: Isaac <no-reply@databricks.com>
@rahulsmahadev

Copy link
Copy Markdown
Contributor Author

@sohailshk could you take a look when you have a chance? You changed UpdateSpec in #4307; this fixes rename_field onto the name of a void field the same way Java's renameField does.

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant