Skip to content

Add update_partition_statistics() to Table and Transaction - #4397

Open
rahulsmahadev wants to merge 1 commit into
apache:mainfrom
rahulsmahadev:rahul/update-partition-statistics
Open

rahulsmahadev wants to merge 1 commit into
apache:mainfrom
rahulsmahadev:rahul/update-partition-statistics

Conversation

@rahulsmahadev

Copy link
Copy Markdown
Contributor

Rationale for this change

PyIceberg already models partition statistics in the table metadata: PartitionStatisticsFile, TableMetadata.partition_statistics, and the SetPartitionStatisticsUpdate and RemovePartitionStatisticsUpdate table updates (#2192). Table and Transaction have no builder for these updates; callers currently need to construct the update objects and commit them through the catalog API.

Java has this API. Table.updatePartitionStatistics() and Transaction.updatePartitionStatistics() return an UpdatePartitionStatistics with setPartitionStatistics(file) and removePartitionStatistics(snapshotId) (api/src/main/java/org/apache/iceberg/UpdatePartitionStatistics.java, core/src/main/java/org/apache/iceberg/SetPartitionStatistics.java).

This PR adds the same API to PyIceberg:

  • Table.update_partition_statistics() and Transaction.update_partition_statistics(), next to update_statistics().
  • UpdatePartitionStatistics in pyiceberg/table/update/partition_statistics.py, modelled on UpdateStatistics. Pending operations are applied on commit, for chained and with block operations alike, with sets applied before removals as in Java.
  • set_partition_statistics(file) replaces the file already set for the same snapshot. None is a no-op, as in Java.
  • remove_partition_statistics(snapshot_id) uses the existing RemovePartitionStatisticsUpdate.
table.update_partition_statistics().set_partition_statistics(partition_statistics_file).commit()

with table.update_partition_statistics() as update:
    update.set_partition_statistics(partition_statistics_file)
    update.remove_partition_statistics(snapshot_id)

Related open PRs: #4087 fixes UpdateStatistics.remove_statistics, which replaces the pending operations instead of appending to them. This PR does not copy that pattern. #4364 removes partition statistics when their snapshots are removed. This PR changes only the RemovePartitionStatisticsUpdate handler in pyiceberg/table/update/__init__.py, a different part of that file from the snapshot removal handler that #4364 changes, and does not touch the snapshot expiry code. #4364 only removes entries that exist, so the no-op change does not affect its path.

Are these changes tested?

Yes. Five tests in tests/table/test_statistics.py use the catalog fixture (in-memory and SQL catalogs) and go through the public API:

  • test_update_partition_statistics: set, replace the file for the same snapshot, set in a with block, remove, and reload the table from the catalog.
  • test_update_partition_statistics_set_and_remove_in_one_update: operations on different snapshots are both applied. On the same snapshot the removal wins in either call order, with or without an existing file. Repeated sets keep the last file, and repeated removals are no-ops.
  • test_update_partition_statistics_in_transaction: pending changes are visible in transaction.table_metadata, but not in table.metadata or a catalog reload, until the transaction commits. An exception rolls them back.
  • test_remove_partition_statistics_without_file_is_noop: removing a snapshot without a file, once or repeated, commits and leaves the metadata unchanged.
  • test_set_partition_statistics_none_is_noop: set_partition_statistics(None) changes nothing.

The low-level handler is covered by test_remove_partition_statistics_update_without_file_is_noop in tests/table/test_init.py.

All five tests fail on main with AttributeError: the transaction test reports that Transaction has no update_partition_statistics attribute, and the other four report the missing attribute on Table. All five pass with this change. The low-level test in tests/table/test_init.py fails on main with ValueError and passes with this change. I also checked that a copy of the UpdateStatistics pattern, where removal discards the pending sets, fails the set-and-remove test.

Are there any user-facing changes?

Yes. New methods Table.update_partition_statistics() and Transaction.update_partition_statistics(), and a new UpdatePartitionStatistics class. The docs are in mkdocs/docs/api.md. Two behaviours follow Java:

  • Removing partition statistics that are absent is a no-op, matching Java. This also changes the existing RemovePartitionStatisticsUpdate handler, which previously raised ValueError.
  • Within a builder, sets are applied before removals, matching Java. Removing a snapshot's statistics wins over setting them, regardless of call order.

Please add the changelog label.

This pull request and its description were written by Isaac.

Add an UpdatePartitionStatistics builder with set_partition_statistics() and
remove_partition_statistics(), so callers no longer build update objects by
hand. Sets apply before removals and absent removals are no-ops, as in Java.
The RemovePartitionStatisticsUpdate handler therefore no longer raises.

Co-authored-by: Isaac <no-reply@databricks.com>
@rahulsmahadev

Copy link
Copy Markdown
Contributor Author

@rambleraptor, this adds Table and Transaction builders for the partition-statistics updates you introduced in #2192. Could you take a look at the API, and at the missing-file and operation-order semantics, when you have a chance?

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant