Skip to content

Default ascending sort fields to nulls-first - #4390

Open
rahulsmahadev wants to merge 1 commit into
apache:mainfrom
rahulsmahadev:rahul/asc-sort-nulls-first
Open

rahulsmahadev wants to merge 1 commit into
apache:mainfrom
rahulsmahadev:rahul/asc-sort-nulls-first

Conversation

@rahulsmahadev

Copy link
Copy Markdown
Contributor

Rationale for this change

Two defaults give ascending sort fields the nulls-last order, which disagrees with Java and with SortField's own default for ascending fields. The same ascending field can therefore be recorded with different null orders depending on how it is built.

Table.update_sort_order().asc(name, transform) with no null_order records nulls-last:

tbl.update_sort_order().asc("user_id", IdentityTransform()).commit()
tbl.sort_order().fields[0].null_order  # NullOrder.NULLS_LAST, expected NULLS_FIRST

SortField built with a str direction also gets nulls-last, while the enum member gets nulls-first:

SortField(source_id=1, transform=IdentityTransform(), direction="asc").null_order  # NULLS_LAST
SortField(source_id=1, transform=IdentityTransform(), direction=SortDirection.ASC).null_order  # NULLS_FIRST

Java's SortOrderBuilder.asc(String) uses NullOrder.NULLS_FIRST (its Javadoc says "ascending with nulls first"), and desc(String) uses NULLS_LAST. SortField picks nulls-first for SortDirection.ASC and nulls-last otherwise. So desc() agrees in all three places, and only the ascending default in UpdateSortOrder.asc() differs. The spec requires null-order in every sort field and does not define a default, so Java and SortField are the reference.

In SortField, the set_null_order validator chooses the default by comparing the raw direction with SortDirection.ASC. SortDirection is a plain Enum, so the str "asc" never matches and gets the descending default. Pydantic converts the value to the enum afterwards, so the field ends up ascending with nulls-last.

This change:

  • UpdateSortOrder.asc() defaults to NullOrder.NULLS_FIRST. desc() keeps NULLS_LAST.
  • set_null_order converts the direction to SortDirection before it chooses the default.

An explicit null order is used as given. Existing sort orders are not changed, because their null order is already stored. The asc() default came in with UpdateSortOrder in #2552. Calls that omit null_order now request nulls-first, also on existing tables, where a matching existing sort order may be reused. Pass NullOrder.NULLS_LAST explicitly to keep the previous behaviour.

Are these changes tested?

Yes. test_update_sort_order_default_null_order in tests/catalog/test_catalog_behaviors.py runs through the catalog fixture (memory, SQL, and SQL without rowcount catalogs). It commits asc() and desc() with no null order and checks the stored sort order, then commits explicit null orders and checks that they are kept. test_deserialize_sort_field_str_direction in tests/table/test_sorting.py checks that "asc" and "desc" validate to the same SortField as the enum members, with the matching default null order. Both fail on main with NullOrder.NULLS_LAST where NullOrder.NULLS_FIRST is expected, and both pass with this change. The integration tests in tests/integration/test_sort_order_update.py pass explicit null orders, so they do not depend on the default.

Are there any user-facing changes?

Yes. update_sort_order().asc() without a null order now records nulls-first (it recorded nulls-last), and SortField with a str direction now gets the same default as the enum member. Please add the changelog label.

This pull request and its description were written by Isaac.

asc() defaulted to nulls-last, and SortField with a str direction did too: the
validator compared the raw str with the enum. Java and SortField default ascending
fields to nulls-first, so default asc() to nulls-first and normalize the direction.
desc() and explicit null orders are unchanged.

Co-authored-by: Isaac <no-reply@databricks.com>
@rahulsmahadev

Copy link
Copy Markdown
Contributor Author

@kevinjqliu could you take a look when you have a chance? You merged #2552, which added UpdateSortOrder; this makes its asc() default nulls-first, as Java and SortField already do.

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant