Repository navigation
Default ascending sort fields to nulls-first - #4390
Open
rahulsmahadev wants to merge 1 commit into
Open
rahulsmahadev wants to merge 1 commit into
rahulsmahadev wants to merge 1 commit into
Conversation
asc() defaulted to nulls-last, and SortField with a str direction did too: the validator compared the raw str with the enum. Java and SortField default ascending fields to nulls-first, so default asc() to nulls-first and normalize the direction. desc() and explicit null orders are unchanged. Co-authored-by: Isaac <no-reply@databricks.com>
Contributor
Author
|
@kevinjqliu could you take a look when you have a chance? You merged #2552, which added |
This branch has not been deployed
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Rationale for this change
Two defaults give ascending sort fields the nulls-last order, which disagrees with Java and with
SortField's own default for ascending fields. The same ascending field can therefore be recorded with different null orders depending on how it is built.Table.update_sort_order().asc(name, transform)with nonull_orderrecords nulls-last:SortFieldbuilt with a str direction also gets nulls-last, while the enum member gets nulls-first:Java's
SortOrderBuilder.asc(String)usesNullOrder.NULLS_FIRST(its Javadoc says "ascending with nulls first"), anddesc(String)usesNULLS_LAST.SortFieldpicks nulls-first forSortDirection.ASCand nulls-last otherwise. Sodesc()agrees in all three places, and only the ascending default inUpdateSortOrder.asc()differs. The spec requiresnull-orderin every sort field and does not define a default, so Java andSortFieldare the reference.In
SortField, theset_null_ordervalidator chooses the default by comparing the rawdirectionwithSortDirection.ASC.SortDirectionis a plainEnum, so the str"asc"never matches and gets the descending default. Pydantic converts the value to the enum afterwards, so the field ends up ascending with nulls-last.This change:
UpdateSortOrder.asc()defaults toNullOrder.NULLS_FIRST.desc()keepsNULLS_LAST.set_null_orderconverts the direction toSortDirectionbefore it chooses the default.An explicit null order is used as given. Existing sort orders are not changed, because their null order is already stored. The
asc()default came in withUpdateSortOrderin #2552. Calls that omitnull_ordernow request nulls-first, also on existing tables, where a matching existing sort order may be reused. PassNullOrder.NULLS_LASTexplicitly to keep the previous behaviour.Are these changes tested?
Yes.
test_update_sort_order_default_null_orderintests/catalog/test_catalog_behaviors.pyruns through thecatalogfixture (memory, SQL, and SQL without rowcount catalogs). It commitsasc()anddesc()with no null order and checks the stored sort order, then commits explicit null orders and checks that they are kept.test_deserialize_sort_field_str_directionintests/table/test_sorting.pychecks that"asc"and"desc"validate to the sameSortFieldas the enum members, with the matching default null order. Both fail on main withNullOrder.NULLS_LASTwhereNullOrder.NULLS_FIRSTis expected, and both pass with this change. The integration tests intests/integration/test_sort_order_update.pypass explicit null orders, so they do not depend on the default.Are there any user-facing changes?
Yes.
update_sort_order().asc()without a null order now records nulls-first (it recorded nulls-last), andSortFieldwith a str direction now gets the same default as the enum member. Please add thechangeloglabel.This pull request and its description were written by Isaac.