Repository navigation
Use the botocore_session credentials in PyArrowFileIO for S3 - #4394
Open
David-Banquet wants to merge 1 commit into
Open
David-Banquet wants to merge 1 commit into
David-Banquet wants to merge 1 commit into
Conversation
The botocore_session catalog property reached the catalog clients but not the FileIO, which fell back to the default credential chain and only failed on the first data file write. When no explicit S3 credentials or role are set, the S3 filesystem now takes the session's frozen credentials, and they are part of the fs_by_scheme cache key so refreshed credentials get a new filesystem. Closes apache#2657 Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
This branch has not been deployed
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Closes #2657.
This follows @Fokko's proposal on #2657, PyArrow part only. Today
botocore_sessionreaches the catalog clients but not the FileIO, which falls back to the default credential chain, so a misconfiguration only shows up on the first data file write.Changes in
PyArrowFileIO:botocore_sessionis set and nos3.access-key-id/client.access-key-idis configured, the S3 filesystem gets the session's frozen credentials. Explicit properties keep precedence, which also covers credentials vended by a REST catalog.s3.role-arnorclient.role-arn/client.assume-role.arnis set, so the existing role assumption keeps working as before.fs_by_schemecache key. Eachnew_input/new_output/deletecallsget_frozen_credentials(), which refreshes them when they are about to expire, so a new token gets a newS3FileSystemand long-running processes keep working after the first token expires.BOTOCORE_SESSIONmoves topyiceberg.iosopyarrow.pycan use it without importingpyiceberg.catalog;pyiceberg.catalogre-exports it.FsspecFileIOis left for a follow-up, as suggested. #4349 (docs) can still go in first; I'll adjust or drop its note once this one lands.Testing:
tests/io/test_pyarrow.py: the session credentials reachS3FileSystem, explicit ones win, and changing the session's credentials creates a second filesystem. Both fail on main.make test: 4694 passed.AWS_ACCESS_KEY_IDin the environment (the repro from REST catalog, S3tables with botocore session #2657):appendfails on main withINVALID_ACCESS_KEY_IDinCreateMultipartUpload, and succeeds with this change.