Skip to content

Use the botocore_session credentials in PyArrowFileIO for S3 - #4394

Open
David-Banquet wants to merge 1 commit into
apache:mainfrom
David-Banquet:fix-2657-botocore-session-fileio
Open

David-Banquet wants to merge 1 commit into
apache:mainfrom
David-Banquet:fix-2657-botocore-session-fileio

Conversation

@David-Banquet

Copy link
Copy Markdown
Contributor

Closes #2657.

This follows @Fokko's proposal on #2657, PyArrow part only. Today botocore_session reaches the catalog clients but not the FileIO, which falls back to the default credential chain, so a misconfiguration only shows up on the first data file write.

Changes in PyArrowFileIO:

  • When botocore_session is set and no s3.access-key-id / client.access-key-id is configured, the S3 filesystem gets the session's frozen credentials. Explicit properties keep precedence, which also covers credentials vended by a REST catalog.
  • The session is also skipped when s3.role-arn or client.role-arn / client.assume-role.arn is set, so the existing role assumption keeps working as before.
  • The frozen credentials are part of the fs_by_scheme cache key. Each new_input / new_output / delete calls get_frozen_credentials(), which refreshes them when they are about to expire, so a new token gets a new S3FileSystem and long-running processes keep working after the first token expires.
  • BOTOCORE_SESSION moves to pyiceberg.io so pyarrow.py can use it without importing pyiceberg.catalog; pyiceberg.catalog re-exports it.

FsspecFileIO is left for a follow-up, as suggested. #4349 (docs) can still go in first; I'll adjust or drop its note once this one lands.

Testing:

  • 2 unit tests in tests/io/test_pyarrow.py: the session credentials reach S3FileSystem, explicit ones win, and changing the session's credentials creates a second filesystem. Both fail on main.
  • make test: 4694 passed.
  • Against S3 Tables with a REST catalog and bogus AWS_ACCESS_KEY_ID in the environment (the repro from REST catalog, S3tables with botocore session #2657): append fails on main with INVALID_ACCESS_KEY_ID in CreateMultipartUpload, and succeeds with this change.

The botocore_session catalog property reached the catalog clients but not
the FileIO, which fell back to the default credential chain and only failed
on the first data file write. When no explicit S3 credentials or role are
set, the S3 filesystem now takes the session's frozen credentials, and they
are part of the fs_by_scheme cache key so refreshed credentials get a new
filesystem.

Closes apache#2657

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

REST catalog, S3tables with botocore session

1 participant