Skip to content

Build: Use stdlib compression.zstd on Python 3.14+ and build CPython 3.15 wheels - #4395

Open
raufer wants to merge 1 commit into
apache:mainfrom
raufer:python-3.15-stdlib-zstd
Open

raufer wants to merge 1 commit into
apache:mainfrom
raufer:python-3.15-stdlib-zstd

Conversation

@raufer

@raufer raufer commented Nov 16, 2026

Copy link
Copy Markdown

Rationale for this change

CPython 3.15 is out (3.15.2), but PyIceberg cannot be installed on it from wheels:

Python 3.14 added a zstd module to the standard library: compression.zstd, from PEP 784. PyIceberg only needs one-shot compress and decompress (the Avro codec and Puffin blobs), so on 3.14+ the third-party package is not needed at all.

What changes

  • pyiceberg/table/puffin.py uses compression.zstd.decompress on Python 3.14+. Below 3.14 it keeps zstandard.ZstdDecompressor().decompress.
  • pyiceberg/avro/codecs/zstandard_codec.py uses compression.zstd.compress and decompress on 3.14+. Below 3.14 the existing zstandard code is unchanged, only indented under the version check (git diff -w shows the real change).
  • In pyproject.toml, the dependency becomes zstandard>=0.13.0; python_version < '3.14'. uv.lock is regenerated with uv lock.
  • The wheel build adds 3.15: CIBW_PROJECT_REQUIRES_PYTHON becomes >=3.11,<3.16, and 3.15 joins the setup-python list. cibuildwheel v4.3.4 already builds cp315 by default.
  • CIBW_TEST_SKIP: "cp315-*" is set for now. The test step syncs the locked dev group, which pins pyarrow 25.0.1 and numpy 2.4.6 through pyspark[connect], and neither has a cp315 wheel. Moving those pins is a separate change, and the skip can go once it lands.

The version check follows the existing sys.version_info >= (3, 14) pattern (uuid7 in catalog/rest/__init__.py), so mypy checks each branch for the interpreter it runs on.

PEP 784 makes compression.zstd an optional module, built when libzstd is available; python-build-standalone (what uv installs) ships it on 3.14 and 3.15. A CPython built without it fails at import time, as a missing zstandard does today.

Are these changes tested?

The existing tests cover both paths: Puffin's test_read_compressed_zstd, and the zstd/zstandard manifest tests in tests/utils/test_manifest.py. Locally, on macOS arm64:

Python zstandard installed tests/avro tests/table/test_puffin.py tests/utils/test_manifest.py tests/table/test_metadata.py
3.13.7 yes (marker keeps it) 281 passed
3.14.10 no 281 passed
3.15.2 no 281 passed

ruff (v0.14.3) and mypy (1.18.2) are clean on the changed files under 3.13 and 3.15.

Are there any user-facing changes?

On Python 3.14+, pip install pyiceberg no longer installs zstandard. On 3.15, pyiceberg installs from a wheel.

🤖 Generated with Claude Code

…3.15 wheels

zstandard publishes no cp315 wheel; Python 3.14 ships compression.zstd (PEP 784).

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant