Skip to content

Avro: Cache the Avro to Iceberg schema conversion per schema string - #4384

Open
zn-danielbariudin wants to merge 1 commit into
apache:mainfrom
zn-danielbariudin:avro-cache-schema-conversion
Open

zn-danielbariudin wants to merge 1 commit into
apache:mainfrom
zn-danielbariudin:avro-cache-schema-conversion

Conversation

@zn-danielbariudin

Copy link
Copy Markdown

Closes #3662

Rationale for this change

Every manifest in a table carries the same Avro schema string in its header, but AvroFileHeader.get_schema() parsed and converted that string to an Iceberg Schema on every file open. During scan planning this repeats the same conversion once per manifest.

This caches the conversion by schema string with functools.lru_cache, so each distinct schema is converted once per process.

  • Schema is a frozen model, so sharing the cached instance between readers is safe.
  • lru_cache keeps at most 128 schema strings; a process sees only a few distinct manifest schemas.

Builds on #3663 by @vishnuprakaz (picked up with his 👍 on the issue); he is credited as co-author on the commit.

Benchmark

plan_files() on an unpartitioned table with 150 manifests (local SqlCatalog, 7 runs in one process), Python 3.14.7, Apple M5 Pro:

Before After
Schema conversions in one plan_files() 151 1
plan_files() median ~56 ms ~30 ms
plan_files() first call ~96 ms ~67 ms
Benchmark script
import statistics
import sys
import tempfile
import time

import pyarrow as pa
from pyiceberg.avro.file import AvroSchemaConversion
from pyiceberg.catalog.sql import SqlCatalog

manifests = int(sys.argv[1])
repeats = 7

warehouse = tempfile.mkdtemp()
catalog = SqlCatalog("bench", uri=f"sqlite:///{warehouse}/catalog.db", warehouse=f"file://{warehouse}")
catalog.create_namespace("ns")
table = catalog.create_table("ns.t", schema=pa.schema([("id", pa.int64())]))
for i in range(manifests):
    table.append(pa.table({"id": pa.array([i], pa.int64())}))

calls = 0
original = AvroSchemaConversion.avro_to_iceberg


def counting(self, avro_schema):
    global calls
    calls += 1
    return original(self, avro_schema)


AvroSchemaConversion.avro_to_iceberg = counting

timings = []
for run in range(repeats):
    start = time.perf_counter()
    tasks = list(table.scan().plan_files())
    timings.append((time.perf_counter() - start) * 1000)
    if run == 0:
        first_run_calls = calls

print(f"manifests={manifests} data_files={len(tasks)}")
print(f"conversions: first plan_files={first_run_calls}, all {repeats} runs={calls}")
print(f"plan_files ms: first={timings[0]:.1f} median={statistics.median(timings):.1f} min={min(timings):.1f}")

Run with uv run python bench_plan_files.py 150.

Are these changes tested?

Yes. Added test_get_schema_reuses_conversion_for_identical_schema_strings in tests/avro/test_file.py: it checks the converted schema, that two headers with the same schema string return the same instance, and that a different string returns a different one. The full unit suite passes locally.

Are there any user-facing changes?

No API changes. Scan planning is faster on tables with many manifests.

@zn-danielbariudin

Copy link
Copy Markdown
Author

@Fokko @kevinjqliu could you take a look when you have a moment? Small change for apache/iceberg-python#3662: it caches the Avro schema conversion, and plan_files() on 150 manifests goes from ~56 ms to ~30 ms. Thanks

@Fokko

Fokko commented Nov 15, 2026

Copy link
Copy Markdown
Contributor

I'm torn on this one. There is a reasonable speedup, but the tradeoff is on increased memory. Also, hashing the schema is not cheap. Can you maybe profile the memory?

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Avro schema is re-converted to an Iceberg schema on every manifest read during scan planning

2 participants