Skip to content

Avro schema is re-converted to an Iceberg schema on every manifest read during scan planning #3662

Description

@vishnuprakaz

Feature Request / Improvement

AvroFileHeader.get_schema() (pyiceberg/avro/file.py) runs the full avro_to_iceberg conversion every time an
Avro file is opened.

The inefficiency

  • Every manifest under a spec embeds an identical Avro schema string.
  • So during scan planning, that same conversion is repeated once per manifest.
  • The cost grows with the manifest count, even though only a couple of distinct schemas are ever involved.

Proposed fix

  • The conversion depends only on the schema string, so the result can be cached (keyed on that string).

Measured impact

  • scan().plan_files() on an unpartitioned 150-manifest table: ~86 ms → ~48 ms (~1.8× faster).
  • The saving grows as the number of manifests increases.

I am willing to contribute for this improvement.

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions