What is the problem the feature request solves?
Native Parquet scans expose bytes_scanned, but that counter does not distinguish projected data from footer, page-index, or Bloom-filter reads, and it does not reveal the larger GET ranges produced by object-store coalescing. A metadata-only scan and a warm metadata-cache hit can therefore be difficult to distinguish in the Spark SQL UI.
For example, two logical 64-byte reads can be serviced by one coalesced 524,416-byte GET. Both paths can report the same 128 logical bytes while placing very different demands on storage.
Describe the potential solution
Expose counters at the Parquet reader, remote object-store API, and metadata-cache boundaries, and propagate them to Spark SQL metrics. Document which byte counters overlap, the difference between requested and consumed bytes, encrypted-footer behavior, backend scope, and the limitations of cancellation snapshots. Preserve the existing bytes_scanned meaning and correct documentation that describes it as complete filesystem traffic.
Tests should cover coalescing, projection and pruning, cache hits and misses, encrypted and malformed footers, partial response consumption, and real delegation to local/custom range readers. Measure the additional SQL accumulator work on a job with many tasks.
Additional context
This tracks the metrics-only scope of #5453. Object-store registration isolation is tracked separately in #5502 and implemented by prerequisite #5503. Producer cancellation and final-snapshot waiting are tracked separately in #5504 and implemented by #5505; they are not part of this metrics request.
What is the problem the feature request solves?
Native Parquet scans expose
bytes_scanned, but that counter does not distinguish projected data from footer, page-index, or Bloom-filter reads, and it does not reveal the larger GET ranges produced by object-store coalescing. A metadata-only scan and a warm metadata-cache hit can therefore be difficult to distinguish in the Spark SQL UI.For example, two logical 64-byte reads can be serviced by one coalesced 524,416-byte GET. Both paths can report the same 128 logical bytes while placing very different demands on storage.
Describe the potential solution
Expose counters at the Parquet reader, remote object-store API, and metadata-cache boundaries, and propagate them to Spark SQL metrics. Document which byte counters overlap, the difference between requested and consumed bytes, encrypted-footer behavior, backend scope, and the limitations of cancellation snapshots. Preserve the existing
bytes_scannedmeaning and correct documentation that describes it as complete filesystem traffic.Tests should cover coalescing, projection and pruning, cache hits and misses, encrypted and malformed footers, partial response consumption, and real delegation to local/custom range readers. Measure the additional SQL accumulator work on a job with many tasks.
Additional context
This tracks the metrics-only scope of #5453. Object-store registration isolation is tracked separately in #5502 and implemented by prerequisite #5503. Producer cancellation and final-snapshot waiting are tracked separately in #5504 and implemented by #5505; they are not part of this metrics request.