Skip to content

feat: Improve PK MOR read performance with batch queue and cached column metadata#304

Open
lxy-9602 wants to merge 3 commits into
alibaba:mainfrom
lxy-9602:pk-mor-read-efficiency
Open

feat: Improve PK MOR read performance with batch queue and cached column metadata#304
lxy-9602 wants to merge 3 commits into
alibaba:mainfrom
lxy-9602:pk-mor-read-efficiency

Conversation

@lxy-9602
Copy link
Copy Markdown
Collaborator

@lxy-9602 lxy-9602 commented May 26, 2026

Purpose

No Linked issue.

Summary

This PR optimizes PK table read performance with two main improvements that reduce CPU overhead in the hot path:

  1. Batch queue for Producer-Consumer communication — reduces mutex contention from per-row to per-batch granularity
  2. Pre-cached column metadata in ColumnarRowRef — eliminates virtual function calls and checked_cast for field access

Tests

API and Format

Documentation

Generative AI tooling

Generated-by: Claude-4.7-Opus

@zjw1111 zjw1111 requested a review from Copilot May 26, 2026 09:53
Copilot AI reviewed May 26, 2026
View reviewed changes
Copy link
Copy Markdown

Copilot AI left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Copilot encountered an error and was unable to review this pull request. You can try again by re-requesting a review.

@zjw1111 zjw1111 requested a review from Copilot May 26, 2026 10:27
Copilot AI reviewed May 26, 2026
View reviewed changes
Copy link
Copy Markdown

Copilot AI left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Copilot encountered an error and was unable to review this pull request. You can try again by re-requesting a review.

@zjw1111 zjw1111 requested a review from Copilot May 26, 2026 13:22
Copilot AI reviewed May 26, 2026
View reviewed changes
Copy link
Copy Markdown

Copilot AI left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

Copilot reviewed 7 out of 7 changed files in this pull request and generated 2 comments.

Comment thread src/paimon/core/io/async_key_value_producer_and_consumer.cpp
Comment thread src/paimon/common/data/columnar/columnar_batch_context.h
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

Copilot code review Copilot Copilot left review comments

At least 1 approving review is required to merge this pull request.

Assignees

No one assigned

Labels

None yet

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

2 participants

@lxy-9602