Skip to content

[client] Implement adaptive fetch rate control for LogScanner#3007

Open
swuferhong wants to merge 1 commit intoapache:mainfrom
swuferhong:partition-fetch-freq
Open

[client] Implement adaptive fetch rate control for LogScanner#3007
swuferhong wants to merge 1 commit intoapache:mainfrom
swuferhong:partition-fetch-freq

Conversation

@swuferhong
Copy link
Copy Markdown
Contributor

(The sections below can be removed for hotfixes or typos)
-->

Purpose

Linked issue: close #3006

For partitioned tables with many inactive partitions, the LogFetcher
previously sent fetch requests to all subscribed buckets equally,
wasting CPU and network resources on empty partitions.

Introduce BucketFetchRateController that uses exponential backoff to
reduce fetch frequency for buckets that consistently return no data.
Buckets that return data are always fetched at full frequency, and a
single non-empty fetch immediately resets the backoff.

Backoff schedule: 1, 2, 4, 8, 16, 32 rounds (configurable max).

New config options:

  • client.scanner.log.adaptive-fetch.enabled (default: true)
  • client.scanner.log.adaptive-fetch.max-skip-rounds (default: 32)

Brief change log

Tests

API and Format

Documentation

@swuferhong swuferhong force-pushed the partition-fetch-freq branch from dbe43c8 to 0f38996 Compare April 7, 2026 02:13
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

[client] Implement adaptive fetch rate control for LogScanner to reduce overhead on partitioned tables

1 participant