Skip to content

HDDS-16179. Dry run command for container balancer with profiles - #11255

Draft
sreejasahithi wants to merge 3 commits into
apache:masterfrom
sreejasahithi:HDDS-16179
Draft

sreejasahithi wants to merge 3 commits into
apache:masterfrom
sreejasahithi:HDDS-16179

Conversation

@sreejasahithi

Copy link
Copy Markdown
Contributor

What changes were proposed in this pull request?

This PR adds a dry-run subcommand to ozone admin containerbalancer so users can estimate how much data the Container Balancer would move, how many iterations it would need, and how long it could take(upper bound), without starting the balancer.

Flow:

  • Fetch datanode usage from SCM(via the existing getDatanodeUsageInfo RPC)
  • Reads balancer limits, timeouts, and profile presets from local Ozone configuration (with optional CLI overrides)
  • Analyze the cluster and estimate one or more throttling profiles

For each profile is reports:

  • bytes to move
  • per-iteration throughput
  • estimated iterations
  • upper-bound duration (with a +30% planning buffer)

Key changes:

  • Adds ozone admin containerbalancer dry-run with optional --profile slow|medium|fast. If --profile is omitted, all three profiles are estimated.
  • Introduces SLOW, MEDIUM, and FAST throttling profiles with new hdds.container.balancer.profile.* config keys for datanode involvement and per-node move limits.
  • Adds ContainerBalancerAdvisor and ContainerBalancerEstimation for dry-run cluster analysis and estimation.
  • Extracts shared CLI options into ContainerBalancerConfigOptions, reused by start and dry-run (start behaviour unchanged).
  • When all three profiles are estimated, a validation failure in one profile (e.g. FAST) does not block SLOW/MEDIUM results.

Note: start does not support --profile yet.

What is the link to the Apache JIRA

HDDS-16179

How was this patch tested?

Unit tests: TestContainerBalancerAdvisor, TestContainerBalancerSubCommand
Manual tested the dry-run command in a cluster:

  1. ozone admin containerbalancer dry-run
Profile: SLOW
 Based on:
   Datanode involvement:     40%
   Max entering target:      10 GB / node
   Max leaving source:       10 GB / node
   Max per iteration:        500 GB
   Move timeout:             65 min
   Balancing interval:       70 min
 Bytes to move:            73.62 GB
 Per iteration (estimate): ~10 GB
 Estimated iterations:     8 (planning estimate: 11, includes +30% buffer)
 Estimated duration:       upper bound ~18.0 hours  (planning estimate: ~1.0 days, includes +30% buffer)
                           (assumes full move timeout + interval each cycle)
Profile: MEDIUM
 Based on:
   Datanode involvement:     40%
   Max entering target:      26 GB / node
   Max leaving source:       26 GB / node
   Max per iteration:        500 GB
   Move timeout:             65 min
   Balancing interval:       70 min
 Bytes to move:            73.62 GB
 Per iteration (estimate): ~26 GB
 Estimated iterations:     3 (planning estimate: 4, includes +30% buffer)
 Estimated duration:       upper bound ~6.8 hours (planning estimate: ~9.0 hours, includes +30% buffer)
                           (assumes full move timeout + interval each cycle)
Profile: FAST
 Based on:
   Datanode involvement:     40%
   Max entering target:      100 GB / node
   Max leaving source:       100 GB / node
   Max per iteration:        500 GB
   Move timeout:             65 min
   Balancing interval:       70 min
 Bytes to move:            73.62 GB
 Per iteration (estimate): ~73.62 GB
 Estimated iterations:     1 (planning estimate: 2, includes +30% buffer)
 Estimated duration:       upper bound ~2.3 hours (planning estimate: ~4.5 hours, includes +30% buffer)
                           (assumes full move timeout + interval each cycle)
  1. ozone admin containerbalancer dry-run --threshold 5
Profile: SLOW
Based on:
  Datanode involvement:     40%
  Max entering target:      10 GB / node
  Max leaving source:       10 GB / node
  Max per iteration:        500 GB
  Move timeout:             65 min
  Balancing interval:       70 min
Bytes to move:            111.59 GB
Per iteration (estimate): ~10 GB
Estimated iterations:     12 (planning estimate: 16, includes +30% buffer)
Estimated duration:       upper bound ~1.1 days (planning estimate: ~1.5 days, includes +30% buffer)
                          (assumes full move timeout + interval each cycle)
Profile: MEDIUM
Based on:
  Datanode involvement:     40%
  Max entering target:      26 GB / node
  Max leaving source:       26 GB / node
  Max per iteration:        500 GB
  Move timeout:             65 min
  Balancing interval:       70 min
Bytes to move:            111.59 GB
Per iteration (estimate): ~26 GB
Estimated iterations:     5 (planning estimate: 7, includes +30% buffer)
Estimated duration:       upper bound ~11.3 hours (planning estimate: ~15.8 hours, includes +30% buffer)
                          (assumes full move timeout + interval each cycle)
Profile: FAST
Based on:
  Datanode involvement:     40%
  Max entering target:      100 GB / node
  Max leaving source:       100 GB / node
  Max per iteration:        500 GB
  Move timeout:             65 min
  Balancing interval:       70 min
Bytes to move:            111.59 GB
Per iteration (estimate): ~100 GB
Estimated iterations:     2 (planning estimate: 3, includes +30% buffer)
Estimated duration:       upper bound ~4.5 hours (planning estimate: ~6.8 hours, includes +30% buffer)
                          (assumes full move timeout + interval each cycle)
  1. ozone admin containerbalancer dry-run --profile MEDIUM
Profile: MEDIUM
Based on:
 Datanode involvement:     40%
 Max entering target:      26 GB / node
 Max leaving source:       26 GB / node
 Max per iteration:        500 GB
 Move timeout:             65 min
 Balancing interval:       70 min
Bytes to move:            73.62 GB
Per iteration (estimate): ~26 GB
Estimated iterations:     3 (planning estimate: 4, includes +30% buffer)
Estimated duration:       upper bound ~6.8 hours (planning estimate: ~9.0 hours, includes +30% buffer)
                         (assumes full move timeout + interval each cycle)
  1. ozone admin containerbalancer dry-run --threshold 5 --max-size-entering-target-in-gb 15 --max-size-leaving-source-in-gb 15 --balancing-iteration-interval-minutes 30 --profile SLOW
Profile: SLOW
 Based on:
   Datanode involvement:     40%
   Max entering target:      15 GB / node
   Max leaving source:       15 GB / node
   Max per iteration:        500 GB
   Move timeout:             65 min
   Balancing interval:       30 min
 Bytes to move:            111.59 GB
 Per iteration (estimate): ~15 GB
 Estimated iterations:     8 (planning estimate: 11, includes +30% buffer)
 Estimated duration:       upper bound ~12.7 hours (planning estimate: ~17.4 hours, includes +30% buffer)
                           (assumes full move timeout + interval each cycle)
  1. ozone admin containerbalancer dry-run --max-size-to-move-per-iteration-in-gb 70 --threshold 5
Profile: SLOW
 Based on:
   Datanode involvement:     40%
   Max entering target:      10 GB / node
   Max leaving source:       10 GB / node
   Max per iteration:        70 GB
   Move timeout:             65 min
   Balancing interval:       70 min
 Bytes to move:            111.46 GB
 Per iteration (estimate): ~10 GB
 Estimated iterations:     12 (planning estimate: 16, includes +30% buffer)
 Estimated duration:       upper bound ~1.1 days (planning estimate: ~1.5 days, includes +30% buffer)
                           (assumes full move timeout + interval each cycle)
Profile: MEDIUM
 Based on:
   Datanode involvement:     40%
   Max entering target:      26 GB / node
   Max leaving source:       26 GB / node
   Max per iteration:        70 GB
   Move timeout:             65 min
   Balancing interval:       70 min
 Bytes to move:            111.46 GB
 Per iteration (estimate): ~26 GB
 Estimated iterations:     5 (planning estimate: 7, includes +30% buffer)
 Estimated duration:       upper bound ~11.3 hours (planning estimate: ~15.8 hours, includes +30% buffer)
                           (assumes full move timeout + interval each cycle)
Profile: FAST
 Based on:
   Datanode involvement:     40%
   Max entering target:      100 GB / node
   Max leaving source:       100 GB / node
   Max per iteration:        70 GB
   Move timeout:             65 min
   Balancing interval:       70 min
 Estimation failed: max-size-entering-target must be less than or equal to max-size-to-move-per-iteration.

Green CI: https://github.com/sreejasahithi/ozone/actions/runs/35199786999

@sreejasahithi sreejasahithi changed the title HDDS 16179. Dry run command for container balancer with profiles HDDS-16179. Dry run command for container balancer with profiles Sep 18, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants