Add concurrent job queue support#10
Conversation
The job queue now runs up to MAX_CONCURRENT_JOBS (default 3) benchmark jobs in parallel instead of one at a time. This allows running benchmarks for multiple models simultaneously when each targets a different vLLM endpoint on separate Brev instances.
|
Warning Review limit reached
Next review available in: 43 minutes Enable usage-based reviews in Billing to review now. Otherwise, wait until the next included review is available. How can I continue?After more reviews become available, a review can be triggered using the To avoid repeated limits, reduce automatic review volume by pausing incremental auto-reviews earlier, using label-based review opt-in, excluding WIP or generated PR titles, or requesting reviews manually when the PR is ready. If your team needs uninterrupted high-volume reviews, an organization admin can enable usage-based reviews. How do review limits work?CodeRabbit enforces per-developer PR review limits for each organization. Most developers receive the normal plan review availability. For paid Pro and Pro+ PR reviews, CodeRabbit uses adaptive limits for sustained high-volume activity. When a developer's recent PR review activity reaches the 95th percentile or higher among CodeRabbit users, additional reviews become available more gradually as earlier reviews age out of the rolling window. Please refer docs for additional details. Review details⚙️ Run configurationConfiguration used: Organization UI Review profile: CHILL Plan: Enterprise Run ID: 📒 Files selected for processing (1)
📝 WalkthroughWalkthroughChangesConcurrent job execution
Estimated code review effort: 4 (Complex) | ~45 minutes Suggested reviewers: Sequence Diagram(s)sequenceDiagram
participant JobWorker
participant asyncio.create_task
participant _run_job
participant _job_semaphore
participant _active_jobs
JobWorker->>asyncio.create_task: Spawn _run_job for queued job
asyncio.create_task->>_run_job: Start job task
_run_job->>_active_jobs: Register job_id task
_run_job->>_job_semaphore: Acquire execution slot
_job_semaphore-->>_run_job: Permit bounded execution
🚥 Pre-merge checks | ✅ 5✅ Passed checks (5 passed)
✨ Finishing Touches🧪 Generate unit tests (beta)
Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out. Comment |
![coderabbitai[bot]](https://avatars.githubusercontent.com/in/347564?s=60&v=4){kind=small}
There was a problem hiding this comment.
Actionable comments posted: 3
🤖 Prompt for all review comments with AI agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
Inline comments:
In `@src/coding_agent_bench/api.py`:
- Around line 29-30: Update the MAX_CONCURRENT_JOBS initialization and
_job_semaphore setup to parse the environment value with a clear configuration
error for non-integer input, and reject values below 1 before creating the
semaphore. Ensure invalid settings fail explicitly at startup rather than
causing runtime blocking or an unhandled conversion error.
- Around line 293-300: Update the _shutting_down branch of the
asyncio.CancelledError handler to invoke job cleanup with signal=True, ensuring
shutdown cleanup signals the job pod and does not leave child task pods
orphaned. Preserve the existing error construction, status update, and
cancellation propagation.
- Line 333: Update the job-draining logic around _run_job so it does not create
one asyncio Task for every queued job before capacity is available. Use a
bounded worker-pool pattern or acquire a concurrency slot before spawning each
task, and ensure _active_jobs registration and shutdown cancellation remain
consistent for jobs still waiting in the backlog.
🪄 Autofix (Beta)
Fix all unresolved CodeRabbit comments on this PR:
- Push a commit to this branch (recommended)
- Create a new PR with the fixes
ℹ️ Review info
⚙️ Run configuration
Configuration used: Organization UI
Review profile: CHILL
Plan: Enterprise
Run ID: 64bc4fb7-05b7-428b-ab65-22e5dd34c037
📒 Files selected for processing (1)
src/coding_agent_bench/api.py
- Validate MAX_CONCURRENT_JOBS env var at startup (reject non-int and < 1) - Pass signal=True during shutdown cleanup to avoid orphaned task pods - Track background tasks in a set to prevent GC and satisfy RUF006
1 participant
Summary
MAX_CONCURRENT_JOBS(default 3) benchmark jobs in parallel instead of one at a timeasyncio.Semaphoreto gate concurrent jobs, with an env var to tune the limitChanges
_active_jobtuple with_active_jobsdict keyed by job_idasyncio.create_task()instead ofawait, so it doesn't block on one jobRelated
Depends on taagarwa-rh/harbor#1 — adds resource requests to Harbor's OpenShift build pods so the scheduler spreads them across nodes instead of stacking them on one. Without that fix, running multiple concurrent jobs could cause disk pressure from accumulated build images. With both changes together, 2-3 parallel benchmark jobs should run safely.
Test plan
server_urlvalues/uioc get events --field-selector reason=Evicted)cc @taagarwa-rh @rounakbende10