feat(interception): bound concurrent model requests per server

[faresobeid]

What

InterceptionServerConfig.max_concurrent_requests: PositiveInt | None = None. When set, an asyncio.Semaphore bounds the upstream model requests in flight across every session registered on that server:

• rendered (non-streaming) calls hold a slot around client.get_response(...) only — response interceptors / @stops run after release;
• streamed calls acquire before client.relay(...) and release when the upstream connection closes (reply.close()), which now sits on one AsyncExitStack with the slot so the two always free together, on every exit path (a finally in _stream closes the stack before record_call);
• tool execution, replay-cache hits, coalesced SDK retries, /v1/models and aux routes take no slot; None keeps today's behavior.

The limit is per server: it applies to the server and static interception shapes (static lists one InterceptionServerConfig per server). The elastic pool mints servers with the default config and stays unlimited; a pool-wide knob is a separate change.

The same concern was bundled into the closed #2552 with unrelated changes; this reimplements only the request limit on current main.

Why

Rollout concurrency does not bound inference load: a recursive harness (rho in the data-flywheel pipeline, ~90 concurrent rollouts) fans out several model requests per rollout, so the provider sees bursts well past the rollout cap and answers with 429s that then surface as rollout errors. A per-server admission limit puts the bound where every request already passes.

Validation

• tests/v1/test_interception.py::test_max_concurrent_requests_bounds_upstream[rendered|streamed]: a real InterceptionServer with max_concurrent_requests=2, five sessions sharing a gated stub Client that counts in-flight upstream requests (the streamed stub decrements only on close()), five concurrent POST /v1/chat/completions over HTTP. Asserts exactly two reach upstream while the rest queue, peak in-flight stays 2, and all five complete with a committed turn. With the limit unset the same probe reaches peak 5.
• uv run pytest tests/v1 -m "not e2e": 88 passed. uv run ruff check, uv run pre-commit run --all-files: clean.

Note

Bound concurrent upstream model requests in InterceptionServer

• Adds optional max_concurrent_requests field (PositiveInt) to InterceptionServerConfig in server.py
• Acquires an asyncio semaphore in the rendered sample handler and streamed relay handler to bound concurrent upstream model requests
• Streamed responses retain their semaphore slot until the relay's close callback runs
• Adds GatedClient test double and concurrency tests in test_interception.py to verify the limit is respected for both rendered and streamed paths
• Behavioral Change: servers without max_concurrent_requests set use a no-op context manager, preserving existing unlimited admission

^{Macroscope summarized 507e414.}

---

Note

Medium Risk
Changes admission control on the central model proxy path; misconfigured limits could throttle or delay rollouts, but default None preserves current behavior.

Overview
Adds optional max_concurrent_requests on InterceptionServerConfig so a single interception server can cap how many upstream model calls are in flight across all registered sessions (None keeps unlimited admission).

Non-streaming turns acquire a semaphore only around client.get_response; streamed turns acquire before client.relay and hold the slot until upstream teardown by pairing the semaphore with reply.close() on an AsyncExitStack (with idempotent cleanup on every exit path). Tool routes, replay-cache hits, and aux/model listing paths are unchanged and do not consume slots.

New integration tests drive five concurrent /v1/chat/completions calls against a real server with limit 2, using a gated stub client to assert queuing, peak in-flight of 2, and successful completion for both rendered and streamed modes.

^{Reviewed by Cursor Bugbot for commit 507e414. Bugbot is set up for automated code reviews on this repo. Configure here.}

[macroscopeapp]

Approvability

Verdict: Approved at 507e414

Macroscope's review found this PR approvable — Adds a localized, opt-in per-server concurrency limit while preserving unlimited admission by default. Both rendered and streamed request paths are covered by integration tests, and unrelated routes remain unchanged.

^{You can add or adjust custom eligibility rules. Learn more.}