Terminate broker when its app-server child exits (fixes wedged/zombie broker hangs)#453
Terminate broker when its app-server child exits (fixes wedged/zombie broker hangs)#453lselva123 wants to merge 1 commit into
Conversation
The broker proxies every request to the single `codex app-server` child it spawns at startup. If that child exits, appClient can no longer serve any request: request() writes to a closed stdin and its promise never resolves, so the caller hangs indefinitely (a submitted turn/task stays at "starting" and is never answered). The broker itself stays alive as a zombie because its listening socket still accepts connections, so ensureBrokerSession() keeps reusing it and every subsequent request wedges the same way. Register an exit handler on appClient.exitPromise that shuts the broker down and exits once the child is gone, so the stale socket/pid-file are removed and the next connection spawns a fresh, working broker. A `terminating` guard keeps the handler from racing an intentional shutdown (broker/shutdown, SIGTERM/SIGINT). Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: 8e7300ed9c
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
| if (terminating) { | ||
| return; |
There was a problem hiding this comment.
Return the in-flight shutdown instead of exiting early
If a second shutdown trigger arrives while the first cleanup is still awaiting appClient.close() or server.close(), this new early return makes that caller continue directly to its process.exit(...) path and abort the original cleanup before it can unlink the broker socket/pid file. This can happen with two broker/shutdown requests, or a SIGTERM during app-server-exit cleanup, leaving stale broker state or an orphaned child; keep and return a shared shutdown promise rather than treating an in-progress shutdown as complete.
Useful? React with 👍 / 👎.
Problem
The app-server broker (
scripts/app-server-broker.mjs) spawns exactly onecodex app-serverchild at startup and proxies every request to it. If that child exits at any point, the broker becomes a zombie that accepts connections but can never serve a request again:AppServerClientBase.handleExit()rejects in-flight requests but does not setclosed, and nothing respawns the child.request()therefore passes theif (this.closed)guard, registers a pending promise, and writes to the dead child's stdin — the promise never resolves.await appClient.request(...)in the broker's data handler hangs forever, the client sees its submittedturn/start/review/startsit at "starting" indefinitely, and it is eventually reported lost.ensureBrokerSession()(whose readiness check is a bare socket connect) keeps reusing the same wedged broker, so every subsequent Codex review/task from that workspace wedges the same way until the broker is killed manually.We observed this repeatedly on a busy multi-session machine: brokers whose app-server child had died (e.g. reaped by an unrelated process sweeper, OOM, or a crash) lingered for many hours with zero children, and every Codex review routed to them hung.
Fix
Register a handler on
appClient.exitPromisethat shuts the broker down and exits (code 1) as soon as the app-server child is gone. This removes the stale unix socket + pid-file, so the nextensureBrokerSession()sees a dead endpoint and spawns a fresh, working broker instead of binding the zombie. Aterminatingflag guards against racing an intentional shutdown (broker/shutdown,SIGTERM/SIGINT).Net effect: a broker whose backing app-server dies now fails closed and self-heals on next use, instead of silently swallowing every future request.
Testing
node --checkpasses.codex app-serverchild, and confirmed the broker process exits and its socket is removed (vs. the unpatched build, where the broker stays up childless and the next request hangs).broker/shutdownpath.Small, self-contained change; no interface or behavior change on the healthy path.