Skip to content

Terminate broker when its app-server child exits (fixes wedged/zombie broker hangs)#453

Open
lselva123 wants to merge 1 commit into
openai:mainfrom
lselva123:fix/broker-self-terminate-on-child-exit
Open

Terminate broker when its app-server child exits (fixes wedged/zombie broker hangs)#453
lselva123 wants to merge 1 commit into
openai:mainfrom
lselva123:fix/broker-self-terminate-on-child-exit

Conversation

@lselva123

Copy link
Copy Markdown

Problem

The app-server broker (scripts/app-server-broker.mjs) spawns exactly one codex app-server child at startup and proxies every request to it. If that child exits at any point, the broker becomes a zombie that accepts connections but can never serve a request again:

  • AppServerClientBase.handleExit() rejects in-flight requests but does not set closed, and nothing respawns the child.
  • The next request() therefore passes the if (this.closed) guard, registers a pending promise, and writes to the dead child's stdin — the promise never resolves.
  • So await appClient.request(...) in the broker's data handler hangs forever, the client sees its submitted turn/start / review/start sit at "starting" indefinitely, and it is eventually reported lost.
  • Because the broker's listening socket is still up, ensureBrokerSession() (whose readiness check is a bare socket connect) keeps reusing the same wedged broker, so every subsequent Codex review/task from that workspace wedges the same way until the broker is killed manually.

We observed this repeatedly on a busy multi-session machine: brokers whose app-server child had died (e.g. reaped by an unrelated process sweeper, OOM, or a crash) lingered for many hours with zero children, and every Codex review routed to them hung.

Fix

Register a handler on appClient.exitPromise that shuts the broker down and exits (code 1) as soon as the app-server child is gone. This removes the stale unix socket + pid-file, so the next ensureBrokerSession() sees a dead endpoint and spawns a fresh, working broker instead of binding the zombie. A terminating flag guards against racing an intentional shutdown (broker/shutdown, SIGTERM/SIGINT).

Net effect: a broker whose backing app-server dies now fails closed and self-heals on next use, instead of silently swallowing every future request.

Testing

  • node --check passes.
  • Spawned a broker from the patched script, killed its codex app-server child, and confirmed the broker process exits and its socket is removed (vs. the unpatched build, where the broker stays up childless and the next request hangs).
  • Verified the guard prevents a double-shutdown on the normal broker/shutdown path.

Small, self-contained change; no interface or behavior change on the healthy path.

The broker proxies every request to the single `codex app-server` child it
spawns at startup. If that child exits, appClient can no longer serve any
request: request() writes to a closed stdin and its promise never resolves, so
the caller hangs indefinitely (a submitted turn/task stays at "starting" and is
never answered). The broker itself stays alive as a zombie because its listening
socket still accepts connections, so ensureBrokerSession() keeps reusing it and
every subsequent request wedges the same way.

Register an exit handler on appClient.exitPromise that shuts the broker down and
exits once the child is gone, so the stale socket/pid-file are removed and the
next connection spawns a fresh, working broker. A `terminating` guard keeps the
handler from racing an intentional shutdown (broker/shutdown, SIGTERM/SIGINT).

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
@lselva123 lselva123 requested a review from a team July 8, 2026 08:08

@chatgpt-codex-connector chatgpt-codex-connector Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: 8e7300ed9c

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

Comment on lines +104 to +105
if (terminating) {
return;

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

P2 Badge Return the in-flight shutdown instead of exiting early

If a second shutdown trigger arrives while the first cleanup is still awaiting appClient.close() or server.close(), this new early return makes that caller continue directly to its process.exit(...) path and abort the original cleanup before it can unlink the broker socket/pid file. This can happen with two broker/shutdown requests, or a SIGTERM during app-server-exit cleanup, leaving stale broker state or an orphaned child; keep and return a shared shutdown promise rather than treating an in-progress shutdown as complete.

Useful? React with 👍 / 👎.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant