Skip to content

fix(runner): sweep detached exec sessions when an agent stops (RIG-4708) - #1805

Merged
trunk-io[bot] merged 3 commits into
compass-agent/rig-4181-cli-stop-kills-agentfrom
compass-agent/rig-4708-stop-sweeps-sessions
Oct 7, 2026
Merged

trunk-io[bot] merged 3 commits into
compass-agent/rig-4181-cli-stop-kills-agentfrom
compass-agent/rig-4708-stop-sweeps-sessions

Conversation

@rigel-mintaka

@rigel-mintaka rigel-mintaka commented Oct 6, 2026 •

Copy link
Copy Markdown
Contributor

This PR is part of a stack containing 2 PRs:

  1. main
  2. fix(runtime): kill the in-container process group when a CLI exec client dies (RIG-4181) #1791
  3. "fix(runner): sweep detached exec sessions when an agent stops (RIG-4708)" (this PR)

Summary

  • The CLI stop wrapper (fix(runtime): kill the in-container process group when a CLI exec client dies (RIG-4181) #1791) kills the exec's process group. Children that called setsid escape it: shell commands run without a tty, and SDK-spawned MCP servers, debug adapters and eval kernels.
  • New optional backend capability runtime.SessionSweeper, implemented by PodmanCLI and AppleContainerCLI. As the agent uid, it runs a POSIX sh /proc sweep that SIGKILLs every non-zombie process outside PID 1's session (and outside its own). It rescans every 100 ms until none remain, with a 10 s bound.
    • It reads whole stat files and parses after the last ) , so a crafted comm cannot fake its session.
    • A kill that fails with EPERM (another uid) is not counted.
    • PID 1's session holds the container keep-alive, so the container stays up for an in-place reload.
  • agentHost.Stop sweeps after stopping the stream and logs a failure. Reload sweeps before relaunch; on failure it marks the session ERRORED and returns the error. Remove is unchanged.
  • RefreshSecrets now takes the container lock around the secret install and re-checks the session, so a sweep cannot kill its execs and a stale refresh cannot overwrite a newer session's secrets.

Stacked on #1791.

Verification

  • go test ./internal/runtime/... ./internal/runner/...: pass. -race ./internal/runner/: pass.
  • e2e TestExecStreamingStopSweepsDetachedSession (podman, real agent image) runs a detached bun holding 300 MiB. It passes -count=5 together with the existing exec-stop tests. With the kill removed, it fails. With the earlier no-wait sweep, it failed 1 of 3 runs.
  • TestReloadSweepFailureMarksErrored: reported READY before the markErrored fix, and passes after it.
  • TestStopAndReloadSweepExecSessions checks the workload id and uid on both paths.
  • golangci-lint: no findings on the changed files.
  • Reviews: correctness (1 high, 2 medium, 2 low, all fixed), then re-reviewed. The reviewer also smoke-ran the script in a PID namespace: a 600 MB target, a comm containing a newline, and a comm containing ) .

Risks

  • Stop now spends up to about 10 s plus one exec per agent. This affects only the CLI backends; microVM and host backends do not implement the sweeper.
  • A process under another uid in a non-PID-1 session survives the sweep by design.

Compatibility

  • Runner-internal only; there are no wire or config changes.

Documentation

  • None needed: no user-facing contract changed.

Refs RIG-4708

rigel-mintaka and others added 3 commits October 6, 2026 15:51
The stop wrapper kills the exec process group, but SDK and shell children
that called setsid escape it. Stop and Reload now run an optional backend
sweep, as the agent uid, that kills every process outside PID 1 session,
keeping the container keep-alive. Stop logs a sweep failure; Reload
returns it so a relaunch never runs beside survivors.

Co-authored-by: Matt Wilkinson <matt@rigel.build>
…IG-4708)

The sweep now waits between rounds (10s bound), so a large process still
exiting is not a failure. It reads whole stat files so a crafted comm
cannot fake its session. A failed Reload sweep marks the session ERRORED.
RefreshSecrets takes the container lock so a sweep cannot kill its execs.

Co-authored-by: Matt Wilkinson <matt@rigel.build>
… (RIG-4708)

Co-authored-by: Matt Wilkinson <matt@rigel.build>
@linear-code

linear-code Bot commented Oct 6, 2026

Copy link
Copy Markdown

RIG-4708

@github-actions

github-actions Bot commented Oct 6, 2026

Copy link
Copy Markdown

Compass engineering docs preview: https://compass-agent-rig-4708-stop.compass-eng-docs.pages.dev

Deployed from compass-agent/rig-4708-stop-sweeps-sessions at 9a1eeb6.

@rigel-mintaka
rigel-mintaka marked this pull request as ready for review October 6, 2026 20:39
@mattwilkinsonn
mattwilkinsonn added this pull request to stack #1819 October 7, 2026 00:17
@mattwilkinsonn

Copy link
Copy Markdown
Contributor

/trunk merge

@trunk-io

trunk-io Bot commented Oct 7, 2026 •

Copy link
Copy Markdown

😎 Stack merged successfully - details.

@trunk-io
trunk-io Bot merged commit 4be37e2 into main Oct 7, 2026
17 checks passed
@trunk-io
trunk-io Bot deleted the compass-agent/rig-4708-stop-sweeps-sessions branch October 7, 2026 02:01
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants