fix(cloud-agent-next): preserve classified assistant terminal failures

docs/superpowers/specs/2026-07-22-cloud-agent-e2e-health-design.md

@@ -0,0 +1,87 @@
+# Cloud Agent E2E Health Reliability Design
+
+## Goal
+
+Make the local fake-LLM smoke matrix a reliable health check for the real Cloud Agent path: Worker, Durable Object, sandbox container, wrapper, Kilo, and persisted run reporting. The harness must distinguish genuine product regressions from expected warm-session preparation chatter and from local container-engine cleanup races.
+
+## Scope
+
+This change is limited to the local E2E harness under `services/cloud-agent-next/test/e2e` and its focused unit tests and documentation. It does not change Worker, wrapper, or UI behavior. It does not add retries.
+
+## Warm-Reuse Health Contract
+
+The `hot` and `cold-hot` scenarios must continue to prove all of the following:
+
+- the follow-up reaches its expected terminal state;
+- the first Kilo event arrives within the existing observation window;
+- the original primary sandbox remains present;
+- no new sandbox container appears; and
+- no live preparation step that represents cold workspace work starts.
+
+Preparation snapshots replayed when a stream connects are historical state, not evidence of new work. Live `attempt_started`, `attempt_completed`, progress, and completion frames are also not sufficient by themselves to prove reprovisioning. The harness will therefore classify only version 2 `preparing` frames with `action: "step_started"`.
+
+The following steps are expected during a warm verification and do not indicate reprovisioning:
+
+- `sandbox_provision`
+- `sandbox_boot`
+- `kilo_server`
+
+Any other live `step_started` preparation step is cold-path work and fails warm reuse. This deliberately fails closed for newly introduced step names: a new step must be explicitly classified before the health check accepts it. Legacy unversioned `preparing` frames also fail warm reuse because the harness cannot prove that they are harmless snapshots or warm verification.
+
+Failure output will report the unexpected preparation step names instead of only printing `noPrepare=false`.
+
+## Matrix Isolation Contract
+
+After every matrix row, the harness will:
+
+1. identify sandbox families created after the matrix baseline;
+2. kill their currently running primary/proxy containers using the existing exact-family matching;
+3. wait for each selected family to disappear; and
+4. poll all non-baseline sandbox containers until none exist continuously for five seconds.
+
+The continuous absence window covers the observed local container-engine `destroy -> create -> destroy` churn after an external kill. If a new non-baseline sandbox appears during the window, the stability timer resets. The total quiescence wait is bounded.
+
+If cleanup does not quiesce, the matrix records an explicit cleanup failure and stops before running another scenario in contaminated state. It does not silently warn and continue, sleep for a fixed interval, or retry a failed product scenario.
+
+Baseline containers are never killed or treated as cleanup failures.
+
+## Components
+
+### Preparation classification
+
+A small pure helper in `lifecycle.ts` will return the unexpected cold preparation steps observed in a stream event list. Both `hot` and `cold-hot` will use it.
+
+### Sandbox quiescence
+
+`sandbox-control.ts` will own a bounded stable-absence polling helper because it already owns Docker discovery and family operations. Its Docker executor and timing inputs will be injectable for deterministic unit tests while production callers use defaults.
+
+`smoke.ts` will use the helper after its existing family cleanup and convert failure into an explicit matrix result before stopping.
+
+### Documentation
+
+The E2E README will describe the warm-reuse and between-row isolation assertions so operators know what a failure means.
+
+## Tests
+
+Focused unit tests will cover:
+
+- snapshots and warm verification steps do not count as cold preparation;
+- cold-path and legacy preparation events do count;
+- the stable-absence window succeeds only after uninterrupted absence;
+- a transient recreation resets the stability window;
+- persistent non-baseline containers time out; and
+- baseline containers are ignored.
+
+Focused verification will run the new unit tests, service typecheck, formatting checks, and `git diff --check`.
+
+Runtime verification will run the complete 15-row matrix against the current local stack with `WORKER_URL=http://localhost:10594`. Completion requires 15/15 scenario results and no matrix-cleanup failure.
+
+## Deferred Work
+
+The following remain separate follow-ups:
+
+- suppressing no-op preparation attempts in production to remove the warm UI flash;
+- retrying matrix scenarios;
+- removing stopped Docker container corpses;
+- adding provisioning elapsed-time landmarks; and
+- cleaning pre-existing proxy containers.

packages/worker-utils/src/cloud-agent-queue-report.ts

@@ -35,6 +35,8 @@ export const CloudAgentRunFailureClassifications = [
   { failureStage: 'agent_activity', failureCode: 'payment_required' },
   { failureStage: 'agent_activity', failureCode: 'model_missing' },
   { failureStage: 'agent_activity', failureCode: 'wrapper_error_after_activity' },
+  { failureStage: 'agent_activity', failureCode: 'payment_required' },
+  { failureStage: 'agent_activity', failureCode: 'model_missing' },
   { failureStage: 'interruption', failureCode: 'user_interrupt' },
   { failureStage: 'interruption', failureCode: 'container_shutdown' },
   { failureStage: 'interruption', failureCode: 'system_interrupt' },

services/cloud-agent-next/src/agent-sandbox/cloudflare/cloudflare-agent-sandbox.test.ts

@@ -1666,23 +1666,40 @@ describe('CloudflareAgentSandbox', () => {
   it('deletes session resources without destroying a shared sandbox', async () => {
     const destroy = vi.fn();
     const deleteSession = vi.fn().mockResolvedValue(undefined);
-    const sandbox = new CloudflareAgentSandbox({} as Env, metadata(), {
-      resolveSandbox: () =>
-        ({
-          getSession: vi.fn().mockResolvedValue({
-            exec: vi.fn().mockResolvedValue({ exitCode: 0, stdout: '', stderr: '' }),
-          }),
-          deleteSession,
-          destroy,
-        }) as unknown as SandboxInstance,
-    });
+    const sandbox = new CloudflareAgentSandbox(
+      {} as Env,
+      metadata({ sandboxId: `usr-${'a'.repeat(48)}` as TestSandboxId }),
+      {
+        resolveSandbox: () =>
+          ({
+            getSession: vi.fn().mockResolvedValue({
+              exec: vi.fn().mockResolvedValue({ exitCode: 0, stdout: '', stderr: '' }),
+            }),
+            deleteSession,
+            destroy,
+          }) as unknown as SandboxInstance,
+      }
+    );
 
     await sandbox.delete('explicit');
 
     expect(deleteSession).toHaveBeenCalledWith('agent_cloudflare');
     expect(destroy).not.toHaveBeenCalled();
   });
 
+  it('destroys an isolated sandbox when deleting its only session', async () => {
+    const destroy = vi.fn().mockResolvedValue(undefined);
+    const deleteSession = vi.fn();
+    const sandbox = new CloudflareAgentSandbox({} as Env, metadata(), {
+      resolveSandbox: () => ({ destroy, deleteSession }) as unknown as SandboxInstance,
+    });
+
+    await sandbox.delete('explicit');
+
+    expect(destroy).toHaveBeenCalledOnce();
+    expect(deleteSession).not.toHaveBeenCalled();
+  });
+
   it('maps infrastructure recovery to destructive Cloudflare sandbox replacement', async () => {
     const destroy = vi.fn().mockResolvedValue(undefined);
     const sandbox = new CloudflareAgentSandbox({} as Env, metadata(), {

services/cloud-agent-next/src/agent-sandbox/cloudflare/cloudflare-agent-sandbox.ts

@@ -24,6 +24,7 @@ import { SANDBOX_SLEEP_AFTER_SECONDS } from '../../core/lease.js';
 import {
   generateSandboxId,
   getSandboxNamespace,
+  isGeneratedSharedSandboxId,
   isOrgInList,
   MANAGED_SCM_OUTBOUND_HANDLER,
 } from '../../sandbox-id.js';
@@ -845,8 +846,9 @@ export class CloudflareAgentSandbox implements AgentSandbox {
   }
 
   async delete(reason: SandboxDeleteReason): Promise<void> {
-    const sandbox = await this.getSandbox();
-    if (reason === 'recovery') {
+    const sandboxId = await this.resolveSandboxId();
+    const sandbox = this.resolveSandbox(sandboxId);
+    if (reason === 'recovery' || !isGeneratedSharedSandboxId(sandboxId)) {
       await sandbox.destroy();
       return;
     }

services/cloud-agent-next/src/kilo/wrapper-client.test.ts

@@ -360,7 +360,13 @@ describe('WrapperClient', () => {
       );
 
       expect(result.kiloSessionId).toBe('kilo_sess_1');
-      expect(transport.request).toHaveBeenNthCalledWith(1, 'POST', '/session/ready', readyRequest);
+      expect(transport.request).toHaveBeenNthCalledWith(
+        1,
+        'POST',
+        '/session/ready',
+        readyRequest,
+        expect.any(String)
+      );
       expect(transport.request).toHaveBeenNthCalledWith(
         2,
         'POST',
@@ -381,7 +387,8 @@ describe('WrapperClient', () => {
             condenseOnComplete: false,
           },
           session: binding,
-        })
+        }),
+        expect.any(String)
       );
       expect('executeSession' in client).toBe(false);
       expect(session.exec).not.toHaveBeenCalled();
@@ -2282,6 +2289,91 @@ describe('WrapperClient', () => {
       await expect(client.health()).rejects.toThrow(WrapperError);
     });
 
+    it('retains the actual curl status and content type for malformed health diagnostics', async () => {
+      const session = createMockSession(command => {
+        const marker = command.match(/__KILO_WRAPPER_RESPONSE_[A-Za-z0-9-]+__/)?.[0];
+        if (!marker) throw new Error('Missing curl response metadata marker');
+        return {
+          exitCode: 0,
+          stdout: `<html>private-wrapper-error</html>\n${marker}500\ttext/html`,
+        };
+      });
+      const withFields = vi.spyOn(logger, 'withFields').mockReturnValue(logger);
+      const logError = vi.spyOn(logger, 'error').mockImplementation(() => logger);
+      const client = new WrapperClient({ session, port: defaultPort });
+
+      await expect(client.health()).rejects.toMatchObject({ code: 'PARSE_ERROR' });
+
+      expect(withFields).toHaveBeenCalledWith(
+        expect.objectContaining({
+          path: '/health',
+          statusCode: 500,
+          contentType: 'text/html',
+        })
+      );
+      expect(JSON.stringify(withFields.mock.calls)).not.toContain('private-wrapper-error');
+      withFields.mockRestore();
+      logError.mockRestore();
+    });
+
+    it('fingerprints malformed health responses without retaining their body', async () => {
+      const body = '<html>token=secret-health-body</html>';
+      const transport: WrapperTransport = {
+        request: vi.fn().mockResolvedValue(
+          new Response(body, {
+            status: 500,
+            headers: { 'Content-Type': 'text/html; charset=utf-8' },
+          })
+        ),
+      };
+      const withFields = vi.spyOn(logger, 'withFields').mockReturnValue(logger);
+      const logError = vi.spyOn(logger, 'error').mockImplementation(() => logger);
+      const client = new WrapperClient({
+        session: createMockSession(createSuccessResponse({})),
+        port: defaultPort,
+        transport,
+      });
+
+      const expectedHash = Array.from(
+        new Uint8Array(await crypto.subtle.digest('SHA-256', new TextEncoder().encode(body))),
+        byte => byte.toString(16).padStart(2, '0')
+      ).join('');
+      let thrown: unknown;
+      try {
+        await client.health();
+      } catch (error) {
+        thrown = error;
+      }
+
+      expect(thrown).toMatchObject({
+        message: 'Failed to parse wrapper response',
+        code: 'PARSE_ERROR',
+      });
+      expect(String(thrown)).not.toContain(body);
+      expect(withFields).toHaveBeenCalledWith(
+        expect.objectContaining({
+          method: 'GET',
+          path: '/health',
+          requestId: expect.any(String),
+          contentType: 'text/html; charset=utf-8',
+          responseBytes: new TextEncoder().encode(body).byteLength,
+          responseSha256: expectedHash,
+          statusCode: 500,
+        })
+      );
+      expect(JSON.stringify(withFields.mock.calls)).not.toContain('secret-health-body');
+      expect(logError).toHaveBeenCalledWith('Failed to parse wrapper HTTP response');
+      expect(transport.request).toHaveBeenCalledWith(
+        'GET',
+        '/health',
+        undefined,
+        expect.any(String)
+      );
+
+      withFields.mockRestore();
+      logError.mockRestore();
+    });
+
     it('handles curl exit codes', async () => {
       const session = createMockSession(createCurlError(28, 'Operation timed out'));
       const client = new WrapperClient({ session, port: defaultPort });