Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
Original file line number Diff line number Diff line change
@@ -0,0 +1,25 @@
# Android emulator SystemServer restart + SystemUI ANR deadlock under host load spikes (2026-07-28)

Observed on attach-oom-11fd r2 while 3 slots were active (host load avg spiked to ~168):

**Symptom chain:** mid-run `adb shell input tap` fails with `Broken pipe (32)` then `Can't find service: input`;
`sys.boot_completed` still reads 1; `uiautomator dump` returns `Killed` (RC=137); qemu process stays alive.
Cause: the guest's system_server restarted under memory/CPU pressure — treat those adb signatures as
"guest rebooting", poll `getprop sys.boot_completed` (bounded loop) instead of assuming an adb wedge.

**Aftermath:** SystemUI can deadlock afterwards: persistent `System UI isn't responding` ANR dialog,
`Wait` taps are no-ops, guest CPU idle (so not starvation), `dumpsys window` shows
`mCurrentFocus=... Application Not Responding: com.android.systemui`, SystemUI in S state, and a shell-user
`kill <pid>` silently does nothing (PID unchanged).

**Recovery (confirmed):** `adb root` works on the kilo AVDs (`restarting adbd as root`), then
`adb shell kill -9 <systemui-pid>`; system_server restarts SystemUI in ~20 s, the ANR dialog clears, and the
app process, adb reverse mappings, login session, and conversation state all survive. No emulator relaunch
(never consume a launch attempt for this). Afterwards Maestro, which also wedges under the load
(`maestro hierarchy` empty/timing out, `maestro test` never writing its first log line), works again.

**Related:** plain `pnpm dev:stop` (runbook-mandated when the slot release leaves the stack up) tears down
the SHARED `kiloclaw-docker-tcp` socat on 127.0.0.1:23750 even though `e2e-slot.sh release` first prints
"Leaving Docker infrastructure running". After cleanup, 23750 having no listener is expected; the next
`dev:start` recreates it on demand. Do not "repair" it by hand (never create a socat you did not start).
The KiloClaw sandbox container itself survives both the release and dev:stop.
Original file line number Diff line number Diff line change
@@ -0,0 +1,14 @@
# Android DocumentsUI picker: stale-dump tap race + reliable sonner toast capture

Two Android-emulator techniques confirmed on attach-oom-11fd (2026-07-28):

1. `uiautomator dump` bounds can describe a view the picker has already left (a tap aimed at them hit the wrong
Recent row). Before an `adb input tap`, cross-check the same node's bounds in a fresh
`maestro --device <serial> hierarchy` — only tap when both agree. Prefer Maestro flows for navigation and
reserve adb taps for the final pick when sub-second screenshot timing matters.
2. `sonner-native` toasts are catchable deterministically with an adb tap immediately followed by burst
screencaps at +0.5s/+1.2s/+2.2s/+3.5s (`adb exec-out screencap -p > f.png`). A `maestro test` flow's
teardown latency (seconds) is too slow — the toast dismisses before the shell regains control. The first
burst frame usually still shows the picker; the second (+1.2s) reliably showed the toast.
3. Files `adb push`ed to /sdcard/Download appear in the picker's Downloads root immediately, but only enter its
Recent view after being picked once — do not rely on Recent containing a freshly pushed fixture.
Original file line number Diff line number Diff line change
@@ -0,0 +1,7 @@
# `dev:start` refuses on `kiloclaw-docker-tcp:23750` even though the shared bridge is healthy — retry once

Symptom: `pnpm dev:start` fails with `Refusing to share occupied worktree service ports: kiloclaw-docker-tcp:23750`, while `curl http://127.0.0.1:23750/_ping` (and even `/v1.44/_ping`) returns `OK` with an `Api-Version` header — the exact thing the runner's own probe requires.

Cause: the reuse branch in `dev/local/cli.ts` only runs when `probeDockerApi(23750)` (`dev/local/docker-api-probe.ts`) succeeds inside a 500 ms window. A momentarily slow Docker daemon or a freshly (re)started `socat` makes that one probe time out, and the runner then reads the occupied port as a foreign conflict instead of a reusable bridge. The port is host-wide and offset-independent, so no `KILO_PORT_OFFSET` change can clear this refusal.

Fix: retry the same `dev:start` command once — the probe is re-run and the second attempt typically prints `Reusing host kiloclaw-docker-tcp on 23750` and proceeds. Only if the refusal repeats, investigate the bridge itself per `kiloclaw-docker-tcp-absent-from-worktree-status.md` (`curl` the unversioned `/_ping`, check `lsof -iTCP:23750`); never kill a foreign-owned `socat`, and never "fix" this by picking another offset.
Original file line number Diff line number Diff line change
@@ -1,7 +1,7 @@
# Long interactive kilo sessions die or wedge — relaunch fresh with a continuation handoff

Symptom: an orchestrator running as `kilo run --interactive` stops making progress: the process died (stream stall, typically 10–15 minutes into a long run), or it sits wedged after a provider error (`--interactive --auto` hard-wedges on these). Log bytes stagnate while the tmux window looks alive.
Symptom: an orchestrator running as `kilo run --interactive` stops making progress: the process died (stream stall, typically 10–15 minutes into a long run), it sits wedged after a provider error (`--interactive --auto` hard-wedges on these), or it dies loudly with a hard, repeated SDK error — observed: `Invalid prompt: The messages do not match the ModelMessage[] schema.` printed on every retry. Log bytes stagnate while the tmux window looks alive; a frozen context counter and queued steers that never flush confirm the silent variant, while the hard-error variant is detectable immediately from the pane with no stagnation timer.

Cause: provider stream stalls kill long kilo runs; interactive sessions surface provider errors as a wedge the session cannot recover from. Nothing about the plan or repository is wrong.
Cause: provider stream stalls kill long kilo runs; interactive sessions surface provider errors as a wedge the session cannot recover from, and a poisoned message history surfaces as the repeated schema error. Nothing about the plan or repository is wrong.

Fix: detect via log-byte stagnation plus pane inspection, not just process exit. Kill the window and relaunch a **fresh** session (never `--continue` or `--session`) with a continuation handoff: the original handoff plus everything observably done — commits (`git log`), PR number and state, review/E2E rounds passed, resources still held — so the new session verifies and continues instead of redoing. Distrust the dead session's last claims: re-verify anything it reported green that is not independently evidenced (a passing check in CI, a commit, a resolved thread). Related: `kilo-run-exits-0-without-verdict.md` covers the same stall killing non-interactive role-agent rounds.
Fix: detect via log-byte stagnation plus pane inspection, not just process exit — and on the hard schema error, kill immediately rather than waiting out a timer. Kill the window and relaunch a **fresh** session (never `--continue` or `--session`: the poisoned history is exactly what must not be carried over) with a continuation handoff: the original handoff plus everything observably done — commits (`git log`), PR number and state, review/E2E rounds passed, resources still held — so the new session verifies and continues instead of redoing. Distrust the dead session's last claims: re-verify anything it reported green that is not independently evidenced (a passing check in CI, a commit, a resolved thread). Related: `kilo-run-exits-0-without-verdict.md` covers the same stall killing non-interactive role-agent rounds.
Original file line number Diff line number Diff line change
@@ -0,0 +1,28 @@
# KiloClaw chat attachment uploads fail locally: R2 media secrets absent machine-wide

> **RESOLVED 2026-07-28 (r2):** the `R2_*_KILOCHAT_MEDIA` pair now exists in the Secrets Store and
> `dev:start` provisions it from that canonical source like any source-backed secret
> (`⊕ secret: R2_ACCESS_KEY_ID_KILOCHAT_MEDIA @from …`); verified in the local secrets-store KV and
> end-to-end (`/v1/attachments/init 200`, chip `ready`, send 201). Everything below is the historical
> r1 record — the "no canonical source" claim is stale. If the symptom ever recurs, first check whether
> the store lost the pair again, then whether the worktree's `.wrangler/state` secrets KV is stale
> (re-run `dev:start`).

Symptom: mobile E2E on the KiloClaw chat composer (`chat/[sandbox-id]/[conversation-id]`): picking an under-limit
file adds a chip that immediately goes to `failed` (a11y `Retry upload for <name>`); kilo-chat logs
`POST /v1/attachments/init 500` with `Error: Secret "R2_ACCESS_KEY_ID_KILOCHAT_MEDIA" not found`.

Cause: `services/kilo-chat/wrangler.jsonc` binds `R2_ACCESS_KEY_ID`/`R2_SECRET_ACCESS_KEY` to Secrets-Store
secrets `R2_*_KILOCHAT_MEDIA` (store 342a86d9e3a94da698e82d0c6e2a36f0) with `MEDIA_BUCKET` declared
`remote: true`. Those secrets have no canonical local source: `.dev.vars.example` documents manual creation
from a personal Cloudflare R2 API token, and `dev:env`/`dev:start` only warn ("Missing and no source value").
Checked 2026-07-28: 0 of 42 worktrees' `services/kilo-chat/.wrangler/state/v3/secrets-store` contain an R2 key;
`pnpm dev:worktree:prepare` + `KILO_PORT_OFFSET=<n> pnpm dev:env -y kilo-chat` does not provision them. Every
chip upload 500s deterministically (initial attempt and in-app retry alike). Attachment E2E assertions that
require a chip to reach `ready`/send/render are environment-blocked until someone creates the two secrets
with a real R2 token. Oversized-rejection assertions (pre-read size gate) are unaffected and fully testable.

Related: a `prep`-style phase that ends with `e2e-slot.sh release` stops the worktree stack (slot/stack are one
resource), so a handoff claiming "stack is up" can be stale by verifier time — check `pnpm dev:status --json`
and restart per the runbook quickstart (with the handoff's KILO_PORT_OFFSET prefix on dev:env/dev:start);
the docker-local KiloClaw sandbox container survives stack restarts.
Original file line number Diff line number Diff line change
@@ -0,0 +1,7 @@
# `kiloclaw-docker-tcp` absent from a worktree's `dev:status` is expected when the shared 23750 bridge answers

Symptom: `pnpm dev:status --json` for a worktree whose stack started cleanly lists no `kiloclaw-docker-tcp` service row, and the `dev:start` argv named `kiloclaw` (which expands the group that includes it). Provisioning or starting a docker-local KiloClaw instance looks under-provisioned.

Cause: `kiloclaw-docker-tcp` is a stateless loopback `socat` proxy to the shared Docker socket on `127.0.0.1:23750` — the one host-wide shared port the runbook allows. When another worktree (or an earlier stack) already owns that listener, this worktree's runner reuses it and records no service row of its own.

Fix: gate on the bridge, not the row: `curl http://127.0.0.1:23750/v1.44/_ping` must print `OK`. Only if that fails, start the bridge yourself per `services/kiloclaw/README.md` (`socat TCP-LISTEN:23750,bind=127.0.0.1,reuseaddr,fork UNIX-CONNECT:/var/run/docker.sock`). Never kill a `socat` owned by another worktree (runbook, Host Networking Safety) — `lsof -iTCP:23750` shows the owning PID and it is shared by every docker-local instance on the host, not just yours.
Original file line number Diff line number Diff line change
@@ -1,7 +1,10 @@
# Worktree slug hashing to port offset 2000 collides with macOS AirPlay Receiver (5000/7000)
# Worktree slug-hash port offset collides with macOS AirPlay Receiver (5000/7000) or another worktree's live stack

Symptom: `pnpm dev:start` fails with `Refusing to share occupied worktree service ports: nextjs:5000` even with no other dev stack running; `lsof -iTCP:5000` shows `ControlCe` (macOS AirPlay Receiver), which also holds 7000.
Symptom: `pnpm dev:start` fails with `Refusing to share occupied worktree service ports: …`. Two observed occupants of the hash-chosen offset:

Cause: `computePortOffset` derives the offset from the worktree slug hash (`KILO_PORT_OFFSET=auto` uses the same hash — it does not probe for free ports); a slug that buckets to 20 lands nextjs on 3000+2000=5000 permanently. Only `dev:status` reads the offset back from the manifest, so `dev:start`/`dev:restart`/`dev:env` all recompute from the environment.
1. Offset 2000 (nextjs 5000) — `lsof -iTCP:5000` shows `ControlCe` (macOS AirPlay Receiver), which also holds 7000, even with no other dev stack running.
2. Offset 2400 — another worktree's live `kilo-dev-*` stack legitimately holds the ports; nothing is wrong with the machine, the hash just bucketed two worktrees onto the same offset.

Fix: prefix every port-computing dev command with an explicit collision-free offset, e.g. `KILO_PORT_OFFSET=2100 pnpm dev:start ...` (2100 clears AirPlay: nextjs 5100, metro 10181, wrangler 10889+). Per-command prefix only — never `export` it; `apps/mobile/e2e/AGENTS.md` forbids the export because stale shell values select the wrong bundle endpoints, and that rule stands. `dev:status`/`dev:seed`/`login.sh` are manifest- or DB-driven and need no prefix. Restate the prefix rule in every device-phase handoff; a `dev:restart` without it silently recomputes ports at the hash offset and breaks the stack's URL wiring.
Cause: `computePortOffset` derives the offset from the worktree slug hash (`KILO_PORT_OFFSET=auto` uses the same hash — it does not probe for free ports); a slug that buckets to 20 lands nextjs on 3000+2000=5000 permanently, and two slugs can bucket to the same offset. Only `dev:status` reads the offset back from the manifest, so `dev:start`/`dev:restart`/`dev:env` all recompute from the environment.

Fix: prefix every port-computing dev command with an explicit collision-free offset, e.g. `KILO_PORT_OFFSET=2100 pnpm dev:start ...` (2100 clears AirPlay: nextjs 5100, metro 10181, wrangler 10889+); 2900 is probe-verified against the same full service-port set (nextjs 5900, metro 10981, kiloclaw 11695, kilo-chat 11708, event-service 11709, session-ingest 11700). Probe before committing to an offset: compute every service port from it and check `lsof` — do not probe only the one port that errored. Per-command prefix only — never `export` it; `apps/mobile/e2e/AGENTS.md` forbids the export because stale shell values select the wrong bundle endpoints, and that rule stands. `dev:status`/`dev:seed`/`login.sh` are manifest- or DB-driven and need no prefix. Restate the prefix rule in every device-phase handoff; a `dev:restart` without it silently recomputes ports at the hash offset and breaks the stack's URL wiring.
147 changes: 135 additions & 12 deletions apps/mobile/src/components/kilo-chat/message-attachment-picker.test.ts
Original file line number Diff line number Diff line change
@@ -1,16 +1,27 @@
import { type AlertButton } from 'react-native';
import * as DocumentPicker from 'expo-document-picker';
import * as ImagePicker from 'expo-image-picker';
import { afterEach, beforeEach, describe, expect, it, vi } from 'vitest';

import { pickCameraImage, pickLibraryImages } from './message-attachment-picker';
import {
materializeAttachment,
pickCameraImage,
pickFiles,
pickLibraryImages,
} from './message-attachment-picker';
import { normalizeAttachmentSelection } from './message-attachment-state';

const fileSystemMock = vi.hoisted(() => ({
File: vi.fn(function File(this: { name: string; size: number; type: string }, uri: string) {
this.name = uri.split('/').at(-1) ?? 'attachment';
this.size = 12;
this.type = 'image/jpeg';
}),
}));
const fileSystemMock = vi.hoisted(() => {
const state = { fileSize: 12 };
return {
state,
File: vi.fn(function File(this: { name: string; size: number; type: string }, uri: string) {
this.name = uri.split('/').at(-1) ?? 'attachment';
this.size = state.fileSize;
this.type = 'image/jpeg';
}),
};
});

const reactNativeMock = vi.hoisted(() => ({
alert: vi.fn(),
Expand Down Expand Up @@ -44,6 +55,7 @@ const requestMediaLibraryPermissionsMock = vi.mocked(
);
const launchCameraMock = vi.mocked(ImagePicker.launchCameraAsync);
const launchImageLibraryMock = vi.mocked(ImagePicker.launchImageLibraryAsync);
const getDocumentAsyncMock = vi.mocked(DocumentPicker.getDocumentAsync);

function deniedPermissionResponse(): ImagePicker.PermissionResponse {
return {
Expand All @@ -59,6 +71,7 @@ describe('message attachment picker permissions', () => {

beforeEach(() => {
vi.clearAllMocks();
fileSystemMock.state.fileSize = 12;
globalThis.fetch = vi.fn(async () => {
await Promise.resolve();
return new Response(new Blob(['x'], { type: 'image/jpeg' }));
Expand Down Expand Up @@ -117,13 +130,123 @@ describe('message attachment picker permissions', () => {
expect(reactNativeMock.alert).not.toHaveBeenCalled();
expect(result).toEqual([
{
input: {
blob: expect.objectContaining({ type: 'image/jpeg' }),
filename: 'photo.jpg',
uri: 'file:///photo.jpg',
filename: 'photo.jpg',
mimeType: 'image/jpeg',
size: 42,
isImage: true,
},
]);
expect(globalThis.fetch).not.toHaveBeenCalled();
});

it('returns normalized document selections without materializing', async () => {
getDocumentAsyncMock.mockResolvedValue({
canceled: false,
assets: [
{
uri: 'file:///notes.pdf',
name: 'notes.pdf',
mimeType: 'application/pdf',
size: 2048,
lastModified: 0,
},
],
});

const result = await pickFiles();

expect(result).toEqual([
{
uri: 'file:///notes.pdf',
filename: 'notes.pdf',
mimeType: 'application/pdf',
size: 2048,
isImage: false,
},
]);
expect(globalThis.fetch).not.toHaveBeenCalled();
});

it('materializeAttachment fetches the file uri and returns a picked attachment', async () => {
const attachment = normalizeAttachmentSelection({
uri: 'file:///photo.jpg',
name: 'photo.jpg',
mimeType: 'image/jpeg',
size: 42,
});

const result = await materializeAttachment(attachment);

expect(globalThis.fetch).toHaveBeenCalledTimes(1);
expect(globalThis.fetch).toHaveBeenCalledWith('file:///photo.jpg');
expect(result).toEqual({
input: {
blob: expect.objectContaining({ type: 'image/jpeg' }),
filename: 'photo.jpg',
mimeType: 'image/jpeg',
},
localUri: 'file:///photo.jpg',
});
});

it('uses the larger of measured file size and reported picker size', async () => {
fileSystemMock.state.fileSize = 4096;
launchImageLibraryMock.mockResolvedValue({
assets: [
{
uri: 'file:///photo.jpg',
fileName: 'photo.jpg',
mimeType: 'image/jpeg',
fileSize: 12,
height: 100,
width: 100,
},
],
canceled: false,
});

const result = await pickLibraryImages();

expect(result).toEqual([
{
uri: 'file:///photo.jpg',
filename: 'photo.jpg',
mimeType: 'image/jpeg',
size: 4096,
isImage: true,
},
]);
expect(globalThis.fetch).not.toHaveBeenCalled();
});

it('uses reported picker size when the file stat is unreadable', async () => {
fileSystemMock.state.fileSize = 0;
launchImageLibraryMock.mockResolvedValue({
assets: [
{
uri: 'file:///photo.jpg',
fileName: 'photo.jpg',
mimeType: 'image/jpeg',
fileSize: 2048,
height: 100,
width: 100,
},
localUri: 'file:///photo.jpg',
],
canceled: false,
});

const result = await pickLibraryImages();

expect(result).toEqual([
{
uri: 'file:///photo.jpg',
filename: 'photo.jpg',
mimeType: 'image/jpeg',
size: 2048,
isImage: true,
},
]);
expect(globalThis.fetch).not.toHaveBeenCalled();
});
});
Loading
Loading