From c94ba9aa420de298b56441710e1f96af96f22df5 Mon Sep 17 00:00:00 2001 From: =?UTF-8?q?Igor=20=C5=A0=C4=87eki=C4=87?= Date: Wed, 29 Jul 2026 13:32:57 +0200 Subject: [PATCH 01/34] chore(workflow): dedupe and prune learnings Delete learnings whose fixes landed in scripts or WORKFLOW.md (slot teardown, steer.sh, dispatch-role.sh, monitor rules, TTY rule, env-facts rule, PR image upload), merge overlapping entries (kilo startup crashes, Android emulator claims, Android cold-start deep link, device-auth pending limit, iOS26 Maestro quirks), and trim stale or contradicting paragraphs. --- .kilo_workflow/WORKFLOW.md | 4 +- ...mserver-restart-systemui-anr-under-load.md | 5 -- ...v-stack-died-mid-round-restart-in-place.md | 19 +++++++ ...tching-kilo-agents-from-another-harness.md | 21 -------- ...2e-slot-owner-must-be-dedicated-session.md | 17 ------- ...-release-stopped-another-sections-stack.md | 49 ------------------- .../github-pr-no-image-upload-api.md | 13 ----- .../handoff-env-facts-must-be-inline.md | 7 --- ...os-simulator-cannot-starve-location-fix.md | 3 -- ...kilo-provider-model-not-found-transient.md | 2 +- .../kilo-run-crashes-on-credential-write.md | 43 ---------------- .../kilo-run-interactive-requires-tty.md | 7 --- .../kilo-run-session-not-found-or-db-race.md | 7 --- .../kilo-run-startup-crash-transient.md | 39 +++++++++++++++ ...claw-chat-attachments-r2-secret-missing.md | 36 +++++--------- ...-docker-tcp-absent-from-worktree-status.md | 2 + ...mobile-android-claim-race-at-visibility.md | 13 ----- ...android-dev-client-cold-start-deep-link.md | 38 +++++++++----- .../mobile-android-emulator-claims.md | 19 +++++++ ...bile-android-emulator-r7-envelope-notes.md | 13 ----- ...-deep-link-reload-outlasts-flow-budgets.md | 23 --------- .../mobile-android-maestro-flow-pitfalls.md | 4 ++ ...bile-android-stale-claims-explicit-port.md | 20 -------- ...le-device-auth-ip-limit-shared-sections.md | 20 -------- .../mobile-device-auth-stuck-pending-rows.md | 49 +++++++++++-------- .../mobile-ios26-maestro-under-load.md | 7 +-- ...-attachments-capability-gate-blocks-e2e.md | 12 ++--- ...maestro-between-flow-springboard-bounce.md | 19 ------- ...hestrator-scrollback-before-intervening.md | 25 ---------- ...mulators-left-booted-after-slot-release.md | 43 ---------------- .../steering-a-running-kilo-session.md | 11 ----- .kilo_workflow/steer.sh | 2 +- 32 files changed, 159 insertions(+), 433 deletions(-) create mode 100644 .kilo_workflow/learnings/dev-stack-died-mid-round-restart-in-place.md delete mode 100644 .kilo_workflow/learnings/dispatching-kilo-agents-from-another-harness.md delete mode 100644 .kilo_workflow/learnings/e2e-slot-owner-must-be-dedicated-session.md delete mode 100644 .kilo_workflow/learnings/e2e-slot-release-stopped-another-sections-stack.md delete mode 100644 .kilo_workflow/learnings/github-pr-no-image-upload-api.md delete mode 100644 .kilo_workflow/learnings/handoff-env-facts-must-be-inline.md delete mode 100644 .kilo_workflow/learnings/kilo-run-crashes-on-credential-write.md delete mode 100644 .kilo_workflow/learnings/kilo-run-interactive-requires-tty.md delete mode 100644 .kilo_workflow/learnings/kilo-run-session-not-found-or-db-race.md create mode 100644 .kilo_workflow/learnings/kilo-run-startup-crash-transient.md delete mode 100644 .kilo_workflow/learnings/mobile-android-claim-race-at-visibility.md create mode 100644 .kilo_workflow/learnings/mobile-android-emulator-claims.md delete mode 100644 .kilo_workflow/learnings/mobile-android-emulator-r7-envelope-notes.md delete mode 100644 .kilo_workflow/learnings/mobile-android-login-deep-link-reload-outlasts-flow-budgets.md delete mode 100644 .kilo_workflow/learnings/mobile-android-stale-claims-explicit-port.md delete mode 100644 .kilo_workflow/learnings/mobile-device-auth-ip-limit-shared-sections.md delete mode 100644 .kilo_workflow/learnings/mobile-maestro-between-flow-springboard-bounce.md delete mode 100644 .kilo_workflow/learnings/monitor-must-read-orchestrator-scrollback-before-intervening.md delete mode 100644 .kilo_workflow/learnings/simulators-left-booted-after-slot-release.md delete mode 100644 .kilo_workflow/learnings/steering-a-running-kilo-session.md diff --git a/.kilo_workflow/WORKFLOW.md b/.kilo_workflow/WORKFLOW.md index 98874ad592..9702f781ce 100644 --- a/.kilo_workflow/WORKFLOW.md +++ b/.kilo_workflow/WORKFLOW.md @@ -77,7 +77,7 @@ Worker roles are steered by re-dispatching them (see Escalation). The interactiv printf '%s' "$AMENDMENT" | .kilo_workflow/steer.sh
-orchestrator - # long or multi-line text via stdin ``` -It prints `running` when the session took the message immediately and `queued` when the message is waiting behind the active turn; either way it is delivered. A non-zero exit means it is **not** delivered — inspect the target rather than sending a second copy. What the script encodes (details in `learnings/steering-a-running-kilo-session.md`): +It prints `running` when the session took the message immediately and `queued` when the message is waiting behind the active turn; either way it is delivered. A non-zero exit means it is **not** delivered — inspect the target rather than sending a second copy. What the script encodes (details in its header comments): - Enter as its own keystroke after the text. A trailing `Enter` in the same `send-keys` call submits short messages but is swallowed by long ones, which then sit unsent in the composer — the "wedged" session that is really an undelivered message. - Bracketed paste, so a multi-line message stays one prompt. An unbracketed paste submits at every newline, and the first fragment gets acted on before the rest arrives. @@ -194,7 +194,7 @@ The `orchestrator` agent definition pins the model and permissions, so the launc After the handoff the planner stops all hands-on work. It has exactly one job — relaunch or unstick the orchestrator when infrastructure fails (a crashed kilo CLI, a dead tmux window, a hung service). Product, logic, design, and review problems are the orchestrator's, handled by its escalation ladder. Check about every 30 minutes; react immediately when the orchestrator's process exits. -A monitor never kills a live dispatch on a judgment call, never edits the worktree, and never writes to another role's dispatch log — a forged `EXITCODE` line corrupts the dispatcher's void-round detection, the exact signal the contract keys on. Before concluding an orchestrator is misbehaving, read its evidence: its pane scrollback (`tmux capture-pane -t -p -S -`), its scratch directory, and the git log. The orchestrator's handoff may route user questions to its own interactive session, so an answer the monitor never saw can exist there (see `learnings/monitor-must-read-orchestrator-scrollback-before-intervening.md`). The only kill-worthy states are the infrastructure failures named above. +A monitor never kills a live dispatch on a judgment call, never edits the worktree, and never writes to another role's dispatch log — a forged `EXITCODE` line corrupts the dispatcher's void-round detection, the exact signal the contract keys on. Before concluding an orchestrator is misbehaving, read its evidence: its pane scrollback (`tmux capture-pane -t -p -S -`), its scratch directory, and the git log. The orchestrator's handoff may route user questions to its own interactive session, so an answer the monitor never saw can exist there — a monitor once killed a healthy implementer and forged its `EXITCODE` line because it never read the scrollback where the user had already answered. The only kill-worthy states are the infrastructure failures named above. A dead orchestrator window is not automatically a crash — check the scratch directory: diff --git a/.kilo_workflow/learnings/android-emulator-systemserver-restart-systemui-anr-under-load.md b/.kilo_workflow/learnings/android-emulator-systemserver-restart-systemui-anr-under-load.md index b121fc0256..729e3aa1cb 100644 --- a/.kilo_workflow/learnings/android-emulator-systemserver-restart-systemui-anr-under-load.md +++ b/.kilo_workflow/learnings/android-emulator-systemserver-restart-systemui-anr-under-load.md @@ -18,8 +18,3 @@ app process, adb reverse mappings, login session, and conversation state all sur (never consume a launch attempt for this). Afterwards Maestro, which also wedges under the load (`maestro hierarchy` empty/timing out, `maestro test` never writing its first log line), works again. -**Related:** plain `pnpm dev:stop` (runbook-mandated when the slot release leaves the stack up) tears down -the SHARED `kiloclaw-docker-tcp` socat on 127.0.0.1:23750 even though `e2e-slot.sh release` first prints -"Leaving Docker infrastructure running". After cleanup, 23750 having no listener is expected; the next -`dev:start` recreates it on demand. Do not "repair" it by hand (never create a socat you did not start). -The KiloClaw sandbox container itself survives both the release and dev:stop. diff --git a/.kilo_workflow/learnings/dev-stack-died-mid-round-restart-in-place.md b/.kilo_workflow/learnings/dev-stack-died-mid-round-restart-in-place.md new file mode 100644 index 0000000000..3c1976252d --- /dev/null +++ b/.kilo_workflow/learnings/dev-stack-died-mid-round-restart-in-place.md @@ -0,0 +1,19 @@ +# Dev stack died mid-round (killed externally or by a stale slot script) — restart in place, don't re-verify from scratch + +Symptom: mid-E2E, `pnpm dev:status` suddenly prints "No dev session running", the dev client +loses Metro, and flows start failing on connection errors — with your slot still held and no +crash logs of your own. Usual culprits: another section's stale pre-fix `e2e-slot.sh` copy +stopping the wrong worktree's stack on release, or a slot-covering placeholder session dying and +the reaper stopping the now-uncovered stack. + +Cause: the stack was stopped out from under you; nothing about your worktree, data, or device is +wrong. Postgres rows, Durable Object storage, and local D1 files all survive a `dev:stop`. + +Fix (verified twice): while the slot is still held, restart the stack in place with the same +port offset and service group from your handoff — `KILO_PORT_OFFSET= pnpm dev:start +--no-attach ` (ports are deterministic per worktree, so everything comes back on the +same URLs). Then restore what rode on it: restart the CLI relay if the round uses one +(`apps/mobile/e2e/remote-cli.sh exec remote` → `Remote connection enabled.`), re-anchor the dev +client with the deep link, and re-run only the iteration that was in flight — completed +iterations' device-local measurements stay valid. Classify the interruption as environment +interference, never as a product failure. diff --git a/.kilo_workflow/learnings/dispatching-kilo-agents-from-another-harness.md b/.kilo_workflow/learnings/dispatching-kilo-agents-from-another-harness.md deleted file mode 100644 index 74e43003d7..0000000000 --- a/.kilo_workflow/learnings/dispatching-kilo-agents-from-another-harness.md +++ /dev/null @@ -1,21 +0,0 @@ -# Dispatching kilo role agents from a non-kilo harness (timeouts, exit codes, wait loops) - -Symptom: a `kilo run --agent ` dispatched from a harness whose shell tool has a command timeout (Claude Code: 10 minutes) gets killed mid-run. Worse, a piped run (`kilo run ... | tee log`) records the exit status of `tee`, not of kilo, so a crashed agent reports `EXITCODE=0` and reads as a clean pass. Separately, an `until grep -q EXITCODE= "$LOG"` wait loop can false-trigger mid-run when the agent read a document (this one included) that mentions the marker, echoing it into the log. - -Cause: three independent traps — the harness command timeout, `$?` after a pipeline referring to the last stage, and a wait marker that is not unique to the wrapper's final append. - -Fix: run the dispatch inside a tmux window, redirect rather than pipe, and append the exit code after the redirected command: - -```bash -cd "$WORKTREE" -env $(env | grep -oE '^(KILO|OPENCODE)[A-Za-z0-9_]*' | sed 's/^/-u /') \ - kilo run "Review the attached plan per your role definition." \ - --agent plan-reviewer --file msg.txt --file "$PLAN" > "$LOG" 2>&1 -echo "EXITCODE=$?" >> "$LOG" -``` - -The message stays a short literal; the handoff and plan travel via `--file` — see `kilo-run-shell-substitution-executes-backticks.md` for why `"$(cat ...)"` is forbidden. - -Do not pass `--model`/`--variant` for role agents — their definitions in `.kilo/agent/` pin the model, and a flag that drifts from the definition silently disagrees with it. - -Keep the message positional **before** the flags: `--file` takes multiple values and swallows a trailing message as a path (`File not found`). Treat the run as done only when the tmux window is gone **or** the marker is the last line: `tail -1 "$LOG" | grep -q '^EXITCODE=[0-9]'`. diff --git a/.kilo_workflow/learnings/e2e-slot-owner-must-be-dedicated-session.md b/.kilo_workflow/learnings/e2e-slot-owner-must-be-dedicated-session.md deleted file mode 100644 index b50a52cc2f..0000000000 --- a/.kilo_workflow/learnings/e2e-slot-owner-must-be-dedicated-session.md +++ /dev/null @@ -1,17 +0,0 @@ -# E2E slot acquired under a shared/long-lived tmux session name leaks the slot - -Symptom: an E2E slot stays held for tens of minutes with no verifier running; `e2e-slot.sh status` -shows the holder as a long-lived session name (e.g. the planner/starter session) while other -sections starve on a 3-slot machine. The slot is never auto-reclaimed because the holding session -never dies. - -Cause: an orchestrator (not a verifier) ran `e2e-slot.sh acquire` itself, under its window's shared -tmux session name instead of a dedicated per-round verifier session. Slot reclamation keys on the -holder's tmux session being gone; a shared session outlives every round, so the hold is effectively -permanent until someone releases it by hand. - -Fix: only the e2e-verifier acquires a slot, always under its own dedicated session name -(`
-e2e-verifier-