[Refactor & Feature] New Adaptive Agent Loop for Long-Horizon tasks - #22
Merged
Conversation
Signal-driven bounded OODA frames: S0 adaptive depth (elastic budget/posture/prefix-commitment/append-only compression/research ledger) + recursive full-loop subagents; S1 persistent orientation; S2 event-driven re-entry; S3 learning self-calibration; governed proactive outbound (Progressive Trust + ApprovalGate); D1 layered orientation + read-only /orient. All new capabilities default-off, bounded, reversible. Adds docs/design/adaptive_depth_ooda.md + README section. 983 tests pass.
The input area was hard-capped at 4 visible rows, so long wrapped input scrolled earlier lines out of view. Make the height adaptive: grow with content up to ~45% of the terminal (reserving room for the status chrome and transcript), never below the original 4-row floor, still content-sized for short drafts. Add Ctrl+X Ctrl+E to compose the draft in $EDITOR (reusing the pre-wired .md tempfile suffix) and return it to the prompt for review before submit. Add a hermetic test for the cap and update the theme test that pinned the fixed max height.
…, guardrails, recovery, tool-failure autonomy) Systematic fix for super-long tasks being cut short, replacing short-turn/hardcoded/progress-decoupled dead rules with single-decision-point + progress-aware + capacity-scaled governance. (1) Context compression: token-utilization-driven Summarize/Archive/Drop (was message-count), Drop token-only + preserves frozen summaries + recent tail (was nuke-to-4). (2) Adaptive budget: progress-gated continuation past the elastic ceiling to a hard cap (agent.iter_ceiling 80->200, iter_hard_cap 500, stall_rounds), no fixed-count kill. (3) Tool-loop guardrails: progress-aware (halt/nudge suppressed while advancing), StagnationGuard denominator counts only real tool results, thresholds configurable + disable switch. (4) RecoveryBudget: configurable, 0 = unlimited deadline (was hardcoded 300s), scaled action budget. (5) Tool failures route through a single RecoveryCoordinator decision point (_evaluate_tool_failures): recoverable failures fed back for autonomous diagnosis at zero recovery-budget cost, only NON_RECOVERABLE halts — removed 4 max_consecutive_tool_failures->break sites. (6) TurnRecoveryState content one-shots re-arm after progress. (7) _should_stop_after_tool_result is policy-driven (removed hardcoded side-effect tool-name list). (8) Removed the dead ReAct state-machine cluster (415 lines); wired ledger-aware _budget_exhausted_response to the live loops. +26 tests, 1004 pass, zero regression.
…TTY owner) The adaptive input height + external-editor features contended with the concurrent streaming worker (patch_stdout) for the TTY: (1) Ctrl+X Ctrl+E open_in_editor spawned $EDITOR (alternate screen + raw mode) with no execution guard, and (2) the dynamic input height (up to 45% of the screen, recomputed every render) reshaped the scroll region under patch_stdout mid-stream — both can crash the terminal, amplified by longer/streamier runs. Fix follows single-TTY-owner: while a task streams (_agent_running), pin the input area to a stable small region (Dimension min=1,max=4,preferred=1) and disable the external editor; adaptive height + editor resume when idle. _input_hidden_rows/overflow hint use the same effective cap and the hint drops the editor suggestion during streaming. +2 tests, 1006 pass, zero regression.
…with seamless ripgrep provisioning Closes the biggest coding-capability gaps vs Claude Code/Codex/Cursor. edit_file: anchored search-replace (non-unique anchor rejected, replace_all, dry_run, multi-edit) routed through the same path-sensitivity/approval/threat-scan governance as file_write; mutating_idempotent. code_search: ripgrep(--json) backend + pure-Python fallback, glob/ignore_case/multiline/max_results, skips VCS/dep/build/cache dirs, redacted structured results; read_only. file_find: recursive glob, skips dep dirs, max_results; read_only. All wired into TOOL_DEFINITIONS/_BRIDGE_TOOLS/TOOL_HANDLERS with x_leapflow metadata (verified read_only vs mutating classification so failed searches never trip the batch-stop gate). ripgrep provisioning is seamless: pure-Python search always works with zero install; a best-effort, cached, non-fatal, no-sudo Homebrew install runs in the background at startup (macOS), and a manual-install hint is surfaced when ripgrep is absent (tools.ripgrep_autoinstall, default on). +26 tests, 1032 pass, zero regression.
…nd code_intel (precise symbols) git_query: read-only structured git inspection (diff/log/status/branch/show) with clipped+redacted output and parsed log/branch fields; read_only, no approval; ref sanitized (no ranges), path passed after -- to avoid option injection. Use scm_sync for pull/push. code_intel: document symbols (outline) — Python via exact AST (class/function/method with line ranges, parent nesting, def header), other languages via keyword-prefix fallback; invalid Python degrades to heuristic-fallback instead of failing; read_only. Both registered with x_leapflow metadata and verified read_only so failed calls never trip the batch-stop gate. +11 tests, 1043 pass, zero regression.
…erminal_session (opt-in persistent shells) test_run/lint_check: structured verification wrappers over shell_run (inherit hardline/danger/approval/timeout/redaction). Auto-detect the runner (pytest/npm/go/cargo; ruff/eslint/go vet/clippy) from project markers, or use tools.test_command / tools.lint_command / an explicit command. Parse pytest pass/fail + failing tests and ruff issue counts. Semantics: ok=true means the runner executed (see success/clean); a failing suite is informative, not a side-effecting tool error — classified read_only so it never trips the batch-stop gate. terminal_session: long-lived interactive shells (open/send/read/close/list) as a session-managed resource per Transport-Lifecycle Separation — never folded into one-shot execution. DISABLED by default (tools.terminal_session_enabled); operator opt-in is the primary gate. Bounded (max sessions, idle TTL), background stdout reader with ring buffer, hardline check on open/send, process-group termination + atexit cleanup. open/send classified external_side_effect (approval), read/list read_only, close mutating. All wired with x_leapflow metadata + config (3 keys, 3 sites + descriptions) + context startup wiring. +11 tests, 1054 pass, zero regression.
… mode, git_write) + document tool suite Polish: code_search gains context_lines (context_before/after, redacted, backend-agnostic post-process). edit_file gains a unified-diff mode — each hunk becomes an anchored (context+removed -> context+added) edit applied via the same unique-anchor replace, so a missing/ambiguous hunk is rejected rather than fuzz-applied; edits is now optional (path only required). New git_write tool: mutating commit (stage_all + message via argv, never echoed/shelled) / branch (create+switch) / checkout (switch, create=-b); classified mutating_once + approval-gated; refs sanitized. Docs: README 'Built-in Coding Tools' section (tool matrix, governance, ripgrep provisioning, semantics, config keys). +8 tests, 1062 pass, zero regression.
… coding norms Beyond the tool layer, strengthen coding + tool-calling across the pipeline. P0 (Act/Observe hardening): - A1 pre-execution argument validation + in-turn self-repair: missing required params return a structured invalid_arguments result (with accepted schema) before execution; marked non-failing and policy-free so it never trips the batch-stop gate or failure budgets (agent.validate_tool_args). - B1 advisory post-edit syntax check: edit_file/file_write (overwrite) annotate .py results with syntax_ok/syntax_error via AST; never blocks the write (tools.verify_edits). - B2 structure-aware result truncation: _truncate_result_for_budget shrinks the largest string fields head+tail (tail errors survive) and keeps valid JSON instead of a naive cut; replaces all 6 truncation sites. - Review fix: _compact_error now preserves structured repair hints (error_type/missing/accepted_parameters/match_count) so A1 + edit_file anchor errors + code_search reach the model. P1 (Orient + norms): - repo_map tool (read-only): compact project orientation — languages, detected test/lint commands, top-level structure, entry points, manifest, VCS branch; cheap (no subprocess/deep walk). - F1/F2 prompt norms: a compact 'Coding & Verification' section in the unified system template (prefer precise tools over ad-hoc shell; verify with syntax_ok + test_run/lint_check; batch independent read-only calls). LLM norms, not rule-based routing. Deferred (needs confirmation): A2 relevance-ranked disclosure (core-loop risk), C2 conventions memory (memory integration). Pre-existing tree-wide ruff debt (132, CI ruff disabled) is untouched; all changed files are ruff-clean. +23 tests, 1085 pass, zero regression.
…ve tree (depth) Root cause of long-task abnormal termination: failed shell_run results were compacted by _compact_error, which discarded stdout/stderr and substituted 'unknown error' (shell_run set no error field), blinding the agent to the real traceback -> it could not diagnose broken scripts -> no progress -> stalled -> RepetitionGuard correctly halted with 'Action failed: unknown error (N consecutive tool failures)'. Fix (failure visibility): - _compact_error now preserves stdout/stderr (head+tail, so the tail traceback survives) and returncode/exit_code on failures — diagnostic output matters most when a tool fails. - shell_run populates 'error' from the stderr tail on non-zero exit, so downstream never falls back to a bare 'unknown error'. Success path unchanged. Also included (file_list recursive tree): file_list gains a depth param (0=flat default, 1-5=recursive tree skipping VCS/dep dirs) via _list_tree; ToolEvidenceBuilder flattens the tree to compact indented paths and prunes list-heavy payloads during compaction; schema/bridge updated. + tests; 1098 pass, zero regression; all changed files ruff-clean.
…d structured session summary
P1 — Align L2/L3 truncation thresholds: max_tool_output_chars 2000→3000 so TrimStage and the execution budget both use 3000 chars; removes the silent double-truncation of results that passed L2 but were re-cut by L3. P1 — file_read truncation_hint: when max_lines limits the read, the result now includes an explicit actionable hint ('Read more with start_line=N or increase max_lines') so the agent never silently misses the tail of a large file. P1 — Shell timeout injectable ceiling: hard-coded 120 s cap replaced by a module-level _max_shell_timeout_s (default 300 s) configurable via LEAPFLOW_MAX_SHELL_TIMEOUT_S; engine wires settings at startup through _configure_tool_defaults(). P2 — memory prefetch preview 100→500 chars so injected memory entries carry 5× more context. P2 — Difficulty-adaptive convergence round: ContextGovernanceController gains convergence_round_ceiling (default 40) and convergence_scale (default 2.0); effective convergence round = min(ceiling, base + round(base × difficulty × scale)), so hard tasks (difficulty≈1) explore up to 36 rounds before converging instead of the fixed 12. Configurable via LEAPFLOW_CONVERGENCE_ROUND_CEILING and LEAPFLOW_CONVERGENCE_SCALE. P2 — Structured session summary: _build_session_summary_context replaced flat 180-char preview with role-aware format: user intent preserved at 400 chars, tool-call turns show the tool list, prose turns preview at 300 chars.
…ine is busy The daemon serializes all engine turns behind a single _engine_lock, so a long task from one client silently blocked every other client's turn (e.g. a second TUI asking a simple question hung on 'thinking' for minutes with no output — the request was queued at the lock before the first LLM call). engine_chat now checks _engine_lock.locked() and, when busy, emits an immediate 'queued' status chunk (with the active request id and a /cancel hint) before awaiting the lock, so the waiting client knows it is queued instead of hanging silently. Serialization behavior is unchanged; this is feedback only (the concurrency fix is P1). Updated the serialization test to filter the new status chunk and assert exactly one turn was queued. +2 tests, 1106 pass, zero regression.
… across subagent frames (P1 Stage 1a) First safe, behavior-preserving increment of P1 concurrency isolation (Approach A). AgentLoopFrame now carries session_id/turn_id/command_id (populated in _build_frame; initialized in __init__ so the 16 read-sites and the subagent save/restore never hit an unset attribute). _install_frame/_restore_per_turn_state now save/restore these three ids around a child frame, closing a latent gap (previously only the 8 per-turn subsystems were isolated, so a subagent could leak its ids to the parent). Key design constraint captured: _cancel_requested is deliberately NOT frame-scoped — it is a cross-frame signal that must propagate into a running child (scoping it would swallow a cancel that arrives mid-child). Tests extended: child-frame isolation asserts ids restore to the parent; frame carries ids. N=1 behavior unchanged; 1106 pass, zero regression.
…ation (P1 Stage 1c) Empirically confirms the concurrency gap and pins the Stage 3 target: two turns run concurrently on one engine currently cross-contaminate (a probe showed one turn's echoed output carrying the other turn's message content), because per-turn substrate is shared on the single engine instance — not only the loop frame but also working memory and prompt assembly. The daemon deliberately serializes turns via _engine_lock today; P0 gives the waiting client immediate 'queued' feedback. The test is xfail(strict=False) so it documents the acceptance criterion without breaking CI, and flips to xpass when Stage 3 lands N>1 isolation. Finding: N>1 needs coordinated isolation of the whole per-turn substrate (frame + working memory + assembly), a larger effort than frame-field isolation alone.
…s (P3-1) Stage 3 phase 1 (additive; daemon still uses one engine). build_session_engine shallow-copies a wired base engine, sharing its stateless/session-keyed services (LLM, DuckDB stores, registry, tool bridge, compressor) by reference, but gives each session a FRESH working memory, idempotency ledger, per-turn subsystems (governance/research-ledger/commitment/usage/recovery), and a clean per-turn state slate. This isolates exactly the substrate that concurrent turns corrupt (working memory + accumulating governance + per-turn state) without duplicating the scattered engine wiring or touching single-turn internals. Empirically validated: two per-session engines run concurrent turns with NO cross-contamination (the scenario the single shared engine fails at, per the Stage 1c xfail). +2 tests, 1108 pass, zero regression. Design: temp/plan/concurrent_turns_stage3.md.
…-2a) Stage 3 phase 2a (infrastructure; not yet wired into engine_chat). Adds SessionRegistry/SessionExecutionContext: maps session_id -> per-session engine (built via the P3-1 factory) + a per-session turn lock. The first session reuses the daemon's base engine (single-session daemon unchanged); additional sessions get isolated engines. Bounded by max_live_sessions with idle-TTL eviction that never evicts the primary session. New config (all via leap config / catalog): daemon.max_concurrent_turns (default 1 = today's serialized behavior), daemon.max_live_sessions (16), daemon.session_idle_ttl_s (1800). Pure infrastructure, unit-tested in isolation (5 tests). engine_chat session-id routing + semaphore activation are P3-2b/P3-4. 1113 pass, zero regression.
…ionRegistry (P3-2b) Wires the P3-2a registry into engine_chat. A non-empty session_id (client-provided kwarg, else the engine's current session) is routed through the registry: the primary/first session reuses the base engine, additional distinct sessions run on isolated per-session engines (fresh working memory + per-turn state), so their state never cross-contaminates. Un-sessioned turns keep using the base engine directly — which is also the primary session's engine — so behavior stays consistent across a ""->real session_id transition and the single-session daemon is unchanged. Turns remain globally serialized via _engine_lock in this phase; bounded cross-session concurrency (semaphore) + per-request approval/cancel routing are P3-3/P3-4. +1 routing test; 1114 pass, zero regression.
Routes approval prompts and cancellation to the correct turn so concurrency (P3-4) is safe. Approval: the single _approval_event_queue slot is replaced by a _approval_route ContextVar set inside each turn's own asyncio task; the globally-shared approval gate reads it deep in tool execution and delivers the prompt to that turn's queue. Concurrent turns run in separate tasks, so the ContextVar isolates them (validated: two concurrent approvals land on their own queues, no cross-delivery). Cancel: engine_cancel(request_id='') now targets the running turn's own engine via a request_id->engine map (may be a per-session engine, not the base) — fixing a latent bug where cancel hit the base engine; with a request_id it targets one turn, without it cancels all active (N=1: the single one). Client/protocol engine_cancel gains an optional request_id (backward-compatible: the TUI's no-arg /cancel still works). +2 tests; 1116 pass, zero regression.
…nAdmission (P3-4) Replaces the daemon's single _engine_lock with TurnAdmission, a semaphore-based readers/writer gate: turns acquire one of N slots (turn_slot); host restart and re-entry dispatch acquire an exclusive window that drains all N slots (exclusive/exclusive_gate), so maintenance never overlaps a turn and concurrent exclusive ops are serialized (no drain deadlock). engine_chat now admits a turn (bounded by daemon.max_concurrent_turns) then serializes within a session via the SessionExecutionContext lock, so different sessions run in parallel while one session's turns stay sequential. N=1 (default) is byte-equivalent to the old mutex; the P0 'queued' feedback triggers when all slots are busy. Per-request approval (ContextVar) + cancel routing (P3-3) make concurrent turns safe. New primitive unit-tested (4); end-to-end test proves two sessions run in parallel under N=2; Stage 1c stays xfail by design (isolation is per-session engines, not a shared engine). 1123 pass, 1 xfailed, zero regression.
…-to-end concurrency (P3-4 activation) Completes Stage 3 end-to-end: the client's engine_chat now accepts an optional session_id and forwards it in the RPC params (omitted when empty, so single-session behavior is unchanged), and the TUI REPL passes its active_session_id. With daemon.max_concurrent_turns > 1, two TUI clients on distinct sessions now route to isolated per-session engines and run concurrently instead of clobbering a shared _current_session_id and serializing. Server dispatch already forwards params as kwargs (same path as enable_thinking); the daemon-side routing (P3-2b), per-request approval/cancel (P3-3), and bounded admission (P3-4) make this safe. +1 client test (forwards session_id when set, omits when empty). User-confirmed client+TUI data-passing change (no layout/interaction change). 1122 pass, 1 xfailed, zero regression.
…e leak (Stage 3 review) Deep-review findings on the Stage 3 changes: 1) engine: S1a regression — __init__ set self._current_session_id = "" a second time (the top of __init__ already sets it to None). The session-creation path (_ensure_session) mints + persists a session only when _current_session_id is None, so the accidental "" silently disabled conversation persistence (fresh engine never created a session; messages persisted under an empty id). Tests missed it because they run with session persistence disabled. Removed the duplicate assignment; None sentinel restored. 2) daemon: engine_chat acquired the per-session lock AFTER registering per-turn state (approval route + _active_engines). A cancellation while waiting on that lock (client disconnects while queued behind a same-session turn) skipped the try/finally, leaking the _active_engines entry. Acquire the lock before the (synchronous) per-turn state setup so the try/finally always cleans up. Machinery validated correct: the guardrails' check() is a pure function of the passed history (the instance _recent_hashes is vestigial), so sharing a guardrail across per-session engines is safe; build_session_engine resets exactly the stateful per-turn subsystems; TurnAdmission N=1 == the old mutex; re-entry dispatch is serial so the _ExclusiveGate is safe. 1122 pass, 1 xfailed, zero regression.
…UIs run concurrently (Stage 3 end-to-end) Closes the end-to-end concurrency gap found in review: two FRESH TUIs previously converged on the daemon's shared base session (both sent an empty session_id, then adopted the base engine's minted id) and therefore serialized + shared a conversation. Now each is a distinct, isolated, concurrent session. Three coherent changes: - cli(TUI): a fresh TUI (no --resume) generates its own distinct session id up front and owns it, so distinct clients route to distinct per-session engines. --resume keeps the requested id unchanged. - daemon: after routing, bind the engine to the routed session_id (isolated engines are pre-bound by the factory; this also binds the primary/base engine to the first session that claims it), keeping routing key, engine session, persistence, and returned metadata consistent so the client never re-routes mid-conversation. - engine: _ensure_session now creates the session row if it does not exist (get_session -> create_session, which is idempotent via ON CONFLICT), covering both an engine-minted id and a client-provided/bound id. Fixes persistence for client-owned ids. Behavioral change (user-approved): two separate TUIs are now independent conversations (like two shells) that run concurrently under daemon.max_concurrent_turns>1, instead of implicitly sharing one. In-process TUIs are unchanged (the engine still mints and the client adopts). +2 tests (daemon session binding; engine create-if-not-exists). 1124 pass, 1 xfailed, zero regression.
Fix the leapd stream failure where engine_chat's per-turn approval ContextVar was set in one per-chunk asyncio Task and reset in another, raising 'was created in a different Context'. _dispatch_stream now captures one contextvars.Context per stream and passes it to every per-chunk create_task call, preserving ContextVar set/reset identity while retaining heartbeat behavior. Add explicit ContextVar lifecycle contracts in daemon approval routing and subagent depth tracking, plus a real UnixRpcServer/socket end-to-end test with two concurrent sessions, mid-stream approvals, and N=2 admission. The new test fails without the server.py context fix and passes with it.
…ism (TC-P0/P1/P2) Replace the hardcoded tool-name classification in DefaultConcurrencyPolicy with the same registry metadata that already drives idempotency and the side-effect batch-stop gate — one source of truth, generalizes to any tool, and closes an unsafe default. TC-P0 (metadata-driven partition, safe default): - DefaultConcurrencyPolicy takes an injected spec_lookup(name)->ToolSpec (DIP; the module no longer imports the registry). partition() classifies each call via execution_policy_for: read_only -> parallel; mutating_idempotent with a non-overlapping path -> parallel (path-overlap check retained); mutating_once / external_side_effect -> sequential. - Unknown / unregistered tool (spec None) -> SEQUENTIAL (was: concurrent fallback) — a new tool never auto-parallelizes just because it is unlisted. - Removed _DEFAULT_PARALLEL_SAFE / _DEFAULT_PATH_SCOPED / _DEFAULT_NEVER_PARALLEL and the _is_mcp_read / _is_stateful name heuristics. engine wires spec_lookup from the singleton TOOL_REGISTRY (gp_-prefix aware). - Known tools are unchanged (reads->parallel, file_write->path-scoped, shell/scm/hub->sequential); MCP tools (no x_leapflow, absent from the import-time registry) now default to sequential (safe) instead of the risky name heuristic. TC-P1 (bounded parallelism): agent.max_parallel_tools (default 8) bounds the concurrent-group asyncio.gather with a per-batch semaphore so a large read batch cannot fan out unbounded IO; 1 forces sequential. TC-P2 (cleanup): name-set dead code removed; MCP classified by declared metadata (or safe sequential default); registry metadata completeness confirmed for built-ins. +10 tests (9 partition classification incl. safe defaults + gp_ prefix; 1 bounded-concurrency peak==cap). 1136 pass, 1 xfailed, zero regression.
…n state Close the Stage 3 product loop: daemon.max_concurrent_turns now defaults to 3 (including the env fallback), while daemon.* config fields are marked restart-required and config mutations warn users to run 'leap daemon restart'. TurnAdmission now tracks structured runtime metrics (max/active/waiting/available), RuntimeLeapService exposes them in daemon.status and queued stream metadata, and the at-capacity message includes capacity plus actionable config/restart guidance. TUI status rendering now separates local command queue from daemon turn admission and /daemon status prints turn capacity. Tests cover default N=3, explicit N=1 queueing, N=4 real-socket concurrency with a fifth queued client, restart-required config warnings, status rendering, and admission metrics.
…n and deferred init - Fix P0 memory cross-session pollution: add session_scope gate to episodic/semantic providers, MemoryQuery, manager passthrough and engine prefetch/sync paths - Fix approval queue hanging on client disconnect: per-request deny on turn end plus configurable TTL pruning (daemon_approval_ttl_s) - Split Context.initialize() into critical/deferred phases; daemon starts on critical path only, deferred components init in background behind a lock-protected _ensure_deferred() gate - Extract ApprovalCoordinator, MonitorCoordinator, SessionCoordinator, ReentryCoordinator and _service_helpers from service.py (1562 -> ~690 lines) with zero RPC contract changes - Conditional startup for monitor/reentry subsystems; idle-aware reentry tick intervals; lazy episodic GC and narrative dir creation - Move approval routing ContextVar to daemon/approval_route.py to remove coordinator->service reverse dependency - Add tests/test_daemon_isolation.py covering memory scope isolation, approval queue lifecycle and no-accumulation guarantees
Harden deferred init responsiveness, daemon recovery probing, and status diagnostics with regression coverage.
|
Caution The consumer version of Gemini Code Assist on GitHub has been sunset. All code review activity has officially ceased. |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
/orient)