Skip to content

Restore retained checkpoints across compatible nodes - #655

Merged
SaladDay merged 6 commits into
mainfrom
restore-checkpoints-across-nodes
Oct 10, 2026
Merged

SaladDay merged 6 commits into
mainfrom
restore-checkpoints-across-nodes

Conversation

@SaladDay

@SaladDay SaladDay commented Oct 10, 2026 •

Copy link
Copy Markdown
Collaborator

A settled retained checkpoint can resume on another online node with the same ready deployment generation, archive domain and execution class. Core commits the target route and capacity reservation before native work, then wakes the destination’s existing lifecycle loop. Allocation, Device and Session identities stay unchanged; unavailable compatible capacity does not shorten RAM retention.

The provider and node protocols carry checkpoint qualification and exact closure of restore attempts proven never dispatched to native execution. Microsandbox uses an explicit private archive store, durable admission journals and exact native cleanup. A fresh request whose wire deadline expires before native dispatch can close its attempt for retry; unknown native outcomes cannot. Observing an absent target avoids repeatedly reading the archive. Settled offline sources do not prevent compatible recovery or cleanup; unknown writers retain ownership.

Validation:

  • Real PostgreSQL tests cover capacity, concurrent routing, unsent restore recovery, stale receipts, offline-source wake/expiry cleanup, and expiry winning before Turn admission. Execution, node, adapter, helper with official SDK/FFI, installer, race and EN/zh documentation checks pass. Strict health JSON tests cover static/managed hello and heartbeat with checkpoint qualification.
  • Actual Core → Runtime → native Codex and Claude both pass automatic 300-second idle suspension, physical source-compute removal, source-node-offline recovery on another host, unchanged allocation/device/generation/native Session/PID/start time, and the next model/tool/MCP Turn. Codex also passes persistent npm/Python native dependencies and Skill checks, controlled-deadline expiry with real cleanup, original-Session cold continuation, and stopped-write backup restoration into an isolated database and filesystem. Normal public deletion and physical resource absence pass.
  • Actual helper interruption, exact cleanup, late-request rejection, allocation-lock deadline and fresh wire-deadline closure gates pass. Absence or helper death never qualifies an unknown restore for replay. Tests do not establish every crash state or a full 24-hour retention wait.
  • Controlled PostgreSQL measurements (10 ticker phases per existing/absent destination lane) reduce median route-commit-to-Resume-entry latency from 1.99/2.82 seconds to 46/46 milliseconds. A 256 MiB warm-cache absent-target observation avoids the entire archive read (367 ms → 0.784 ms). These do not measure successful native restore or guarantee latency under backlog.
  • Same-database input acceptance → Turn start in the live Codex fixture is 1.70 seconds for cold continuation and 12.40 seconds for cross-node RAM restoration, excluding model/tool execution. These are individual qualification samples, not SLOs; isolated SDK import/restore timings remain a measurement follow-up.
  • Full-diff independent review, targeted delta reviews and final Fable advisor closure found no release-blocking issue after the live gates. Final head 48cb2aff passes all 25 applicable CI checks including the aggregate check; its production code is unchanged from live-qualified 473dd6f7.

Targets must already hold the exact generation. Host kernel/CPU-profile changes can change checkpoint compatibility; retain eligible nodes during maintenance. No historical generation auto-installation, unknown-writer takeover, early cold fallback or external TCP-connection continuity is promised. Unknown dispatched restores can remain unavailable until deletion or retention expiry; cleanup waits for an eligible node and exact settlement.


View with [code]smith Autofix with [code]smith
Need help on this PR? Tag @codesmith-bot with what you need. Autofix is disabled.

@SaladDay
SaladDay marked this pull request as ready for review October 10, 2026 19:57
@SaladDay
SaladDay merged commit 601da24 into main Oct 10, 2026
25 checks passed
@SaladDay
SaladDay deleted the restore-checkpoints-across-nodes branch October 10, 2026 19:57
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant