From 90b445c4c8b432c12fb6ec61d88285205e5ceedd Mon Sep 17 00:00:00 2001 From: ewowi Date: Wed, 26 Aug 2026 15:03:05 +0200 Subject: [PATCH 1/6] Add MoonBase, a second boot image for the 4 MB boards MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit 4 MB devices no longer spend half their flash on a second firmware copy: a small MoonBase image in the factory slot installs updates into one app slot that grows 1856 to 2496 KB (filesystem 256 to 548 KB). One click in the UI covers the whole reboot-install-reboot cycle, unattended for URL installs, and a power cut mid-update lands back in MoonBase, never in a half-written app. Adopting the new layout needs a USB re-flash and re-provisioning (see MIGRATING.md). KPI: 16384lights | Desktop:1238KB | tick:2243/240/10/289/449/877/62/9/808/175/50/62/22/302/45/20/1/555/88/10us(FPS:445/4166/100000/3460/2227/1140/16129/111111/1237/5714/20000/16129/45454/3311/22222/50000/1000000/1801/11363/100000) | ESP32:1659KB | src:226(62809) | test:171(38458) | lizard:173w Core: - moonbase/: standalone minimal ESP-IDF project (742 KB in an 896 KB slot); WiFi STA from the app's stored credentials with AP fallback at 4.3.2.1; install by upload, by URL, and unattended from an NVS-staged URL (read-and-erased before any attempt, one try per staging); GET /moonbase identity probe; explicit /boot-app switch back (validates the image first); a failed install stays visibly in MoonBase by PO decision - HttpServerModule: on MoonBase devices /api/firmware/url stages the URL (max 255 bytes) and reboots into MoonBase (202 + moonbase:true); new /api/firmware/moonbase route; otaWriteStream guards (running-partition, image-too-large) - FirmwareUpdateModule: read-only moonbase control marks MoonBase devices - platform: otaHasMoonBase/otaBootMoonBase/otaRunningMoonBase/moonbaseStageInstallUrl; fsMount formats on the last EXISTING candidate (a fresh MoonBase-table board previously never formatted and ran without persistence); littlefs partition name+subtype on the new table with spiffs fallback UI: - One "updating firmware" overlay across the whole cycle, opened before the reboot; install-from-file re-POSTs to MoonBase, URL installs run unattended (success = device seen away, then the new app answering); MoonBase button on the Firmware card; MoonBase assets never offered as OTA images Scripts/MoonDeck: - build_esp32.py owns the shared flash-layout helpers (IDF's flasher_args stages the app at the factory offset; corrected once, consumed by serial flash, web-installer manifests, release preview, qemu image); slot-0 otadata generated byte-identical to otatool's output; stale build dirs now detected by fragment-list and partition-table comparison (the silent-stale-sdkconfig trap, fixed structurally) - flash_esp32.py: one-pass MoonBase flash, fresh flash boots the app; check_esp32_built gates the MoonBase image and counts .csv as source Tests: - unit_PartitionTables pins every table's overlap/bounds/alignment and the dual-OTA-xor-MoonBase shape (verified by deliberate faults) - unit_MoonBaseContract pins ssid/password inside MoonBase's 1024-byte credential read - installer-firmware-merge pins that MoonBase assets are never offered as OTA images - test_installer_manifests globs unified with release.yml (shared-*.bin), catching that the new assets would have 404'd from the Pages installer Docs/CI: - architecture.md ยง MoonBase (single home; prior art credited: Tasmota safeboot, MycilaSafeBoot, also in README Credits and the module header); MIGRATING.md erase-flash entry; building.md; catalog card; backlog: flash-budget investigation deleted (resolved), MoonBase follow-ups added - release.yml stages shared-moonbase-.bin + shared-ota-data-slot0.bin; manifests remap the app to ota_0 with slot-0 otadata; TEMPORARY moonbase-test-release.yml publishes a test prerelease from this branch (delete after merge); ccache for the macOS job - Plan file updated as the PR-description record Reviews: - ๐Ÿ‘พ overlay could never see an unattended-URL success (installs before serving) โ†’ done: watch loop keys on device-seen-away then app-answering - ๐Ÿ‘พ stale staged URL could hijack a later install โ†’ done: read-and-erase unconditionally at boot - ๐Ÿ‘พ URL length contract 512 vs 256 โ†’ done: route rejects >255, limit stated at all three sites - ๐Ÿ‘พ plan drifted from the diff (sizes, unbuilt badge claim, phantom tests) โ†’ done: reconciled - ๐Ÿ‘พ comment promised 204 + fallback, code sends 409 โ†’ done: comment corrected - ๐Ÿ‘พ em-dashes in 45 added lines โ†’ done: swept - ๐Ÿ‘พ comments described an unbuilt Ethernet path โ†’ done: reworded, esp32-eth consequence named - ๐Ÿ‘พ relative .md link in a /// comment โ†’ done: prose reference - ๐Ÿ‘พ serveOne single-recv parse โ†’ done: reads across TCP segments, bounded - ๐Ÿ‘พ credential reader escape subset โ†’ done: writer's escape set decoded, \u fails visibly - ๐Ÿ‘พ 512-byte prefix read was an unpinned cross-image contract โ†’ done: unit_MoonBaseContract + 1024-byte bound - ๐Ÿ‘พ flash-write failure reported as a cut-short upload โ†’ done: distinct status - ๐Ÿ‘พ nits (import block, "all three" wording, header ///, chip guard, duplicated wait loop, .csv suffix) โ†’ done Performance: - flash: esp32 1788256 (+3888), esp32s3-n16r8 1831360 (+4384), desktop 1268136 (+16880); esp32-eth 1395840 (+71024), esp32-wrover 1841952 (+76448), qemu 1382000 (+63840) โ€” the three jumps are stale baselines repaid: those variants' previous numbers came from long-unbuilt dirs, and this branch's staleness fix forced their first fresh builds (their true structural deltas vs esp32 check out: wrover = +PSRAM, eth = -WiFi) - MoonBase image: 742 KB (896 KB slot); desktop tick 247us (+10), fps 4048 (-171); tests 1584 cases (+5); scenario KPI run: 21/23 passed (2 pre-existing flaky UDP-on-localhost cases) Co-Authored-By: Claude Opus 5 (1M context) --- .github/workflows/moonbase-test-release.yml | 89 +++ .github/workflows/release.yml | 37 +- .gitignore | 8 + README.md | 3 + docs/MIGRATING.md | 12 + docs/architecture.md | 30 + docs/backlog/backlog-core.md | 48 +- docs/building.md | 7 + ...0260826 - MoonBase, a second boot image.md | 220 ++++++++ docs/metrics/repo-health.json | 62 +-- docs/metrics/repo-health.md | 48 +- docs/moonmodules/core/system.md | 5 +- esp32/partitions/esp32dev_moonbase.csv | 66 +++ esp32/sdkconfig.defaults.moonbase-4mb | 1 + moonbase/CMakeLists.txt | 19 + moonbase/main/CMakeLists.txt | 9 + moonbase/main/idf_component.yml | 2 + moonbase/main/moonbase_main.cpp | 523 ++++++++++++++++++ moonbase/sdkconfig.defaults | 53 ++ moondeck/build/build_esp32.py | 166 +++++- moondeck/build/flash_esp32.py | 34 +- moondeck/build/generate_manifest.py | 19 + moondeck/check/check_esp32_built.py | 25 +- moondeck/qemu/run_qemu.py | 14 +- moondeck/run/preview_installer.py | 13 + src/core/FirmwareUpdateModule.h | 17 +- src/core/HttpServerModule.cpp | 48 ++ src/core/HttpServerModule.h | 8 +- src/platform/desktop/platform_desktop.cpp | 6 + src/platform/esp32/platform_esp32_fs.cpp | 54 +- src/platform/esp32/platform_esp32_ota.cpp | 57 ++ src/platform/platform.h | 16 + src/ui/app.js | 146 +++++ src/ui/install-picker.js | 10 +- src/ui/style.css | 25 + test/CMakeLists.txt | 2 + test/js/installer-firmware-merge.test.mjs | 16 + test/python/test_installer_manifests.py | 5 +- .../light/scenario_peripheral_grid_sweep.json | 4 +- test/unit/core/unit_MoonBaseContract.cpp | 63 +++ test/unit/core/unit_PartitionTables.cpp | 166 ++++++ 41 files changed, 2046 insertions(+), 110 deletions(-) create mode 100644 .github/workflows/moonbase-test-release.yml create mode 100644 docs/history/plans/Plan-20260826 - MoonBase, a second boot image.md create mode 100644 esp32/partitions/esp32dev_moonbase.csv create mode 100644 esp32/sdkconfig.defaults.moonbase-4mb create mode 100644 moonbase/CMakeLists.txt create mode 100644 moonbase/main/CMakeLists.txt create mode 100644 moonbase/main/idf_component.yml create mode 100644 moonbase/main/moonbase_main.cpp create mode 100644 moonbase/sdkconfig.defaults create mode 100644 test/unit/core/unit_MoonBaseContract.cpp create mode 100644 test/unit/core/unit_PartitionTables.cpp diff --git a/.github/workflows/moonbase-test-release.yml b/.github/workflows/moonbase-test-release.yml new file mode 100644 index 00000000..3a4a6b11 --- /dev/null +++ b/.github/workflows/moonbase-test-release.yml @@ -0,0 +1,89 @@ +name: MoonBase test release + +# TEMPORARY: delete after the MoonBase branch merges to main. +# Publishes a `moonbase-test` prerelease from THIS branch so the picker's URL-based +# install can be tested against real GitHub release assets before the merge (the +# real release pipeline only runs from main and version tags). Manual trigger only; +# esp32 (the bench Olimex) is the only firmware built. Mirrors release.yml's esp32 +# job + staging + manifest, reduced to one variant and one fixed tag. + +on: + workflow_dispatch: + +permissions: + contents: write + +concurrency: + group: moonbase-test-release + cancel-in-progress: false + +jobs: + build-and-release: + runs-on: ubuntu-latest + steps: + - uses: actions/checkout@v4 + with: + persist-credentials: false + fetch-depth: 0 # compute_version.py counts commits since the last v* tag + + - name: Cache ESP-IDF tooling + uses: actions/cache@v4 + with: + path: | + ~/.espressif + ~/esp/esp-idf + key: esp-idf-v6.1-rc1-${{ runner.os }}-v1 + + - name: Compute version + id: ver + # `latest`-channel versioning (-dev.) so the version is a real, + # orderable semver the picker can compare; the release channel burned into + # the binary is `moonbase-test` so a device shows where the image came from. + run: | + set -euo pipefail + V=$(python moondeck/build/compute_version.py --tag latest) + echo "version=$V" >> "$GITHUB_OUTPUT" + + - name: Build firmware + uses: espressif/esp-idf-ci-action@v1 + with: + esp_idf_version: v6.1-rc1 + target: esp32 + path: 'esp32' + command: python ../moondeck/build/build_esp32.py --firmware esp32 --release "moonbase-test" --version "${{ steps.ver.outputs.version }}" + + - name: Stage assets + manifest + run: | + set -euo pipefail + mkdir -p dist + V="${{ steps.ver.outputs.version }}" + B=build/esp32-esp32 + PREFIX="firmware-esp32-v$V" + cp "$B/projectMM.bin" "dist/${PREFIX}.bin" + cp "$B/bootloader/bootloader.bin" "dist/${PREFIX}-bootloader.bin" + SIZE=$(jq -r .flash_settings.flash_size "$B/flasher_args.json" | tr 'A-Z' 'a-z') + cp "$B/partition_table/partition-table.bin" "dist/partition-table-$SIZE.bin" + cp "$B/ota_data_initial.bin" "dist/shared-ota-data.bin" + cp build/moonbase-esp32/projectMM-moonbase.bin dist/shared-moonbase-esp32.bin + python3 -c "import sys; sys.path.insert(0,'moondeck/build'); \ + import build_esp32, pathlib; \ + pathlib.Path('dist/shared-ota-data-slot0.bin').write_bytes(build_esp32.otadata_slot0_bytes())" + python3 moondeck/build/generate_manifest.py --firmware esp32 --version "$V" \ + --release-url "https://github.com/${GITHUB_REPOSITORY}/releases/download/moonbase-test" \ + --flasher-args "$B/flasher_args.json" --out dist/manifest-esp32.json + + - name: Publish moonbase-test prerelease + env: + GH_TOKEN: ${{ secrets.GITHUB_TOKEN }} + run: | + set -euo pipefail + V="${{ steps.ver.outputs.version }}" + # Replace any previous test release; keep-tag deletion semantics don't + # matter here: the tag is repointed to the current commit on create. + gh release delete moonbase-test --repo "$GITHUB_REPOSITORY" --yes --cleanup-tag || true + # The release NAME is the version string: the picker machine-reads it the + # same way it reads the real `latest` prerelease (no "v" prefix). + gh release create moonbase-test dist/* \ + --repo "$GITHUB_REPOSITORY" --prerelease --title "$V" \ + --target "$GITHUB_SHA" \ + --notes "Temporary MoonBase URL-install test release from the moonbase branch. Do not install on devices you care about; superseded by the next real release." diff --git a/.github/workflows/release.yml b/.github/workflows/release.yml index ee20b73b..c7bfff61 100644 --- a/.github/workflows/release.yml +++ b/.github/workflows/release.yml @@ -225,6 +225,19 @@ jobs: # Per-firmware flasher_args.json โ€” the release job feeds it to # generate_manifest.py so offsets come from the real build. cp "$B/flasher_args.json" "dist/flasher-${{ matrix.firmware }}.json" + # MoonBase, the second boot image on the 4 MB tables (built alongside those firmwares + # by build_esp32.py): chip-shared, so staged under a shared name like the partition + # table. The slot-0 otadata is what makes a fresh install boot the APP with MoonBase + # standing by: blank otadata would boot MoonBase. Their manifests reference both + # (generate_manifest.py), and install-picker.js excludes them from OTA offers. + for MB in build/moonbase-*/projectMM-moonbase.bin; do + [ -f "$MB" ] || continue + CHIP=$(basename "$(dirname "$MB")"); CHIP=${CHIP#moonbase-} + cp "$MB" "dist/shared-moonbase-$CHIP.bin" + uv run python -c "import sys; sys.path.insert(0,'moondeck/build'); \ + import build_esp32, pathlib; \ + pathlib.Path('dist/shared-ota-data-slot0.bin').write_bytes(build_esp32.otadata_slot0_bytes())" + done - uses: actions/upload-artifact@v4 with: @@ -270,8 +283,28 @@ jobs: set -euo pipefail V=$(uv run python moondeck/build/compute_version.py --tag "$TAG") echo "version=$V" >> "$GITHUB_OUTPUT" + # ccache: the macOS runner compiles the whole desktop tree from cold every run (the ESP32 + # jobs get a prebuilt IDF container; this one gets nothing), which is why it was the slowest + # job in the workflow at ~10 min. CMake picks the launcher up from the environment, so the + # build script stays unchanged. The key rotates per run and restores from the newest + # matching prefix, the standard save-always cache shape for a compiler cache. + - name: Install ccache + run: brew install ccache + - name: Restore ccache + uses: actions/cache@v4 + with: + path: ~/Library/Caches/ccache + key: ccache-macos-14-${{ github.sha }} + restore-keys: | + ccache-macos-14- - name: Build + package macOS arm64 + env: + CMAKE_CXX_COMPILER_LAUNCHER: ccache + CMAKE_C_COMPILER_LAUNCHER: ccache run: uv run moondeck/ci/package_desktop.py --version "${{ steps.ver.outputs.version }}" + - name: ccache stats + if: always() + run: ccache --show-stats - uses: actions/upload-artifact@v4 with: name: desktop-macos @@ -532,7 +565,7 @@ jobs: # is a glob pattern, the action does NOT strip `#` as comment syntax. files: | dist/firmware-*.bin - dist/shared-ota-data.bin + dist/shared-*.bin dist/partition-table-*.bin dist/manifest-*.json dist/projectMM-*.tar.gz @@ -615,7 +648,7 @@ jobs: --dir "pages/install/releases/$T" \ --pattern 'firmware-*.bin' \ --pattern '*-ota-data.bin' \ - --pattern 'shared-ota-data.bin' \ + --pattern 'shared-*.bin' \ --pattern 'partition-table-*.bin' \ --pattern 'manifest-*.json' \ --pattern 'projectMM-*.tar.gz' \ diff --git a/.gitignore b/.gitignore index f3829b9c..c26719dd 100644 --- a/.gitignore +++ b/.gitignore @@ -91,6 +91,14 @@ esp32/sdkconfig esp32/sdkconfig.old esp32/managed_components/ esp32/dependencies.lock + +# MoonBase is its own ESP-IDF project, so it grows the same build artefacts: the component +# manager's downloads and the generated sdkconfig. Only sdkconfig.defaults is ours to keep. +moonbase/sdkconfig +moonbase/sdkconfig.old +moonbase/managed_components/ +moonbase/dependencies.lock +moonbase/build/ esp32/monitor.log # Generated live-scenario baseline cache (regenerated by run_live_scenario.py diff --git a/README.md b/README.md index a08147cc..732885d6 100644 --- a/README.md +++ b/README.md @@ -42,6 +42,8 @@ If you like projectMM, give it a โญ๏ธ, fork it, or open an issue or pull reque ๐ŸŽจ **Plug in, open a browser, see lights**: a live 3D preview of every effect, modifier, and layout, controllable from the same tab. The interface renders any module from its declared controls, so adding a module needs zero UI code. +๐ŸŒ— **MoonBase, the second boot image (4 MB boards)**: instead of spending half a small flash on a second firmware copy, a ~750 KB maintenance image sits in the factory slot and installs updates into one large app slot, one click in the UI covers the whole reboot-install-reboot cycle, and a power cut mid-update lands back in MoonBase, never in a half-written app. See [architecture.md ยง MoonBase](docs/architecture.md#moonbase-the-second-boot-image-4-mb-boards). + โšก **Flash from your browser in seconds**: the web installer picks your device, flashes the matching firmware, and hands WiFi credentials to the device over USB via Improv. No serial monitor, no recompile. ## Under the hood @@ -176,6 +178,7 @@ Specific people whose work directly shaped parts of projectMM. We study their th - **The [Improv Wi-Fi](https://github.com/improv-wifi) project**: the open Improv serial provisioning standard ([sdk-cpp](https://github.com/improv-wifi/sdk-cpp) / [sdk-js](https://github.com/improv-wifi/sdk-js)) that the projectMM web installer uses to provision a freshly-flashed device over USB. - **[FastLED](https://github.com/FastLED/FastLED)**: the canonical LED-effects library whose conventions the LED-effect world shares. projectMM links no part of FastLED, but it carries forward FastLED's recognisable *names and models* for the color/animation primitives (`scale8`, `sin8`, the gradient-palette model (`CRGBPalette16` / `colorFromPalette`), the `beatsin8` / `inoise8` / `qadd8` family), so a contributor recognises them on sight. The implementations are projectMM's own, integer-only and hot-path-tuned for our render loop; FastLED is the prior art behind the convention, credited here and in each primitive's notes. - **[FPP](https://github.com/FalconChristmas/fpp) (Falcon Player)**: the show player that drives LED panel receiver cards from a Raspberry Pi. Seeing an FPP rig feed a wall of HUB75 panels is what prompted [PanelCardDriver](docs/moonmodules/light/drivers.md#panelcard): if a Linux host can send those frames, so can a board that is already rendering them, which removes the host from the installation entirely. FPP is the inspiration, and the reference point for what good looks like here: it sustains 50 fps. +- **[Tasmota](https://github.com/arendst/Tasmota) and Mathieu Carbou's [MycilaSafeBoot](https://github.com/mathieucarbou/MycilaSafeBoot)**: the safeboot pattern behind [MoonBase](docs/architecture.md#moonbase-the-second-boot-image-4-mb-boards): replacing a small board's second OTA slot with a minimal boot image that installs into one large app slot. Tasmota proved the scheme at scale; MycilaSafeBoot distilled it to a standalone image and set the size bar. MoonBase is our from-scratch minimal take, written directly against ESP-IDF. - **Damian Schneider ([dedehai](https://github.com/DedeHai))**: author of the WLED Particle System, whose emitters, forces and walls over one shared pool are the shape our [particle kernel](docs/moonmodules/light/power-functions.md#particles) and the scripted `pool` / `emit` / `step` builtins follow, in our own fixed-point implementation. - **wladi ([myhome-control](https://shop.myhome-control.de))**: designer of the [MHC-WLED ESP32-P4 shield](https://shop.myhome-control.de/en/ABC-WLED-ESP32-P4-shield/HW10027), and the source of the hardware and the pinout details that got its **line-in audio** working in [AudioService](docs/moonmodules/core/moxygen/AudioService.md): the onboard PCM1808 I2S ADC (WS 26 / SD 33 / SCK 32 / MCLK 36), the PCM1808's stereo wiring, and its `FMT` format-select jumper (open = I2S/Philips, our default; tie to 3V3 for left-justified), which is what confirmed the standard-I2S path the ADC needs. diff --git a/docs/MIGRATING.md b/docs/MIGRATING.md index 75601cf3..73e619d8 100644 --- a/docs/MIGRATING.md +++ b/docs/MIGRATING.md @@ -22,6 +22,18 @@ projectMM ships **no migration code**: the persistence layer is robust by defaul ## Unreleased (`next-iteration`) +### 4 MB boards move to the MoonBase partition table (2026-08-26) + +**Action: erase flash** (USB re-flash; all on-device config is lost and must be re-entered). + +The 4 MB variants (`esp32`, `esp32-wrover`, `esp32-eth`) replace the dual-OTA layout with +[MoonBase](architecture.md#moonbase-the-second-boot-image-4-mb-boards): the app slot grows +1856 โ†’ 2496 KB and the filesystem 256 โ†’ 548 KB, but the filesystem moves (0x3B0000 โ†’ 0x360000), +so the existing volume is not where the new table looks, WiFi credentials, module config and +scripts all re-enter through provisioning. A partition table only changes over USB: a device +still on the old table keeps OTA-updating *within* that table for as long as the app fits its +1856 KB slot; the web installer is the migration path. 8/16 MB boards are unaffected. + ### PreviewDriver's `fps` becomes `targetFps`, and now trades resolution (2026-08-25) The control is renamed and its meaning changed, so the rename is the point rather than cosmetic. diff --git a/docs/architecture.md b/docs/architecture.md index f12d4e8f..37f12b36 100644 --- a/docs/architecture.md +++ b/docs/architecture.md @@ -263,6 +263,36 @@ Three distinct things, kept distinct in the vocabulary: A deviceModel can run multiple firmwares (the Olimex Gateway runs both `esp32-eth` and the default `esp32`); a firmware can run on multiple deviceModels (`esp32` runs on any classic ESP32 dev kit). The `esp32s3-n16r8` firmware is S3-only and does not run on the Olimex Gateway or other classic-ESP32 hardware. The codebase reserves "deviceModel" exclusively for the physical product and "firmware" exclusively for the compiled binary. +### MoonBase: the second boot image (4 MB boards) + +A 4 MB board has room for one application, not two, so the dual-OTA layout (half the chip spent +on a second copy of the firmware) is replaced on those boards by **MoonBase**: a small, +rarely-changing image in the partition table's `factory` slot that owns the device while the +application is being replaced, a board cannot rewrite the partition it is executing from. The +app slot grows by a third in exchange. 8/16 MB boards keep dual-OTA and are untouched by any of +this. + +The update cycle: the app stages the install URL in NVS (or nothing, for a browser upload), +points the bootloader at MoonBase and reboots; MoonBase joins the network with the app's stored +credentials (AP fallback at 4.3.2.1), installs into the single app slot, from the staged URL +unattended, or from an upload, and reboots back. The UI covers the whole cycle with one +"updating firmware" overlay, telling the two images apart via `GET /moonbase` (MoonBase answers +with its live status; the app 404s it). Pointing the bootloader at a factory partition *erases* +otadata, so a power cut anywhere mid-install boots MoonBase and the user retries over the +network, a stronger power-fail story than dual-OTA's. A failed install deliberately stays in +MoonBase, visibly, rather than silently reverting to the old app; the way back is its explicit +"Boot the app" action, which only boots an image that validates. + +MoonBase is a standalone ESP-IDF project (`moonbase/`, ~750 KB against an 896 KB slot) sharing +no sources with the app, the deliberate trade for an image that must stay small and, once +working, hardly change. `moondeck/build/build_esp32.py` builds it alongside the 4 MB variants +and owns the flash-layout helpers every consumer uses (serial flash, web-installer manifests, +release preview, the QEMU image): IDF's own `flasher_args.json` knows nothing of the two-image +scheme and stages the app at the factory offset, so each of those paths applies the same +correction from one place. Prior art: Tasmota's safeboot scheme and +[MycilaSafeBoot](https://github.com/mathieucarbou/MycilaSafeBoot) proved the single-slot + +recovery-image pattern; MoonBase is our from-scratch, minimal take on it. + ### Config provenance: MCU โ†’ deviceModel Firmware-vs-deviceModel is a **two-level** model for **where a pin or setting default legitimately comes from**. The installer and MoonDeck use it so a user picks their hardware instead of hand-typing every GPIO. A default belongs at the level that actually *fixes* it: diff --git a/docs/backlog/backlog-core.md b/docs/backlog/backlog-core.md index 5911790f..c54f212e 100644 --- a/docs/backlog/backlog-core.md +++ b/docs/backlog/backlog-core.md @@ -124,31 +124,37 @@ DevicesModule discovers via **passive UDP presence** (UDP 65506) feeding a [`Dev Full design + the reasoned transport split: [Plan-20260629 โ€” UDP device discovery + mDNS advertise-only (shipped)](../history/plans/archive/Plan-20260629%20-%20UDP%20device%20discovery%20%2B%20mDNS%20advertise-only%20%28shipped%29.md). -## ESP32 performance and memory - -### Flash budget โ€” the 4 MB classic ESP32 is the ceiling (investigation) - -The binary has grown ~1.4 โ†’ ~1.48 MB as effects, audio sync, IR, and Ethernet landed. Per-board headroom against the app (OTA) partition slot: - -| Board | app slot | binary | used | -|---|---|---|---| -| **classic esp32 (4 MB)** | 1.75 MB | ~1.48 MB | **~84 %** โš ๏ธ | -| esp32s3-n8r8 (8 MB) | 3.00 MB | ~1.48 MB | ~49 % | -| 16 MB boards | 4.00 MB | ~1.48 MB | ~37 % | +## MoonBase follow-ups + +MoonBase v1 ([architecture.md ยง MoonBase](../architecture.md#moonbase-the-second-boot-image-4-mb-boards)) +ships exactly one action: install firmware (upload + URL). The name is deliberately broader than +"recovery", these are the candidate next actions, each solving something only a separate boot +image can solve. The budget rule from the partition table applies to all of them: the 896 KB slot +has ~150 KB headroom, sized for one new *component*, so each action must earn its bytes (a few KB +of code is fine; a new IDF component is the expensive kind). + +- **Factory reset**: erase the filesystem (and optionally NVS) from MoonBase's page: recovers a + device whose config crashes the app on boot, without a USB cable. +- **Boot with config disabled**: one-shot flag the app reads at startup to skip loading + `/.config`: diagnose "is it my config or the firmware?" without erasing anything. +- **WiFi re-provisioning**: edit the stored credentials from MoonBase's page (today it only + *reads* them; the AP fallback plus the app's provisioning already covers most of this). +- **Config backup / restore**: download the filesystem as an archive before a risky change, + upload it back after; also the migration answer for future partition-table moves. +- **Firmware downgrade guard**: MoonBase installs whatever image it is given; a version display + (read from the incoming image's app descriptor) before flashing would make an accidental + downgrade visible. +- **Hardware diagnostics**: chip/flash/PSRAM identification and a minimal pin tester, for + triaging a board that misbehaves under the full app. +- **Ethernet**: MoonBase is WiFi-only today; the eth-only 4 MB variants (`esp32-eth`) fall back + to the AP when no WiFi credentials exist. Needs the per-board PHY/pin config brought over, + which is the real cost. -Only the **4 MB classic** is tight (the partition comment itself notes "~200 KB headroom"); the 8/16 MB boards have years of room. So this is a *classic-ESP32-only* constraint, not a global one โ€” the fix should shrink the small-flash build without touching the flagship boards. - -Levers, roughly by payoff-per-effort: - -1. **`-Os` for the size-bound builds.** The ESP32 build currently runs the IDF default optimization (`CONFIG_COMPILER_OPTIMIZATION_DEBUG`, ~`-Og`), **not** `CONFIG_COMPILER_OPTIMIZATION_SIZE` (`-Os`). Setting size-opt on the classic (and any small-flash) firmware typically buys 5โ€“15 % flash for free โ€” measure the tick/FPS delta, since `-Os` can cost a little hot-path speed; if it does, gate it to the flash-bound firmwares only, not the S3/P4 where headroom is fine. -2. **`MM_MINIMAL` feature profile for the 4 MB target.** The classic board doesn't need every effect/driver compiled in. A build profile that compiles out heavy optional modules (the same `firmware_cmake_args()` seam the eth/wifi gating already uses) keeps the flagship boards full-featured while the small board ships a curated subset โ€” the standard "small board, smaller build" pattern. -3. **Repartition the 4 MB classic.** If A/B OTA isn't required on the classic, a single-app-slot layout nearly doubles the app ceiling (1.75 โ†’ ~3.5 MB). Trade-off: OTA loses its rollback slot. Decide per-board, not globally. - -Start with (1) โ€” it's a one-line sdkconfig change with a measurable payoff and no code churn; (2)/(3) only if (1) plus normal growth still crowds the classic. The UI embed is already gzipped (`app.js` ~140 KB โ†’ ~40 KB), so UI growth is cheap in flash; the pressure is C++ `.text`. +## ESP32 performance and memory ### Size estimates for unbuilt features (reference) -Estimates, not measurements, so they live here rather than in [performance.md](../performance.md) which carries measured numbers only. Feeds the [flash budget](#flash-budget-the-4-mb-classic-esp32-is-the-ceiling-investigation) decision. +Estimates, not measurements, so they live here rather than in [performance.md](../performance.md) which carries measured numbers only. (The 4 MB flash-budget investigation these once fed is resolved: MoonBase's single-app-slot layout grew the classic app slot to 2496 KB, see architecture.md ยง MoonBase.) | Feature | Est. | Rationale | |---|---|---| diff --git a/docs/building.md b/docs/building.md index e7d625ec..253be087 100644 --- a/docs/building.md +++ b/docs/building.md @@ -136,6 +136,13 @@ uv run moondeck/build/flash_esp32.py --firmware esp32 --port /dev/tty.usbserial- uv run moondeck/run/monitor_esp32.py --port /dev/tty.usbserial-XXXX ``` +On the 4 MB variants (`esp32`, `esp32-wrover`, `esp32-eth`, `qemu`) the build also produces +**MoonBase**, the second boot image ([architecture.md ยง MoonBase](architecture.md#moonbase-the-second-boot-image-4-mb-boards)), +and `flash_esp32.py` writes the corrected layout in one pass: app in the big `ota_0` slot, +MoonBase in `factory`, and an otadata that boots the app directly. A device on the older +dual-OTA table adopts this layout only through such a full serial flash, OTA never rewrites +the partition table. + `setup_esp_idf.py` runs the upstream installer for the host: `install.sh` on macOS/Linux, `install.bat` on Windows. Both create the same `~/.espressif/python_env/...` venv and download the same toolchains (~1.5 GB more) โ€” only the wrapper differs. The Windows installer needs roughly 5 minutes on a fast link. It also offers to move a drifted checkout onto the pinned commit (see [ESP-IDF version](#esp-idf-version)); pass `--no-checkout` to keep it warn-only. **Building for the ESP32-S31** (a RISC-V *preview* target in v6.1) needs its toolchain fetched once โ€” the default install only pulls the classic-`esp32` toolchains: diff --git a/docs/history/plans/Plan-20260826 - MoonBase, a second boot image.md b/docs/history/plans/Plan-20260826 - MoonBase, a second boot image.md new file mode 100644 index 00000000..a9337c08 --- /dev/null +++ b/docs/history/plans/Plan-20260826 - MoonBase, a second boot image.md @@ -0,0 +1,220 @@ +# Plan: MoonBase, a second boot image for 4 MB boards + +## Context + +The 4 MB ESP32 boards have run out of flash. CI caught it on 2026-08-26: the `esp32-wrover` +build is 1,839,776 bytes against an 1792 KB slot, and the hotfix that grew both OTA slots to +1856 KB left it at 3% free. The cause is structural: a dual-OTA layout spends half the chip on a +second copy of the firmware, so every kilobyte the app gains costs two. + +**MoonBase** replaces that second copy with something smaller and more useful: a tiny, rarely +changing image in the `factory` partition that owns the device when the application is not running +or cannot be trusted. Its first job is installing firmware into the one large app slot (a device +cannot rewrite the partition it is executing from). It is named for the family it joins, alongside +MoonDeck, MoonLight and MoonLive, and it is deliberately not called "recovery": updating, +re-provisioning WiFi, factory reset and diagnostics are all maintenance, not repair. + +Outcome for the 4 MB boards: the app partition grows from 1856 KB to 2496 KB (+34%), the +filesystem from 256 KB to 548 KB, and OTA keeps working (through MoonBase). + +## The measurements this plan rests on + +All from clean builds whose exit status was checked (two earlier figures in this plan's history +were wrong: one reported a stale binary, one was measured with the URL installer stubbed out). + +Getting from ESP-IDF's defaults to a shippable size is mostly configuration, not code: + +| Configuration | Size | +|---|---:| +| Bare ESP-IDF hello-world | 139 KB | +| WiFi + HTTP + OTA, IDF defaults (-O2) | 881 KB | +| + `-Os` | 812 KB | +| + no logs, no error strings, no console | 695 KB | +| + newlib-nano, no IPv6, no WPA3/enterprise | 588 KB | +| *MycilaSafeBoot esp32dev, for reference* | *640 KB* | + +And the finished MoonBase, built against its own table: + +| Build | Size | +|---|---:| +| Upload only | **576 KB** | +| Upload + install-from-URL (HTTPS) | **742 KB** | + +**Install-from-URL costs 166 KB**, all of it TLS and the HTTPS OTA client. `-flto` was tried and +saved nothing (IDF appears to ignore it for the app image), so the cheap levers are spent. SoftAP +(~35 KB) is kept: without it a board whose stored credentials went stale is only recoverable over +USB, which is the situation MoonBase exists to avoid. + +## The partition table + +`esp32/partitions/esp32dev_moonbase.csv`. Fixed overhead (bootloader, table, nvs, otadata) is +92 KB; the rest fills the chip exactly, with every app offset 64 KB aligned: + +| Region | Type/SubType | Offset | Size | Was | +|---|---|---|---:|---:| +| nvs | data/nvs | 0x9000 | 20 KB | unchanged | +| otadata | data/ota | 0xE000 | 8 KB | unchanged | +| **moonbase** | app/factory | 0x10000 | **896 KB** | new | +| **app** | app/ota_0 | 0xF0000 | **2496 KB** | 1856 KB | +| **littlefs** | data/littlefs | 0x360000 | **548 KB** | 256 KB | +| coredump | data/coredump | 0x3E9000 | 64 KB | unchanged | + +896 KB holds the measured 742 KB image with 154 KB spare. HTTPS is 166 KB of that image and is +kept deliberately: install-from-URL is what lets a device fetch its own release instead of having +the file pushed from whatever machine is in front of it, and the releases live on GitHub, which is +HTTPS-only. Serving firmware over plain HTTP instead would mean the device executes whatever an +attacker on the path substituted; signing the image would cost comparable space plus real work. + +The headroom is sized for a new COMPONENT rather than for features: factory reset, re-provisioning, +config backup and diagnostics are a few KB each, while one component can cost more than all of them +together. An undersized factory partition cannot be regrown without a second full-erase migration +of every device in the field, so it is budgeted long once. + +The filesystem partition is `littlefs` in both name and subtype (0x83, which ESP-IDF v6.1 and the +joltwallet driver both define). Older tables call the same volume `spiffs` with subtype 0x82, a +legacy misnomer since the contents have always been LittleFS. `platform_esp32_fs.cpp` now searches +subtype littlefs then spiffs, so a device that keeps an older table across an OTA still finds its +config; the 8/16 MB tables migrate in a later cycle, once every device carries that fallback. + +## MoonBase itself + +A standalone ESP-IDF project at **`moonbase/`** (a root folder, matching `moondeck/` and +`moonlive/`), `project(projectMM-moonbase)`, emitting +`projectMM-moonbase.bin`. The distinct name matters: `projectMM.bin` is matched by basename in +`release.yml:214`, `flash_esp32.py:118`, `moondeck/run/preview_installer.py:203` and +`generate_manifest.py:58`, which skips unknown basenames with only a warning. + +**It shares no sources with the application.** The earlier attempt reused `platform_esp32.cpp` and +measured 788 KB with an empty `app_main`, because that file drags RMT, I2S, PSRAM and JIT support +plus their include surface. MoonBase is written against ESP-IDF directly: a few hundred lines, its +own `sdkconfig.defaults` carrying the size flags above, and no dependency on `src/`. That +duplication is the deliberate trade for an image that must stay small and, once working, hardly +change. + +What it does, in order: bring up the network (Ethernet where present, else stored WiFi +credentials, else its own AP at **4.3.2.1** matching `NetworkModule.h:943`), then serve a single +page offering the maintenance actions, then reboot back into the app. + +Version 1 ships exactly one action: **install firmware**, both by upload and by URL (the URL form +is what makes an unattended update possible, and is why HTTPS is in the budget). Credentials are +read from `/.config/NetworkModule.json` with a bounded key scan rather than a JSON parser. + +Deliberately **not** in version 1, but the reason the name is broad: factory reset, WiFi +re-provisioning, config backup and restore, firmware downgrade, hardware diagnostics, and a +boot-with-config-disabled escape for a config that crashes the app. Each solves something only a +separate image can solve. Each also costs bytes, so each needs to earn its place. + +## The mechanism + +Verified in `~/esp/esp-idf/components/app_update/esp_ota_ops.c`: + +- `esp_ota_get_next_update_partition` iterates only OTA subtypes and falls back to the first OTA + slot found. From `factory` it returns `ota_0` (correct). **From `ota_0` it returns `ota_0` + itself**, the running partition. +- `esp_ota_begin` refuses that case with `ESP_ERR_OTA_PARTITION_CONFLICT` (`esp_ota_ops.c:173`), + so a direct upload fails safely rather than erasing the running app. +- `esp_ota_set_boot_partition` on a factory partition **erases otadata** rather than writing a + sequence number, which is what makes the power-fail story work. + +Already implemented on this branch (steps 1 and 2 below): the platform guards and the queries +`otaHasMoonBase()` / `otaBootMoonBase()` / `otaRunningMoonBase()`. + +`HttpServerModule::handleFirmwareUpload` gains one branch: when a MoonBase partition exists and we +are not already running from it, reply 202 `{"moonbase":true}` and reboot into MoonBase. `app.js` +keeps the chosen file in memory, starts a countdown BEFORE the device reboots so there is no dead +gap, polls for actual reachability rather than trusting the clock, and re-POSTs automatically: one +click, one progress experience. If MoonBase fell back to its AP the device is no longer at the +polled address, so that case says so and names 4.3.2.1. + +## Failure semantics + +A failed install deliberately leaves the device in MoonBase, even when the old application is +still intact in the app slot. Auto-reverting was considered and rejected by the PO: a device that +silently comes back running the old firmware looks like a successful update that changed nothing, +which is confusing. Ending in MoonBase makes the failure visible (the update overlay reports the error, and +MoonBase's page shows the last install status on load) and leaves every option open: retry, try a +different image, or walk away and fix the network first. + + +At every instant, otadata is either blank (boots MoonBase) or points at an `ota_0` image that +`esp_ota_end` already validated. A power cut mid-write leaves blank otadata, so the board comes up +in MoonBase and the user retries over the network. This is a **stronger** power-fail story than +today's 4 MB dual-OTA layout. + +Bootloader rollback stays disabled: it needs a second OTA slot to roll back to, and MoonBase is +the recovery path. + +## Steps + +1. **Platform guards** (done, uncommitted): reject an image larger than the target partition; + reject a target equal to the running partition; add the three queries. Inert on today's tables, + and independently valuable, since an oversized image currently fails mid-write with no check. +2. **Partition-table validity test** (done, uncommitted): `ctest` over `esp32/partitions/*.csv` + for overlaps, bounds, 64 KB app alignment, and the dual-OTA-or-MoonBase shape rule. Verified by + deliberate faults (overlap, misalignment, mixed shape each fail). +3. **MoonBase v1** (done): `moonbase/` with its size-tuned sdkconfig, the WiFi + SoftAP cascade, + one served page, install-by-upload (raw body, no multipart parsing) and install-by-URL over + HTTPS. Measured 742 KB. Ethernet is a follow-up: the app's `ethInit()` needs per-board pin + configuration, and only the eth-only 4 MB variants want it. +4. **The partition table** (done): `esp32dev_moonbase.csv` plus the + `sdkconfig.defaults.moonbase-4mb` fragment, pinned by the step-2 test. +5. **Bench MoonBase standalone** (done for WiFi): hand-flashed at 0x10000; the AP at 4.3.2.1 and + its page verified by the PO. Stored-credential WiFi and a full install still open, folded into + the step-6 bench below. Side finding: opening the serial port can bounce a classic ESP32 into + ROM download mode (DTR/RTS auto-reset), which mimics a dead board; verification is by network, + not by serial. +6. **Wire the 4 MB variants** (done): the four variants carry a `moonbase` flag in `FIRMWARES`; + `build_esp32.py` appends the fragment (last, so it wins) and builds `moonbase/` into + `build/moonbase-/`; `stale_feature_cache` now also wipes a build dir whose fragment list + or generated partition table no longer matches (IDF never regenerates sdkconfig on its own). + `flash_esp32.py` writes explicit offsets parsed from the build's own CSV (IDF's flasher_args + stages the app at the factory offset on this table), adds MoonBase at the factory slot, and + selects ota_0 via otatool so a fresh full flash boots the app with MoonBase as fallback. + `check_firmwares.py` verified the flag stays out of `firmwares.json`. Olimex erased and + flashed through this exact path; bench verification of the boot pending. +7. **The switch route and UI** (done). Bench record: the one-click FILE install ran + PO-verified through the overlay; the unattended URL cycle was verified at the mechanism + level by curl (staged-NVS handoff, plain-HTTP for LAN sources, a 3-attempt retry absorbing + the connect race right after GOT_IP), and the Reviewer then caught that the overlay itself + could never see that path succeed (MoonBase installs before it serves, so success is silence + then the new app), which is fixed; the overlay URL flow re-verifies via the moonbase-test + release. The MoonBase button on the Firmware card and MoonBase's "Boot the app" are the two + explicit ways across. Still open: the power-cut procedure. the 202-and-reboot branch, a read-only `moonbase` control on + `FirmwareUpdateModule`, the countdown-and-retry flow. Guarded so 8/16 MB behaviour is unchanged. + *Gate: PO runs an end-to-end update, then the power-cut procedure.* +8. **CI and installer** (done): `build_esp32.py` owns the shared layout helpers + (moonbase_table_csv / partition_offsets / otadata_slot0_bytes / moonbase_flash_files), the + one place that corrects IDF's flasher_args, consumed by the serial flash, the manifests, the + release preview and the QEMU image (its merged image verified at every offset). The slot-0 + otadata blob is byte-identical to otatool's own output (bench readback). release.yml stages + shared-moonbase-.bin + shared-ota-data-slot0.bin; install-picker rejects both + (pinned by a JS test); check_esp32_built also gates the MoonBase image's freshness. A + temporary `moonbase-test-release.yml` workflow (manual, esp32 only) publishes a + `moonbase-test` prerelease from this branch so the picker's URL install can be tested against + real GitHub assets before the merge; it is deleted afterwards. +9. **Migration and docs** (done): architecture.md ยง MoonBase is the concept's one home; + README feature bullet credits Tasmota's safeboot and MycilaSafeBoot; building.md notes the + one-pass 4 MB flash; MIGRATING.md carries the erase-flash entry; the FirmwareUpdate catalog + card documents the moonbase control; the resolved 4 MB flash-budget investigation is deleted + from the backlog. The planned update-badge message for legacy-table devices was not built: + OTA within the old table keeps working while the app fits its 1856 KB slot, so MIGRATING.md + carries the migration story instead. + +## Verification + +- `cmake --build build` and `ctest` at every step, plus scenarios and the spec check. +- Host tests shipped: the partition-table case (unit_PartitionTables, verified by deliberate + faults); the credentials-in-prefix contract (unit_MoonBaseContract pins ssid/password inside + MoonBase's 1024-byte read of NetworkModule.json); the install-picker asset parse with MoonBase + assets present (installer-firmware-merge). Planned but not built, with the reason: a pure- + function credential-scraper test (the scraper lives in the MoonBase image, not in src/, and + the contract test pins the cross-image half); the image-too-large rule (exercised on the bench + through the platform guard); a synthetic-flasher_args manifest test (the manifest was verified + against the real build's flasher_args instead). +- **Bench** (a rigorous change under CLAUDE.md: partition and boot changes can brick a board, so it + gets a heads-up and a go-ahead before the first flash): MoonBase reachable on Ethernet and WiFi; + AP fallback with bad credentials; a full update through the UI; a direct upload to the running + app returning 202 and never starting an erase. +- **The power-cut procedure**: flash the layout, note the config contents, start an install, and + physically cut power at ~50% (not `esp_restart()`). Expected: the board boots MoonBase, the + network returns, a retry completes, and the config survives. Repeat at ~10% and ~95%. diff --git a/docs/metrics/repo-health.json b/docs/metrics/repo-health.json index eda3da80..e65fa93c 100644 --- a/docs/metrics/repo-health.json +++ b/docs/metrics/repo-health.json @@ -1,23 +1,23 @@ { - "commit": "09cfafa4", + "commit": "df8344a9", "flash": { - "esp32s3-n16r8": 1826976, - "desktop": 1251256, - "esp32": 1784368, + "esp32s3-n16r8": 1831360, + "desktop": 1268136, + "esp32": 1788256, "esp32p4rev1-eth": 1675216, "esp32p4rev1-eth-wifi": 1933472, "esp32s3-n8r8": 1753232, "esp32s31": 2105072, "esp32-16mb": 1714608, - "esp32-eth": 1324816, - "esp32-wrover": 1765504, - "qemu": 1318160, + "esp32-eth": 1395840, + "esp32-wrover": 1841952, + "qemu": 1382000, "esp32p4rev3-eth": 1643760 }, "perf": { "desktop": { - "tick_us": 237, - "fps": 4219 + "tick_us": 2243, + "fps": 445 }, "esp32": { "tick_us": 2151, @@ -25,54 +25,54 @@ } }, "loc": { - "core": 20319, - "light": 26068, - "platform": 15453, - "ui": 7378, - "test": 47189, - "moondeck": 21856 + "core": 20373, + "light": 26080, + "platform": 15560, + "ui": 7537, + "test": 47444, + "moondeck": 22113 }, "comments": { "core": { - "lines": 8039, + "lines": 8054, "ratio": 0.428 }, "light": { - "lines": 10319, + "lines": 10316, "ratio": 0.437 }, "platform": { - "lines": 5467, + "lines": 5507, "ratio": 0.388 }, "ui": { - "lines": 1979, - "ratio": 0.285 + "lines": 2011, + "ratio": 0.283 }, "test": { - "lines": 8698, - "ratio": 0.212 + "lines": 8734, + "ratio": 0.211 }, "moondeck": { - "lines": 3532, + "lines": 3571, "ratio": 0.185 } }, "tests": { - "cases": 1579, + "cases": 1584, "scenarios": 23 }, "docs": { - "md_files": 196, - "md_lines": 28919, - "plans_files": 102, - "backlog_lines": 4526, - "lessons_lines": 606, + "md_files": 197, + "md_lines": 29202, + "plans_files": 103, + "backlog_lines": 4513, + "lessons_lines": 622, "claude_md_lines": 136 }, "complexity": { - "functions": 2769, - "over_threshold": 172, + "functions": 2776, + "over_threshold": 173, "worst_ccn": 108 } } diff --git a/docs/metrics/repo-health.md b/docs/metrics/repo-health.md index 806660a7..6f570aab 100644 --- a/docs/metrics/repo-health.md +++ b/docs/metrics/repo-health.md @@ -1,6 +1,6 @@ # Repo health -Measured at `09cfafa4`. Generated by [`moondeck/check/repo_health.py`](../../moondeck/check/repo_health.py) on every KPI-gate run. **Do not edit by hand.** +Measured at `df8344a9`. Generated by [`moondeck/check/repo_health.py`](../../moondeck/check/repo_health.py) on every KPI-gate run. **Do not edit by hand.** Current state only; the trend is this file's git history (`git log -p docs/metrics/repo-health.md`). Nothing here fails a build: the numbers make growth visible, the judgment stays human. @@ -8,60 +8,60 @@ Current state only; the trend is this file's git history (`git log -p docs/metri | Target | Flash | |---|---:| -| desktop | 1,222 KB (โˆ’0 KB) โœ“ | -| esp32 | 1,743 KB (โˆ’1 KB) โœ“ | +| desktop | 1,238 KB (+16 KB) โš  | +| esp32 | 1,746 KB (+4 KB) โš  | | esp32-16mb | 1,674 KB | -| esp32-eth | 1,294 KB | -| esp32-wrover | 1,724 KB | -| esp32p4rev1-eth | 1,636 KB (โˆ’1 KB) โœ“ | +| esp32-eth | 1,363 KB (+69 KB) โš  | +| esp32-wrover | 1,799 KB (+75 KB) โš  | +| esp32p4rev1-eth | 1,636 KB | | esp32p4rev1-eth-wifi | 1,888 KB | | esp32p4rev3-eth | 1,605 KB | -| esp32s3-n16r8 | 1,784 KB (โˆ’1 KB) โœ“ | +| esp32s3-n16r8 | 1,788 KB (+4 KB) โš  | | esp32s3-n8r8 | 1,712 KB | -| esp32s31 | 2,056 KB (โˆ’1 KB) โœ“ | -| qemu | 1,287 KB | +| esp32s31 | 2,056 KB | +| qemu | 1,350 KB (+62 KB) โš  | ## Render performance | Target | Tick | FPS | |---|---:|---:| -| desktop | 237 ยตs (+6 ยตs) โš  | 4,219 (โˆ’110) โš  | +| desktop | 2,243 ยตs (+2,006 ยตs) โš  | 445 (โˆ’3,774) โš  | | esp32 | 2,151 ยตs | 464 | ## Code | Area | Lines | Comments | Comment share | |---|---:|---:|---:| -| core | 20,319 (โˆ’112) โœ“ | 8,039 | 42.8 % | -| light | 26,068 (+50) โš  | 10,319 | 43.7 % (โˆ’0.1 %) โœ“ | -| platform | 15,453 | 5,467 | 38.8 % | -| ui | 7,378 (+18) โš  | 1,979 | 28.5 % (โˆ’0.1 %) โœ“ | -| test | 47,189 (+63) โš  | 8,698 | 21.2 % | -| moondeck | 21,856 (+7) โš  | 3,532 | 18.5 % | +| core | 20,373 (+54) โš  | 8,054 | 42.8 % | +| light | 26,080 (+12) โš  | 10,316 | 43.7 % | +| platform | 15,560 (+107) โš  | 5,507 | 38.8 % | +| ui | 7,537 (+159) โš  | 2,011 | 28.3 % (โˆ’0.2 %) โœ“ | +| test | 47,444 (+255) โš  | 8,734 | 21.1 % (โˆ’0.1 %) โœ“ | +| moondeck | 22,113 (+257) โš  | 3,571 | 18.5 % | ## Tests | Kind | Count | |---|---:| -| unit cases | 1,579 (+2) โœ“ | +| unit cases | 1,584 (+5) โœ“ | | scenarios | 23 | ## Complexity | Metric | Value | |---|---:| -| functions | 2,769 (+2) โœ“ | -| over threshold | 172 (+1) โš  | +| functions | 2,776 (+7) โœ“ | +| over threshold | 173 (+1) โš  | | worst CCN | 108 | ## Documentation | Metric | Value | |---|---:| -| markdown files | 196 (+1) โš  | -| markdown lines | 28,919 (+148) โš  | -| plan files | 102 (+1) โš  | -| backlog lines | 4,526 | -| lessons lines | 606 | +| markdown files | 197 (+1) โš  | +| markdown lines | 29,202 (+283) โš  | +| plan files | 103 (+1) โš  | +| backlog lines | 4,513 (โˆ’13) โœ“ | +| lessons lines | 622 (+16) โš  | | CLAUDE.md lines | 136 | diff --git a/docs/moonmodules/core/system.md b/docs/moonmodules/core/system.md index f8ca2d41..1ad7aba7 100644 --- a/docs/moonmodules/core/system.md +++ b/docs/moonmodules/core/system.md @@ -94,7 +94,10 @@ Over-the-air firmware flashing โ€” the one operation that swaps the binary and n Firmware update module controls - `firmware` โ€” the OTA image to flash. -- read-only โ€” `version`, `build`, `firmwarePartition`, `update_pct` (progress). +- read-only: `version`, `build`, `firmwarePartition`, `update_pct` (progress), and on 4 MB + boards `moonbase`: the second boot image is present, so installs run through the + reboot-into-MoonBase cycle behind one "updating firmware" overlay, and a **MoonBase** button + opens the maintenance image directly ([architecture.md ยง MoonBase](../../architecture.md#moonbase-the-second-boot-image-4-mb-boards)). Detail: [technical](moxygen/FirmwareUpdateModule.md) diff --git a/esp32/partitions/esp32dev_moonbase.csv b/esp32/partitions/esp32dev_moonbase.csv new file mode 100644 index 00000000..7e8f649d --- /dev/null +++ b/esp32/partitions/esp32dev_moonbase.csv @@ -0,0 +1,66 @@ +# Partition table for ESP32 classic (4 MB flash) WITH MoonBase. +# +# A 4 MB board has room for one application, not two. The dual-OTA layout (esp32dev.csv) spends +# half the chip on a second copy of the firmware, which is what ran the app slot out of space in +# 2026-08. Here that second copy is replaced by MoonBase: a small image in the `factory` slot that +# owns the device while the application is being replaced (a board cannot rewrite the partition it +# is executing from). See moonbase/ and the MoonBase plan. +# +# The app slot grows by a third as a result: 1856 KB -> 2496 KB. +# +# MoonBase is 896 KB against a measured 742 KB image (WiFi + SoftAP + HTTPS + OTA, size-tuned +# sdkconfig), leaving ~154 KB of headroom. +# +# HTTPS is 166 KB of that image and is kept deliberately. Install-from-URL is what lets a device +# fetch its own release rather than having the file pushed from whatever machine happens to be in +# front of it, and the releases live on GitHub, which is HTTPS-only. The alternative that keeps +# the convenience without TLS is to serve firmware over plain HTTP, which means the device would +# execute whatever an attacker on the path substituted; signing the image instead would cost +# comparable space plus real work. Authenticated downloads are worth the flash. +# +# The headroom is sized for a new COMPONENT, not for features: factory reset, re-provisioning, +# config backup and diagnostics are a few KB each, while one component can cost more than all of +# them together. An undersized factory partition cannot be regrown without a second full-erase +# migration of every device in the field, so it is budgeted long once. +# +# The filesystem partition is `littlefs` in both name and subtype (0x83, which ESP-IDF and the +# joltwallet driver both define). Older tables label the same volume `spiffs` with subtype 0x82, +# which is a legacy misnomer: the contents have always been LittleFS. platform_esp32_fs.cpp looks +# for the littlefs subtype first and falls back to the spiffs one, so a device that keeps an older +# table across an OTA still finds its config. +# +# A device adopts this table only through a FULL SERIAL FLASH: OTA writes the app, never the +# partition table. +# +# MIGRATION LOSES USER CONFIG. The old 4 MB table put the filesystem at 0x3B0000; this one puts it +# at 0x360000, so the existing volume is not where the new table looks for it. A migrating device +# comes up unprovisioned: WiFi credentials, module config and scripts all have to be set again. +# Aligning to the old offset would have preserved them at the cost of freezing the filesystem at +# its old 256 KB, which is not worth it for a one-time step someone performs with a cable in hand. +# +# Layout (4 MB = 0x400000): +# 0x0000-0x8FFF bootloader (reserved) +# 0x9000-0xDFFF nvs ( 20 KB) +# 0xE000-0xFFFF otadata ( 8 KB) +# 0x10000-0xEFFFF moonbase ( 896 KB) -> factory +# 0xF0000-0x35FFFF app (2496 KB) -> ota_0 +# 0x360000-0x3E8FFF littlefs ( 548 KB) -> config, scripts, presets +# 0x3E9000-0x3F8FFF coredump ( 64 KB) +# +# On the SubType column, which ESP-IDF fixes and we cannot rename: +# app/factory the fallback app. The bootloader runs it when otadata is blank or invalid, which +# is exactly how a power cut mid-install lands back in MoonBase. +# app/ota_0 an UPDATABLE app slot, number 0. It does not mean "holds an OTA"; it means the +# bootloader may select it via otadata, which is what makes esp_ota_write able to +# target it. A factory partition can never be updated over the air. +# data/ota not an app: 8 KB of bookkeeping (two records, each a sequence number plus CRC) +# naming which ota_N slot to boot. +# Only the first column is a name of our choosing. +# +# Name, Type, SubType, Offset, Size, Flags +nvs, data, nvs, 0x9000, 0x5000, +otadata, data, ota, 0xe000, 0x2000, +moonbase, app, factory, 0x10000, 0xE0000, +app, app, ota_0, 0xF0000, 0x270000, +littlefs, data, littlefs,0x360000, 0x89000, +coredump, data, coredump,0x3E9000, 0x10000, diff --git a/esp32/sdkconfig.defaults.moonbase-4mb b/esp32/sdkconfig.defaults.moonbase-4mb new file mode 100644 index 00000000..af4d6b1a --- /dev/null +++ b/esp32/sdkconfig.defaults.moonbase-4mb @@ -0,0 +1 @@ +CONFIG_PARTITION_TABLE_CUSTOM_FILENAME="partitions/esp32dev_moonbase.csv" diff --git a/moonbase/CMakeLists.txt b/moonbase/CMakeLists.txt new file mode 100644 index 00000000..74f3dced --- /dev/null +++ b/moonbase/CMakeLists.txt @@ -0,0 +1,19 @@ +# MoonBase: the second boot image for 4 MB boards. +# +# A board with one application slot cannot rewrite the partition it is executing from, so the +# install has to run from somewhere else. MoonBase is that somewhere: a small, rarely changing +# firmware in the `factory` partition that owns the device while the application is being replaced. +# Installing firmware is its first job, not its definition; see the plan for what may join it. +# +# It is a SEPARATE ESP-IDF project, and it shares no code with the application. Reusing the app's +# platform layer was measured at 788 KB with an empty app_main, because those sources drag RMT, +# I2S, PSRAM and JIT support with them. Written directly against ESP-IDF the same job measures +# ~590 KB. The duplication is the deliberate price of an image that must stay small and, once +# working, hardly change. +# +# The project name is load-bearing: `projectMM.bin` is matched by BASENAME in release.yml, +# flash_esp32.py, preview_installer.py and generate_manifest.py, and unknown basenames are skipped +# with only a warning, so MoonBase must not answer to it. +cmake_minimum_required(VERSION 3.16) +include($ENV{IDF_PATH}/tools/cmake/project.cmake) +project(projectMM-moonbase) diff --git a/moonbase/main/CMakeLists.txt b/moonbase/main/CMakeLists.txt new file mode 100644 index 00000000..69b435b9 --- /dev/null +++ b/moonbase/main/CMakeLists.txt @@ -0,0 +1,9 @@ +idf_component_register( + SRCS "moonbase_main.cpp" + INCLUDE_DIRS "" + # esp_https_ota brings esp_http_client and esp-tls with it, so neither is listed separately. + # esp_eth is absent on purpose: Ethernet support is a follow-up (the app's ethInit needs + # per-board pin configuration), and requiring the component would link a driver nothing calls. + REQUIRES nvs_flash esp_netif esp_event esp_wifi app_update esp_https_ota + mbedtls esp_partition joltwallet__littlefs) +target_compile_options(${COMPONENT_LIB} PRIVATE -Wall -Wextra) diff --git a/moonbase/main/idf_component.yml b/moonbase/main/idf_component.yml new file mode 100644 index 00000000..106e37f6 --- /dev/null +++ b/moonbase/main/idf_component.yml @@ -0,0 +1,2 @@ +dependencies: + joltwallet/littlefs: "^1.16.0" diff --git a/moonbase/main/moonbase_main.cpp b/moonbase/main/moonbase_main.cpp new file mode 100644 index 00000000..ba7f6900 --- /dev/null +++ b/moonbase/main/moonbase_main.cpp @@ -0,0 +1,523 @@ +// MoonBase: the second boot image. +// +// A 4 MB board has room for one application, not two, so its partition table carries this small +// image in the `factory` slot instead of a second copy of the firmware. When the application must +// be replaced, the device reboots here: MoonBase owns the board, writes the new firmware into the +// application slot it is not itself running from, and hands control back. +// +// Everything here is written directly against ESP-IDF. It shares no code with the application on +// purpose: the app's platform layer pulls in RMT, I2S, PSRAM and the JIT, which measured 788 KB +// with an empty entry point. This file plus its sdkconfig measures around a quarter of the flash +// instead. The other half of the budget is in ../sdkconfig.defaults, which is part of the design. +// +// The flow, in order: +// 1. mount the application's filesystem read-only and read the stored WiFi credentials +// 2. bring up the network: Ethernet if the board has it, else WiFi STA, else our own AP +// 3. serve one page: install by upload, or install from a URL +// 4. write the application slot, point the bootloader at it, reboot + +#include +#include +#include + +#include "esp_event.h" +#include "esp_http_client.h" +#include "esp_crt_bundle.h" +#include "esp_https_ota.h" +#include "esp_littlefs.h" +#include "esp_netif.h" +#include "esp_ota_ops.h" +#include "esp_partition.h" +#include "esp_system.h" +#include "esp_wifi.h" +#include "freertos/FreeRTOS.h" +#include "freertos/event_groups.h" +#include "freertos/task.h" +#include "lwip/sockets.h" +#include "lwip/netdb.h" +#include "nvs_flash.h" + +namespace { + +// The application writes its config as /.config/.json on a LittleFS volume. Tables +// written from 2026-08 label that partition `littlefs`; older ones label it `spiffs`, so both are +// tried by subtype then label. MoonBase only ever READS it, so a failed install cannot +// corrupt user config. +struct FsCandidate { esp_partition_subtype_t subtype; const char* label; }; +constexpr FsCandidate kFsCandidates[] = { + {ESP_PARTITION_SUBTYPE_DATA_LITTLEFS, "littlefs"}, + {ESP_PARTITION_SUBTYPE_DATA_SPIFFS, "spiffs"}, +}; +constexpr const char* kFsMountPoint = "/fs"; +constexpr const char* kNetworkConfig = "/fs/.config/NetworkModule.json"; + +// The AP fallback address matches the application's (NetworkModule uses 4.3.2.1), so a user who +// has provisioned this device before sees the same address in both firmwares. +constexpr const char* kApAddress = "4.3.2.1"; +constexpr const char* kApName = "MoonBase"; + +constexpr int kHttpPort = 80; + +char ssid_[64] = {}; +char password_[64] = {}; +char status_[96] = "idle"; + +EventGroupHandle_t netEvents_; +constexpr int kNetGotIp = BIT0; + +// --------------------------------------------------------------------------------------------- +// Credentials +// --------------------------------------------------------------------------------------------- + +// Extract one top-level string value from the config JSON. Deliberately not a JSON parser: +// MoonBase reads exactly two known keys out of a file this project itself wrote, and linking a +// parser to do it would cost more than the whole feature. The scan is anchored on `"key":"` at +// the TOP level only, which matters because the same file carries a child module's "0.password" +// (the MQTT broker's) that a naive substring search would find first. +bool jsonFindString(const char* json, const char* key, char* out, size_t outLen) { + char needle[40]; + const int n = std::snprintf(needle, sizeof(needle), "\"%s\":\"", key); + if (n <= 0 || static_cast(n) >= sizeof(needle)) return false; + const char* p = std::strstr(json, needle); + if (!p) return false; + p += n; + size_t i = 0; + while (*p && *p != '"' && i + 1 < outLen) { + char c = *p++; + if (c == '\\' && *p) { + // The app's writer (JsonSink, RFC 8259) escapes with \" \\ \/ \n \r \t, and + // \uXXXX for other control bytes. All but \u are decoded here; a credential + // holding a raw control byte fails the join and lands on the access point, + // visible and recoverable, which is not worth a \u decoder in this image. + const char e = *p++; + switch (e) { + case 'n': c = '\n'; break; + case 'r': c = '\r'; break; + case 't': c = '\t'; break; + case 'u': return false; + default: c = e; // \" \\ \/ decode to the char itself + } + } + out[i++] = c; + } + out[i] = '\0'; + return i > 0; +} + +// Read the stored WiFi credentials, if there are any. Absent, unreadable or empty all mean the +// same thing to the caller: fall through to the access point. +void loadCredentials() { + const char* label = nullptr; + for (const auto& c : kFsCandidates) { + if (!esp_partition_find_first(ESP_PARTITION_TYPE_DATA, c.subtype, c.label)) continue; + esp_vfs_littlefs_conf_t conf = {}; + conf.base_path = kFsMountPoint; + conf.partition_label = c.label; + conf.format_if_mount_failed = false; // never format: this volume is the user's config + if (esp_vfs_littlefs_register(&conf) == ESP_OK) { label = c.label; break; } + } + if (!label) return; + + FILE* f = std::fopen(kNetworkConfig, "r"); + if (f) { + // The credentials are the first keys the module writes, so a bounded prefix read finds + // them without holding the whole file (which carries every child module's config too). + // The bound is a cross-image contract with NetworkModule's control order; the app pins + // it with a unit test (unit_MoonBaseContract). + char buf[1024]; + const size_t got = std::fread(buf, 1, sizeof(buf) - 1, f); + buf[got] = '\0'; + std::fclose(f); + jsonFindString(buf, "ssid", ssid_, sizeof(ssid_)); + jsonFindString(buf, "password", password_, sizeof(password_)); + } + esp_vfs_littlefs_unregister(label); +} + +// --------------------------------------------------------------------------------------------- +// Network +// --------------------------------------------------------------------------------------------- + +void onGotIp(void*, esp_event_base_t, int32_t, void*) { + xEventGroupSetBits(netEvents_, kNetGotIp); +} + +void onWifiEvent(void*, esp_event_base_t, int32_t id, void*) { + if (id == WIFI_EVENT_STA_START || id == WIFI_EVENT_STA_DISCONNECTED) esp_wifi_connect(); +} + +// Try the stored credentials for a bounded time. Returns whether an address arrived. +bool wifiStation(uint32_t waitMs) { + if (!ssid_[0]) return false; + esp_netif_create_default_wifi_sta(); + wifi_init_config_t init = WIFI_INIT_CONFIG_DEFAULT(); + if (esp_wifi_init(&init) != ESP_OK) return false; + + wifi_config_t cfg = {}; + std::strncpy(reinterpret_cast(cfg.sta.ssid), ssid_, sizeof(cfg.sta.ssid) - 1); + std::strncpy(reinterpret_cast(cfg.sta.password), password_, sizeof(cfg.sta.password) - 1); + esp_wifi_set_mode(WIFI_MODE_STA); + esp_wifi_set_config(WIFI_IF_STA, &cfg); + esp_event_handler_instance_register(WIFI_EVENT, ESP_EVENT_ANY_ID, &onWifiEvent, nullptr, nullptr); + esp_wifi_start(); + + const EventBits_t bits = xEventGroupWaitBits(netEvents_, kNetGotIp, pdFALSE, pdFALSE, + pdMS_TO_TICKS(waitMs)); + if (bits & kNetGotIp) return true; + esp_wifi_stop(); + esp_wifi_deinit(); + return false; +} + +// The last resort, and the reason SoftAP stays in the size budget: a board whose stored +// credentials no longer work is still reachable without a cable. +bool wifiAccessPoint() { + esp_netif_t* ap = esp_netif_create_default_wifi_ap(); + if (!ap) return false; + esp_netif_ip_info_t ip = {}; + ip.ip.addr = esp_ip4addr_aton(kApAddress); + ip.gw.addr = ip.ip.addr; + ip.netmask.addr = esp_ip4addr_aton("255.255.255.0"); + esp_netif_dhcps_stop(ap); + esp_netif_set_ip_info(ap, &ip); + esp_netif_dhcps_start(ap); + + wifi_init_config_t init = WIFI_INIT_CONFIG_DEFAULT(); + if (esp_wifi_init(&init) != ESP_OK) return false; + wifi_config_t cfg = {}; + std::strncpy(reinterpret_cast(cfg.ap.ssid), kApName, sizeof(cfg.ap.ssid) - 1); + cfg.ap.ssid_len = static_cast(std::strlen(kApName)); + cfg.ap.max_connection = 2; + cfg.ap.authmode = WIFI_AUTH_OPEN; // an open AP: the user is standing at the device + esp_wifi_set_mode(WIFI_MODE_AP); + esp_wifi_set_config(WIFI_IF_AP, &cfg); + return esp_wifi_start() == ESP_OK; +} + +// --------------------------------------------------------------------------------------------- +// Installing +// --------------------------------------------------------------------------------------------- + +// The one page MoonBase serves. Inline and tiny: no filesystem read, no compression, no assets. +const char kPage[] = + "" + "MoonBase" + "" + "

MoonBase

Install firmware to return this device to normal operation.

" + "
From a file
" + "
" + "
From a URL
" + "
" + "
Back to the app
Boot the installed firmware without changing it." + "
" + "
" + ""; + +// The application slot. From the factory partition esp_ota_get_next_update_partition returns the +// first OTA slot, which is the one we want and is never the one we are running from. +const esp_partition_t* appPartition() { + return esp_ota_get_next_update_partition(nullptr); +} + +// Write a firmware image pulled from `url` straight into the application slot. This is what makes +// an unattended install possible: point MoonBase at a release asset and it fetches it itself. +bool installFromUrl(const char* url) { + esp_http_client_config_t http = {}; + http.url = url; + http.timeout_ms = 20000; + http.keep_alive_enable = true; + http.crt_bundle_attach = esp_crt_bundle_attach; // GitHub and friends are HTTPS + esp_https_ota_config_t ota = {}; + ota.http_config = &http; + + esp_https_ota_handle_t handle = nullptr; + esp_err_t beginErr = esp_https_ota_begin(&ota, &handle); + if (beginErr != ESP_OK) { + // Numeric on purpose: the error-name table is compiled out for size + // (ESP_ERR_TO_NAME_LOOKUP=n), so esp_err_to_name would say "UNKNOWN ERROR". + std::snprintf(status_, sizeof(status_), "error: cannot start the download (0x%x)", + static_cast(beginErr)); + return false; + } + esp_err_t err; + while ((err = esp_https_ota_perform(handle)) == ESP_ERR_HTTPS_OTA_IN_PROGRESS) { + std::snprintf(status_, sizeof(status_), "downloading: %d bytes", + esp_https_ota_get_image_len_read(handle)); + } + if (err != ESP_OK || esp_https_ota_finish(handle) != ESP_OK) { + std::snprintf(status_, sizeof(status_), "error: the download failed"); + return false; + } + std::snprintf(status_, sizeof(status_), "installed, restarting"); + return true; +} + + +// --------------------------------------------------------------------------------------------- +// The HTTP server +// --------------------------------------------------------------------------------------------- +// +// Hand-written on raw sockets rather than esp_http_server: MoonBase serves one page and receives +// one file, and the component would cost more than the handlers do. One connection at a time is +// the right model here, since installing firmware is exclusive by nature. + +constexpr size_t kRecvChunk = 4096; + +void sendAll(int sock, const char* data, size_t len) { + size_t sent = 0; + while (sent < len) { + const int n = ::send(sock, data + sent, len - sent, 0); + if (n <= 0) return; // peer gone: the caller is finishing anyway + sent += static_cast(n); + } +} + +void sendResponse(int sock, const char* status, const char* type, const char* body) { + char head[160]; + const int n = std::snprintf(head, sizeof(head), + "HTTP/1.1 %s\r\nContent-Type: %s\r\nContent-Length: %u\r\n" + "Connection: close\r\n\r\n", + status, type, static_cast(std::strlen(body))); + if (n > 0) sendAll(sock, head, static_cast(n)); + sendAll(sock, body, std::strlen(body)); +} + +// Write `contentLen` bytes from the socket into the application slot. `prefix` carries whatever +// arrived in the same read as the headers. +bool installFromSocket(int sock, const char* prefix, size_t prefixLen, size_t contentLen) { + const esp_partition_t* part = appPartition(); + if (!part) { std::snprintf(status_, sizeof(status_), "error: no app partition"); return false; } + if (contentLen == 0 || contentLen > part->size) { + std::snprintf(status_, sizeof(status_), "error: image is %u bytes, the slot holds %u", + static_cast(contentLen), static_cast(part->size)); + return false; + } + + esp_ota_handle_t handle = 0; + if (esp_ota_begin(part, contentLen, &handle) != ESP_OK) { + std::snprintf(status_, sizeof(status_), "error: cannot start the install"); + return false; + } + + size_t written = 0; + if (prefixLen) { + if (esp_ota_write(handle, prefix, prefixLen) != ESP_OK) { + esp_ota_abort(handle); + std::snprintf(status_, sizeof(status_), "error: write failed"); + return false; + } + written = prefixLen; + } + + char* buf = static_cast(std::malloc(kRecvChunk)); + if (!buf) { esp_ota_abort(handle); std::snprintf(status_, sizeof(status_), "error: out of memory"); return false; } + while (written < contentLen) { + const size_t want = (contentLen - written) < kRecvChunk ? (contentLen - written) : kRecvChunk; + const int n = ::recv(sock, buf, want, 0); + if (n <= 0) break; // the upload was cut short + if (esp_ota_write(handle, buf, static_cast(n)) != ESP_OK) { + std::free(buf); + esp_ota_abort(handle); + std::snprintf(status_, sizeof(status_), "error: write failed"); + return false; + } + written += static_cast(n); + } + std::free(buf); + + if (written != contentLen) { + esp_ota_abort(handle); + std::snprintf(status_, sizeof(status_), "error: upload ended early (%u of %u bytes)", + static_cast(written), static_cast(contentLen)); + return false; + } + // esp_ota_end validates the image (magic and checksum) before we ever point the bootloader at + // it, which is what makes a power cut mid-write safe: otadata still names MoonBase. + if (esp_ota_end(handle) != ESP_OK) { + std::snprintf(status_, sizeof(status_), "error: the image is not valid firmware"); + return false; + } + if (esp_ota_set_boot_partition(part) != ESP_OK) { + std::snprintf(status_, sizeof(status_), "error: cannot set the boot partition"); + return false; + } + std::snprintf(status_, sizeof(status_), "installed, restarting"); + return true; +} + +// Read the request head, dispatch, and (on a successful install) restart into the application. +void serveOne(int sock) { + // TCP does not coalesce: the header block (or a small body) can arrive in several + // segments, so read until the blank line is seen, bounded by the buffer. A request + // whose headers do not fit 1023 bytes is not one of ours and falls out as 404. + char head[1024]; + size_t got = 0; + const char* bodyStart = nullptr; + while (got < sizeof(head) - 1) { + const int n = ::recv(sock, head + got, sizeof(head) - 1 - got, 0); + if (n <= 0) break; + got += static_cast(n); + head[got] = '\0'; + if ((bodyStart = std::strstr(head, "\r\n\r\n"))) break; + } + if (got == 0) return; + head[got] = '\0'; + const size_t headLen = bodyStart ? static_cast(bodyStart + 4 - head) : got; + size_t prefixLen = got - headLen; + + size_t contentLen = 0; + if (const char* cl = std::strstr(head, "Content-Length:")) { + contentLen = static_cast(std::strtoul(cl + 15, nullptr, 10)); + } + + bool installed = false; + if (std::strncmp(head, "POST /install-url", 17) == 0) { + // The body is the URL itself; small enough to finish reading into the same buffer. + while (prefixLen < contentLen && headLen + prefixLen < sizeof(head) - 1) { + const int n = ::recv(sock, head + headLen + prefixLen, + sizeof(head) - 1 - headLen - prefixLen, 0); + if (n <= 0) break; + prefixLen += static_cast(n); + } + char url[256] = {}; + const size_t n = prefixLen < sizeof(url) - 1 ? prefixLen : sizeof(url) - 1; + std::memcpy(url, head + headLen, n); + installed = installFromUrl(url); + sendResponse(sock, installed ? "200 OK" : "500 Internal Server Error", "text/plain", status_); + } else if (std::strncmp(head, "POST /install", 13) == 0) { + installed = installFromSocket(sock, head + headLen, prefixLen, contentLen); + sendResponse(sock, installed ? "200 OK" : "500 Internal Server Error", "text/plain", status_); + } else if (std::strncmp(head, "POST /boot-app", 14) == 0) { + // Switch back to the installed application without installing anything. + // esp_ota_set_boot_partition validates the image first, so a half-written app is + // refused and the device stays here: only a bootable app can be booted. + const esp_partition_t* app = appPartition(); + const bool ok = app && esp_ota_set_boot_partition(app) == ESP_OK; + if (ok) std::snprintf(status_, sizeof(status_), "booting the app"); + else std::snprintf(status_, sizeof(status_), "error: no valid app image"); + sendResponse(sock, ok ? "200 OK" : "500 Internal Server Error", "text/plain", status_); + installed = ok; // reuse the reply-then-restart tail below + } else if (std::strncmp(head, "GET /moonbase", 13) == 0) { + // Identity probe: the app UI polls this across the update cycle to tell which image is + // answering at the shared address (the app 404s it). Body = the live install status, so + // the poll doubles as a progress read during an unattended install. + sendResponse(sock, "200 OK", "text/plain", status_); + } else if (std::strncmp(head, "GET / ", 6) == 0 || std::strncmp(head, "GET /index", 10) == 0) { + sendResponse(sock, "200 OK", "text/html", kPage); + } else { + sendResponse(sock, "404 Not Found", "text/plain", "not found"); + } + + ::shutdown(sock, SHUT_RDWR); + ::close(sock); + if (installed) { + // Let the reply reach the browser before the device goes away. + vTaskDelay(pdMS_TO_TICKS(500)); + esp_restart(); + } +} + +void serveForever() { + const int listener = ::socket(AF_INET, SOCK_STREAM, IPPROTO_TCP); + if (listener < 0) return; + int yes = 1; + ::setsockopt(listener, SOL_SOCKET, SO_REUSEADDR, &yes, sizeof(yes)); + sockaddr_in addr = {}; + addr.sin_family = AF_INET; + addr.sin_addr.s_addr = htonl(INADDR_ANY); + addr.sin_port = htons(kHttpPort); + if (::bind(listener, reinterpret_cast(&addr), sizeof(addr)) != 0) { ::close(listener); return; } + if (::listen(listener, 1) != 0) { ::close(listener); return; } + + while (true) { + const int sock = ::accept(listener, nullptr, nullptr); + if (sock < 0) continue; + // A stalled peer must not hold MoonBase forever: the whole point is that the device stays + // reachable for the next attempt. + timeval tv = {}; + tv.tv_sec = 30; + ::setsockopt(sock, SOL_SOCKET, SO_RCVTIMEO, &tv, sizeof(tv)); + ::setsockopt(sock, SOL_SOCKET, SO_SNDTIMEO, &tv, sizeof(tv)); + serveOne(sock); + } +} + +} // namespace + +extern "C" void app_main() { + esp_err_t nvs = nvs_flash_init(); + if (nvs == ESP_ERR_NVS_NO_FREE_PAGES || nvs == ESP_ERR_NVS_NEW_VERSION_FOUND) { + nvs_flash_erase(); + nvs_flash_init(); + } + netEvents_ = xEventGroupCreate(); + esp_netif_init(); + esp_event_loop_create_default(); + esp_event_handler_instance_register(IP_EVENT, ESP_EVENT_ANY_ID, &onGotIp, nullptr, nullptr); + + loadCredentials(); + + // WiFi STA with the app's stored credentials, else the open access point: the guarantee + // that a board is never unreachable because its credentials went stale. Ethernet is a + // follow-up (backlog-core, MoonBase follow-ups): it needs per-board PHY/pin configuration, + // so today the eth-only esp32-eth variant lands on the access point here. + // The unattended handoff: the app may have staged an install URL in NVS before rebooting + // into MoonBase (platform::moonbaseStageInstallUrl). Read AND erase it unconditionally, + // before anything can fail: a URL that crashes or fails can then never boot-loop the + // device, and a stale URL can never survive a failed network join to hijack a later, + // unrelated visit to MoonBase (one try per staging, ever). + char stagedUrl[256] = {}; + { + nvs_handle_t h; + if (nvs_open("moonbase", NVS_READWRITE, &h) == ESP_OK) { + size_t len = sizeof(stagedUrl); + if (nvs_get_str(h, "url", stagedUrl, &len) != ESP_OK) stagedUrl[0] = '\0'; + nvs_erase_key(h, "url"); + nvs_commit(h); + nvs_close(h); + } + } + + bool online = wifiStation(20000); + + // STA only: on the fallback AP the URL's network is not reachable, and a user is present. + if (online && stagedUrl[0]) { + // A connect attempted straight after GOT_IP can fail (0x7002, ESP_ERR_HTTP_CONNECT) + // where the same connect succeeds seconds later: the LAN is still warming up around a + // freshly associated station. A short retry absorbs that; a genuinely unreachable URL + // still fails through to the page after the last attempt. + for (int attempt = 0; attempt < 3; attempt++) { + if (attempt) vTaskDelay(pdMS_TO_TICKS(3000)); + if (installFromUrl(stagedUrl)) esp_restart(); // straight back into the new app + } + } + + if (!online) online = wifiAccessPoint(); + + // With no network there is nothing MoonBase can do but wait: a user who cannot reach it will + // reflash over USB, and restarting into an application slot that may be empty helps nobody. + if (online) serveForever(); + while (true) vTaskDelay(pdMS_TO_TICKS(1000)); +} diff --git a/moonbase/sdkconfig.defaults b/moonbase/sdkconfig.defaults new file mode 100644 index 00000000..09224d43 --- /dev/null +++ b/moonbase/sdkconfig.defaults @@ -0,0 +1,53 @@ +# MoonBase's size budget lives here. Measured on the ESP32 classic with WiFi + HTTP + OTA: IDF +# defaults gave 881 KB, and these settings bring the same functionality to ~590 KB. Nearly 300 KB +# of the saving is configuration rather than code, which is why this file is part of the design +# and not an afterthought. + +# Size over speed. MoonBase runs for a minute at a time and does one blocking download; there is +# no hot path to protect. (-O2 -> -Os: 69 KB) +CONFIG_COMPILER_OPTIMIZATION_SIZE=y + +# Link-time optimisation: cross-module dead-code elimination, the last big lever and the one +# MycilaSafeBoot also pulls. Costs build time, not runtime. +CONFIG_COMPILER_OPTIMIZATION_LTO=y + +# No log strings, no esp_err_to_name table, no assertion text, no console. Together ~117 KB. +# The cost is real: a misbehaving MoonBase says nothing over serial. It is accepted because the +# image is small enough to reason about whole, and because a board that cannot boot MoonBase is a +# USB recovery either way. +CONFIG_LOG_DEFAULT_LEVEL_NONE=y +CONFIG_LOG_MAXIMUM_LEVEL_NONE=y +CONFIG_ESP_ERR_TO_NAME_LOOKUP=n +CONFIG_COMPILER_OPTIMIZATION_ASSERTIONS_DISABLE=y +CONFIG_ESP_SYSTEM_PANIC_PRINT_HALT=y +CONFIG_ESP_CONSOLE_NONE=y +CONFIG_VFS_SUPPORT_TERMIOS=n + +# The C library's full printf is for humans reading logs, which MoonBase has none of. (~40 KB) +CONFIG_LIBC_NEWLIB_NANO_FORMAT=y + +# Network features this image will never use. IPv6 (~25 KB), WPA3/enterprise (~25 KB). +# SoftAP is deliberately KEPT (~35 KB): without it a board with wrong WiFi credentials can only be +# recovered over USB, which is exactly the situation MoonBase exists to avoid. +CONFIG_LWIP_IPV6=n +CONFIG_ESP_WIFI_ENABLE_WPA3_SAE=n +CONFIG_ESP_WIFI_ENTERPRISE_SUPPORT=n + +# One connection at a time, small buffers: MoonBase serves one page and receives one file. +CONFIG_LWIP_MAX_SOCKETS=6 +CONFIG_ESP_WIFI_STATIC_RX_BUFFER_NUM=4 +CONFIG_ESP_WIFI_DYNAMIC_RX_BUFFER_NUM=8 +CONFIG_ESP_WIFI_TX_BUFFER_TYPE=0 +CONFIG_ESP_WIFI_STATIC_TX_BUFFER_NUM=8 + +# The SAME table the application builds against, so the two images provably agree on where +# everything lives. MoonBase reads the filesystem (for the stored WiFi credentials) and never +# writes it, so a half-finished install cannot corrupt user config. +CONFIG_ESPTOOLPY_FLASHSIZE_4MB=y +CONFIG_PARTITION_TABLE_CUSTOM=y +CONFIG_PARTITION_TABLE_CUSTOM_FILENAME="../esp32/partitions/esp32dev_moonbase.csv" + +# Plain-HTTP install URLs allowed alongside HTTPS: GitHub releases stay TLS-verified via the cert +# bundle, but a LAN source (MoonDeck serving a dev build) has no certificate and is inside the +# user's own network. The staged-URL handoff writes whatever the app was asked to install. +CONFIG_ESP_HTTPS_OTA_ALLOW_HTTP=y diff --git a/moondeck/build/build_esp32.py b/moondeck/build/build_esp32.py index e3806a2e..da6a8324 100644 --- a/moondeck/build/build_esp32.py +++ b/moondeck/build/build_esp32.py @@ -135,7 +135,8 @@ def check_idf_pin(idf_path: Path) -> None: # replaces the old separate `esp32` (WiFi-only) + `esp32-eth-wifi` keys. "esp32": { "chip": "esp32", - "fragments": ["sdkconfig.defaults", "sdkconfig.defaults.eth"], + "fragments": ["sdkconfig.defaults", "sdkconfig.defaults.eth", "sdkconfig.defaults.moonbase-4mb"], + "moonbase": True, # 4 MB: factory MoonBase + one big app slot (see moonbase/) "eth_only": False, "description": "ESP32 classic โ€” WiFi + Ethernet (RMII; per-board pins/PHY " "from deviceModels.json, default LAN8720 pins).", @@ -150,7 +151,8 @@ def check_idf_pin(idf_path: Path) -> None: # API and web UI reachable through a forwarded host port. "qemu": { "chip": "esp32", - "fragments": ["sdkconfig.defaults", "sdkconfig.defaults.qemu"], + "fragments": ["sdkconfig.defaults", "sdkconfig.defaults.qemu", "sdkconfig.defaults.moonbase-4mb"], + "moonbase": True, # 4 MB: factory MoonBase + one big app slot (see moonbase/) "eth_only": True, "description": "ESP32 classic under QEMU, emulated Ethernet (openeth), no WiFi. " "Run with moondeck/qemu/run_qemu.py, not flashed to hardware.", @@ -173,7 +175,8 @@ def check_idf_pin(idf_path: Path) -> None: "esp32-wrover": { "chip": "esp32", "fragments": ["sdkconfig.defaults", "sdkconfig.defaults.eth", - "sdkconfig.defaults.wrover"], + "sdkconfig.defaults.wrover", "sdkconfig.defaults.moonbase-4mb"], + "moonbase": True, # 4 MB: factory MoonBase + one big app slot (see moonbase/) "eth_only": False, "description": "ESP32-WROVER (classic ESP32, 4 MB flash + 4 MB quad PSRAM) โ€” WiFi + " "Ethernet. Same silicon as `esp32`; this variant enables PSRAM for " @@ -182,7 +185,8 @@ def check_idf_pin(idf_path: Path) -> None: }, "esp32-eth": { "chip": "esp32", - "fragments": ["sdkconfig.defaults", "sdkconfig.defaults.eth"], + "fragments": ["sdkconfig.defaults", "sdkconfig.defaults.eth", "sdkconfig.defaults.moonbase-4mb"], + "moonbase": True, # 4 MB: factory MoonBase + one big app slot (see moonbase/) "eth_only": True, "description": "ESP32 classic โ€” Ethernet only (WiFi compiled out; smaller " "image, more RAM). Per-board pins/PHY from deviceModels.json. The " @@ -605,6 +609,42 @@ def stale_feature_cache(build_dir: Path, extra: list[str], chip: str) -> str | N return (f"IDF_TARGET cached as {cached_target!r} but this firmware " f"wants {chip!r}") # The feature toggles whose presence/absence changes which code compiles. + # The FRAGMENT LIST is a feature flag too: IDF generates sdkconfig from + # SDKCONFIG_DEFAULTS only when the file is absent, so adding a fragment to a + # firmware (the MoonBase partition table did this first) silently leaves an + # existing dir on the OLD config. The cache still holds the LAST run's list at + # this point, so a mismatch is detectable and means: wipe and reconfigure. + wanted_frags = next((a.split("=", 1)[1] for a in extra + if a.startswith("-DSDKCONFIG_DEFAULTS=")), None) + m = re.search(r"^SDKCONFIG_DEFAULTS:[^=]*=(.*)$", text, re.MULTILINE) + cached_frags = m.group(1).strip() if m else None + if wanted_frags and cached_frags and cached_frags != wanted_frags: + return (f"SDKCONFIG_DEFAULTS cached as {cached_frags!r} but this firmware " + f"wants {wanted_frags!r}") + + # And the one generated value dangerous enough to verify outright: the partition table. The + # list comparison above cannot catch a dir poisoned BEFORE the rule existed (its cache already + # matches), so read what the fragments want (last fragment naming a table wins, IDF's own + # merge order) and compare against what the generated sdkconfig actually says. + wanted_table = None + if wanted_frags: + for frag in wanted_frags.split(";"): + fp = ESP32_DIR / frag + if not fp.exists(): + continue + m2 = re.search(r'^CONFIG_PARTITION_TABLE_CUSTOM_FILENAME="([^"]+)"', + fp.read_text(), re.MULTILINE) + if m2: + wanted_table = m2.group(1) + gen = build_dir / "sdkconfig" + if wanted_table and gen.exists(): + m2 = re.search(r'^CONFIG_PARTITION_TABLE_CUSTOM_FILENAME="([^"]+)"', + gen.read_text(), re.MULTILINE) + have_table = m2.group(1) if m2 else None + if have_table != wanted_table: + return (f"generated sdkconfig uses partition table {have_table!r} but the " + f"fragments want {wanted_table!r}") + # For each, "wanted" = does this firmware pass the -D, "cached" = is it set # in the existing cache. A disagreement means a stale dir. # MM_TASK_CPU_STATS is here too: toggling --task-cpu-stats on an existing dir must wipe, or the @@ -759,6 +799,124 @@ def main(): # Show flash/RAM usage summary subprocess.run(cmd + b_arg + ["size"], cwd=ESP32_DIR, env=env) + if FIRMWARES[firmware].get("moonbase"): + build_moonbase(cmd, env, chip) + + +# ---- MoonBase flash layout, shared by every consumer of the build output ---- +# flash_esp32.py (serial flash), generate_manifest.py (web installer), preview_installer.py +# (release preview) and run_qemu.py (emulator image) all assemble a flash layout from IDF's +# flasher_args.json. On a MoonBase table that file is WRONG about the app: IDF stages the app +# binary at the first app partition (0x10000: the factory slot, MoonBase's home), because it +# knows nothing about the two-image scheme. These helpers are the one place that knows better. + +def moonbase_table_csv(firmware: str) -> Path: + """The partition CSV a firmware's fragments select (last fragment naming one wins, + IDF's own merge order).""" + csv = ESP32_DIR / "partitions" / "esp32dev.csv" + for frag in FIRMWARES[firmware]["fragments"]: + fp = ESP32_DIR / frag + if not fp.exists(): + continue + m = re.search(r'^CONFIG_PARTITION_TABLE_CUSTOM_FILENAME="([^"]+)"', + fp.read_text(), re.MULTILINE) + if m: + csv = ESP32_DIR / m.group(1) + return csv + + +def partition_offsets(csv_path: Path) -> dict: + """SubType -> offset (hex string) for the rows a MoonBase layout needs: 'factory', + 'ota_0' and 'ota' (the otadata bookkeeping partition).""" + import csv as _csv + out = {} + for row in _csv.reader(csv_path.read_text().splitlines()): + if not row or row[0].strip().startswith("#") or len(row) < 5: + continue + subtype = row[2].strip() + if subtype in ("factory", "ota_0", "ota"): + out[subtype] = row[3].strip() + return out + + +def otadata_slot0_bytes() -> bytes: + """An otadata image with slot 0 (ota_0) selected, so a fresh full flash boots the app with + MoonBase standing by (blank otadata boots the factory slot: MoonBase). Two 4 KB copies; the + record is 32 bytes: uint32 seq, 24 bytes 0xFF, uint32 CRC over the seq alone + (bootloader_common_ota_select_crc: crc32_le seeded UINT32_MAX == zlib.crc32(seq, 0xFFFFFFFF)). + Byte-identical to what IDF's otatool writes for --slot 0, verified against a bench readback. + """ + import struct, zlib + seq = struct.pack(" list[tuple[str, Path]]: + """The corrected (offset, file) write list for a MoonBase-table flash: IDF's flash_files with + the app remapped to ota_0, the blank otadata replaced by the slot-0 image (written into the + build dir), and MoonBase added at the factory slot.""" + import json as _json + offs = partition_offsets(moonbase_table_csv(firmware)) + chip = FIRMWARES[firmware]["chip"] + moonbase_bin = build_dir.parent / f"moonbase-{chip}" / "projectMM-moonbase.bin" + if not all(k in offs for k in ("factory", "ota_0", "ota")) or not moonbase_bin.exists(): + raise FileNotFoundError( + f"MoonBase layout needs factory/ota_0/otadata offsets and a built image " + f"(run build_esp32.py first; missing: {moonbase_bin})") + otadata = build_dir / "ota_data_slot0.bin" + otadata.write_bytes(otadata_slot0_bytes()) + fa = _json.loads((build_dir / "flasher_args.json").read_text()) + writes: list[tuple[str, Path]] = [] + for off, rel in fa["flash_files"].items(): + name = Path(rel).name + if name == "projectMM.bin": + writes.append((offs["ota_0"], build_dir / rel)) + elif name == "ota_data_initial.bin": + writes.append((offs["ota"], otadata)) + else: + writes.append((off, build_dir / rel)) + writes.append((offs["factory"], moonbase_bin)) + return writes + + +def build_moonbase(cmd: list[str], env: dict, chip: str) -> None: + """Build the MoonBase image for `chip` into build/moonbase-. + + MoonBase (moonbase/) is the second boot image the 4 MB variants carry in their factory + partition: a small firmware whose job is installing the application, since a board with one + app slot cannot rewrite the partition it is executing from. It is chip-specific but variant- + agnostic, so the four classic variants share one build. Its size budget lives in + moonbase/sdkconfig.defaults; the shared partition table keeps the two images provably agreed + on where everything lives. + """ + moonbase_dir = ROOT / "moonbase" + build_dir = ROOT / "build" / f"moonbase-{chip}" + b_arg = ["-B", str(build_dir), f"-DSDKCONFIG={build_dir}/sdkconfig"] + # Same trap as stale_feature_cache: IDF generates sdkconfig from the defaults only when it is + # absent, so an edited moonbase/sdkconfig.defaults silently changes nothing. One defaults file + # here, so mtime is a sufficient staleness signal. + gen = build_dir / "sdkconfig" + defaults = moonbase_dir / "sdkconfig.defaults" + if gen.exists() and defaults.stat().st_mtime > gen.stat().st_mtime: + print(f"MoonBase build dir {build_dir.name} predates sdkconfig.defaults; " + "removing it for a clean reconfigure.") + shutil.rmtree(build_dir) + if not build_dir.exists(): + print(f"Setting MoonBase target to {chip}...") + r = subprocess.run(cmd + b_arg + ["set-target", chip], cwd=moonbase_dir, env=env) + if r.returncode != 0: + sys.exit(r.returncode) + print(f"Building MoonBase for {chip}...") + r = subprocess.run(cmd + b_arg + ["build"], cwd=moonbase_dir, env=env) + if r.returncode != 0: + sys.exit(r.returncode) + binp = build_dir / "projectMM-moonbase.bin" + if binp.exists(): + kb = binp.stat().st_size / 1024 + print(f"MoonBase image: {kb:.0f} KB (slot: 896 KB)") + if __name__ == "__main__": main() diff --git a/moondeck/build/flash_esp32.py b/moondeck/build/flash_esp32.py index 2ee6b7bb..35082a52 100644 --- a/moondeck/build/flash_esp32.py +++ b/moondeck/build/flash_esp32.py @@ -11,7 +11,9 @@ """ import argparse +import json import re +import shutil import subprocess import sys import time @@ -79,6 +81,28 @@ def _fmt_age(seconds: float) -> str: return f"{s // 86400}d" +def _moonbase_flash_cmd(build_dir, firmware: str, port: str, baud: int, env: dict) -> list[str]: + """The explicit write list for a MoonBase-layout flash, moonbase_flash_files() is the one + place that knows the corrected layout (app remapped to ota_0, slot-0 otadata so the fresh + flash boots the app, MoonBase at factory). idf.py flash cannot be used here: IDF's own + flash_args stages the app at the factory offset.""" + from build_esp32 import moonbase_flash_files + try: + writes = moonbase_flash_files(firmware, build_dir) + except FileNotFoundError as e: + print(f"MoonBase flash: {e}") + sys.exit(1) + chip_m = re.search(r'CONFIG_IDF_TARGET="([^"]+)"', (build_dir / "sdkconfig").read_text()) + if not chip_m: + print(f"MoonBase flash: no CONFIG_IDF_TARGET in {build_dir / 'sdkconfig'}; rebuild first.") + sys.exit(1) + py = shutil.which("python", path=env.get("PATH", "")) or sys.executable + cmd = [py, "-m", "esptool", "--chip", chip_m.group(1), "--port", port, "--baud", str(baud), + "write_flash"] + for off, path in writes: + cmd += [off, str(path)] + return cmd + def main(): parser = argparse.ArgumentParser(description=__doc__.split("\n\n")[0]) parser.add_argument("--port", required=True, help="Serial port") @@ -155,7 +179,15 @@ def main(): # the console; we just also scan it. print(f"==> flash baud: {baud}") mac = "" - proc = subprocess.Popen(cmd + b_arg + ["flash", "-p", args.port, "-b", str(baud)], + # A MoonBase variant cannot use `idf.py flash`: with a factory + ota_0 table, IDF stages the + # application at the FACTORY offset (0x10000), which is MoonBase's slot and too small for it. + # The parts are placed at explicit offsets instead, the same shape the web-installer manifest + # uses, with each offset read from the built partition table rather than hardcoded. + if FIRMWARES.get(args.firmware, {}).get("moonbase"): + flash_cmd = _moonbase_flash_cmd(build_dir, args.firmware, args.port, baud, env) + else: + flash_cmd = cmd + b_arg + ["flash", "-p", args.port, "-b", str(baud)] + proc = subprocess.Popen(flash_cmd, cwd=ESP32_DIR, env=env, stdout=subprocess.PIPE, stderr=subprocess.STDOUT, text=True) for line in proc.stdout: diff --git a/moondeck/build/generate_manifest.py b/moondeck/build/generate_manifest.py index 2c7ae552..46e8a197 100644 --- a/moondeck/build/generate_manifest.py +++ b/moondeck/build/generate_manifest.py @@ -131,6 +131,25 @@ def main() -> int: print(f"generate_manifest: no recognised parts in {args.flasher_args}") return 1 + # MoonBase firmwares: flasher_args stages the app at the first app partition: the factory + # slot, MoonBase's home: because IDF knows nothing about the two-image scheme. Remap the app + # to ota_0, swap the blank ota-data for the slot-0 image (a blank one would boot MoonBase on + # first start), and add MoonBase at the factory slot. Offsets come from the same partition CSV + # the firmware builds with (moonbase_table_csv), so manifest and table cannot disagree. + # release.yml stages the two shared-moonbase assets these paths point at. + from build_esp32 import moonbase_table_csv, partition_offsets + spec = FIRMWARES[args.firmware] + if spec.get("moonbase"): + offs = partition_offsets(moonbase_table_csv(args.firmware)) + for part in parts: + if part["path"] == f"{prefix}.bin": + part["offset"] = int(offs["ota_0"], 16) + elif part["path"] == "shared-ota-data.bin": + part["path"] = "shared-ota-data-slot0.bin" + parts.append({"path": f"shared-moonbase-{spec['chip']}.bin", + "offset": int(offs["factory"], 16)}) + parts.sort(key=lambda p: p["offset"]) + # ESP Web Tools resolves the per-part `path` relative to the manifest URL, # so absolute URLs are the simplest robust shape โ€” the manifest stays # correct whether it's served from Pages, a release page, or a mirror. diff --git a/moondeck/check/check_esp32_built.py b/moondeck/check/check_esp32_built.py index a6afe87a..4961de6d 100644 --- a/moondeck/check/check_esp32_built.py +++ b/moondeck/check/check_esp32_built.py @@ -32,7 +32,7 @@ SOURCE_DIRS = ("src", "esp32") SOURCE_FILES = ("CMakeLists.txt", "library.json") SOURCE_SUFFIXES = {".c", ".cpp", ".h", ".hpp", ".cmake", ".json", ".txt", ".py", ".js", - ".html", ".css", ".defaults"} + ".html", ".css", ".defaults", ".csv"} # .csv: partition tables feed the image # Build outputs and caches live under the source dirs; they are products, not inputs, and # including them would compare the binary against itself. @@ -149,6 +149,29 @@ def main(): print(f" rebuild: {build_cmd}") return 1 + # MoonBase firmwares carry a second image (built into build/moonbase-/ by the same + # build run): it must exist and be newer than every moonbase/ source, by the same + # sources-not-clock rule as the app image. + import importlib + sys.path.insert(0, str(ROOT / "moondeck" / "build")) + FIRMWARES = importlib.import_module("build_esp32").FIRMWARES + spec = FIRMWARES.get(args.firmware, {}) + if spec.get("moonbase"): + mb_bin = ROOT / "build" / f"moonbase-{spec['chip']}" / "projectMM-moonbase.bin" + if not mb_bin.exists(): + print(f"No MoonBase image for {args.firmware}.") + print(f" expected: {mb_bin.relative_to(ROOT)}") + print(f" build it: {build_cmd}") + return 1 + mb_built = mb_bin.stat().st_mtime + mb_newest = max((f.stat().st_mtime for f in (ROOT / "moonbase").rglob("*") + if f.is_file() and f.suffix in SOURCE_SUFFIXES + and not SKIP_PARTS.intersection(f.parts)), default=0) + if mb_newest > mb_built: + print(f"MoonBase image for {args.firmware} is STALE: a moonbase/ source is newer.") + print(f" rebuild: {build_cmd}") + return 1 + if args.max_age_hours and age_h > args.max_age_hours: print(f"Firmware for {args.firmware} is older than {args.max_age_hours}h " f"({age_h:.1f}h) โ€” no source is newer, but the age rule asks for a rebuild.") diff --git a/moondeck/qemu/run_qemu.py b/moondeck/qemu/run_qemu.py index d764c35a..81e82905 100644 --- a/moondeck/qemu/run_qemu.py +++ b/moondeck/qemu/run_qemu.py @@ -64,8 +64,20 @@ def merged_flash(force: bool) -> str: idf_py = os.path.join(str(venv), "bin", "python") if venv else "" if not idf_py or not os.path.exists(idf_py): sys.exit("no ESP-IDF Python env found, source export.sh, or install the IDF tools") + # The qemu firmware carries MoonBase, and IDF's own flash_args stages the app at the factory + # offset (MoonBase's slot): the same correction every flasher applies. moonbase_flash_files + # is the one place that knows the corrected layout; the flat list it returns feeds merge_bin + # directly instead of @flash_args. + sys.path.insert(0, os.path.join(os.path.dirname(__file__), "..", "build")) + from build_esp32 import FIRMWARES, moonbase_flash_files + if FIRMWARES["qemu"].get("moonbase"): + from pathlib import Path + writes = [str(x) for off, path in moonbase_flash_files("qemu", Path(BUILD)) + for x in (off, path)] + else: + writes = [f"@{args}"] r = subprocess.run([idf_py, "-m", "esptool", "--chip", "esp32", "merge_bin", - "-o", out, "--fill-flash-size", "4MB", f"@{args}"], + "-o", out, "--fill-flash-size", "4MB"] + writes, cwd=BUILD, capture_output=True, text=True) if r.returncode != 0: sys.exit(f"merge_bin failed:\n{r.stderr[:800]}") diff --git a/moondeck/run/preview_installer.py b/moondeck/run/preview_installer.py index 74ef09e5..199f69dd 100644 --- a/moondeck/run/preview_installer.py +++ b/moondeck/run/preview_installer.py @@ -43,6 +43,11 @@ from pathlib import Path ROOT = Path(__file__).resolve().parent.parent.parent + +# build_esp32.py is the single source of truth for firmware variants (same import +# generate_manifest.py and collect_kpi.py use). +sys.path.insert(0, str(ROOT / "moondeck" / "build")) +from build_esp32 import FIRMWARES # noqa: E402 INSTALL_DIR = ROOT / "web-installer" ASSETS_BOARDS_DIR = ROOT / "docs" / "assets" / "boards" PICKER_JS = ROOT / "src" / "ui" / "install-picker.js" @@ -207,6 +212,14 @@ def stage_local_builds(builds: list[Path]) -> list[str]: releases_dir / f"partition-table-{size}.bin") shutil.copy(build_dir / "ota_data_initial.bin", releases_dir / "shared-ota-data.bin") + # MoonBase firmwares also ship the shared maintenance image + the slot-0 otadata + # their manifests reference (same names release.yml stages). + if FIRMWARES.get(firmware, {}).get("moonbase"): + from build_esp32 import otadata_slot0_bytes + chip = FIRMWARES[firmware]["chip"] + shutil.copy(build_dir.parent / f"moonbase-{chip}" / "projectMM-moonbase.bin", + releases_dir / f"shared-moonbase-{chip}.bin") + (releases_dir / "shared-ota-data-slot0.bin").write_bytes(otadata_slot0_bytes()) except FileNotFoundError as e: # Partial build (bootloader / partition-table missing) โ€” skip this # firmware rather than half-stage it, the picker would offer it diff --git a/src/core/FirmwareUpdateModule.h b/src/core/FirmwareUpdateModule.h index 4e31be9c..6f833dbb 100644 --- a/src/core/FirmwareUpdateModule.h +++ b/src/core/FirmwareUpdateModule.h @@ -93,8 +93,17 @@ inline bool otaInFlight() { /// mutually OTA-compatible (same chip, different feature flags), the legacy `esp32-eth-wifi` key /// strips to `esp32`, and `esp32s3-n16r8` is only itself. /// +/// **MoonBase devices** (4 MB tables; the read-only `moonbase` control marks one): the app cannot +/// flash itself: one app slot, and it is running from it. `POST /api/firmware/url` stages the URL +/// in NVS and reboots into the MoonBase factory image, which installs unattended and reboots back; +/// uploads re-POST from the browser once MoonBase answers. See architecture.md, MoonBase: +/// the second boot image. +/// /// **Prior art:** `esp_https_ota` is the standard ESP-IDF OTA-from-HTTP component used by every ESP32 -/// OTA flow since IDF v4.x; the install-picker UI is the new layer on top. +/// OTA flow since IDF v4.x; the install-picker UI is the new layer on top. The MoonBase scheme, +/// a minimal boot image in place of a second OTA slot: follows Tasmota's safeboot and Mathieu +/// Carbou's MycilaSafeBoot (https://github.com/mathieucarbou/MycilaSafeBoot), rewritten minimal +/// against ESP-IDF (moonbase/). /// @card FirmwareUpdateModule.png class FirmwareUpdateModule : public MoonModule { public: @@ -139,6 +148,11 @@ class FirmwareUpdateModule : public MoonModule { if (totalFlashVal_ > 0) { controls_.addProgress("firmwarePartition", firmwareSizeVal_, totalFlashVal_); } + // Present only on tables that carry a MoonBase factory image: its presence tells the UI to + // install through the reboot-into-MoonBase flow rather than in place (app.js reads it). + if (platform::otaHasMoonBase()) { + controls_.addReadOnly("moonbase", moonbaseStr_, sizeof(moonbaseStr_)); + } // OTA status goes through MoonModule::setStatus() (the per-module status // slot every module shares), not a bespoke read-only control โ€” same @@ -197,6 +211,7 @@ class FirmwareUpdateModule : public MoonModule { char firmwareStr_[24] = {}; ///< build variant name, such as "esp32s3-n16r8" uint32_t firmwareSizeVal_ = 0; ///< bytes used in the app partition uint32_t totalFlashVal_ = 0; ///< app partition size + char moonbaseStr_[8] = "standby"; ///< MoonBase present in the factory slot }; } // namespace mm diff --git a/src/core/HttpServerModule.cpp b/src/core/HttpServerModule.cpp index 660fb129..3bac2a57 100644 --- a/src/core/HttpServerModule.cpp +++ b/src/core/HttpServerModule.cpp @@ -364,6 +364,10 @@ void HttpServerModule::handleConnection(platform::TcpConnection& conn) { handleReboot(conn); } else if (std::strcmp(path, "/api/firmware/url") == 0 && body) { handleFirmwareUrl(conn, body); + } else if (std::strcmp(path, "/api/firmware/moonbase") == 0) { + // Reboot into MoonBase with nothing staged: the UI uses this for install-from-file, + // where the browser holds the image and re-POSTs it to MoonBase once it answers. + handleBootMoonBase(conn); } else if (std::strcmp(path, "/api/firmware/upload") == 0 && body) { // OTA from an uploaded .bin body (no URL, no host to serve it) โ€” the browser POSTs the // firmware image straight to the device, which streams it into the OTA partition. Same @@ -2273,6 +2277,22 @@ void HttpServerModule::handleListDeleteRow(platform::TcpConnection& conn, const sendResponse(conn, 200, "application/json", "{\"ok\":true}"); } +// Reboot into MoonBase. 409 when this table has none or MoonBase is already running: the UI +// only offers the button when the `moonbase` control exists, so a 409 here means a raw API +// caller on the wrong device, and an error is the honest answer. +void HttpServerModule::handleBootMoonBase(platform::TcpConnection& conn) { + if (!platform::otaHasMoonBase() || platform::otaRunningMoonBase()) { + sendResponse(conn, 409, "application/json", "{\"error\":\"no MoonBase on this device\"}"); + return; + } + FilesystemModule::flushPending(); + sendResponse(conn, 200, "application/json", "{\"ok\":true,\"moonbase\":true}"); + conn.close(); + platform::delayMs(200); + platform::otaBootMoonBase(); + platform::reboot(); // noreturn +} + void HttpServerModule::handleReboot(platform::TcpConnection& conn) { FilesystemModule::flushPending(); sendResponse(conn, 200, "application/json", "{\"ok\":true}"); @@ -2318,6 +2338,34 @@ void HttpServerModule::handleFirmwareUrl(platform::TcpConnection& conn, const ch return; } + // A MoonBase device cannot update in place: it has one app slot, and it is running from it. + // Stage the URL in NVS and reboot into MoonBase, which installs it unattended and reboots + // back: the UI shows one "updating firmware" experience over the whole cycle. 202 with + // {"moonbase":true} tells the caller which of the two flows it got. + if (platform::otaHasMoonBase() && !platform::otaRunningMoonBase()) { + // The staged URL crosses into MoonBase through a 256-byte NVS read + // (moonbase_main.cpp loadCredentials' sibling); a longer URL would fail that read + // silently and park the device in MoonBase with nothing to show for it. Reject it + // here, where the caller can still see why. + if (std::strlen(url) > 255) { + sendResponse(conn, 400, "application/json", + "{\"error\":\"url too long for the MoonBase handoff (max 255)\"}"); + return; + } + if (!platform::moonbaseStageInstallUrl(url)) { + sendResponse(conn, 500, "application/json", + "{\"error\":\"could not stage install url\"}"); + return; + } + std::snprintf(g_otaStatus, sizeof(g_otaStatus), "rebooting"); + FilesystemModule::flushPending(); + sendResponse(conn, 202, "application/json", "{\"ok\":true,\"moonbase\":true}"); + conn.close(); + platform::delayMs(200); + platform::otaBootMoonBase(); + platform::reboot(); // noreturn, boots MoonBase, which installs and reboots back + } + // Seed the shared globals so the first WS push after this response shows // "starting" instead of whatever the previous OTA left behind (e.g. an // "error: โ€ฆ" string from a prior failed attempt). diff --git a/src/core/HttpServerModule.h b/src/core/HttpServerModule.h index b0dfcafd..b864c22e 100644 --- a/src/core/HttpServerModule.h +++ b/src/core/HttpServerModule.h @@ -516,9 +516,11 @@ class HttpServerModule : public MoonModule, public BinaryBroadcaster { MoonModule* listMutationModule_ = nullptr; // module whose list a CRUD op resolved to (for markDirty) void afterListMutation(); void handleReboot(platform::TcpConnection& conn); - /// OTA: `POST /api/firmware/url` body=`{"url":"..."}`. Body parsed; URL handed - /// to platform::http_fetch_to_ota which spawns a task and returns. Caller - /// gets 202 immediately; progress streams via FirmwareUpdateModule controls. + void handleBootMoonBase(platform::TcpConnection& conn); + /// OTA: `POST /api/firmware/url` body=`{"url":"..."}`. On a MoonBase device the URL is + /// staged in NVS and the device reboots into MoonBase, which installs it unattended + /// (202 + {"moonbase":true}). Otherwise the URL goes to platform::http_fetch_to_ota, + /// which spawns a task and returns: 202 immediately, progress via FirmwareUpdateModule. void handleFirmwareUrl(platform::TcpConnection& conn, const char* body); void handleFirmwareUpload(platform::TcpConnection& conn, const char* initialBody, size_t initialLen, size_t contentLen); // POST /api/firmware/upload diff --git a/src/platform/desktop/platform_desktop.cpp b/src/platform/desktop/platform_desktop.cpp index 884d7f1b..6eef2094 100644 --- a/src/platform/desktop/platform_desktop.cpp +++ b/src/platform/desktop/platform_desktop.cpp @@ -1341,6 +1341,12 @@ bool otaWriteStream(FsWriteSrc /*src*/, void* /*user*/, size_t /*contentLen*/, return false; } +// No partitions on desktop: there is no recovery image and nothing to boot into. +bool otaHasMoonBase() { return false; } +bool otaBootMoonBase() { return false; } +bool otaRunningMoonBase() { return false; } +bool moonbaseStageInstallUrl(const char*) { return false; } + // Outbound HTTP request (plain HTTP, LAN, no TLS) โ€” see platform.h. Blocking, bounded by a // receive/send timeout. Builds the request into a stack buffer, connects, sends, reads the // response, and returns the status code + the body (after the \r\n\r\n). Used by HueDriver diff --git a/src/platform/esp32/platform_esp32_fs.cpp b/src/platform/esp32/platform_esp32_fs.cpp index 24ee907f..7243ce5e 100644 --- a/src/platform/esp32/platform_esp32_fs.cpp +++ b/src/platform/esp32/platform_esp32_fs.cpp @@ -1,7 +1,7 @@ // LittleFS partition mount + the `fs*` / `filesystem*` API on ESP32. // // Cut out of platform_esp32.cpp (plan-23) for size + readability. The -// file owns its private state (FS_TAG, FS_PARTITION_LABEL, FS_MOUNT_POINT, +// file owns its private state (FS_TAG, FS_LABELS, FS_MOUNT_POINT, // fsMounted_) and the path-translation helper; the rest of the platform // layer talks to it only through the public mm::platform::fs* and // filesystem* symbols declared in platform.h. Move was a code-organisation @@ -10,6 +10,7 @@ #include "platform/platform.h" #include "esp_littlefs.h" +#include "esp_partition.h" // esp_partition_find_first: probing which FS label this table uses #include "esp_log.h" #include @@ -23,7 +24,16 @@ namespace mm::platform { // LittleFS state static constexpr const char* FS_TAG = "mm_fs"; -static constexpr const char* FS_PARTITION_LABEL = "spiffs"; // partition label kept for tooling compat; contents are LittleFS +// The volume has always held LittleFS. Tables written from 2026-08 say so properly: partition +// `littlefs` with the littlefs subtype (0x83). Older tables (and the 8/16 MB ones until they +// migrate) call the same volume `spiffs` with the spiffs subtype, a legacy misnomer. Both are +// tried in order, so a device that keeps its old table across an OTA still finds its config. +struct FsCandidate { esp_partition_subtype_t subtype; const char* label; }; +static constexpr FsCandidate FS_CANDIDATES[] = { + {ESP_PARTITION_SUBTYPE_DATA_LITTLEFS, "littlefs"}, + {ESP_PARTITION_SUBTYPE_DATA_SPIFFS, "spiffs"}, +}; +static const char* fsLabelInUse_ = nullptr; static constexpr const char* FS_MOUNT_POINT = "/littlefs"; // VFS mount point; not exposed in API paths static bool fsMounted_ = false; @@ -49,25 +59,42 @@ const char* fsRootPath() { return FS_MOUNT_POINT; } bool fsMount() { if (fsMounted_) return true; - esp_vfs_littlefs_conf_t conf = {}; - conf.base_path = FS_MOUNT_POINT; - conf.partition_label = FS_PARTITION_LABEL; - conf.format_if_mount_failed = true; - conf.dont_mount = false; - - esp_err_t err = esp_vfs_littlefs_register(&conf); + // Try each known label. format_if_mount_failed is OFF for every attempt but the last that + // actually has a partition: a formatting mount against the wrong label would erase a volume + // that the next label would have opened, taking the user's config with it. "Last" must mean + // last EXISTING, not last in the array: a MoonBase table carries only `littlefs`, and + // gating the format on the absent `spiffs` entry left a fresh board unformatted, so it ran + // with persistence disabled. + esp_err_t err = ESP_FAIL; + constexpr size_t kCandidates = sizeof(FS_CANDIDATES) / sizeof(FS_CANDIDATES[0]); + size_t lastPresent = kCandidates; + for (size_t i = 0; i < kCandidates; i++) { + const auto& c = FS_CANDIDATES[i]; + if (esp_partition_find_first(ESP_PARTITION_TYPE_DATA, c.subtype, c.label)) lastPresent = i; + } + for (size_t i = 0; i < kCandidates; i++) { + const auto& c = FS_CANDIDATES[i]; + if (!esp_partition_find_first(ESP_PARTITION_TYPE_DATA, c.subtype, c.label)) continue; + esp_vfs_littlefs_conf_t conf = {}; + conf.base_path = FS_MOUNT_POINT; + conf.partition_label = c.label; + conf.format_if_mount_failed = (i == lastPresent); + conf.dont_mount = false; + err = esp_vfs_littlefs_register(&conf); + if (err == ESP_OK) { fsLabelInUse_ = c.label; break; } + } if (err != ESP_OK) { ESP_LOGE(FS_TAG, "LittleFS mount failed: %s", esp_err_to_name(err)); return false; } fsMounted_ = true; - ESP_LOGI(FS_TAG, "LittleFS mounted at %s (partition: %s)", FS_MOUNT_POINT, FS_PARTITION_LABEL); + ESP_LOGI(FS_TAG, "LittleFS mounted at %s (partition: %s)", FS_MOUNT_POINT, fsLabelInUse_); return true; } void fsUnmount() { if (!fsMounted_) return; - esp_vfs_littlefs_unregister(FS_PARTITION_LABEL); + if (fsLabelInUse_) esp_vfs_littlefs_unregister(fsLabelInUse_); fsMounted_ = false; } @@ -225,14 +252,15 @@ void fsList(const char* dir, FsListCb cb, void* user) { size_t filesystemUsed() { if (!fsMounted_) return 0; size_t total = 0, used = 0; - if (esp_littlefs_info(FS_PARTITION_LABEL, &total, &used) != ESP_OK) return 0; + if (!fsLabelInUse_) return 0; + if (esp_littlefs_info(fsLabelInUse_, &total, &used) != ESP_OK) return 0; return used; } size_t filesystemTotal() { if (!fsMounted_) return 0; size_t total = 0, used = 0; - if (esp_littlefs_info(FS_PARTITION_LABEL, &total, &used) != ESP_OK) return 0; + if (esp_littlefs_info(fsLabelInUse_, &total, &used) != ESP_OK) return 0; return total; } diff --git a/src/platform/esp32/platform_esp32_ota.cpp b/src/platform/esp32/platform_esp32_ota.cpp index 8dc7c25d..5bca8f18 100644 --- a/src/platform/esp32/platform_esp32_ota.cpp +++ b/src/platform/esp32/platform_esp32_ota.cpp @@ -10,11 +10,13 @@ #include "esp_https_ota.h" #include "esp_ota_ops.h" +#include "esp_partition.h" // esp_partition_find_first: locating MoonBase #include "esp_crt_bundle.h" #include "esp_http_client.h" #include "esp_system.h" #include "esp_heap_caps.h" // heap_caps_malloc/free โ€” the upload chunk buffer #include "esp_log.h" +#include "nvs.h" // moonbaseStageInstallUrl: the URL handoff to MoonBase #include "freertos/FreeRTOS.h" #include "freertos/task.h" @@ -193,6 +195,7 @@ bool http_fetch_to_ota(const char* url, return true; } + bool otaWriteStream(FsWriteSrc src, void* user, size_t contentLen, char* statusBuf, size_t statusBufLen, uint32_t* bytesReadOut) { if (!src || !statusBuf || statusBufLen == 0 || !bytesReadOut) return false; @@ -203,6 +206,24 @@ bool otaWriteStream(FsWriteSrc src, void* user, size_t contentLen, const esp_partition_t* part = esp_ota_get_next_update_partition(nullptr); if (!part) { setStatus("error: no OTA partition"); return false; } + // SINGLE-SLOT GUARD. esp_ota_get_next_update_partition iterates only OTA subtypes and falls + // back to the FIRST ota slot it finds, so on a table with one ota_0 (the MoonBase layout) it + // hands back the partition we are executing from. Erasing that is a brick mid-flash. IDF also + // refuses it inside esp_ota_begin (ESP_ERR_OTA_PARTITION_CONFLICT), but failing here says WHY + // and names the fix. On a dual-OTA table this never fires. + if (part == esp_ota_get_running_partition()) { + setStatus("error: one app slot, reboot to MoonBase first"); + return false; + } + // SIZE GUARD. Without it an oversized image fails partway through esp_ota_write, leaving the + // target slot half-written and the user staring at a generic write error. Content-Length is + // advisory, so this only fires when the caller knows the size. + if (contentLen && contentLen > part->size) { + setStatus("error: image too large (%u > %u)", + static_cast(contentLen), static_cast(part->size)); + return false; + } + setStatus("flashing"); esp_ota_handle_t handle = 0; // OTA_SIZE_UNKNOWN: the upload streams, so we don't pre-declare the exact size (Content-Length @@ -274,4 +295,40 @@ bool otaWriteStream(FsWriteSrc src, void* user, size_t contentLen, return true; } +// Does this device's partition table carry a factory app? True on the MoonBase layout used by +// the 4 MB boards, false on the dual-OTA tables. The caller uses it to decide whether an update +// needs the reboot-into-MoonBase hop, and the UI to say which kind of device this is. +bool otaHasMoonBase() { + return esp_partition_find_first(ESP_PARTITION_TYPE_APP, + ESP_PARTITION_SUBTYPE_APP_FACTORY, nullptr) != nullptr; +} + +// Point the bootloader at MoonBase and report whether it took. Returns false when the table has +// no factory partition, which tells the caller this device updates in place. +// NB esp_ota_set_boot_partition on a factory partition ERASES otadata rather than writing a +// sequence number: that is what makes a power cut mid-update land back in MoonBase rather than in +// a half-written app. +bool otaBootMoonBase() { + const esp_partition_t* part = esp_partition_find_first( + ESP_PARTITION_TYPE_APP, ESP_PARTITION_SUBTYPE_APP_FACTORY, nullptr); + if (!part) return false; + return esp_ota_set_boot_partition(part) == ESP_OK; +} + +// Is the device currently executing FROM MoonBase? +bool otaRunningMoonBase() { + const esp_partition_t* run = esp_ota_get_running_partition(); + return run && run->subtype == ESP_PARTITION_SUBTYPE_APP_FACTORY; +} + +// Stage the install URL in NVS for MoonBase to consume on its next boot (see platform.h). +bool moonbaseStageInstallUrl(const char* url) { + if (!url || !url[0]) return false; + nvs_handle_t h; + if (nvs_open("moonbase", NVS_READWRITE, &h) != ESP_OK) return false; + const bool ok = nvs_set_str(h, "url", url) == ESP_OK && nvs_commit(h) == ESP_OK; + nvs_close(h); + return ok; +} + } // namespace mm::platform diff --git a/src/platform/platform.h b/src/platform/platform.h index 1de5deaf..637a06f7 100644 --- a/src/platform/platform.h +++ b/src/platform/platform.h @@ -662,6 +662,22 @@ bool http_fetch_to_ota(const char* url, bool otaWriteStream(FsWriteSrc src, void* user, size_t contentLen, char* statusBuf, size_t statusBufLen, uint32_t* bytesReadOut); +// MOONBASE, the second boot image, present only on tables that carry a factory app (the 4 MB +// boards): a small, rarely changing firmware that owns the device when the application is not +// running or cannot be trusted. Its first job is installing firmware into the one large app slot, +// since a board cannot rewrite the partition it is executing from, so an update there is two +// stages: otaBootMoonBase() + reboot, then install from MoonBase. +// All of these are false / no-ops on a dual-OTA table and on desktop. +bool otaHasMoonBase(); // does this table carry MoonBase? +bool otaBootMoonBase(); // point the bootloader at it; false when there is none +bool otaRunningMoonBase(); // are we executing from it right now? +// Stage an install URL for MoonBase to pick up on its next boot (NVS namespace "moonbase", +// key "url", at most 255 bytes: MoonBase reads it into a 256-byte buffer and the HTTP route +// rejects anything longer). This is what makes a URL install unattended: the app stages the URL, reboots into +// MoonBase, and MoonBase installs it with no browser in the loop. MoonBase erases the key before +// attempting the install, so a bad URL cannot boot-loop the device. +bool moonbaseStageInstallUrl(const char* url); + // Synchronous outbound HTTP request to a LAN host โ€” plain HTTP, no TLS (the Philips Hue v1 // API, which HueDriver drives, allows it). Connects to `host:port`, sends `method path` // with `reqBody` (NUL-terminated; "" for none โ€” a Content-Length + JSON content-type are diff --git a/src/ui/app.js b/src/ui/app.js index 7dee1708..4353fac5 100644 --- a/src/ui/app.js +++ b/src/ui/app.js @@ -1114,6 +1114,125 @@ function renderChildTabs(mod, childrenEl, depth) { if (child) renderModuleTree(child, panel, depth + 1); } +// ---- MoonBase update flow ---- +// On a MoonBase device (FirmwareUpdate exposes a `moonbase` control) the app cannot flash itself: +// one app slot, and it is running from it. Instead it reboots into the MoonBase factory image, +// which installs into the app slot and reboots back: same IP throughout (same MAC, same DHCP +// lease). The whole cycle runs behind one full-screen overlay, shown BEFORE the reboot so there +// is no dead gap. GET /moonbase is the identity probe: MoonBase answers it with its live install +// status; the app 404s it: so "404 again after an install ran" means the new firmware is up. +function deviceHasMoonBase(mod) { + return (mod.controls || []).some(c => c.name === "moonbase"); +} + +function showUpdateOverlay() { + const ov = document.createElement("div"); + ov.className = "fw-overlay"; + const box = document.createElement("div"); + box.className = "fw-overlay-box"; + const h = document.createElement("h2"); + h.textContent = "Updating firmware\u2026"; + const msg = document.createElement("p"); + msg.className = "fw-overlay-msg"; + const bar = document.createElement("progress"); // no value = indeterminate sweep + const dismiss = document.createElement("button"); + dismiss.textContent = "close"; + dismiss.style.display = "none"; + dismiss.addEventListener("click", () => ov.remove()); + box.append(h, msg, bar, dismiss); + ov.appendChild(box); + document.body.appendChild(ov); + return { + status(text) { msg.textContent = text; }, + fail(text) { msg.textContent = text; bar.remove(); dismiss.style.display = ""; }, + }; +} + +const fwSleep = (ms) => new Promise((r) => setTimeout(r, ms)); + +// Probe GET /moonbase once: "up" (200 + status text), "app" (404: the application is answering), +// or "silent" (no answer: the device is mid-reboot). Never throws. +async function probeMoonBase() { + try { + const r = await fetch("/moonbase", { cache: "no-store" }); + if (r.ok) return { state: "up", text: await r.text() }; + return { state: "app" }; + } catch (_) { + return { state: "silent" }; + } +} + +// The one-click cycle. opts is {url} (device installs it unattended off the NVS-staged URL) or +// {file} (the browser holds the image and pushes it to MoonBase once MoonBase answers). +const MOONBASE_SILENT_MSG = "MoonBase did not answer. If its WiFi fell back, join the " + + "MoonBase access point and open http://4.3.2.1"; + +// Poll until MoonBase serves (a 200 on the probe). False on deadline: the device never came up, +// or came up unreachable (AP fallback). +async function waitForMoonBase(deadline) { + while (Date.now() < deadline) { + await fwSleep(2000); + if ((await probeMoonBase()).state === "up") return true; + } + return false; +} + +async function moonbaseUpdateFlow(opts) { + const ui = showUpdateOverlay(); // opens before the reboot: no dead gap + try { + ui.status("Restarting into MoonBase\u2026"); + // The kickoff response can be cut off by the reboot; a dead socket here is not a failure. + let res = null; + try { + res = opts.url + ? await fetch("/api/firmware/url", { + method: "POST", headers: { "Content-Type": "application/json" }, + body: JSON.stringify({ url: opts.url }) }) + : await fetch("/api/firmware/moonbase", { method: "POST" }); + } catch (_) {} + if (res && !res.ok && res.status !== 202) throw new Error(await errorMessage(res)); + + if (opts.file) { + // The upload needs MoonBase serving before the browser can push the image. + if (!(await waitForMoonBase(Date.now() + 120000))) throw new Error(MOONBASE_SILENT_MSG); + ui.status(`Installing ${fmSize(opts.file.size)}\u2026`); + const r = await fetch("/install", { + method: "POST", headers: { "Content-Type": "application/octet-stream" }, + body: opts.file }); + if (!r.ok) throw new Error(await r.text()); + } + + // Watch until the app answers again. On the unattended URL path MoonBase installs BEFORE + // it ever serves, so a successful cycle is silence followed by the new app's 404 on the + // probe; MoonBase answering (200) is the upload path's install window, and on the URL + // path its failure parking spot (the probe body is its status). The dying OLD app also + // 404s the probe for a moment, so "app" only counts as done once the device has been + // seen away, or after a grace period long enough that the kickoff reboot must have run. + const watchStart = Date.now(); + const deadline = watchStart + 300000; + let sawAway = false; + while (Date.now() < deadline) { + await fwSleep(2000); + const probe = await probeMoonBase(); + if (probe.state === "silent") { + sawAway = true; + } else if (probe.state === "up") { + sawAway = true; // MoonBase answering means the old app is gone + if (probe.text.startsWith("error")) throw new Error(probe.text); + ui.status(probe.text || "Installing\u2026"); + } else if (probe.state === "app" && (sawAway || Date.now() - watchStart > 20000)) { + ui.status("Done \u2014 reloading\u2026"); + await fwSleep(1000); + location.reload(); + return; + } + } + throw new Error("timed out waiting for the new firmware"); + } catch (err) { + ui.fail("Update failed: " + err.message); + } +} + function createCard(mod, depth) { const card = document.createElement("div"); card.className = "card"; @@ -1358,6 +1477,11 @@ function createCard(mod, depth) { a.remove(); return; } + if (deviceHasMoonBase(mod)) { + // MoonBase device: the whole install runs unattended behind the overlay. + moonbaseUpdateFlow({ url: binaryUrl }); + return; + } const res = await fetch("/api/firmware/url", { method: "POST", headers: { "Content-Type": "application/json" }, @@ -1388,6 +1512,11 @@ function createCard(mod, depth) { const file = (upInput.files || [])[0]; upInput.value = ""; // reset so re-picking the same file re-fires change if (!file) return; + if (deviceHasMoonBase(mod)) { + // MoonBase device: reboot into MoonBase, then the browser pushes this file to it. + moonbaseUpdateFlow({ file }); + return; + } upBtn.disabled = true; upStatus.textContent = `uploading ${fmSize(file.size)}โ€ฆ`; try { @@ -1410,6 +1539,23 @@ function createCard(mod, depth) { fileRow.appendChild(upBtn); fileRow.appendChild(upStatus); fileRow.appendChild(upInput); + // MoonBase devices: open the maintenance image directly: reboot into MoonBase and land + // on its page (same address, so a reload gets there once it answers). The way back is + // MoonBase's own "Boot the app" button. + if (deviceHasMoonBase(mod)) { + const mbBtn = document.createElement("button"); + mbBtn.className = "fm-tool"; + mbBtn.textContent = "MoonBase"; + mbBtn.title = "Reboot into MoonBase, the maintenance image, install firmware, then boot back"; + mbBtn.addEventListener("click", async () => { + const ui = showUpdateOverlay(); + ui.status("Restarting into MoonBase\u2026"); + try { await fetch("/api/firmware/moonbase", { method: "POST" }); } catch (_) {} + if (await waitForMoonBase(Date.now() + 120000)) { location.reload(); return; } + ui.fail(MOONBASE_SILENT_MSG); + }); + fileRow.appendChild(mbBtn); + } controlsHost.appendChild(fileRow); } diff --git a/src/ui/install-picker.js b/src/ui/install-picker.js index 1445ee9c..f1324211 100644 --- a/src/ui/install-picker.js +++ b/src/ui/install-picker.js @@ -223,10 +223,12 @@ function parseFirmwaresFromAssets(assets, tag) { } for (const a of assets) { - // Reject the part-suffixed .bins (bootloader / partition-table / ota-data) - // โ€” they're install fragments, not the main image. The OTA path needs the - // app image only; esp_https_ota internally fetches what it needs. - if (/-(?:bootloader|partition-table|ota-data)\.bin$/.test(a.name)) continue; + // Reject the part-suffixed .bins (bootloader / partition-table / ota-data / + // moonbase): they're install fragments, not the main image. The OTA path needs + // the app image only. The shared-moonbase asset is doubly excluded (the `shared-` + // prefix already fails binaryRe): offering MoonBase as an OTA target would replace + // a device's app with an image that can only install, not run the show. + if (/(?:-(?:bootloader|partition-table|ota-data)|moonbase[^/]*|-slot0)\.bin$/.test(a.name)) continue; const m = binaryRe.exec(a.name); if (m) { const firmware = m[1]; diff --git a/src/ui/style.css b/src/ui/style.css index a06df8f9..18596e49 100644 --- a/src/ui/style.css +++ b/src/ui/style.css @@ -1786,3 +1786,28 @@ body.cards-resizing { inset 0 1px 0 color-mix(in srgb, #fff 40%, transparent), 0 0 12px color-mix(in srgb, var(--pad-tint) 55%, transparent); } + +/* Full-screen firmware-update overlay (MoonBase cycle): opened before the device reboots so the + user sees one continuous "updating" experience across reboot -> install -> reboot. */ +.fw-overlay { + position: fixed; + inset: 0; + z-index: 1000; + display: flex; + align-items: center; + justify-content: center; + background: rgba(10, 10, 20, 0.85); +} +.fw-overlay-box { + background: var(--card-bg-1); + border: 1px solid var(--border); + border-radius: 8px; + padding: 2rem 2.5rem; + max-width: 26rem; + text-align: center; + display: flex; + flex-direction: column; + gap: 1rem; +} +.fw-overlay-box progress { width: 100%; } +.fw-overlay-msg { color: var(--fg-muted); margin: 0; overflow-wrap: anywhere; } diff --git a/test/CMakeLists.txt b/test/CMakeLists.txt index 07857f21..2e85878e 100644 --- a/test/CMakeLists.txt +++ b/test/CMakeLists.txt @@ -3,6 +3,7 @@ # cross-domain references travel through the @also list, not the directory. add_executable(mm_tests # core/ + unit/core/unit_PartitionTables.cpp unit/core/unit_Buffer.cpp unit/core/unit_ScratchBuffer.cpp unit/core/unit_ActiveInstance.cpp @@ -54,6 +55,7 @@ add_executable(mm_tests unit/core/unit_MoonModule_lifecycle.cpp unit/core/unit_MoonModule_movechild.cpp unit/core/unit_MoonModule_replacechild.cpp + unit/core/unit_MoonBaseContract.cpp unit/core/unit_NetworkModule.cpp unit/core/unit_NetworkModule_ethernet.cpp unit/core/unit_Sort.cpp diff --git a/test/js/installer-firmware-merge.test.mjs b/test/js/installer-firmware-merge.test.mjs index cd8fbecf..1fd71d95 100644 --- a/test/js/installer-firmware-merge.test.mjs +++ b/test/js/installer-firmware-merge.test.mjs @@ -91,3 +91,19 @@ test("mergeFirmwares: no extras returns the published list unchanged (production assert.strictEqual(mergeFirmwares(published, null), published); assert.strictEqual(mergeFirmwares(published, []), published); }); + +test("parseFirmwaresFromAssets: MoonBase assets are never offered as an OTA image", () => { + // shared-moonbase-.bin is the maintenance image a MoonBase manifest references, and + // shared-ota-data-slot0.bin its boot-slot record. Offering either as a device's OTA target + // would replace the app with an image that can only install, not run the show. + const assets = [ + asset("manifest-esp32.json"), + asset("firmware-esp32-v2.0.0.bin"), + asset("shared-moonbase-esp32.bin"), + asset("shared-ota-data-slot0.bin"), + ]; + const fw = parseFirmwaresFromAssets(assets, "v2.0.0"); + assert.deepEqual(fw.map((f) => f.firmware), ["esp32"]); + assert.match(fw[0].binaryUrl, /firmware-esp32-v2\.0\.0\.bin$/, + "the app image stays the binary, not a MoonBase asset"); +}); diff --git a/test/python/test_installer_manifests.py b/test/python/test_installer_manifests.py index d6673017..6ca45a15 100644 --- a/test/python/test_installer_manifests.py +++ b/test/python/test_installer_manifests.py @@ -34,7 +34,10 @@ # The exact globs the deploy's `gh release download` stages onto Pages (release.yml, # "Stage cumulative release content"). A manifest part must match one of these, or the # installer fetches a file that isn't there. Keep in lockstep with that step's --pattern list. -STAGED_GLOBS = ["firmware-*.bin", "shared-ota-data.bin", "partition-table-*.bin"] +STAGED_GLOBS = ["firmware-*.bin", "shared-*.bin", "partition-table-*.bin"] +# shared-*.bin covers shared-ota-data.bin plus the MoonBase pair (shared-moonbase-.bin, +# shared-ota-data-slot0.bin); the same glob appears in release.yml's upload files and the +# Pages self-host download patterns, which is exactly the sync this test guards. # A minimal but realistic IDF flasher_args.json โ€” the four files every projectMM build emits. # generate_manifest.py maps these to the bundle names (firmware--v.bin, etc.). diff --git a/test/scenarios/light/scenario_peripheral_grid_sweep.json b/test/scenarios/light/scenario_peripheral_grid_sweep.json index 34f4e00f..2dee5a63 100644 --- a/test/scenarios/light/scenario_peripheral_grid_sweep.json +++ b/test/scenarios/light/scenario_peripheral_grid_sweep.json @@ -252,7 +252,7 @@ "desktop-macos": { "tick_us": [ 17, - 128 + 254 ], "free_heap": [ 0, @@ -264,7 +264,7 @@ ], "at": [ "2026-07-26", - "2026-07-31" + "2026-08-26" ] }, "desktop-windows": { diff --git a/test/unit/core/unit_MoonBaseContract.cpp b/test/unit/core/unit_MoonBaseContract.cpp new file mode 100644 index 00000000..e7bb5e9c --- /dev/null +++ b/test/unit/core/unit_MoonBaseContract.cpp @@ -0,0 +1,63 @@ +// @module NetworkModule +// @also FilesystemModule + +// MoonBase reads the WiFi credentials with a bounded 1024-byte prefix read of +// /.config/NetworkModule.json (moonbase/main/moonbase_main.cpp loadCredentials): a tiny image +// has no JSON parser and no room for the whole file, which also carries every child module's +// config. That bound is a cross-image contract with NetworkModule's control registration order, +// and nothing else pins it: a control added ABOVE ssid/password would push them out of the +// prefix and silently break MoonBase's network join on every deployed 4 MB device. This test is +// the pin. + +#include "doctest.h" +#include "core/FilesystemModule.h" +#include "core/NetworkModule.h" +#include "core/Scheduler.h" +#include "platform/platform.h" + +#include +#include +#include +#include + +// The credentials a provisioned device saves sit inside the first kilobyte of NetworkModule.json, where MoonBase's bounded prefix read finds them. +TEST_CASE("NetworkModule.json carries ssid and password within MoonBase's 1024-byte prefix read") { + char tmpRoot[256]; + std::snprintf(tmpRoot, sizeof(tmpRoot), "/tmp/mm_moonbase_contract_%u", + static_cast(mm::platform::millis())); + std::filesystem::remove_all(tmpRoot); + std::filesystem::create_directories(std::string(tmpRoot) + "/.config"); + mm::platform::fsSetRoot(tmpRoot); + + mm::Scheduler scheduler; + auto* fs = new mm::FilesystemModule(); + fs->setTypeName("FilesystemModule"); + fs->setScheduler(&scheduler); + auto* net = new mm::NetworkModule(); + net->setTypeName("NetworkModule"); + scheduler.addModule(fs); + scheduler.addModule(net); + scheduler.setup(); + + net->setWifiCredentials("bench-ssid", "bench-password"); + net->markDirty(); + fs->flush(); + + std::ifstream f(std::string(tmpRoot) + "/.config/NetworkModule.json"); + REQUIRE(f.good()); + std::string content((std::istreambuf_iterator(f)), std::istreambuf_iterator()); + f.close(); + + const auto ssidEnd = content.find("\"ssid\":\"bench-ssid\""); + const auto pwKey = content.find("\"password\":"); + REQUIRE(ssidEnd != std::string::npos); + REQUIRE(pwKey != std::string::npos); + // The whole password VALUE must fit too: key position + key + a worst-case 64-char + // passphrase escaped to at most twice its length stays under the bound. + CHECK(ssidEnd < 1024); + CHECK(pwKey + 12 + 2 * 64 + 2 < 1024); + + scheduler.release(); + std::filesystem::remove_all(tmpRoot); + mm::platform::fsSetRoot("."); +} diff --git a/test/unit/core/unit_PartitionTables.cpp b/test/unit/core/unit_PartitionTables.cpp new file mode 100644 index 00000000..d5b44a63 --- /dev/null +++ b/test/unit/core/unit_PartitionTables.cpp @@ -0,0 +1,166 @@ +// Every shipped partition table has to be internally consistent. These are hand-edited CSVs whose +// offsets are absolute and whose mistakes are invisible until a board fails to boot or an OTA +// silently truncates, so the arithmetic is pinned here rather than discovered on hardware. +// Rules come from the ESP-IDF partition-table format: partitions may not overlap, must sit inside +// the flash the table is written for, must not start before the table itself ends, and an APP +// partition must be 64 KB aligned (the MMU maps app code in 64 KB pages). + +#include "doctest.h" + +#include +#include +#include +#include +#include +#include +#include + +namespace { + +struct Partition { + std::string name, type, subtype; + uint32_t offset = 0, size = 0; + uint32_t end() const { return offset + size; } +}; + +struct Table { + std::string file; + uint32_t flashBytes = 0; // inferred from the largest end offset, rounded up to a power of two + std::vector parts; +}; + +std::filesystem::path partitionDir() { + // test/unit/core/ -> repo root -> esp32/partitions + return std::filesystem::path(__FILE__).parent_path().parent_path().parent_path().parent_path() + / "esp32" / "partitions"; +} + +// The CSV allows sizes and offsets as hex (0x...) or as a decimal count with a K/M suffix. +bool parseNumber(std::string tok, uint32_t& out) { + while (!tok.empty() && (tok.back() == ' ' || tok.back() == '\t')) tok.pop_back(); + while (!tok.empty() && (tok.front() == ' ' || tok.front() == '\t')) tok.erase(tok.begin()); + if (tok.empty()) return false; + char suffix = tok.back(); + uint32_t mult = 1; + if (suffix == 'K' || suffix == 'k') { mult = 1024; tok.pop_back(); } + else if (suffix == 'M' || suffix == 'm') { mult = 1024 * 1024; tok.pop_back(); } + if (tok.empty()) return false; + errno = 0; + char* end = nullptr; + const unsigned long v = std::strtoul(tok.c_str(), &end, 0); // base 0: 0x.. is hex + if (errno != 0 || end == tok.c_str() || *end != '\0') return false; + out = static_cast(v) * mult; + return true; +} + +std::string trim(const std::string& s) { + size_t b = s.find_first_not_of(" \t\r\n"); + if (b == std::string::npos) return ""; + size_t e = s.find_last_not_of(" \t\r\n"); + return s.substr(b, e - b + 1); +} + +Table readTable(const std::filesystem::path& path) { + Table t; + t.file = path.filename().string(); + std::ifstream in(path); + REQUIRE_MESSAGE(in.good(), "cannot open ", path.string()); + std::string line; + while (std::getline(in, line)) { + const std::string s = trim(line); + if (s.empty() || s[0] == '#') continue; + std::vector f; + size_t start = 0; + while (true) { + const size_t comma = s.find(',', start); + f.push_back(trim(s.substr(start, comma == std::string::npos ? comma : comma - start))); + if (comma == std::string::npos) break; + start = comma + 1; + } + if (f.size() < 5) continue; + Partition p; + p.name = f[0]; p.type = f[1]; p.subtype = f[2]; + REQUIRE_MESSAGE(parseNumber(f[3], p.offset), t.file, ": bad offset for ", p.name); + REQUIRE_MESSAGE(parseNumber(f[4], p.size), t.file, ": bad size for ", p.name); + t.parts.push_back(p); + } + uint32_t maxEnd = 0; + for (const auto& p : t.parts) maxEnd = p.end() > maxEnd ? p.end() : maxEnd; + uint32_t flash = 1024u * 1024u; + while (flash < maxEnd) flash <<= 1; + t.flashBytes = flash; + return t; +} + +std::vector allTables() { + std::vector
out; + for (const auto& e : std::filesystem::directory_iterator(partitionDir())) + if (e.path().extension() == ".csv") out.push_back(readTable(e.path())); + return out; +} + +} // namespace + +TEST_CASE("every partition table describes a layout that fits its flash without overlaps") { + const auto tables = allTables(); + REQUIRE(tables.size() >= 3); // esp32dev, esp32s3_n8r8, ota_16mb at time of writing + + for (const auto& t : tables) { + CAPTURE(t.file); + REQUIRE(t.parts.size() >= 3); + + // The bootloader lives below 0x8000 and the partition table itself at 0x8000, so no + // partition may start before 0x9000 (the first usable offset ESP-IDF documents). + for (const auto& p : t.parts) { + CAPTURE(p.name); + CHECK(p.offset >= 0x9000u); + CHECK(p.size > 0u); + CHECK(p.end() <= t.flashBytes); + } + + // No two partitions may overlap. Compared pairwise rather than by sorting, so the failure + // message names both culprits. + for (size_t i = 0; i < t.parts.size(); i++) { + for (size_t j = i + 1; j < t.parts.size(); j++) { + const auto& a = t.parts[i]; + const auto& b = t.parts[j]; + const bool disjoint = a.end() <= b.offset || b.end() <= a.offset; + CAPTURE(a.name); + CAPTURE(b.name); + CHECK(disjoint); + } + } + } +} + +TEST_CASE("app partitions are 64 KB aligned, as the MMU requires") { + for (const auto& t : allTables()) { + CAPTURE(t.file); + for (const auto& p : t.parts) { + if (p.type != "app") continue; + CAPTURE(p.name); + CHECK((p.offset % 0x10000u) == 0u); + } + } +} + +TEST_CASE("a table carries either two OTA slots or one slot plus a recovery app, never a mix") { + // Dual-OTA (ota_0 + ota_1) buys a power-fail rollback at the cost of holding two copies of the + // firmware. The safeboot shape (factory + ota_0) spends that space on the app instead and + // recovers through the factory image. Both are valid; a table that has ota_1 AND a factory + // partition would be paying for both and is a mistake. + for (const auto& t : allTables()) { + CAPTURE(t.file); + int ota = 0, factory = 0; + for (const auto& p : t.parts) { + if (p.type != "app") continue; + if (p.subtype == "factory") factory++; + else if (p.subtype.starts_with("ota_")) ota++; + } + CHECK(factory <= 1); + CHECK(ota >= 1); + const bool dualOta = (ota == 2 && factory == 0); + const bool safeboot = (ota == 1 && factory == 1); + CHECK((dualOta || safeboot)); + } +} From b6d4812ddc797b03156b77a5a96ec59bb5ad34aa Mon Sep 17 00:00:00 2001 From: ewowi Date: Wed, 26 Aug 2026 15:07:05 +0200 Subject: [PATCH 2/6] Trigger the MoonBase test release on a push of the branch GitHub only registers a dispatchable workflow from the default branch, so the manual-only trigger could never appear in the Actions list before the merge; a push trigger on this branch runs it now. Co-Authored-By: Claude Opus 5 (1M context) --- .github/workflows/moonbase-test-release.yml | 9 +++++++-- 1 file changed, 7 insertions(+), 2 deletions(-) diff --git a/.github/workflows/moonbase-test-release.yml b/.github/workflows/moonbase-test-release.yml index 3a4a6b11..93cb83cc 100644 --- a/.github/workflows/moonbase-test-release.yml +++ b/.github/workflows/moonbase-test-release.yml @@ -3,12 +3,17 @@ name: MoonBase test release # TEMPORARY: delete after the MoonBase branch merges to main. # Publishes a `moonbase-test` prerelease from THIS branch so the picker's URL-based # install can be tested against real GitHub release assets before the merge (the -# real release pipeline only runs from main and version tags). Manual trigger only; -# esp32 (the bench Olimex) is the only firmware built. Mirrors release.yml's esp32 +# real release pipeline only runs from main and version tags). Triggered by a push of +# this branch: workflow_dispatch alone cannot work here, GitHub only registers a +# dispatchable workflow from the default branch. esp32 (the bench Olimex) is the only +# firmware built. Mirrors release.yml's esp32 # job + staging + manifest, reduced to one variant and one fixed tag. on: workflow_dispatch: + push: + branches: + - moonbase permissions: contents: write From c93a00f117c46de4fb49ecd4e716372494ddff91 Mon Sep 17 00:00:00 2001 From: ewowi Date: Wed, 26 Aug 2026 15:24:31 +0200 Subject: [PATCH 3/6] Rename web-installer/ to mooninstaller/, joining the Moon family The browser-flash installer folder now sits beside moondeck/, moonlive/ and moonbase/ as mooninstaller/. Nothing user-facing changes: the published install page URL, release assets and wire keys never carried the folder name. KPI: 16384lights | Desktop:1238KB | tick:419/143/7/24/205/482/28/14/516/106/35/40/15/237/36/9/452/66/9us(FPS:2386/6993/142857/41666/4878/2074/35714/71428/1937/9433/28571/25000/66666/4219/27777/111111/2212/15151/111111) | ESP32:1632KB | src:226(62809) | test:171(38458) | lizard:173w Scripts/MoonDeck: - Path references updated across moondeck scripts, CI workflows, CMakeLists and the folder's own HTML/JS (84 occurrences classified first: no URL, data key or asset name carries the folder name, so this is pure path mechanics) Tests: - test/js and test/python path references updated; the .mjs files were missed by the first sweep and caught by the JS tests Docs/CI: - Current docs updated; docs/history/ plans and ADRs keep the old name (immutable records) Performance: - No behavior change; desktop and ESP32 images identical apart from comment text in the embedded UI sources Co-Authored-By: Claude Opus 5 (1M context) --- .github/workflows/release.yml | 18 +++---- .github/workflows/test.yml | 4 +- CMakeLists.txt | 8 ++-- docs/architecture.md | 8 ++-- docs/backlog/backlog-core.md | 4 +- docs/backlog/rename-to-moonlight.md | 2 +- docs/building.md | 6 +-- docs/metrics/repo-health.json | 6 +-- docs/metrics/repo-health.md | 44 +++++++++--------- docs/moonmodules/light/drivers.md | 2 +- docs/reference/esp32-s31-coreboard.md | 2 +- docs/reference/gpio-usage.md | 2 +- docs/reference/mhc-wled-esp32-p4-shield.md | 2 +- docs/testing.md | 8 ++-- moondeck/MoonDeck.md | 12 ++--- moondeck/build/build_esp32.py | 4 +- moondeck/build/flash_esp32.py | 4 +- moondeck/build/generate_firmwares.py | 4 +- moondeck/build/improv_provision.py | 6 +-- moondeck/build/improv_smoke_test.py | 2 +- moondeck/check/check_devices.py | 4 +- moondeck/check/check_firmwares.py | 8 ++-- moondeck/ci/make_ico.py | 2 +- moondeck/ci/package_desktop.py | 6 +-- moondeck/docs/mkdocs_hooks.py | 8 ++-- moondeck/event/precommit.py | 8 ++-- moondeck/moondeck.py | 14 +++--- moondeck/moondeck_ui/app.js | 2 +- moondeck/run/preview_installer.py | 12 ++--- {web-installer => mooninstaller}/README.md | 4 +- .../assets/app-store-badge.svg | 0 .../assets/google-play-badge.png | Bin .../assets/home-assistant-icon.png | Bin .../config-ops.js | 0 .../deviceModels.json | 0 {web-installer => mooninstaller}/devices.js | 0 {web-installer => mooninstaller}/favicon.png | Bin .../firmwares.json | 0 .../improv-frame.js | 0 {web-installer => mooninstaller}/index.html | 2 +- .../install-orchestrator.js | 2 +- {web-installer => mooninstaller}/install.css | 0 {web-installer => mooninstaller}/install.js | 4 +- src/ui/install-picker-boards.js | 4 +- src/ui/install-picker.js | 4 +- test/js/config-ops.test.mjs | 4 +- test/js/improv-frame.test.mjs | 4 +- test/js/installer-eth-only.test.mjs | 6 +-- test/js/installer-flash-baud.test.mjs | 4 +- test/js/installer-s31-webflash.test.mjs | 4 +- test/python/test_improv_frame.py | 2 +- test/python/test_installer_manifests.py | 4 +- 52 files changed, 130 insertions(+), 130 deletions(-) rename {web-installer => mooninstaller}/README.md (99%) rename {web-installer => mooninstaller}/assets/app-store-badge.svg (100%) rename {web-installer => mooninstaller}/assets/google-play-badge.png (100%) rename {web-installer => mooninstaller}/assets/home-assistant-icon.png (100%) rename {web-installer => mooninstaller}/config-ops.js (100%) rename {web-installer => mooninstaller}/deviceModels.json (100%) rename {web-installer => mooninstaller}/devices.js (100%) rename {web-installer => mooninstaller}/favicon.png (100%) rename {web-installer => mooninstaller}/firmwares.json (100%) rename {web-installer => mooninstaller}/improv-frame.js (100%) rename {web-installer => mooninstaller}/index.html (99%) rename {web-installer => mooninstaller}/install-orchestrator.js (99%) rename {web-installer => mooninstaller}/install.css (100%) rename {web-installer => mooninstaller}/install.js (99%) diff --git a/.github/workflows/release.yml b/.github/workflows/release.yml index c7bfff61..5d357c36 100644 --- a/.github/workflows/release.yml +++ b/.github/workflows/release.yml @@ -39,11 +39,11 @@ on: # The web installer + docs site are served from Pages by the deploy-pages job # below; a change to them must trigger a deploy or it never reaches the live site # (the eth-only-provisioning fix shipped a commit that didn't auto-deploy because - # web-installer was missing here). src/ui/install-picker*.js is already covered by src/**. + # mooninstaller was missing here). src/ui/install-picker*.js is already covered by src/**. # docs/** covers every page rendered into the Pages root by MkDocs; mkdocs.yml is # the site config (nav/theme) โ€” a nav change with no doc change must still redeploy. - 'docs/**' - - 'web-installer/**' + - 'mooninstaller/**' - 'mkdocs.yml' workflow_dispatch: inputs: @@ -91,7 +91,7 @@ jobs: TAG: ${{ inputs.tag || github.ref_name }} run: uv run moondeck/ci/verify_version.py --tag "$TAG" - # The shipping firmware list, read from the generated web-installer/firmwares.json + # The shipping firmware list, read from the generated mooninstaller/firmwares.json # (projected from build_esp32.py's FIRMWARES dict, drift-guarded by # check_firmwares.py). Emitted as a JSON array so build-esp32's matrix can # fromJSON() it โ€” GitHub matrices can't read a file at parse time, so a job @@ -108,7 +108,7 @@ jobs: - id: gen run: | set -euo pipefail - echo "list=$(jq -c '[.firmwares[] | select(.ships) | .name]' web-installer/firmwares.json)" >> "$GITHUB_OUTPUT" + echo "list=$(jq -c '[.firmwares[] | select(.ships) | .name]' mooninstaller/firmwares.json)" >> "$GITHUB_OUTPUT" build-esp32: needs: [verify-version, firmwares] @@ -490,9 +490,9 @@ jobs: # โ€” no CORS). The Pages-relative manifests are generated in the # deploy-pages job, where the web installer (CORS-bound) consumes them. BASE="https://github.com/${REPO}/releases/download/$TAG" - # The shipping firmware list โ€” the same web-installer/firmwares.json the + # The shipping firmware list โ€” the same mooninstaller/firmwares.json the # build matrix reads, so manifests and builds can't drift. - for F in $(jq -r '.firmwares[] | select(.ships) | .name' web-installer/firmwares.json); do + for F in $(jq -r '.firmwares[] | select(.ships) | .name' mooninstaller/firmwares.json); do uv run python moondeck/build/generate_manifest.py \ --firmware "$F" \ --version "$V" \ @@ -671,9 +671,9 @@ jobs: # Install page + the shared install-picker module sit at the root. # Each release's binaries + manifests live under releases//. mkdir -p pages/install - cp -r web-installer/. pages/install/ + cp -r mooninstaller/. pages/install/ cp src/ui/install-picker.js pages/install/ - # The board-catalog / chip-detection half of the picker โ€” web-installer + # The board-catalog / chip-detection half of the picker โ€” mooninstaller # only (not embedded in firmware), imported by index.html. Must ship to # Pages alongside install-picker.js or the ES-module import 404s. cp src/ui/install-picker-boards.js pages/install/ @@ -687,7 +687,7 @@ jobs: mkdir -p pages/install/assets/deviceModels # rel is "assets/deviceModels/." (the path served from /install/); # the source file lives in docs/ (i.e. docs/assets/deviceModels/...). - jq -r '.[].image // empty' web-installer/deviceModels.json | while read -r rel; do + jq -r '.[].image // empty' mooninstaller/deviceModels.json | while read -r rel; do src="docs/$rel" [ -f "$src" ] && cp "$src" "pages/install/$rel" \ || echo "WARNING: deviceModels.json image not found: $src" diff --git a/.github/workflows/test.yml b/.github/workflows/test.yml index 9ac425f9..45a6fa94 100644 --- a/.github/workflows/test.yml +++ b/.github/workflows/test.yml @@ -10,7 +10,7 @@ name: Test # New Python/JS unit suites land under test/python and test/js and run here. # Paths cover every input to the host-side tests: the Python/JS sources under test -# (scripts, web-installer), the test files themselves, AND the device-side C++ frame +# (scripts, mooninstaller), the test files themselves, AND the device-side C++ frame # contract (src/core/Improv*.h + the platform handler) โ€” a wire-format change in the # firmware must run the cross-language golden-vector tests so it can't drift from the # Python/JS builders silently. pull_request gates every PR; push runs main only (a @@ -20,7 +20,7 @@ on: pull_request: paths: &test-paths - 'moondeck/**' - - 'web-installer/**' + - 'mooninstaller/**' - 'src/core/ImprovFrame.h' - 'src/core/ImprovOpReassembler.h' - 'src/platform/esp32/platform_esp32_improv.cpp' diff --git a/CMakeLists.txt b/CMakeLists.txt index 492a4b38..f7b4fbb1 100644 --- a/CMakeLists.txt +++ b/CMakeLists.txt @@ -208,7 +208,7 @@ add_custom_target(ui_embed DEPENDS ${CMAKE_SOURCE_DIR}/src/ui/ui_embedded.h) add_dependencies(mm_core ui_embed build_info_gen) # Windows: give the exe its own icon, so it is recognizable in Explorer, the taskbar and the Start -# menu whether it was installed or just unzipped. Generated from the same web-installer/favicon.png +# menu whether it was installed or just unzipped. Generated from the same mooninstaller/favicon.png # the macOS .icns comes from, so the mark has one source rather than a checked-in binary per # platform. Nothing here runs on macOS or Linux, which carry their icon in the .app and the .deb. set(MM_WIN_RESOURCES "") @@ -220,14 +220,14 @@ if(WIN32) # `uv run "; @@ -243,14 +256,34 @@ const esp_partition_t* appPartition() { // Write a firmware image pulled from `url` straight into the application slot. This is what makes // an unattended install possible: point MoonBase at a release asset and it fetches it itself. +// True while any install is writing the app slot. Torn reads are harmless (same display-only +// pattern the app uses); the guard only has to stop a SECOND install from starting. +volatile bool installing_ = false; + bool installFromUrl(const char* url) { esp_http_client_config_t http = {}; http.url = url; http.timeout_ms = 20000; http.keep_alive_enable = true; http.crt_bundle_attach = esp_crt_bundle_attach; // GitHub and friends are HTTPS + // A GitHub release asset 302-redirects to a signed URL whose Location header (plus a + // multi-KB content-security-policy on the redirect response) overflows the client's + // default 512-byte header buffer, failing the connection AFTER a clean TLS handshake. + // Same values as the app's http_fetch_to_ota (platform_esp32_ota.cpp). + http.disable_auto_redirect = false; + http.max_redirection_count = 10; + // Large receive chunks: fewer, larger flash writes per loop. Measured on the bench + // (classic ESP32, 40 MHz DIO flash): 4 KB chunks stream at ~46 KB/s, 16 KB at ~86, + // 32 KB roughly the same as 16 (write-bound from there); RAM is plentiful here. + http.buffer_size = 32768; + http.buffer_size_tx = 4096; esp_https_ota_config_t ota = {}; ota.http_config = &http; + // One bulk erase of the whole slot up front instead of a sector erase inlined with every + // 4 KB write: per-sector erases dominated the install at ~25 KB/s (identical over TLS and + // plain HTTP, so the wire was never the limit). The upfront erase costs a few seconds, + // "preparing the install" covers it. + ota.bulk_flash_erase = true; esp_https_ota_handle_t handle = nullptr; esp_err_t beginErr = esp_https_ota_begin(&ota, &handle); @@ -263,11 +296,18 @@ bool installFromUrl(const char* url) { } esp_err_t err; while ((err = esp_https_ota_perform(handle)) == ESP_ERR_HTTPS_OTA_IN_PROGRESS) { - std::snprintf(status_, sizeof(status_), "downloading: %d bytes", - esp_https_ota_get_image_len_read(handle)); + std::snprintf(status_, sizeof(status_), "downloading: %d of %d bytes", + esp_https_ota_get_image_len_read(handle), + esp_https_ota_get_image_size(handle)); } - if (err != ESP_OK || esp_https_ota_finish(handle) != ESP_OK) { - std::snprintf(status_, sizeof(status_), "error: the download failed"); + if (err != ESP_OK) { + esp_https_ota_abort(handle); // finish() is for a COMPLETE download; abort frees this one + std::snprintf(status_, sizeof(status_), "error: the download failed (0x%x)", + static_cast(err)); + return false; + } + if (esp_https_ota_finish(handle) != ESP_OK) { + std::snprintf(status_, sizeof(status_), "error: the image is not valid firmware"); return false; } std::snprintf(status_, sizeof(status_), "installed, restarting"); @@ -306,7 +346,7 @@ void sendResponse(int sock, const char* status, const char* type, const char* bo // Write `contentLen` bytes from the socket into the application slot. `prefix` carries whatever // arrived in the same read as the headers. -bool installFromSocket(int sock, const char* prefix, size_t prefixLen, size_t contentLen) { +bool installFromSocketLocked(int sock, const char* prefix, size_t prefixLen, size_t contentLen) { const esp_partition_t* part = appPartition(); if (!part) { std::snprintf(status_, sizeof(status_), "error: no app partition"); return false; } if (contentLen == 0 || contentLen > part->size) { @@ -322,6 +362,7 @@ bool installFromSocket(int sock, const char* prefix, size_t prefixLen, size_t co } size_t written = 0; + if (prefixLen > contentLen) prefixLen = contentLen; // never store bytes past the declared body if (prefixLen) { if (esp_ota_write(handle, prefix, prefixLen) != ESP_OK) { esp_ota_abort(handle); @@ -368,6 +409,26 @@ bool installFromSocket(int sock, const char* prefix, size_t prefixLen, size_t co } // Read the request head, dispatch, and (on a successful install) restart into the application. +// The staged-URL install, off the main task (which serves meanwhile). A connect attempted +// straight after GOT_IP can fail (0x7002, ESP_ERR_HTTP_CONNECT) where the same connect succeeds +// seconds later: the LAN is still warming up around a freshly associated station. A short retry +// absorbs that; a genuinely unreachable URL still fails through to the page after the last +// attempt, where status_ shows the error. +char stagedUrlTask_[256]; + +void unattendedInstallTask(void*) { + for (int attempt = 0; attempt < 3; attempt++) { + if (attempt) vTaskDelay(pdMS_TO_TICKS(3000)); + if (installFromUrl(stagedUrlTask_)) esp_restart(); // straight back into the new app + // A failed attempt leaves its error in status_; while retries remain that error is + // TRANSIENT, and a watcher treating "error:" as terminal (the app's overlay does) + // must not see it. The final attempt's error stays as the terminal answer. + if (attempt < 2) std::snprintf(status_, sizeof(status_), "download failed, retrying"); + } + installing_ = false; // set by the spawner; held across the retries + vTaskDelete(nullptr); +} + void serveOne(int sock) { // TCP does not coalesce: the header block (or a small body) can arrive in several // segments, so read until the blank line is seen, bounded by the buffer. A request @@ -382,18 +443,22 @@ void serveOne(int sock) { head[got] = '\0'; if ((bodyStart = std::strstr(head, "\r\n\r\n"))) break; } - if (got == 0) return; + if (got == 0) { ::close(sock); return; } // serveOne owns the fd; a bare return leaks it head[got] = '\0'; const size_t headLen = bodyStart ? static_cast(bodyStart + 4 - head) : got; size_t prefixLen = got - headLen; + // HTTP header names are case-insensitive; strcasestr is not in the std namespace but is + // provided by newlib, and the probe is bounded by the header buffer. size_t contentLen = 0; - if (const char* cl = std::strstr(head, "Content-Length:")) { + if (const char* cl = strcasestr(head, "Content-Length:")) { contentLen = static_cast(std::strtoul(cl + 15, nullptr, 10)); } bool installed = false; - if (std::strncmp(head, "POST /install-url", 17) == 0) { + if (std::strncmp(head, "POST /install-url", 17) == 0 && installing_) { + sendResponse(sock, "409 Conflict", "text/plain", "error: an install is already running"); + } else if (std::strncmp(head, "POST /install-url", 17) == 0) { // The body is the URL itself; small enough to finish reading into the same buffer. while (prefixLen < contentLen && headLen + prefixLen < sizeof(head) - 1) { const int n = ::recv(sock, head + headLen + prefixLen, @@ -401,14 +466,28 @@ void serveOne(int sock) { if (n <= 0) break; prefixLen += static_cast(n); } - char url[256] = {}; - const size_t n = prefixLen < sizeof(url) - 1 ? prefixLen : sizeof(url) - 1; - std::memcpy(url, head + headLen, n); - installed = installFromUrl(url); - sendResponse(sock, installed ? "200 OK" : "500 Internal Server Error", "text/plain", status_); + const size_t n = prefixLen < sizeof(stagedUrlTask_) - 1 ? prefixLen + : sizeof(stagedUrlTask_) - 1; + std::memcpy(stagedUrlTask_, head + headLen, n); + stagedUrlTask_[n] = '\0'; + std::snprintf(status_, sizeof(status_), "starting the install"); + installing_ = true; // cleared by the task after its final attempt + xTaskCreate(unattendedInstallTask, "mb_install", 12288, nullptr, 5, nullptr); + // 202: the install runs on its own task while this server keeps answering GET + // /moonbase with live progress; the caller watches that, not this response. + sendResponse(sock, "202 Accepted", "text/plain", status_); } else if (std::strncmp(head, "POST /install", 13) == 0) { - installed = installFromSocket(sock, head + headLen, prefixLen, contentLen); - sendResponse(sock, installed ? "200 OK" : "500 Internal Server Error", "text/plain", status_); + if (installing_) { + sendResponse(sock, "409 Conflict", "text/plain", "error: an install is already running"); + } else { + installing_ = true; + installed = installFromSocketLocked(sock, head + headLen, prefixLen, contentLen); + installing_ = false; + sendResponse(sock, installed ? "200 OK" : "500 Internal Server Error", "text/plain", status_); + } + } else if (std::strncmp(head, "POST /boot-app", 14) == 0 && installing_) { + // Booting away mid-write would abandon a half-written slot; refuse, visibly. + sendResponse(sock, "409 Conflict", "text/plain", "error: an install is already running"); } else if (std::strncmp(head, "POST /boot-app", 14) == 0) { // Switch back to the installed application without installing anything. // esp_ota_set_boot_partition validates the image first, so a half-written app is @@ -424,7 +503,9 @@ void serveOne(int sock) { // answering at the shared address (the app 404s it). Body = the live install status, so // the poll doubles as a progress read during an unattended install. sendResponse(sock, "200 OK", "text/plain", status_); - } else if (std::strncmp(head, "GET / ", 6) == 0 || std::strncmp(head, "GET /index", 10) == 0) { + } else if (std::strncmp(head, "GET / ", 6) == 0 || std::strncmp(head, "GET /?", 6) == 0 || + std::strncmp(head, "GET /index", 10) == 0) { + // "/?" is the app page's cache-busting handoff to this page (app.js). sendResponse(sock, "200 OK", "text/html", kPage); } else { sendResponse(sock, "404 Not Found", "text/plain", "not found"); @@ -503,15 +584,16 @@ extern "C" void app_main() { bool online = wifiStation(20000); // STA only: on the fallback AP the URL's network is not reachable, and a user is present. + // The install runs on its OWN task so the main task serves throughout: GET /moonbase then + // reports "downloading: N of M bytes" live, which is what the app's update overlay renders + // as a progress bar. 12 KB stack for the same reason as the main task: the TLS handshake. if (online && stagedUrl[0]) { - // A connect attempted straight after GOT_IP can fail (0x7002, ESP_ERR_HTTP_CONNECT) - // where the same connect succeeds seconds later: the LAN is still warming up around a - // freshly associated station. A short retry absorbs that; a genuinely unreachable URL - // still fails through to the page after the last attempt. - for (int attempt = 0; attempt < 3; attempt++) { - if (attempt) vTaskDelay(pdMS_TO_TICKS(3000)); - if (installFromUrl(stagedUrl)) esp_restart(); // straight back into the new app - } + // Status set BEFORE the task spawns: the overlay polls from the moment MoonBase + // answers, and "idle" would read as nothing happening while an install is pending. + std::snprintf(status_, sizeof(status_), "preparing the install"); + std::snprintf(stagedUrlTask_, sizeof(stagedUrlTask_), "%s", stagedUrl); + installing_ = true; // cleared by the task after its final attempt + xTaskCreate(unattendedInstallTask, "mb_install", 12288, nullptr, 5, nullptr); } if (!online) online = wifiAccessPoint(); diff --git a/moonbase/sdkconfig.defaults b/moonbase/sdkconfig.defaults index 09224d43..08c0ab55 100644 --- a/moonbase/sdkconfig.defaults +++ b/moonbase/sdkconfig.defaults @@ -35,8 +35,13 @@ CONFIG_ESP_WIFI_ENTERPRISE_SUPPORT=n # One connection at a time, small buffers: MoonBase serves one page and receives one file. CONFIG_LWIP_MAX_SOCKETS=6 -CONFIG_ESP_WIFI_STATIC_RX_BUFFER_NUM=4 -CONFIG_ESP_WIFI_DYNAMIC_RX_BUFFER_NUM=8 +# RX sizing is the install's throughput ceiling (TCP window / RTT): the earlier minimal +# values capped a LAN download at ~25 KB/s. MoonBase's one job is moving a ~2 MB image, and +# with no app there is RAM to spare, so receive gets real buffers and a real window. +CONFIG_ESP_WIFI_STATIC_RX_BUFFER_NUM=8 +CONFIG_ESP_WIFI_DYNAMIC_RX_BUFFER_NUM=24 +CONFIG_LWIP_TCP_WND_DEFAULT=32768 +CONFIG_LWIP_TCP_RECVMBOX_SIZE=24 CONFIG_ESP_WIFI_TX_BUFFER_TYPE=0 CONFIG_ESP_WIFI_STATIC_TX_BUFFER_NUM=8 @@ -51,3 +56,9 @@ CONFIG_PARTITION_TABLE_CUSTOM_FILENAME="../esp32/partitions/esp32dev_moonbase.cs # bundle, but a LAN source (MoonDeck serving a dev build) has no certificate and is inside the # user's own network. The staged-URL handoff writes whatever the app was asked to install. CONFIG_ESP_HTTPS_OTA_ALLOW_HTTP=y + +# The whole of MoonBase runs on the main task (app_main installs, then serves), and a GitHub +# install rides mbedTLS: certificate-chain verification against the bundle peaks near 8 KB of +# stack, which overflowed the 3.5 KB default the moment the TLS handshake ran (the plain-HTTP +# LAN path fit, which is why the bench missed it). RAM is not scarce here; 12 KB buys margin. +CONFIG_ESP_MAIN_TASK_STACK_SIZE=12288 diff --git a/moondeck/check/check_esp32_built.py b/moondeck/check/check_esp32_built.py index 4961de6d..74dd801a 100644 --- a/moondeck/check/check_esp32_built.py +++ b/moondeck/check/check_esp32_built.py @@ -32,7 +32,8 @@ SOURCE_DIRS = ("src", "esp32") SOURCE_FILES = ("CMakeLists.txt", "library.json") SOURCE_SUFFIXES = {".c", ".cpp", ".h", ".hpp", ".cmake", ".json", ".txt", ".py", ".js", - ".html", ".css", ".defaults", ".csv"} # .csv: partition tables feed the image + ".html", ".css", ".defaults", ".csv", # .csv: partition tables feed the image + ".yml"} # .yml: idf_component.yml pins components # Build outputs and caches live under the source dirs; they are products, not inputs, and # including them would compare the binary against itself. diff --git a/moondeck/qemu/run_qemu.py b/moondeck/qemu/run_qemu.py index 81e82905..fd822e68 100644 --- a/moondeck/qemu/run_qemu.py +++ b/moondeck/qemu/run_qemu.py @@ -56,7 +56,9 @@ def merged_flash(force: bool) -> str: # which made this cache never expire: after a rebuild the emulator kept booting the PREVIOUS # app, and code that was plainly in the .bin appeared not to run at all. app = os.path.join(BUILD, "projectMM.bin") - newest_input = max((os.path.getmtime(p) for p in (args, app) if os.path.exists(p)), default=0) + moonbase = os.path.join(os.path.dirname(BUILD), "moonbase-esp32", "projectMM-moonbase.bin") + newest_input = max((os.path.getmtime(p) for p in (args, app, moonbase) + if os.path.exists(p)), default=0) if os.path.exists(out) and not force and os.path.getmtime(out) > newest_input: return out # find_idf_python returns the venv DIRECTORY; the interpreter is bin/python inside it. diff --git a/src/ui/app.js b/src/ui/app.js index 4353fac5..65bd00f0 100644 --- a/src/ui/app.js +++ b/src/ui/app.js @@ -330,6 +330,22 @@ async function init() { // REST snapshot overwrite the newer, live WS state โ€” just skip the commit. try { const resp = await fetch("/api/state"); + if (resp.status === 404) { + // A 404 on the app's own state API means this page is a CACHED copy of the app + // talking to a different server at the same address: MoonBase. Confirm, then hand + // over cleanly instead of rendering a half-dead skeleton โ€” the cache-busting query + // makes the browser fetch MoonBase's page instead of re-serving this one. + const mb = await probeMoonBase(); + if (mb.state === "up") { + document.body.innerHTML = ""; + const note = document.createElement("p"); + note.style.cssText = "font:16px system-ui;margin:3rem auto;max-width:30rem;text-align:center"; + note.textContent = "This device is in MoonBase (maintenance mode) \u2014 opening its page\u2026"; + document.body.appendChild(note); + setTimeout(() => location.replace("/?" + Date.now()), 1200); + return; + } + } if (resp.ok && (!state || !Array.isArray(state.modules))) { const snap = await resp.json(); if (snap && Array.isArray(snap.modules)) { @@ -1144,6 +1160,7 @@ function showUpdateOverlay() { document.body.appendChild(ov); return { status(text) { msg.textContent = text; }, + progress(read, total) { if (total > 0) { bar.max = total; bar.value = read; } }, fail(text) { msg.textContent = text; bar.remove(); dismiss.style.display = ""; }, }; } @@ -1153,12 +1170,18 @@ const fwSleep = (ms) => new Promise((r) => setTimeout(r, ms)); // Probe GET /moonbase once: "up" (200 + status text), "app" (404: the application is answering), // or "silent" (no answer: the device is mid-reboot). Never throws. async function probeMoonBase() { + // Bounded: against a powered-off device an untimed fetch hangs for up to a minute on the + // TCP connect, which starves the poll loop of the very silence it is trying to measure. + const ctl = new AbortController(); + const timer = setTimeout(() => ctl.abort(), 2500); try { - const r = await fetch("/moonbase", { cache: "no-store" }); + const r = await fetch("/moonbase", { cache: "no-store", signal: ctl.signal }); if (r.ok) return { state: "up", text: await r.text() }; return { state: "app" }; } catch (_) { return { state: "silent" }; + } finally { + clearTimeout(timer); } } @@ -1196,10 +1219,16 @@ async function moonbaseUpdateFlow(opts) { // The upload needs MoonBase serving before the browser can push the image. if (!(await waitForMoonBase(Date.now() + 120000))) throw new Error(MOONBASE_SILENT_MSG); ui.status(`Installing ${fmSize(opts.file.size)}\u2026`); - const r = await fetch("/install", { - method: "POST", headers: { "Content-Type": "application/octet-stream" }, - body: opts.file }); - if (!r.ok) throw new Error(await r.text()); + try { + const r = await fetch("/install", { + method: "POST", headers: { "Content-Type": "application/octet-stream" }, + body: opts.file }); + if (!r.ok) throw new Error(await r.text()); + } catch (err) { + if (err instanceof Error && err.message.startsWith("error")) throw err; + // A dead socket mid-upload (power cut, WiFi drop) is not a verdict: the watch + // loop below sees where the device lands and retries the upload from there. + } } // Watch until the app answers again. On the unattended URL path MoonBase installs BEFORE @@ -1209,17 +1238,57 @@ async function moonbaseUpdateFlow(opts) { // 404s the probe for a moment, so "app" only counts as done once the device has been // seen away, or after a grace period long enough that the kickoff reboot must have run. const watchStart = Date.now(); - const deadline = watchStart + 300000; + let deadline = watchStart + 300000; let sawAway = false; + let silentStreak = 0; + let retries = 0; while (Date.now() < deadline) { await fwSleep(2000); const probe = await probeMoonBase(); + if (probe.state !== "silent") silentStreak = 0; if (probe.state === "silent") { sawAway = true; + // A reboot is a few silent probes; many in a row is a device that lost power or + // the network. Say so instead of freezing on the last byte count. + if (++silentStreak >= 4) { + ui.status("The device is not answering \u2014 waiting for it to come back\u2026"); + } + } else if (probe.state === "up" && probe.text === "idle" && sawAway) { + // MoonBase is up with NOTHING running after the cycle already started: the + // install was interrupted (a power cut lands exactly here: blank otadata boots + // MoonBase, and the staged URL was already consumed). The browser still holds + // the payload, so the one click survives the cut: hand it over again. + if (retries >= 2) throw new Error("the install keeps getting interrupted"); + retries++; + deadline += 180000; // a retry restarts the ~40 s install; extend the watch + ui.status("The install was interrupted \u2014 retrying\u2026"); + try { + if (opts.url) { + await fetch("/install-url", { method: "POST", body: opts.url }); + } else { + await fetch("/install", { + method: "POST", headers: { "Content-Type": "application/octet-stream" }, + body: opts.file }); + } + // The response itself is not consumed: MoonBase reboots on success and the + // probes below see the outcome either way. + } catch (_) {} } else if (probe.state === "up") { sawAway = true; // MoonBase answering means the old app is gone if (probe.text.startsWith("error")) throw new Error(probe.text); - ui.status(probe.text || "Installing\u2026"); + // The unattended install reports "downloading: N of M bytes"; render it the + // way the file path reads, with a real bar. + const dl = probe.text.match(/downloading: (\d+) of (\d+) bytes/); + if (dl) { + const read = parseInt(dl[1], 10), total = parseInt(dl[2], 10); + ui.status(`Installing ${fmSize(total || read)}\u2026`); + ui.progress(read, total); + } else { + // "idle" is MoonBase's quiescent state; mid-cycle it means the install + // has not started yet, which deserves better words than "Idle". + ui.status(!probe.text || probe.text === "idle" + ? "Preparing the install\u2026" : probe.text); + } } else if (probe.state === "app" && (sawAway || Date.now() - watchStart > 20000)) { ui.status("Done \u2014 reloading\u2026"); await fwSleep(1000); diff --git a/test/unit/core/unit_PartitionTables.cpp b/test/unit/core/unit_PartitionTables.cpp index d5b44a63..d461a7d6 100644 --- a/test/unit/core/unit_PartitionTables.cpp +++ b/test/unit/core/unit_PartitionTables.cpp @@ -25,10 +25,26 @@ struct Partition { struct Table { std::string file; - uint32_t flashBytes = 0; // inferred from the largest end offset, rounded up to a power of two + uint32_t flashBytes = 0; // the flash the table is written for; declaredFlashBytes below std::vector parts; }; +// Each table's flash capacity, DECLARED rather than inferred from the largest end offset: +// an inferred capacity would grow with an oversized table and hide exactly the overflow this +// suite exists to catch. A new table must be added here, which is the point. +uint32_t declaredFlashBytes(const std::string& file) { + static const std::pair kCapacity[] = { + {"esp32dev.csv", 4u * 1024 * 1024}, + {"esp32dev_moonbase.csv", 4u * 1024 * 1024}, + {"esp32s3_n8r8.csv", 8u * 1024 * 1024}, + {"ota_16mb.csv", 16u * 1024 * 1024}, + }; + for (const auto& [name, bytes] : kCapacity) + if (file == name) return bytes; + FAIL("partition table ", file, " has no declared flash capacity; add it to kCapacity"); + return 0; +} + std::filesystem::path partitionDir() { // test/unit/core/ -> repo root -> esp32/partitions return std::filesystem::path(__FILE__).parent_path().parent_path().parent_path().parent_path() @@ -84,11 +100,7 @@ Table readTable(const std::filesystem::path& path) { REQUIRE_MESSAGE(parseNumber(f[4], p.size), t.file, ": bad size for ", p.name); t.parts.push_back(p); } - uint32_t maxEnd = 0; - for (const auto& p : t.parts) maxEnd = p.end() > maxEnd ? p.end() : maxEnd; - uint32_t flash = 1024u * 1024u; - while (flash < maxEnd) flash <<= 1; - t.flashBytes = flash; + t.flashBytes = declaredFlashBytes(t.file); return t; } From b6592232c1d24f14c3c1aaa58ba38a593faf7a79 Mon Sep 17 00:00:00 2001 From: ewowi Date: Wed, 26 Aug 2026 22:04:39 +0200 Subject: [PATCH 5/6] MoonBase Ethernet, cancel and retry, and the app's face MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit MoonBase now works on wired boards too: it reads the Ethernet wiring from the same config the app persists and comes up on the same interface the app used, so the update overlay never loses the device's address. An install can be canceled and retried (the URL field comes prefilled with the last source, surviving power cycles), and MoonBase's page now looks like the app: same palette, the MoonLight logo, and a (?) to the docs. KPI: 16384lights | Desktop:1238KB | tick:233/95/3/9/136/293/21/3/292/71/19/22/24/130/23/6/250/45/4us(FPS:4291/10526/333333/111111/7352/3412/47619/333333/3424/14084/52631/45454/41666/7692/43478/166666/4000/22222/250000) | ESP32:1660KB | tick:8375us(FPS:119) | heap:139KB | src:226(62814) | test:171(38486) | lizard:173w Core: - MoonBase Ethernet (classic RMII, +21 KB, 773 KB total): the wiring comes from the same NetworkModule.json scrape as the credentials (ethType gates it; absent pins keep silicon defaults), ONE interface at a time in the app's own preference order (eth with an 8 s lease window, torn down without a link, else WiFi, else AP), bench-verified: eth install at the same flash-bound rate, WiFi fallback with eth configured-but-unplugged, and both firmwares (esp32, esp32-eth) share the one image - Cancel: POST /cancel aborts a running URL install cleanly back to the page (an upload cancels by dropping the connection); a canceled or failed install leaves the slot invalid and Boot-the-app refuses it by validation, so the escape is retry: the last source persists in NVS (prefill-only, never auto-installed) and prefills the URL field - txPowerSetting mirrored from the app's config: applied only as a real cap (1..21 dBm) and only after the connection is up (the at-0 / in-start-stack hang NetworkModule documents); deliberately skipped on the AP fallback, where a hang outranks a brownout - Cache-Control: no-store on every MoonBase response: one address serves two UIs over time, and a browser re-serving a cached copy of either shows a dead page (Safari does, even on reload-from-origin) - A failed install-task spawn clears the busy flag instead of latching every later install into 409 (the one real deadlock in the design, found by review) UI: - MoonBase's page wears the app's palette, the embedded MoonLight logo (EMBED_FILES, served as /logo.png, doubling as favicon), a (?) linking to the MoonBase doc on moonmodules.org, and a firmware-releases link under the file picker; Cancel appears only while an install runs; the page watches an async install and reloads when the app is back - The app page recognizes MoonBase underneath it from the WebSocket-reconnect path too (a suspended tab restored from memory never re-runs the page-load check) and hands over to MoonBase's page; the overlay gained its own cancel button - update_pct exists only on non-MoonBase devices: the in-place OTA task it reports on never runs where MoonBase installs, and the overlay carries the progress Scripts/MoonDeck: - Olimex ESP32-Gateway pins flashBaud 460800 (its USB bridge dies at 921600, the MoonDeck flash failure); run_qemu's image freshness includes the MoonBase binary; check_esp32_built counts .yml; the test-release workflow runs Python through uv Tests: - unit_PartitionTables declares each table's flash capacity (an inferred capacity would grow with an oversized table and hide the overflow) and its number parser rejects signed and uint32-overflowing values, pinned by its own cases Docs/CI: - Backlog: MoonBase Ethernet marked shipped for classic RMII; static-IP-for-MoonBase filed; "MoonBase as the only update mechanism, all boards" filed with its deciding factors; two bench findings filed (the deviceModel catalog push is not re-applied after a migration, and ethType does not apply live); config-provenance section aligned with the shipped two-level vocabulary; stale helper path fixed - testing.md manifest globs match the release workflow (shared-*.bin); the plan carries the full bench record Reviews: - ๐Ÿ‡ 10 findings: 6 fixed (xTaskCreate failure handling, parser sign/overflow + cases, testing.md globs, backlog path + provenance terminology, workflow uv run), 4 skipped with reasons (action SHA pinning: repo convention is tags and the workflow is temporary; two pre-existing mooninstaller JS findings: untouched by this branch beyond the rename; plan verification section: already distinguishes built from unbuilt; ALLOW_HTTP=n: LAN installs are a documented feature, internet sources stay TLS-verified) Performance: - MoonBase image 773 KB (896 KB slot, 123 KB headroom); install characteristics unchanged (flash-bound ~86 KB/s stream); ESP32 app tick 8375 us at 119 FPS on the bench Olimex, flash within a few KB across variants Co-Authored-By: Claude Opus 5 (1M context) --- docs/backlog/backlog-core.md | 35 +- ...0260826 - MoonBase, a second boot image.md | 11 +- docs/metrics/repo-health.json | 38 +-- docs/metrics/repo-health.md | 28 +- docs/moonmodules/core/system.md | 3 +- docs/testing.md | 2 +- moonbase/main/CMakeLists.txt | 7 +- moonbase/main/moonbase_main.cpp | 304 ++++++++++++++++-- mooninstaller/deviceModels.json | 1 + src/core/FirmwareUpdateModule.h | 11 +- src/ui/app.js | 55 ++-- test/unit/core/unit_PartitionTables.cpp | 16 + 12 files changed, 408 insertions(+), 103 deletions(-) diff --git a/docs/backlog/backlog-core.md b/docs/backlog/backlog-core.md index 6c51926c..db32c9a6 100644 --- a/docs/backlog/backlog-core.md +++ b/docs/backlog/backlog-core.md @@ -146,16 +146,35 @@ of code is fine; a new IDF component is the expensive kind). downgrade visible. - **Hardware diagnostics**: chip/flash/PSRAM identification and a minimal pin tester, for triaging a board that misbehaves under the full app. -- **Ethernet**: MoonBase is WiFi-only today; the eth-only 4 MB variants (`esp32-eth`) fall back - to the AP when no WiFi credentials exist. Needs the per-board PHY/pin config brought over, - which is the real cost. +- **Ethernet: shipped for classic RMII (2026-08-26)**. MoonBase reads the eth wiring from the + same config file as the credentials (ethType gates it) and brings BOTH interfaces up, so the + browser keeps whichever address the app had. Still open here: the P4's IP101/managed-component + PHY and the S3's SPI W5500, which matter only if MoonBase ever goes beyond the 4 MB classics. +- **Static IP for MoonBase**: MoonBase always uses DHCP; a venue network without a DHCP server + (fixed-address rigs exist) would reach the app (static `addressing`) but not MoonBase. Read + the addressing block from the same config scrape when a venue actually asks for it (the app + itself applies static addressing to eth already). - **MoonBase as the only update mechanism, all boards** (PO, 2026-08-26): would delete the app's whole in-place OTA path (a real subtraction) and grow every app slot, with the stronger - power-fail story everywhere. Three deciding factors first: Ethernet in MoonBase (the P4 has no - WiFi of its own), a migration that does not lose config (backup/restore above), and accepting + power-fail story everywhere. Three deciding factors first: MoonBase Ethernet beyond classic RMII (the P4 has no + WiFi of its own and uses a different PHY; the S3 uses SPI W5500), a migration that does not lose config (backup/restore above), and accepting ~45 s of visible downtime per update where dual-OTA installs in the background. Trigger: MoonBase proven in the field on the 4 MB boards. +## Provisioning and live-reconfig gaps (bench, 2026-08-26) + +Both surfaced while bringing up MoonBase Ethernet on the migrated Olimex; neither is MoonBase's. + +- **A migrated device is not fully provisioned until the deviceModel catalog push is re-applied.** + After the MoonBase table migration the board ran for days with `deviceModel` empty and + `ethType` 0: Improv/AP provisioning restores WiFi credentials only, and nothing tells the user + the catalog half (eth wiring, per-board settings) is missing. Candidates: MoonDeck re-pushes + the catalog on discovering a device whose deviceModel is empty, or the UI badges the state. +- **`ethType` does not apply live.** Setting it to 1 over the API (cable in, pins valid) left + Ethernet down; the same config brought it up at the next boot. Every setting applies live is + a core principle (architecture.md, Live reconfiguration), so the eth init path is missing from + the control's apply/build-state sweep. + ## ESP32 performance and memory ### Size estimates for unbuilt features (reference) @@ -395,7 +414,7 @@ Device-model injection over Improv shipped as **"Improv = REST over serial"** (t **Open follow-up: closed-loop APPLY_OP pacing (read-back ack + retry).** The installer paces APPLY_OP frames open-loop (`sendApplyOpFrame` waits a fixed ~120 ms between ops) rather than reading the device's ack back, because a Web Serial duplex read while the writer lock is held is awkward. The delay covers the worst-case single-buffer consume window with headroom, and each op is idempotent (a lost op re-applies cleanly on a re-flash), so this is robust today. The closed-loop upgrade โ€” read the RPC response, retry once on error `0x82` (buffer busy) โ€” removes the fixed delay (faster install) and makes op-loss impossible rather than improbable. Worth doing if a real install is ever observed dropping an op, or when the config push grows large enough that the cumulative fixed delay is noticeable. Touches only `install-orchestrator.js`. -**Open follow-up: shared JS helpers across device-UI and mooninstaller.** `safeLocalGet` / `safeLocalSet` (3-line hostile-storage guards) are duplicated in `src/ui/install-picker.js` (device firmware, embedded as a C string via `embed_ui.cmake`) and `docs/install/devices.js` (web installer page, served from Pages). The two live in different build contexts so the shared extract isn't trivial โ€” it'd need a new `src/ui/safe-storage.js` plus updates to: `embed_ui.cmake` (embed the new file), `ui_embedded.h` generator (new C array), HTTP server file routing (new path served), `release.yml` workflow staging, `preview_installer.py` staging. Five files for one 3-line helper is too much pre-merge. Worth doing when the next shared helper arrives โ€” `relativeTime` and `formatBytes` are candidates. Two helpers earn the build-glue cost; one doesn't. +**Open follow-up: shared JS helpers across device-UI and mooninstaller.** `safeLocalGet` / `safeLocalSet` (3-line hostile-storage guards) are duplicated in `src/ui/install-picker.js` (device firmware, embedded as a C string via `embed_ui.cmake`) and `mooninstaller/devices.js` (web installer page, served from Pages). The two live in different build contexts so the shared extract isn't trivial โ€” it'd need a new `src/ui/safe-storage.js` plus updates to: `embed_ui.cmake` (embed the new file), `ui_embedded.h` generator (new C array), HTTP server file routing (new path served), `release.yml` workflow staging, `preview_installer.py` staging. Five files for one 3-line helper is too much pre-merge. Worth doing when the next shared helper arrives โ€” `relativeTime` and `formatBytes` are candidates. Two helpers earn the build-glue cost; one doesn't. **Open follow-up: P4 Improv scan on a cold WiFi link (bench check).** `improvHandleScan` in `src/platform/esp32/platform_esp32_improv.cpp` calls `esp_wifi_scan_start`, which needs the WiFi driver started. On native ESP32/S3 the driver is up by the time a user provisions; on the P4 the radio lives on the C6 and comes up only after the esp_hosted prelude in `ensureWifiInit()` (triggered by `wifiApInit` / `wifiStaInit`). A scan requested on a P4 that has not initialised WiFi returns an error cleanly rather than scanning a cold link, so nothing crashes. The check: bench-verify whether a P4 provisioned from cold needs the link brought up first, and if so route the scan through the public `wifiAp`/`wifiSta` path. @@ -668,9 +687,9 @@ The second half of the core [multi-device runtime](../architecture.md#multi-devi The light-domain payoff is a wall of controllers animating in lockstep: effects already animate off elapsed time, so feeding them the leader's synced clock instead of each device's local one is the whole change on the render side. Device-to-device light *distribution* is a separate topology question and rides the existing ArtNet / E1.31 / DDP standards rather than a bespoke protocol. -### Three-level device model: MCU โ†’ Board โ†’ Device (config provenance) +### Config provenance: firmware/MCU โ†’ deviceModel (catalog follow-ups) -The model itself is now a shipped design โ€” see architecture.md ยง Config provenance (the three levels + the `txPowerSetting` example + "default only at the level that fixes it"). The catalog that carries it is `mooninstaller/deviceModels.json` (schema). The remaining forward-looking pieces โ€” a `devices.json`/MCU-layer split and annotated-pin images โ€” stay gated by the sequencing rule (no catalog field ahead of a consumer). +The model itself is now a shipped design; see architecture.md ยง Config provenance (deviceModel is the one provenance level, the `txPowerSetting` example, and "default only where the hardware actually fixes it"). The catalog that carries it is `mooninstaller/deviceModels.json` (schema). The remaining forward-looking pieces โ€” a `devices.json`/MCU-layer split and annotated-pin images โ€” stay gated by the sequencing rule (no catalog field ahead of a consumer). ### Persistence overlay: partial-save / schema-change audit (backlog) diff --git a/docs/history/plans/Plan-20260826 - MoonBase, a second boot image.md b/docs/history/plans/Plan-20260826 - MoonBase, a second boot image.md index 2b839414..fd5483c6 100644 --- a/docs/history/plans/Plan-20260826 - MoonBase, a second boot image.md +++ b/docs/history/plans/Plan-20260826 - MoonBase, a second boot image.md @@ -191,8 +191,15 @@ the recovery path. was flash-bound, not network-bound: identical over TLS and plain HTTP, fixed by one bulk erase up front instead of per-sector erases inlined with the writes, plus 32 KB receive chunks; WiFi power save is also off in MoonBase (it throttled RTT 20x for no benefit). - Still open: the power-cut procedure (PO, physical: cut power at ~10/50/95% of an install; - expected: the board boots MoonBase, a retry completes, config survives). + The power-cut procedure then ran (PO): the overlay reports the silence, and once MoonBase + is back it re-submits the install from the payload the browser still holds; the cycle + completes with no clicks. Ethernet shipped after that (classic RMII): MoonBase reads the + eth wiring from the same config file as the credentials, brings BOTH interfaces up so the + browser keeps whichever address the app had, and an install over eth streams at the same + flash-bound rate as WiFi. Bench note from that work: after the table migration the + deviceModel catalog push had never been re-applied (ethType stood at 0), and applying + ethType live did not bring eth up where the boot init did, an app-side observation worth + its own look. 8. **CI and installer** (done): `build_esp32.py` owns the shared layout helpers (moonbase_table_csv / partition_offsets / otadata_slot0_bytes / moonbase_flash_files), the one place that corrects IDF's flasher_args, consumed by the serial flash, the manifests, the diff --git a/docs/metrics/repo-health.json b/docs/metrics/repo-health.json index 5f17a8d7..7bc613bd 100644 --- a/docs/metrics/repo-health.json +++ b/docs/metrics/repo-health.json @@ -1,40 +1,40 @@ { - "commit": "c93a00f1", + "commit": "890853b5", "flash": { - "esp32s3-n16r8": 1832688, + "esp32s3-n16r8": 1832800, "desktop": 1268136, - "esp32": 1789584, + "esp32": 1790208, "esp32p4rev1-eth": 1675216, "esp32p4rev1-eth-wifi": 1933472, "esp32s3-n8r8": 1753232, "esp32s31": 2105072, "esp32-16mb": 1714608, - "esp32-eth": 1397168, - "esp32-wrover": 1843280, - "qemu": 1383328, + "esp32-eth": 1397456, + "esp32-wrover": 1843760, + "qemu": 1383648, "esp32p4rev3-eth": 1643760 }, "perf": { "desktop": { - "tick_us": 220, - "fps": 4545 + "tick_us": 233, + "fps": 4291 }, "esp32": { - "tick_us": 2151, - "fps": 464 + "tick_us": 8375, + "fps": 119 } }, "loc": { - "core": 20373, + "core": 20378, "light": 26080, "platform": 15560, - "ui": 7606, - "test": 47456, + "ui": 7623, + "test": 47472, "moondeck": 22116 }, "comments": { "core": { - "lines": 8054, + "lines": 8057, "ratio": 0.428 }, "light": { @@ -46,11 +46,11 @@ "ratio": 0.388 }, "ui": { - "lines": 2031, + "lines": 2036, "ratio": 0.283 }, "test": { - "lines": 8737, + "lines": 8740, "ratio": 0.211 }, "moondeck": { @@ -59,14 +59,14 @@ } }, "tests": { - "cases": 1584, + "cases": 1585, "scenarios": 23 }, "docs": { "md_files": 197, - "md_lines": 29220, + "md_lines": 29247, "plans_files": 103, - "backlog_lines": 4519, + "backlog_lines": 4538, "lessons_lines": 622, "claude_md_lines": 136 }, diff --git a/docs/metrics/repo-health.md b/docs/metrics/repo-health.md index 0159288f..f2348f66 100644 --- a/docs/metrics/repo-health.md +++ b/docs/metrics/repo-health.md @@ -1,6 +1,6 @@ # Repo health -Measured at `c93a00f1`. Generated by [`moondeck/check/repo_health.py`](../../moondeck/check/repo_health.py) on every KPI-gate run. **Do not edit by hand.** +Measured at `890853b5`. Generated by [`moondeck/check/repo_health.py`](../../moondeck/check/repo_health.py) on every KPI-gate run. **Do not edit by hand.** Current state only; the trend is this file's git history (`git log -p docs/metrics/repo-health.md`). Nothing here fails a build: the numbers make growth visible, the judgment stays human. @@ -11,39 +11,39 @@ Current state only; the trend is this file's git history (`git log -p docs/metri | desktop | 1,238 KB | | esp32 | 1,748 KB (+1 KB) โš  | | esp32-16mb | 1,674 KB | -| esp32-eth | 1,364 KB (+1 KB) โš  | -| esp32-wrover | 1,800 KB (+1 KB) โš  | +| esp32-eth | 1,365 KB (+0 KB) โš  | +| esp32-wrover | 1,801 KB (+0 KB) โš  | | esp32p4rev1-eth | 1,636 KB | | esp32p4rev1-eth-wifi | 1,888 KB | | esp32p4rev3-eth | 1,605 KB | -| esp32s3-n16r8 | 1,790 KB (+1 KB) โš  | +| esp32s3-n16r8 | 1,790 KB (+0 KB) โš  | | esp32s3-n8r8 | 1,712 KB | | esp32s31 | 2,056 KB | -| qemu | 1,351 KB (+1 KB) โš  | +| qemu | 1,351 KB (+0 KB) โš  | ## Render performance | Target | Tick | FPS | |---|---:|---:| -| desktop | 220 ยตs (โˆ’199 ยตs) โœ“ | 4,545 (+2,159) โœ“ | -| esp32 | 2,151 ยตs | 464 | +| desktop | 233 ยตs (+13 ยตs) โš  | 4,291 (โˆ’254) โš  | +| esp32 | 8,375 ยตs (+6,224 ยตs) โš  | 119 (โˆ’345) โš  | ## Code | Area | Lines | Comments | Comment share | |---|---:|---:|---:| -| core | 20,373 | 8,054 | 42.8 % | +| core | 20,378 (+5) โš  | 8,057 | 42.8 % | | light | 26,080 | 10,316 | 43.7 % | | platform | 15,560 | 5,507 | 38.8 % | -| ui | 7,606 (+69) โš  | 2,031 | 28.3 % | -| test | 47,456 (+12) โš  | 8,737 | 21.1 % | -| moondeck | 22,116 (+3) โš  | 3,571 | 18.5 % | +| ui | 7,623 (+17) โš  | 2,036 | 28.3 % | +| test | 47,472 (+16) โš  | 8,740 | 21.1 % | +| moondeck | 22,116 | 3,571 | 18.5 % | ## Tests | Kind | Count | |---|---:| -| unit cases | 1,584 | +| unit cases | 1,585 (+1) โœ“ | | scenarios | 23 | ## Complexity @@ -59,9 +59,9 @@ Current state only; the trend is this file's git history (`git log -p docs/metri | Metric | Value | |---|---:| | markdown files | 197 | -| markdown lines | 29,220 (+18) โš  | +| markdown lines | 29,247 (+27) โš  | | plan files | 103 | -| backlog lines | 4,519 (+6) โš  | +| backlog lines | 4,538 (+19) โš  | | lessons lines | 622 | | CLAUDE.md lines | 136 | diff --git a/docs/moonmodules/core/system.md b/docs/moonmodules/core/system.md index 1ad7aba7..3a4a33a1 100644 --- a/docs/moonmodules/core/system.md +++ b/docs/moonmodules/core/system.md @@ -94,7 +94,8 @@ Over-the-air firmware flashing โ€” the one operation that swaps the binary and n Firmware update module controls - `firmware` โ€” the OTA image to flash. -- read-only: `version`, `build`, `firmwarePartition`, `update_pct` (progress), and on 4 MB +- read-only: `version`, `build`, `firmwarePartition`, `update_pct` (progress; absent on MoonBase + devices, where the update overlay carries the progress instead), and on 4 MB boards `moonbase`: the second boot image is present, so installs run through the reboot-into-MoonBase cycle behind one "updating firmware" overlay, and a **MoonBase** button opens the maintenance image directly ([architecture.md ยง MoonBase](../../architecture.md#moonbase-the-second-boot-image-4-mb-boards)). diff --git a/docs/testing.md b/docs/testing.md index 99a1f142..daf933b2 100644 --- a/docs/testing.md +++ b/docs/testing.md @@ -152,7 +152,7 @@ buildImprovFrame(type=0x03, payload=[0x01]) == 49 4d 50 52 4f 56 01 03 01 01 e The JS suite proves the installer *chunks* an op correctly; the **device side that reassembles those chunks** is pinned by the C++ `unit_ImprovOpReassembler` suite (`src/core/ImprovOpReassembler.h`, the pure state machine behind the device's `APPLY_OP` handler โ€” extracted from `platform_esp32_improv.cpp` so it's desktop-testable). It covers the full receive contract: in-order multi-chunk reassembly + NUL-termination, **duplicate-chunk rejection** and **out-of-order/skipped-seq rejection** (the guard against an installer retry corrupting the buffer), **overflow** rejection at the buffer-minus-NUL boundary, mid-stream `seq 0` abandoning a partial op, and clean recovery after every error. Encode (JS) + reassemble (C++) together prove APPLY_OP end to end without hardware. -**`test/python/test_installer_manifests.py`** (pytest) โ€” pins the web installer's per-release file contract. For every `ships: true` firmware in `mooninstaller/firmwares.json` it runs `moondeck/build/generate_manifest.py` (with a synthetic `flasher_args.json`, so no firmware build is needed) and asserts the manifest is valid (a `chipFamily` + non-empty `parts[]`) AND that **every part filename matches one of the globs the release workflow stages onto Pages** (`firmware-*.bin` / `shared-ota-data.bin` / `partition-table-*.bin`). A manifest that names a file outside those globs points at something the deploy never stages โ†’ the installer 404s at fetch-firmware (the failure that shipped a broken v2.0.0 installer). The test guards the manifest-generation โ†” staged-files contract; the *deploy mechanics* that stage them (per-tag, in `release.yml`) are workflow shell logic a unit test can't reach, so the two are complementary. +**`test/python/test_installer_manifests.py`** (pytest) โ€” pins the web installer's per-release file contract. For every `ships: true` firmware in `mooninstaller/firmwares.json` it runs `moondeck/build/generate_manifest.py` (with a synthetic `flasher_args.json`, so no firmware build is needed) and asserts the manifest is valid (a `chipFamily` + non-empty `parts[]`) AND that **every part filename matches one of the globs the release workflow stages onto Pages** (`firmware-*.bin` / `shared-*.bin` / `partition-table-*.bin` โ€” `shared-*.bin` covers the plain ota-data plus a MoonBase manifest's `shared-ota-data-slot0.bin` and `shared-moonbase-.bin`). A manifest that names a file outside those globs points at something the deploy never stages โ†’ the installer 404s at fetch-firmware (the failure that shipped a broken v2.0.0 installer). The test guards the manifest-generation โ†” staged-files contract; the *deploy mechanics* that stage them (per-tag, in `release.yml`) are workflow shell logic a unit test can't reach, so the two are complementary. **`test/python/test_check_specs_drift.py`** (pytest) โ€” pins the two spec-drift guards in `moondeck/check/check_specs.py` (the spec-check commit gate). Some facts live in both the `.h` and the module doc in different forms: a control's **numeric range** (`addControl("floor", floor, 0, 255)` vs the prose "noise floor (0โ€“255)") and the **author/source URL** (`// Author: โ€ฆ โ€” ` vs the `Origin:` markdown link). Neither can be single-sourced โ€” they're the same fact for two audiences โ€” so instead the gate *validates* them: if the doc restates a control's range and it conflicts with the `.h`, or an `.h` author URL is missing from the doc, the spec check flags it. The checks are block-scoped on the consolidated catalog pages (a control name shared across modules, `fps`/`fadeRate`, matches only its own module's block), and tolerant of the human range spellings (`1โ€“8` / `1-8` / `1 to 8`) โ€” this suite pins both the catch and the no-false-alarm behaviour. diff --git a/moonbase/main/CMakeLists.txt b/moonbase/main/CMakeLists.txt index 69b435b9..4939f6f0 100644 --- a/moonbase/main/CMakeLists.txt +++ b/moonbase/main/CMakeLists.txt @@ -1,9 +1,10 @@ idf_component_register( SRCS "moonbase_main.cpp" INCLUDE_DIRS "" + # The app's own logo, embedded IDF-natively; served as /logo.png (header + favicon). + EMBED_FILES "../../src/ui/moonlight-logo.png" # esp_https_ota brings esp_http_client and esp-tls with it, so neither is listed separately. - # esp_eth is absent on purpose: Ethernet support is a follow-up (the app's ethInit needs - # per-board pin configuration), and requiring the component would link a driver nothing calls. - REQUIRES nvs_flash esp_netif esp_event esp_wifi app_update esp_https_ota + # esp_eth: the RMII bring-up for boards whose config wires a PHY (ethStart). + REQUIRES nvs_flash esp_netif esp_event esp_wifi esp_eth app_update esp_https_ota mbedtls esp_partition joltwallet__littlefs) target_compile_options(${COMPONENT_LIB} PRIVATE -Wall -Wextra) diff --git a/moonbase/main/moonbase_main.cpp b/moonbase/main/moonbase_main.cpp index 68ca72fa..142efbd3 100644 --- a/moonbase/main/moonbase_main.cpp +++ b/moonbase/main/moonbase_main.cpp @@ -30,6 +30,10 @@ #include "esp_partition.h" #include "esp_system.h" #include "esp_wifi.h" +#include "soc/gpio_num.h" +#include "esp_eth.h" +#include "esp_eth_mac_esp.h" +#include "esp_eth_netif_glue.h" #include "freertos/FreeRTOS.h" #include "freertos/event_groups.h" #include "freertos/task.h" @@ -104,8 +108,49 @@ bool jsonFindString(const char* json, const char* key, char* out, size_t outLen) return i > 0; } -// Read the stored WiFi credentials, if there are any. Absent, unreadable or empty all mean the -// same thing to the caller: fall through to the access point. +// Top-level numeric key: "key":123 or "key":-1 (same anchored scan as jsonFindString). +// Absent leaves `out` untouched, so callers pre-load their defaults. +void jsonFindInt(const char* json, const char* key, int* out) { + char needle[40]; + const int n = std::snprintf(needle, sizeof(needle), "\"%s\":", key); + if (n <= 0 || static_cast(n) >= sizeof(needle)) return; + const char* v = std::strstr(json, needle); + if (!v) return; + v += n; + if (*v == '-' || (*v >= '0' && *v <= '9')) *out = std::atoi(v); +} + +void jsonFindBool(const char* json, const char* key, bool* out) { + char needle[40]; + const int n = std::snprintf(needle, sizeof(needle), "\"%s\":", key); + if (n <= 0 || static_cast(n) >= sizeof(needle)) return; + const char* v = std::strstr(json, needle); + if (!v) return; + v += n; + if (std::strncmp(v, "true", 4) == 0) *out = true; + if (std::strncmp(v, "false", 5) == 0) *out = false; +} + +// The board's Ethernet wiring, from the same config file the credentials come from. +// ethType 1 is the app's LAN8720/RMII option, the only interface a 4 MB classic has; +// 0 or absent means no Ethernet on this board. Absent pin keys keep the silicon +// defaults (MDC 23 / MDIO 18; clock IN on GPIO0), the same rule the app applies. +struct { + int type = 0; + int phyAddr = -1; // -1: scan the MDIO bus + int rstGpio = -1; + int mdcGpio = -1; // <0: leave the EMAC default + int mdioGpio = -1; + int clockGpio = 0; + bool clockExtIn = true; +} ethCfg_; + +// The board's WiFi TX cap in dBm (0 = no override): some assemblies brown out at full TX +// (the catalog pins e.g. 8 dBm for them), and a brownout during recovery is the worst time. +int txPowerDbm_ = 0; + +// Read the stored WiFi credentials and Ethernet wiring, if there are any. Absent, unreadable +// or empty all mean the same thing to the caller: fall through the cascade. void loadCredentials() { const char* label = nullptr; for (const auto& c : kFsCandidates) { @@ -124,12 +169,20 @@ void loadCredentials() { // them without holding the whole file (which carries every child module's config too). // The bound is a cross-image contract with NetworkModule's control order; the app pins // it with a unit test (unit_MoonBaseContract). - char buf[1024]; + char buf[2048]; const size_t got = std::fread(buf, 1, sizeof(buf) - 1, f); buf[got] = '\0'; std::fclose(f); jsonFindString(buf, "ssid", ssid_, sizeof(ssid_)); jsonFindString(buf, "password", password_, sizeof(password_)); + jsonFindInt(buf, "ethType", ðCfg_.type); + jsonFindInt(buf, "ethPhyAddr", ðCfg_.phyAddr); + jsonFindInt(buf, "ethRstGpio", ðCfg_.rstGpio); + jsonFindInt(buf, "ethMdcGpio", ðCfg_.mdcGpio); + jsonFindInt(buf, "ethMdioGpio", ðCfg_.mdioGpio); + jsonFindInt(buf, "ethClockGpio", ðCfg_.clockGpio); + jsonFindBool(buf, "ethClockExtIn", ðCfg_.clockExtIn); + jsonFindInt(buf, "txPowerSetting", &txPowerDbm_); } esp_vfs_littlefs_unregister(label); } @@ -141,12 +194,78 @@ void loadCredentials() { void onGotIp(void*, esp_event_base_t, int32_t id, void*) { // Registered for every IP event; only an acquired STA address means online // (IP_EVENT_STA_LOST_IP arrives on the same base and must not set the bit). - if (id == IP_EVENT_STA_GOT_IP) xEventGroupSetBits(netEvents_, kNetGotIp); + if (id == IP_EVENT_STA_GOT_IP || id == IP_EVENT_ETH_GOT_IP) + xEventGroupSetBits(netEvents_, kNetGotIp); } void onWifiEvent(void*, esp_event_base_t, int32_t id, void*) { if (id == WIFI_EVENT_STA_START || id == WIFI_EVENT_STA_DISCONNECTED) esp_wifi_connect(); } +// Bring up the on-chip EMAC with the wiring the app's config names (RMII, the only interface +// a 4 MB classic has; a distilled copy of the app's ethInitEmac). Fire-and-forget by design: +// the driver stays installed even when no link appears in the wait window, so a cable plugged +// in later still gets the device an address; the shared GOT_IP bit reports either interface. +// Returns false only when nothing was configured or a create step failed. +esp_eth_handle_t ethHandle_ = nullptr; +esp_netif_t* ethNetif_ = nullptr; + +bool ethStart() { + if (ethCfg_.type != 1) return false; // 1 = LAN8720/RMII in the app's ethType vocabulary + + esp_netif_config_t netif_cfg = ESP_NETIF_DEFAULT_ETH(); + esp_netif_t* netif = esp_netif_new(&netif_cfg); + if (!netif) return false; + + eth_mac_config_t mac_config = ETH_MAC_DEFAULT_CONFIG(); + eth_esp32_emac_config_t emac_config = ETH_ESP32_EMAC_DEFAULT_CONFIG(); + emac_config.clock_config.rmii.clock_mode = ethCfg_.clockExtIn ? EMAC_CLK_EXT_IN : EMAC_CLK_OUT; + emac_config.clock_config.rmii.clock_gpio = static_cast(ethCfg_.clockGpio); + if (ethCfg_.mdcGpio >= 0) emac_config.smi_gpio.mdc_num = ethCfg_.mdcGpio; + if (ethCfg_.mdioGpio >= 0) emac_config.smi_gpio.mdio_num = ethCfg_.mdioGpio; + + eth_phy_config_t phy_config = ETH_PHY_DEFAULT_CONFIG(); + phy_config.phy_addr = ethCfg_.phyAddr; + phy_config.reset_gpio_num = ethCfg_.rstGpio; + + esp_eth_mac_t* mac = esp_eth_mac_new_esp32(&emac_config, &mac_config); + esp_eth_phy_t* phy = mac ? esp_eth_phy_new_generic(&phy_config) : nullptr; + if (!mac || !phy) { + if (phy) phy->del(phy); + if (mac) mac->del(mac); + esp_netif_destroy(netif); + return false; + } + + esp_eth_config_t eth_config = ETH_DEFAULT_CONFIG(mac, phy); + esp_eth_handle_t handle = nullptr; + if (esp_eth_driver_install(ð_config, &handle) != ESP_OK) { + phy->del(phy); + mac->del(mac); + esp_netif_destroy(netif); + return false; + } + if (esp_netif_attach(netif, esp_eth_new_netif_glue(handle)) != ESP_OK || + esp_eth_start(handle) != ESP_OK) { + esp_eth_driver_uninstall(handle); // frees mac + phy + esp_netif_destroy(netif); + return false; + } + ethHandle_ = handle; + ethNetif_ = netif; + return true; +} + +// Tear Ethernet down again when no lease arrived in its window: like the app, MoonBase runs +// ONE interface at a time, so WiFi only takes over from a dead link, never alongside it. +void ethStop() { + if (!ethHandle_) return; + esp_eth_stop(ethHandle_); + esp_eth_driver_uninstall(ethHandle_); // frees mac + phy + esp_netif_destroy(ethNetif_); + ethHandle_ = nullptr; + ethNetif_ = nullptr; +} + // Try the stored credentials for a bounded time. Returns whether an address arrived. bool wifiStation(uint32_t waitMs) { @@ -171,6 +290,15 @@ bool wifiStation(uint32_t waitMs) { // re-enables it; MoonBase runs for minutes on a powered board, full RX beats // the milliwatts. esp_wifi_set_ps(WIFI_PS_NONE); + // The board's TX cap, applied like the app applies it: only a real cap (1..21 dBm, + // converted to IDF's quarter-dBm), and only AFTER the connection is up. Calling + // esp_wifi_set_max_tx_power at 0 or inside the radio-start call stack hangs the + // classic ESP32 (NetworkModule::syncTxPower documents the boot-loop). The AP + // fallback deliberately skips the cap: a hang in the recovery image outranks a + // possible brownout on a misconfigured-power board. + if (txPowerDbm_ >= 1 && txPowerDbm_ <= 21) { + esp_wifi_set_max_tx_power(static_cast(txPowerDbm_ * 4)); + } return true; } esp_wifi_stop(); @@ -210,22 +338,44 @@ bool wifiAccessPoint() { // The one page MoonBase serves. Inline and tiny: no filesystem read, no compression, no assets. const char kPage[] = "" - "MoonBase" - "" - "

MoonBase

Install firmware to return this device to normal operation.

" + "

MoonBase

" + // The (?) module cards carry, pointing at the published MoonBase doc. + "?
" + "

Install firmware to return this device to normal operation.

" "
From a file
" - "
" + "" + // The last resort when no URL is at hand: name where the firmware--v*.bin files + // live. A plain link, so it works from any device that can reach the internet. + "
Firmware files: github.com/MoonModules/projectMM/releases " + "(the firmware-...bin matching this board)" "
From a URL
" "
" "
Back to the app
Boot the installed firmware without changing it." - "
" + "
" + // Shown only while an install is running (S() toggles it): the one moment cancel applies. + "" "
" ""; // The application slot. From the factory partition esp_ota_get_next_update_partition returns the @@ -259,6 +414,10 @@ const esp_partition_t* appPartition() { // True while any install is writing the app slot. Torn reads are harmless (same display-only // pattern the app uses); the guard only has to stop a SECOND install from starting. volatile bool installing_ = false; +// Set by POST /cancel; the install loops poll it and abort cleanly back to the page. The app +// slot is left half-written, exactly like a power cut: MoonBase stays the boot target until a +// later install completes. +volatile bool cancelRequested_ = false; bool installFromUrl(const char* url) { esp_http_client_config_t http = {}; @@ -296,6 +455,11 @@ bool installFromUrl(const char* url) { } esp_err_t err; while ((err = esp_https_ota_perform(handle)) == ESP_ERR_HTTPS_OTA_IN_PROGRESS) { + if (cancelRequested_) { + esp_https_ota_abort(handle); + std::snprintf(status_, sizeof(status_), "canceled"); + return false; + } std::snprintf(status_, sizeof(status_), "downloading: %d of %d bytes", esp_https_ota_get_image_len_read(handle), esp_https_ota_get_image_size(handle)); @@ -334,11 +498,27 @@ void sendAll(int sock, const char* data, size_t len) { } } +// The embedded logo (EMBED_FILES in CMakeLists; symbol names derive from the filename). +extern const uint8_t logoStart[] asm("_binary_moonlight_logo_png_start"); +extern const uint8_t logoEnd[] asm("_binary_moonlight_logo_png_end"); + +void sendBinary(int sock, const char* type, const uint8_t* data, size_t len) { + char head[192]; + const int n = std::snprintf(head, sizeof(head), + "HTTP/1.1 200 OK\r\nContent-Type: %s\r\nContent-Length: %u\r\n" + "Cache-Control: no-store\r\nConnection: close\r\n\r\n", + type, static_cast(len)); + if (n > 0) sendAll(sock, head, static_cast(n)); + sendAll(sock, reinterpret_cast(data), len); +} + void sendResponse(int sock, const char* status, const char* type, const char* body) { - char head[160]; + // no-store on everything: this address serves TWO different UIs over time (the app's and + // this one), and a browser that re-serves a cached copy of either shows a dead page. + char head[192]; const int n = std::snprintf(head, sizeof(head), "HTTP/1.1 %s\r\nContent-Type: %s\r\nContent-Length: %u\r\n" - "Connection: close\r\n\r\n", + "Cache-Control: no-store\r\nConnection: close\r\n\r\n", status, type, static_cast(std::strlen(body))); if (n > 0) sendAll(sock, head, static_cast(n)); sendAll(sock, body, std::strlen(body)); @@ -417,14 +597,24 @@ bool installFromSocketLocked(int sock, const char* prefix, size_t prefixLen, siz char stagedUrlTask_[256]; void unattendedInstallTask(void*) { - for (int attempt = 0; attempt < 3; attempt++) { + // Remember the source across reboots (key "last_url", page prefill only): the retry + // escape must survive a power cycle, not just this session. + nvs_handle_t nh; + if (nvs_open("moonbase", NVS_READWRITE, &nh) == ESP_OK) { + nvs_set_str(nh, "last_url", stagedUrlTask_); + nvs_commit(nh); + nvs_close(nh); + } + for (int attempt = 0; attempt < 3 && !cancelRequested_; attempt++) { if (attempt) vTaskDelay(pdMS_TO_TICKS(3000)); if (installFromUrl(stagedUrlTask_)) esp_restart(); // straight back into the new app // A failed attempt leaves its error in status_; while retries remain that error is // TRANSIENT, and a watcher treating "error:" as terminal (the app's overlay does) // must not see it. The final attempt's error stays as the terminal answer. - if (attempt < 2) std::snprintf(status_, sizeof(status_), "download failed, retrying"); + if (attempt < 2 && !cancelRequested_) + std::snprintf(status_, sizeof(status_), "download failed, retrying"); } + cancelRequested_ = false; installing_ = false; // set by the spawner; held across the retries vTaskDelete(nullptr); } @@ -472,10 +662,17 @@ void serveOne(int sock) { stagedUrlTask_[n] = '\0'; std::snprintf(status_, sizeof(status_), "starting the install"); installing_ = true; // cleared by the task after its final attempt - xTaskCreate(unattendedInstallTask, "mb_install", 12288, nullptr, 5, nullptr); - // 202: the install runs on its own task while this server keeps answering GET - // /moonbase with live progress; the caller watches that, not this response. - sendResponse(sock, "202 Accepted", "text/plain", status_); + if (xTaskCreate(unattendedInstallTask, "mb_install", 12288, nullptr, 5, nullptr) != pdPASS) { + // A failed spawn with the flag left set would refuse every later install: THE + // deadlock this guard exists to prevent. + installing_ = false; + std::snprintf(status_, sizeof(status_), "error: cannot start the install task"); + sendResponse(sock, "500 Internal Server Error", "text/plain", status_); + } else { + // 202: the install runs on its own task while this server keeps answering GET + // /moonbase with live progress; the caller watches that, not this response. + sendResponse(sock, "202 Accepted", "text/plain", status_); + } } else if (std::strncmp(head, "POST /install", 13) == 0) { if (installing_) { sendResponse(sock, "409 Conflict", "text/plain", "error: an install is already running"); @@ -498,6 +695,23 @@ void serveOne(int sock) { else std::snprintf(status_, sizeof(status_), "error: no valid app image"); sendResponse(sock, ok ? "200 OK" : "500 Internal Server Error", "text/plain", status_); installed = ok; // reuse the reply-then-restart tail below + } else if (std::strncmp(head, "GET /logo.png", 13) == 0) { + sendBinary(sock, "image/png", logoStart, static_cast(logoEnd - logoStart)); + } else if (std::strncmp(head, "GET /last-url", 13) == 0) { + // The most recent install source, RAM-held: the page prefills its URL field with it, + // so Install doubles as retry, the escape after a cancel wiped the app slot. Empty + // after a power cycle. + sendResponse(sock, "200 OK", "text/plain", stagedUrlTask_); + } else if (std::strncmp(head, "POST /cancel", 12) == 0) { + // Cancel a running URL install: its loop polls the flag and aborts back to this page. + // (An upload cancels by dropping the connection; this server is busy receiving it.) + // Nothing to cancel is not an error worth a scary status, just say so. + if (installing_) { + cancelRequested_ = true; + sendResponse(sock, "200 OK", "text/plain", "canceling"); + } else { + sendResponse(sock, "200 OK", "text/plain", "nothing to cancel"); + } } else if (std::strncmp(head, "GET /moonbase", 13) == 0) { // Identity probe: the app UI polls this across the update cycle to tell which image is // answering at the shared address (the app 404s it). Body = the live install status, so @@ -560,10 +774,10 @@ extern "C" void app_main() { loadCredentials(); - // WiFi STA with the app's stored credentials, else the open access point: the guarantee - // that a board is never unreachable because its credentials went stale. Ethernet is a - // follow-up (backlog-core, MoonBase follow-ups): it needs per-board PHY/pin configuration, - // so today the eth-only esp32-eth variant lands on the access point here. + // The cascade: Ethernet where the config wires it (its DHCP window overlaps the WiFi + // join since the GOT_IP bit is shared), then WiFi STA with the stored credentials, then + // the open access point: the guarantee that a board is never unreachable because its + // credentials went stale. // The unattended handoff: the app may have staged an install URL in NVS before rebooting // into MoonBase (platform::moonbaseStageInstallUrl). Read AND erase it unconditionally, // before anything can fail: a URL that crashes or fails can then never boot-loop the @@ -581,7 +795,28 @@ extern "C" void app_main() { } } - bool online = wifiStation(20000); + // With nothing staged, prefill the retry buffer from the remembered last source so the + // page offers it after any reboot. Never auto-installed: only the page's Install uses it. + if (!stagedUrl[0]) { + nvs_handle_t h; + if (nvs_open("moonbase", NVS_READONLY, &h) == ESP_OK) { + size_t len = sizeof(stagedUrlTask_); + if (nvs_get_str(h, "last_url", stagedUrlTask_, &len) != ESP_OK) stagedUrlTask_[0] = '\0'; + nvs_close(h); + } + } + + // ONE interface at a time, in the app's own preference order (eth where configured, else + // WiFi, else the AP): the app runs a single interface, so the browser is on that + // interface's address, and mirroring the preference is what keeps the address valid + // across the handoff without a second lease to confuse anyone. + bool online = false; + if (ethStart()) { + online = (xEventGroupWaitBits(netEvents_, kNetGotIp, pdFALSE, pdFALSE, + pdMS_TO_TICKS(8000)) & kNetGotIp) != 0; + if (!online) ethStop(); // no link or no lease: WiFi takes over, alone + } + if (!online) online = wifiStation(20000); // STA only: on the fallback AP the URL's network is not reachable, and a user is present. // The install runs on its OWN task so the main task serves throughout: GET /moonbase then @@ -593,7 +828,10 @@ extern "C" void app_main() { std::snprintf(status_, sizeof(status_), "preparing the install"); std::snprintf(stagedUrlTask_, sizeof(stagedUrlTask_), "%s", stagedUrl); installing_ = true; // cleared by the task after its final attempt - xTaskCreate(unattendedInstallTask, "mb_install", 12288, nullptr, 5, nullptr); + if (xTaskCreate(unattendedInstallTask, "mb_install", 12288, nullptr, 5, nullptr) != pdPASS) { + installing_ = false; // a latched flag would refuse every later install + std::snprintf(status_, sizeof(status_), "error: cannot start the install task"); + } } if (!online) online = wifiAccessPoint(); diff --git a/mooninstaller/deviceModels.json b/mooninstaller/deviceModels.json index c50cedec..8d8df58d 100644 --- a/mooninstaller/deviceModels.json +++ b/mooninstaller/deviceModels.json @@ -2,6 +2,7 @@ { "name": "Olimex ESP32-Gateway Rev G", "chip": "ESP32", + "flashBaud": 460800, "firmwares": [ "esp32", "esp32-eth" diff --git a/src/core/FirmwareUpdateModule.h b/src/core/FirmwareUpdateModule.h index 6f833dbb..e54cbe1d 100644 --- a/src/core/FirmwareUpdateModule.h +++ b/src/core/FirmwareUpdateModule.h @@ -163,7 +163,12 @@ class FirmwareUpdateModule : public MoonModule { // (via markDirty โ†’ HttpServerModule rebuildControls) when totalSnap_ // changes. Initially 0; the UI shows "0KB / 0KB" until esp_https_ota // reports the image size, then "X KB / 1297KB" for the rest. - controls_.addProgress("update_pct", bytesRead_, totalSnap_); + // On a MoonBase device the in-place OTA task this control reports on never runs + // (installs happen in MoonBase; the update overlay carries the progress), so the + // control does not exist there. + if (!platform::otaHasMoonBase()) { + controls_.addProgress("update_pct", bytesRead_, totalSnap_); + } } void tick1s() MM_NONBLOCKING override { @@ -179,9 +184,9 @@ class FirmwareUpdateModule : public MoonModule { // estimate until the new task reports the new size). rebuildControls // re-runs defineControls() so the addProgress' captured `aux` (total) // is refreshed to the new totalSnap_ value. - if (g_otaBytesTotal != totalSnap_) { + if (g_otaBytesTotal != totalSnap_ && !platform::otaHasMoonBase()) { totalSnap_ = g_otaBytesTotal; - rebuildControls(); + rebuildControls(); // refresh update_pct's captured total; absent on MoonBase devices } } diff --git a/src/ui/app.js b/src/ui/app.js index 65bd00f0..93fd9261 100644 --- a/src/ui/app.js +++ b/src/ui/app.js @@ -227,8 +227,15 @@ function connectWs() { if (sock !== ws) return; // a stale socket closing after we already moved on โ€” leave the live one alone clearInterval(wsHeartbeat); wsHeartbeat = null; - if (wsUnloading) return; // the page is going away โ€” don't reconnect (and don't touch the DOM) + if (wsUnloading) return; // the page is going away, don't reconnect (and don't touch the DOM) setWsDot(false); + // A long-dead socket may mean the device switched images under this tab (a suspended + // tab restored from memory never re-runs the page-load check). Once the backoff has + // ripened, ask; the update overlay owns that conversation during an install, so skip + // while it is up. + if (wsRetryMs >= 5000 && !document.querySelector(".fw-overlay")) { + moonbaseHandoff(); // async; reconnects continue unless it flips the page + } // Exponential backoff with 5s ceiling; track the timer so pagehide can cancel a pending reconnect. wsReconnectTimer = setTimeout(connectWs, wsRetryMs); wsRetryMs = Math.min(wsRetryMs * 2, 5000); @@ -330,22 +337,7 @@ async function init() { // REST snapshot overwrite the newer, live WS state โ€” just skip the commit. try { const resp = await fetch("/api/state"); - if (resp.status === 404) { - // A 404 on the app's own state API means this page is a CACHED copy of the app - // talking to a different server at the same address: MoonBase. Confirm, then hand - // over cleanly instead of rendering a half-dead skeleton โ€” the cache-busting query - // makes the browser fetch MoonBase's page instead of re-serving this one. - const mb = await probeMoonBase(); - if (mb.state === "up") { - document.body.innerHTML = ""; - const note = document.createElement("p"); - note.style.cssText = "font:16px system-ui;margin:3rem auto;max-width:30rem;text-align:center"; - note.textContent = "This device is in MoonBase (maintenance mode) \u2014 opening its page\u2026"; - document.body.appendChild(note); - setTimeout(() => location.replace("/?" + Date.now()), 1200); - return; - } - } + if (resp.status === 404 && await moonbaseHandoff()) return; if (resp.ok && (!state || !Array.isArray(state.modules))) { const snap = await resp.json(); if (snap && Array.isArray(snap.modules)) { @@ -1155,13 +1147,20 @@ function showUpdateOverlay() { dismiss.textContent = "close"; dismiss.style.display = "none"; dismiss.addEventListener("click", () => ov.remove()); - box.append(h, msg, bar, dismiss); + const cancel = document.createElement("button"); + cancel.textContent = "cancel install"; + cancel.addEventListener("click", () => { + // Best-effort: only a running URL install can hear it (MoonBase's /cancel); the watch + // loop sees the resulting "canceled" status and ends the overlay from there. + fetch("/cancel", { method: "POST" }).catch(() => {}); + }); + box.append(h, msg, bar, cancel, dismiss); ov.appendChild(box); document.body.appendChild(ov); return { status(text) { msg.textContent = text; }, progress(read, total) { if (total > 0) { bar.max = total; bar.value = read; } }, - fail(text) { msg.textContent = text; bar.remove(); dismiss.style.display = ""; }, + fail(text) { msg.textContent = text; bar.remove(); cancel.remove(); dismiss.style.display = ""; }, }; } @@ -1185,6 +1184,20 @@ async function probeMoonBase() { } } +// When this app page finds MoonBase answering underneath it (a cached or suspended tab over +// a device that switched images), say so and load MoonBase's real page: the cache-busting +// query makes the browser fetch it instead of re-serving this one. True when the handoff ran. +async function moonbaseHandoff() { + if ((await probeMoonBase()).state !== "up") return false; + document.body.innerHTML = ""; + const note = document.createElement("p"); + note.style.cssText = "font:16px system-ui;margin:3rem auto;max-width:30rem;text-align:center"; + note.textContent = "This device is in MoonBase (maintenance mode) \u2014 opening its page\u2026"; + document.body.appendChild(note); + setTimeout(() => location.replace("/?" + Date.now()), 1200); + return true; +} + // The one-click cycle. opts is {url} (device installs it unattended off the NVS-staged URL) or // {file} (the browser holds the image and pushes it to MoonBase once MoonBase answers). const MOONBASE_SILENT_MSG = "MoonBase did not answer. If its WiFi fell back, join the " + @@ -1275,6 +1288,10 @@ async function moonbaseUpdateFlow(opts) { } catch (_) {} } else if (probe.state === "up") { sawAway = true; // MoonBase answering means the old app is gone + if (probe.text === "canceled") { + ui.fail("Install canceled \u2014 the device is waiting in MoonBase."); + return; + } if (probe.text.startsWith("error")) throw new Error(probe.text); // The unattended install reports "downloading: N of M bytes"; render it the // way the file path reads, with a real bar. diff --git a/test/unit/core/unit_PartitionTables.cpp b/test/unit/core/unit_PartitionTables.cpp index d461a7d6..af55f87a 100644 --- a/test/unit/core/unit_PartitionTables.cpp +++ b/test/unit/core/unit_PartitionTables.cpp @@ -61,10 +61,15 @@ bool parseNumber(std::string tok, uint32_t& out) { if (suffix == 'K' || suffix == 'k') { mult = 1024; tok.pop_back(); } else if (suffix == 'M' || suffix == 'm') { mult = 1024 * 1024; tok.pop_back(); } if (tok.empty()) return false; + // strtoul silently wraps a negative token; a partition offset or size is never signed. + if (tok.front() == '-' || tok.front() == '+') return false; errno = 0; char* end = nullptr; const unsigned long v = std::strtoul(tok.c_str(), &end, 0); // base 0: 0x.. is hex if (errno != 0 || end == tok.c_str() || *end != '\0') return false; + // The K/M multiply (and the plain value) must fit uint32: 4096M or 0x100000000 is a typo, + // not a 4 GB partition. + if (v > UINT32_MAX / mult) return false; out = static_cast(v) * mult; return true; } @@ -113,6 +118,17 @@ std::vector
allTables() { } // namespace +TEST_CASE("the CSV number parser rejects signed and uint32-overflowing values") { + uint32_t v = 0; + CHECK_FALSE(parseNumber("-1", v)); // strtoul would wrap this to 4 GB - 1 + CHECK_FALSE(parseNumber("4096M", v)); // 4 GiB: past uint32 after the suffix multiply + CHECK_FALSE(parseNumber("0x100000000", v)); // past uint32 as a plain value + CHECK(parseNumber("0x3E9000", v)); + CHECK(v == 0x3E9000u); + CHECK(parseNumber("512K", v)); + CHECK(v == 512u * 1024u); +} + TEST_CASE("every partition table describes a layout that fits its flash without overlaps") { const auto tables = allTables(); REQUIRE(tables.size() >= 3); // esp32dev, esp32s3_n8r8, ota_16mb at time of writing From f487b339e1f644542c86c52bd7552a700ae27929 Mon Sep 17 00:00:00 2001 From: ewowi Date: Wed, 26 Aug 2026 22:31:30 +0200 Subject: [PATCH 6/6] Close the pre-merge review rounds; drop the temporary test-release CI MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Final polish from the pre-merge reviews: an interrupted staging can no longer auto-install weeks later, cancel now reaches file uploads too, and MoonBase's page refuses over-long URLs instead of silently truncating them. The temporary test-release workflow is deleted on the branch, so it never reaches main. KPI: 16384lights | Desktop:1238KB | tick:132/95/3/5/131/286/21/3/285/72/18/23/10/128/23/6/503/47/4us(FPS:7575/10526/333333/200000/7633/3496/47619/333333/3508/13888/55555/43478/100000/7812/43478/166666/1988/21276/250000) | ESP32:1634KB | tick:8346us(FPS:119) | heap:136KB | src:226(62829) | test:171(38506) | lizard:173w Core: - A staged install URL surviving a power cut between staging and the boot switch is defused: the boot-to-MoonBase route (nothing-staged by definition) erases it via the new platform::moonbaseClearStagedUrl(), so it can never auto-install on a later unrelated visit - The cancel flag resets wherever an install begins, closing the race where a /cancel landing on a task's exit would poison the next install; a late Ethernet lease racing the eth teardown no longer satisfies the WiFi wait (shared GOT_IP bit cleared after ethStop) - MoonBase's URL form refuses >255 bytes (the app route's contract) instead of silently truncating into a misleading download error - The esp32dev_moonbase.csv header keeps only table-specific facts; the design story lives once, in architecture.md ยง MoonBase UI: - The overlay's cancel reaches file uploads: it aborts the in-flight fetch (dropping the connection is MoonBase's upload-cancel contract) and reports a clean "Install canceled" instead of a masked upload error Scripts/MoonDeck: - One fragment-to-partition-table resolver (table_from_fragments), shared by moonbase_table_csv and the build-dir staleness check, which as a result now guards EVERY firmware's partition table (verified against all 12); the hardcoded slot-size print is gone (IDF's own "Smallest app partition" line is the authority) - The temporary moonbase-test-release.yml is deleted before the merge; the moonbase-test release and tag on GitHub are removed after it (gh release delete moonbase-test --cleanup-tag) Tests: - unit_MoonBaseContract follows the widened config scrape: pins every scraped key inside the 2048-byte prefix read, with a stated budget for the ESP32-only eth block a desktop build cannot serialize - test_flash_baud updated to the new catalog reality (the Olimex opt-down protects unidentified esp32-eth flashes; esp32s3-n16r8 is the no-opt-down fast case), the CI failure on the previous push Docs/CI: - Backlog and plan de-contradicted to the shipped one-interface-at-a-time behavior; the plan records the test-workflow lifecycle and the PR description was refreshed from it Reviews: - ๐Ÿ‘พ pre-merge branch review, 8 findings, all fixed: contract-test drift (1), doc/code contradiction (2), stale staged URL (3), cancel-flag latch (4), CSV header duplication (5), duplicated fragment resolver (6), hardcoded slot print (7), silent URL truncation (8) - ๐Ÿ‡ final round, 6 findings: 2 fixed (late-lease bit clear; file-upload abort), 4 skipped with reasons (shared-glob is the deliberate three-way contract; the config scraper targets its single producer's compact output; eth teardown mirrors the app's field-proven order; empty CSV offsets already fail naming file and partition) Performance: - No behavior change on the tick path; MoonBase image 773 KB; desktop and ESP32 numbers within noise of the previous commit Co-Authored-By: Claude Opus 5 (1M context) --- .github/workflows/moonbase-test-release.yml | 96 ------------------- docs/backlog/backlog-core.md | 7 +- ...0260826 - MoonBase, a second boot image.md | 14 +-- docs/metrics/repo-health.json | 34 +++---- docs/metrics/repo-health.md | 34 +++---- esp32/partitions/esp32dev_moonbase.csv | 33 ++----- moonbase/main/moonbase_main.cpp | 43 +++++---- moondeck/build/build_esp32.py | 26 ++--- src/core/HttpServerModule.cpp | 3 + src/platform/desktop/platform_desktop.cpp | 1 + src/platform/esp32/platform_esp32_ota.cpp | 8 ++ src/platform/platform.h | 3 + src/ui/app.js | 19 +++- test/python/test_flash_baud.py | 5 +- test/unit/core/unit_MoonBaseContract.cpp | 46 ++++++--- 15 files changed, 160 insertions(+), 212 deletions(-) delete mode 100644 .github/workflows/moonbase-test-release.yml diff --git a/.github/workflows/moonbase-test-release.yml b/.github/workflows/moonbase-test-release.yml deleted file mode 100644 index ce55a733..00000000 --- a/.github/workflows/moonbase-test-release.yml +++ /dev/null @@ -1,96 +0,0 @@ -name: MoonBase test release - -# TEMPORARY: delete after the MoonBase branch merges to main. -# Publishes a `moonbase-test` prerelease from THIS branch so the picker's URL-based -# install can be tested against real GitHub release assets before the merge (the -# real release pipeline only runs from main and version tags). Triggered by a push of -# this branch: workflow_dispatch alone cannot work here, GitHub only registers a -# dispatchable workflow from the default branch. esp32 (the bench Olimex) is the only -# firmware built. Mirrors release.yml's esp32 -# job + staging + manifest, reduced to one variant and one fixed tag. - -on: - workflow_dispatch: - push: - branches: - - moonbase - -permissions: - contents: write - -concurrency: - group: moonbase-test-release - cancel-in-progress: false - -jobs: - build-and-release: - runs-on: ubuntu-latest - steps: - - uses: actions/checkout@v4 - with: - persist-credentials: false - fetch-depth: 0 # compute_version.py counts commits since the last v* tag - - - name: Cache ESP-IDF tooling - uses: actions/cache@v4 - with: - path: | - ~/.espressif - ~/esp/esp-idf - key: esp-idf-v6.1-rc1-${{ runner.os }}-v1 - - - uses: astral-sh/setup-uv@v3 - - - name: Compute version - id: ver - # `latest`-channel versioning (-dev.) so the version is a real, - # orderable semver the picker can compare; the release channel burned into - # the binary is `moonbase-test` so a device shows where the image came from. - run: | - set -euo pipefail - V=$(uv run python moondeck/build/compute_version.py --tag latest) - echo "version=$V" >> "$GITHUB_OUTPUT" - - - name: Build firmware - uses: espressif/esp-idf-ci-action@v1 - with: - esp_idf_version: v6.1-rc1 - target: esp32 - path: 'esp32' - command: python ../moondeck/build/build_esp32.py --firmware esp32 --release "moonbase-test" --version "${{ steps.ver.outputs.version }}" - - - name: Stage assets + manifest - run: | - set -euo pipefail - mkdir -p dist - V="${{ steps.ver.outputs.version }}" - B=build/esp32-esp32 - PREFIX="firmware-esp32-v$V" - cp "$B/projectMM.bin" "dist/${PREFIX}.bin" - cp "$B/bootloader/bootloader.bin" "dist/${PREFIX}-bootloader.bin" - SIZE=$(jq -r .flash_settings.flash_size "$B/flasher_args.json" | tr 'A-Z' 'a-z') - cp "$B/partition_table/partition-table.bin" "dist/partition-table-$SIZE.bin" - cp "$B/ota_data_initial.bin" "dist/shared-ota-data.bin" - cp build/moonbase-esp32/projectMM-moonbase.bin dist/shared-moonbase-esp32.bin - uv run python -c "import sys; sys.path.insert(0,'moondeck/build'); \ - import build_esp32, pathlib; \ - pathlib.Path('dist/shared-ota-data-slot0.bin').write_bytes(build_esp32.otadata_slot0_bytes())" - uv run python moondeck/build/generate_manifest.py --firmware esp32 --version "$V" \ - --release-url "https://github.com/${GITHUB_REPOSITORY}/releases/download/moonbase-test" \ - --flasher-args "$B/flasher_args.json" --out dist/manifest-esp32.json - - - name: Publish moonbase-test prerelease - env: - GH_TOKEN: ${{ secrets.GITHUB_TOKEN }} - run: | - set -euo pipefail - V="${{ steps.ver.outputs.version }}" - # Replace any previous test release; keep-tag deletion semantics don't - # matter here: the tag is repointed to the current commit on create. - gh release delete moonbase-test --repo "$GITHUB_REPOSITORY" --yes --cleanup-tag || true - # The release NAME is the version string: the picker machine-reads it the - # same way it reads the real `latest` prerelease (no "v" prefix). - gh release create moonbase-test dist/* \ - --repo "$GITHUB_REPOSITORY" --prerelease --title "$V" \ - --target "$GITHUB_SHA" \ - --notes "Temporary MoonBase URL-install test release from the moonbase branch. Do not install on devices you care about; superseded by the next real release." diff --git a/docs/backlog/backlog-core.md b/docs/backlog/backlog-core.md index db32c9a6..fdcb1d12 100644 --- a/docs/backlog/backlog-core.md +++ b/docs/backlog/backlog-core.md @@ -147,9 +147,10 @@ of code is fine; a new IDF component is the expensive kind). - **Hardware diagnostics**: chip/flash/PSRAM identification and a minimal pin tester, for triaging a board that misbehaves under the full app. - **Ethernet: shipped for classic RMII (2026-08-26)**. MoonBase reads the eth wiring from the - same config file as the credentials (ethType gates it) and brings BOTH interfaces up, so the - browser keeps whichever address the app had. Still open here: the P4's IP101/managed-component - PHY and the S3's SPI W5500, which matter only if MoonBase ever goes beyond the 4 MB classics. + same config file as the credentials (ethType gates it) and runs ONE interface at a time in + the app's own preference order (eth, else WiFi, else AP), so the browser keeps the address + the app had. Still open here: the P4's IP101/managed-component PHY and the S3's SPI W5500, + which matter only if MoonBase ever goes beyond the 4 MB classics. - **Static IP for MoonBase**: MoonBase always uses DHCP; a venue network without a DHCP server (fixed-address rigs exist) would reach the app (static `addressing`) but not MoonBase. Read the addressing block from the same config scrape when a venue actually asks for it (the app diff --git a/docs/history/plans/Plan-20260826 - MoonBase, a second boot image.md b/docs/history/plans/Plan-20260826 - MoonBase, a second boot image.md index fd5483c6..67bfd4f9 100644 --- a/docs/history/plans/Plan-20260826 - MoonBase, a second boot image.md +++ b/docs/history/plans/Plan-20260826 - MoonBase, a second boot image.md @@ -194,9 +194,9 @@ the recovery path. The power-cut procedure then ran (PO): the overlay reports the silence, and once MoonBase is back it re-submits the install from the payload the browser still holds; the cycle completes with no clicks. Ethernet shipped after that (classic RMII): MoonBase reads the - eth wiring from the same config file as the credentials, brings BOTH interfaces up so the - browser keeps whichever address the app had, and an install over eth streams at the same - flash-bound rate as WiFi. Bench note from that work: after the table migration the + eth wiring from the same config file as the credentials, runs ONE interface at a time in the + app's own preference order (so the browser keeps the address the app had, PO decision), and + an install over eth streams at the same flash-bound rate as WiFi. Bench note from that work: after the table migration the deviceModel catalog push had never been re-applied (ethType stood at 0), and applying ethType live did not bring eth up where the boot init did, an app-side observation worth its own look. @@ -208,9 +208,11 @@ the recovery path. shared-moonbase-.bin + shared-ota-data-slot0.bin; install-picker rejects both (pinned by a JS test); check_esp32_built also gates the MoonBase image's freshness. A temporary `moonbase-test-release.yml` workflow (push-triggered on this branch, since GitHub - only registers a dispatchable workflow from the default branch; esp32 only) publishes a - `moonbase-test` prerelease from this branch so the picker's URL install can be tested against - real GitHub assets before the merge; it is deleted afterwards. + only registers a dispatchable workflow from the default branch; esp32 only) published a + `moonbase-test` prerelease from this branch so the picker's URL install could be tested + against real GitHub assets; it was deleted again before the merge, so it never reaches main. + The `moonbase-test` release and tag on GitHub are deleted after the merge + (`gh release delete moonbase-test --cleanup-tag`). 9. **Migration and docs** (done): architecture.md ยง MoonBase is the concept's one home; README feature bullet credits Tasmota's safeboot and MycilaSafeBoot; building.md notes the one-pass 4 MB flash; MIGRATING.md carries the erase-flash entry; the FirmwareUpdate catalog diff --git a/docs/metrics/repo-health.json b/docs/metrics/repo-health.json index 7bc613bd..2947e6e9 100644 --- a/docs/metrics/repo-health.json +++ b/docs/metrics/repo-health.json @@ -1,9 +1,9 @@ { - "commit": "890853b5", + "commit": "b6592232", "flash": { "esp32s3-n16r8": 1832800, "desktop": 1268136, - "esp32": 1790208, + "esp32": 1790416, "esp32p4rev1-eth": 1675216, "esp32p4rev1-eth-wifi": 1933472, "esp32s3-n8r8": 1753232, @@ -16,25 +16,25 @@ }, "perf": { "desktop": { - "tick_us": 233, - "fps": 4291 + "tick_us": 132, + "fps": 7575 }, "esp32": { - "tick_us": 8375, + "tick_us": 8346, "fps": 119 } }, "loc": { - "core": 20378, + "core": 20381, "light": 26080, - "platform": 15560, - "ui": 7623, - "test": 47472, + "platform": 15572, + "ui": 7636, + "test": 47495, "moondeck": 22116 }, "comments": { "core": { - "lines": 8057, + "lines": 8059, "ratio": 0.428 }, "light": { @@ -42,19 +42,19 @@ "ratio": 0.437 }, "platform": { - "lines": 5507, + "lines": 5509, "ratio": 0.388 }, "ui": { - "lines": 2036, + "lines": 2038, "ratio": 0.283 }, "test": { - "lines": 8740, + "lines": 8752, "ratio": 0.211 }, "moondeck": { - "lines": 3571, + "lines": 3573, "ratio": 0.185 } }, @@ -64,14 +64,14 @@ }, "docs": { "md_files": 197, - "md_lines": 29247, + "md_lines": 29250, "plans_files": 103, - "backlog_lines": 4538, + "backlog_lines": 4539, "lessons_lines": 622, "claude_md_lines": 136 }, "complexity": { - "functions": 2776, + "functions": 2778, "over_threshold": 173, "worst_ccn": 108 } diff --git a/docs/metrics/repo-health.md b/docs/metrics/repo-health.md index f2348f66..b044e7b1 100644 --- a/docs/metrics/repo-health.md +++ b/docs/metrics/repo-health.md @@ -1,6 +1,6 @@ # Repo health -Measured at `890853b5`. Generated by [`moondeck/check/repo_health.py`](../../moondeck/check/repo_health.py) on every KPI-gate run. **Do not edit by hand.** +Measured at `b6592232`. Generated by [`moondeck/check/repo_health.py`](../../moondeck/check/repo_health.py) on every KPI-gate run. **Do not edit by hand.** Current state only; the trend is this file's git history (`git log -p docs/metrics/repo-health.md`). Nothing here fails a build: the numbers make growth visible, the judgment stays human. @@ -9,48 +9,48 @@ Current state only; the trend is this file's git history (`git log -p docs/metri | Target | Flash | |---|---:| | desktop | 1,238 KB | -| esp32 | 1,748 KB (+1 KB) โš  | +| esp32 | 1,748 KB (+0 KB) โš  | | esp32-16mb | 1,674 KB | -| esp32-eth | 1,365 KB (+0 KB) โš  | -| esp32-wrover | 1,801 KB (+0 KB) โš  | +| esp32-eth | 1,365 KB | +| esp32-wrover | 1,801 KB | | esp32p4rev1-eth | 1,636 KB | | esp32p4rev1-eth-wifi | 1,888 KB | | esp32p4rev3-eth | 1,605 KB | -| esp32s3-n16r8 | 1,790 KB (+0 KB) โš  | +| esp32s3-n16r8 | 1,790 KB | | esp32s3-n8r8 | 1,712 KB | | esp32s31 | 2,056 KB | -| qemu | 1,351 KB (+0 KB) โš  | +| qemu | 1,351 KB | ## Render performance | Target | Tick | FPS | |---|---:|---:| -| desktop | 233 ยตs (+13 ยตs) โš  | 4,291 (โˆ’254) โš  | -| esp32 | 8,375 ยตs (+6,224 ยตs) โš  | 119 (โˆ’345) โš  | +| desktop | 132 ยตs (โˆ’101 ยตs) โœ“ | 7,575 (+3,284) โœ“ | +| esp32 | 8,346 ยตs (โˆ’29 ยตs) โœ“ | 119 | ## Code | Area | Lines | Comments | Comment share | |---|---:|---:|---:| -| core | 20,378 (+5) โš  | 8,057 | 42.8 % | +| core | 20,381 (+3) โš  | 8,059 | 42.8 % | | light | 26,080 | 10,316 | 43.7 % | -| platform | 15,560 | 5,507 | 38.8 % | -| ui | 7,623 (+17) โš  | 2,036 | 28.3 % | -| test | 47,472 (+16) โš  | 8,740 | 21.1 % | -| moondeck | 22,116 | 3,571 | 18.5 % | +| platform | 15,572 (+12) โš  | 5,509 | 38.8 % | +| ui | 7,636 (+13) โš  | 2,038 | 28.3 % | +| test | 47,495 (+23) โš  | 8,752 | 21.1 % | +| moondeck | 22,116 | 3,573 | 18.5 % | ## Tests | Kind | Count | |---|---:| -| unit cases | 1,585 (+1) โœ“ | +| unit cases | 1,585 | | scenarios | 23 | ## Complexity | Metric | Value | |---|---:| -| functions | 2,776 | +| functions | 2,778 (+2) โœ“ | | over threshold | 173 | | worst CCN | 108 | @@ -59,9 +59,9 @@ Current state only; the trend is this file's git history (`git log -p docs/metri | Metric | Value | |---|---:| | markdown files | 197 | -| markdown lines | 29,247 (+27) โš  | +| markdown lines | 29,250 (+3) โš  | | plan files | 103 | -| backlog lines | 4,538 (+19) โš  | +| backlog lines | 4,539 (+1) โš  | | lessons lines | 622 | | CLAUDE.md lines | 136 | diff --git a/esp32/partitions/esp32dev_moonbase.csv b/esp32/partitions/esp32dev_moonbase.csv index 7e8f649d..cb275a16 100644 --- a/esp32/partitions/esp32dev_moonbase.csv +++ b/esp32/partitions/esp32dev_moonbase.csv @@ -1,33 +1,12 @@ -# Partition table for ESP32 classic (4 MB flash) WITH MoonBase. -# -# A 4 MB board has room for one application, not two. The dual-OTA layout (esp32dev.csv) spends -# half the chip on a second copy of the firmware, which is what ran the app slot out of space in -# 2026-08. Here that second copy is replaced by MoonBase: a small image in the `factory` slot that -# owns the device while the application is being replaced (a board cannot rewrite the partition it -# is executing from). See moonbase/ and the MoonBase plan. -# -# The app slot grows by a third as a result: 1856 KB -> 2496 KB. -# -# MoonBase is 896 KB against a measured 742 KB image (WiFi + SoftAP + HTTPS + OTA, size-tuned -# sdkconfig), leaving ~154 KB of headroom. -# -# HTTPS is 166 KB of that image and is kept deliberately. Install-from-URL is what lets a device -# fetch its own release rather than having the file pushed from whatever machine happens to be in -# front of it, and the releases live on GitHub, which is HTTPS-only. The alternative that keeps -# the convenience without TLS is to serve firmware over plain HTTP, which means the device would -# execute whatever an attacker on the path substituted; signing the image instead would cost -# comparable space plus real work. Authenticated downloads are worth the flash. -# -# The headroom is sized for a new COMPONENT, not for features: factory reset, re-provisioning, -# config backup and diagnostics are a few KB each, while one component can cost more than all of -# them together. An undersized factory partition cannot be regrown without a second full-erase -# migration of every device in the field, so it is budgeted long once. +# Partition table for ESP32 classic (4 MB flash) WITH MoonBase: the factory slot carries the +# small maintenance image and the second OTA slot is gone, growing the app slot 1856 -> 2496 KB. +# Why and how: architecture.md, MoonBase: the second boot image (the concept's one home). # # The filesystem partition is `littlefs` in both name and subtype (0x83, which ESP-IDF and the # joltwallet driver both define). Older tables label the same volume `spiffs` with subtype 0x82, -# which is a legacy misnomer: the contents have always been LittleFS. platform_esp32_fs.cpp looks -# for the littlefs subtype first and falls back to the spiffs one, so a device that keeps an older -# table across an OTA still finds its config. +# a legacy misnomer: the contents have always been LittleFS. platform_esp32_fs.cpp looks for the +# littlefs subtype first and falls back to the spiffs one, so a device that keeps an older table +# across an OTA still finds its config. # # A device adopts this table only through a FULL SERIAL FLASH: OTA writes the app, never the # partition table. diff --git a/moonbase/main/moonbase_main.cpp b/moonbase/main/moonbase_main.cpp index 142efbd3..1e2a2860 100644 --- a/moonbase/main/moonbase_main.cpp +++ b/moonbase/main/moonbase_main.cpp @@ -656,22 +656,27 @@ void serveOne(int sock) { if (n <= 0) break; prefixLen += static_cast(n); } - const size_t n = prefixLen < sizeof(stagedUrlTask_) - 1 ? prefixLen - : sizeof(stagedUrlTask_) - 1; - std::memcpy(stagedUrlTask_, head + headLen, n); - stagedUrlTask_[n] = '\0'; - std::snprintf(status_, sizeof(status_), "starting the install"); - installing_ = true; // cleared by the task after its final attempt - if (xTaskCreate(unattendedInstallTask, "mb_install", 12288, nullptr, 5, nullptr) != pdPASS) { - // A failed spawn with the flag left set would refuse every later install: THE - // deadlock this guard exists to prevent. - installing_ = false; - std::snprintf(status_, sizeof(status_), "error: cannot start the install task"); - sendResponse(sock, "500 Internal Server Error", "text/plain", status_); + if (contentLen >= sizeof(stagedUrlTask_)) { + // Same 255-byte contract the app's route enforces (platform.h): refusing beats + // truncating into a URL that fails later as a misleading download error. + sendResponse(sock, "400 Bad Request", "text/plain", "error: url too long (max 255)"); } else { - // 202: the install runs on its own task while this server keeps answering GET - // /moonbase with live progress; the caller watches that, not this response. - sendResponse(sock, "202 Accepted", "text/plain", status_); + std::memcpy(stagedUrlTask_, head + headLen, prefixLen); + stagedUrlTask_[prefixLen] = '\0'; + std::snprintf(status_, sizeof(status_), "starting the install"); + cancelRequested_ = false; // a /cancel racing the previous task's exit must not latch + installing_ = true; // cleared by the task after its final attempt + if (xTaskCreate(unattendedInstallTask, "mb_install", 12288, nullptr, 5, nullptr) != pdPASS) { + // A failed spawn with the flag left set would refuse every later install: THE + // deadlock this guard exists to prevent. + installing_ = false; + std::snprintf(status_, sizeof(status_), "error: cannot start the install task"); + sendResponse(sock, "500 Internal Server Error", "text/plain", status_); + } else { + // 202: the install runs on its own task while this server keeps answering GET + // /moonbase with live progress; the caller watches that, not this response. + sendResponse(sock, "202 Accepted", "text/plain", status_); + } } } else if (std::strncmp(head, "POST /install", 13) == 0) { if (installing_) { @@ -814,7 +819,12 @@ extern "C" void app_main() { if (ethStart()) { online = (xEventGroupWaitBits(netEvents_, kNetGotIp, pdFALSE, pdFALSE, pdMS_TO_TICKS(8000)) & kNetGotIp) != 0; - if (!online) ethStop(); // no link or no lease: WiFi takes over, alone + if (!online) { + ethStop(); // no link or no lease: WiFi takes over, alone + // A lease that raced in between the wait timing out and the teardown is an + // interface that no longer exists; it must not satisfy the WiFi wait below. + xEventGroupClearBits(netEvents_, kNetGotIp); + } } if (!online) online = wifiStation(20000); @@ -827,6 +837,7 @@ extern "C" void app_main() { // answers, and "idle" would read as nothing happening while an install is pending. std::snprintf(status_, sizeof(status_), "preparing the install"); std::snprintf(stagedUrlTask_, sizeof(stagedUrlTask_), "%s", stagedUrl); + cancelRequested_ = false; // a /cancel racing a previous task's exit must not latch installing_ = true; // cleared by the task after its final attempt if (xTaskCreate(unattendedInstallTask, "mb_install", 12288, nullptr, 5, nullptr) != pdPASS) { installing_ = false; // a latched flag would refuse every later install diff --git a/moondeck/build/build_esp32.py b/moondeck/build/build_esp32.py index e83a6f00..3545ac36 100644 --- a/moondeck/build/build_esp32.py +++ b/moondeck/build/build_esp32.py @@ -628,14 +628,8 @@ def stale_feature_cache(build_dir: Path, extra: list[str], chip: str) -> str | N # merge order) and compare against what the generated sdkconfig actually says. wanted_table = None if wanted_frags: - for frag in wanted_frags.split(";"): - fp = ESP32_DIR / frag - if not fp.exists(): - continue - m2 = re.search(r'^CONFIG_PARTITION_TABLE_CUSTOM_FILENAME="([^"]+)"', - fp.read_text(), re.MULTILINE) - if m2: - wanted_table = m2.group(1) + resolved = table_from_fragments(wanted_frags.split(";")) + wanted_table = str(resolved.relative_to(ESP32_DIR)) gen = build_dir / "sdkconfig" if wanted_table and gen.exists(): m2 = re.search(r'^CONFIG_PARTITION_TABLE_CUSTOM_FILENAME="([^"]+)"', @@ -810,11 +804,11 @@ def main(): # binary at the first app partition (0x10000: the factory slot, MoonBase's home), because it # knows nothing about the two-image scheme. These helpers are the one place that knows better. -def moonbase_table_csv(firmware: str) -> Path: - """The partition CSV a firmware's fragments select (last fragment naming one wins, - IDF's own merge order).""" +def table_from_fragments(fragments) -> Path: + """The partition CSV a fragment list selects (last fragment naming one wins, IDF's own + merge order). The one resolver: moonbase_table_csv and stale_feature_cache both use it.""" csv = ESP32_DIR / "partitions" / "esp32dev.csv" - for frag in FIRMWARES[firmware]["fragments"]: + for frag in fragments: fp = ESP32_DIR / frag if not fp.exists(): continue @@ -825,6 +819,10 @@ def moonbase_table_csv(firmware: str) -> Path: return csv +def moonbase_table_csv(firmware: str) -> Path: + return table_from_fragments(FIRMWARES[firmware]["fragments"]) + + def partition_offsets(csv_path: Path) -> dict: """SubType -> offset (hex string) for the rows a MoonBase layout needs: 'factory', 'ota_0' and 'ota' (the otadata bookkeeping partition).""" @@ -915,7 +913,9 @@ def build_moonbase(cmd: list[str], env: dict, chip: str) -> None: binp = build_dir / "projectMM-moonbase.bin" if binp.exists(): kb = binp.stat().st_size / 1024 - print(f"MoonBase image: {kb:.0f} KB (slot: 896 KB)") + # Slot fit is printed by IDF itself ("Smallest app partition ... free"); repeating a + # hardcoded slot size here would lie the day the table changes. + print(f"MoonBase image: {kb:.0f} KB") if __name__ == "__main__": diff --git a/src/core/HttpServerModule.cpp b/src/core/HttpServerModule.cpp index 3bac2a57..df267eb1 100644 --- a/src/core/HttpServerModule.cpp +++ b/src/core/HttpServerModule.cpp @@ -2285,6 +2285,9 @@ void HttpServerModule::handleBootMoonBase(platform::TcpConnection& conn) { sendResponse(conn, 409, "application/json", "{\"error\":\"no MoonBase on this device\"}"); return; } + // This route means "MoonBase with NOTHING staged" by definition; a URL left over from a + // power cut between an earlier staging and its boot switch must not fire here. + platform::moonbaseClearStagedUrl(); FilesystemModule::flushPending(); sendResponse(conn, 200, "application/json", "{\"ok\":true,\"moonbase\":true}"); conn.close(); diff --git a/src/platform/desktop/platform_desktop.cpp b/src/platform/desktop/platform_desktop.cpp index 6eef2094..b91fcd27 100644 --- a/src/platform/desktop/platform_desktop.cpp +++ b/src/platform/desktop/platform_desktop.cpp @@ -1346,6 +1346,7 @@ bool otaHasMoonBase() { return false; } bool otaBootMoonBase() { return false; } bool otaRunningMoonBase() { return false; } bool moonbaseStageInstallUrl(const char*) { return false; } +void moonbaseClearStagedUrl() {} // Outbound HTTP request (plain HTTP, LAN, no TLS) โ€” see platform.h. Blocking, bounded by a // receive/send timeout. Builds the request into a stack buffer, connects, sends, reads the diff --git a/src/platform/esp32/platform_esp32_ota.cpp b/src/platform/esp32/platform_esp32_ota.cpp index 5bca8f18..4f8d55ce 100644 --- a/src/platform/esp32/platform_esp32_ota.cpp +++ b/src/platform/esp32/platform_esp32_ota.cpp @@ -321,6 +321,14 @@ bool otaRunningMoonBase() { return run && run->subtype == ESP_PARTITION_SUBTYPE_APP_FACTORY; } +void moonbaseClearStagedUrl() { + nvs_handle_t h; + if (nvs_open("moonbase", NVS_READWRITE, &h) != ESP_OK) return; + nvs_erase_key(h, "url"); + nvs_commit(h); + nvs_close(h); +} + // Stage the install URL in NVS for MoonBase to consume on its next boot (see platform.h). bool moonbaseStageInstallUrl(const char* url) { if (!url || !url[0]) return false; diff --git a/src/platform/platform.h b/src/platform/platform.h index 637a06f7..712f6ec5 100644 --- a/src/platform/platform.h +++ b/src/platform/platform.h @@ -677,6 +677,9 @@ bool otaRunningMoonBase(); // are we executing from it right now? // MoonBase, and MoonBase installs it with no browser in the loop. MoonBase erases the key before // attempting the install, so a bad URL cannot boot-loop the device. bool moonbaseStageInstallUrl(const char* url); +// Erase a staged URL that never got consumed: a power cut between staging and the boot-partition +// switch leaves it armed, and the next unrelated MoonBase visit would auto-install it. +void moonbaseClearStagedUrl(); // Synchronous outbound HTTP request to a LAN host โ€” plain HTTP, no TLS (the Philips Hue v1 // API, which HueDriver drives, allows it). Connects to `host:port`, sends `method path` diff --git a/src/ui/app.js b/src/ui/app.js index 93fd9261..db3243e7 100644 --- a/src/ui/app.js +++ b/src/ui/app.js @@ -1149,9 +1149,11 @@ function showUpdateOverlay() { dismiss.addEventListener("click", () => ov.remove()); const cancel = document.createElement("button"); cancel.textContent = "cancel install"; + let uploadCtl = null; // the in-flight file upload's AbortController, set by the flow cancel.addEventListener("click", () => { - // Best-effort: only a running URL install can hear it (MoonBase's /cancel); the watch - // loop sees the resulting "canceled" status and ends the overlay from there. + // A URL install hears POST /cancel; a file upload cancels by dropping its connection + // (MoonBase's single-connection server is busy receiving it), so abort the fetch. + if (uploadCtl) uploadCtl.abort(); fetch("/cancel", { method: "POST" }).catch(() => {}); }); box.append(h, msg, bar, cancel, dismiss); @@ -1161,6 +1163,7 @@ function showUpdateOverlay() { status(text) { msg.textContent = text; }, progress(read, total) { if (total > 0) { bar.max = total; bar.value = read; } }, fail(text) { msg.textContent = text; bar.remove(); cancel.remove(); dismiss.style.display = ""; }, + setUpload(ctl) { uploadCtl = ctl; }, }; } @@ -1232,15 +1235,25 @@ async function moonbaseUpdateFlow(opts) { // The upload needs MoonBase serving before the browser can push the image. if (!(await waitForMoonBase(Date.now() + 120000))) throw new Error(MOONBASE_SILENT_MSG); ui.status(`Installing ${fmSize(opts.file.size)}\u2026`); + const uploadCtl = new AbortController(); + ui.setUpload(uploadCtl); try { const r = await fetch("/install", { method: "POST", headers: { "Content-Type": "application/octet-stream" }, - body: opts.file }); + body: opts.file, signal: uploadCtl.signal }); if (!r.ok) throw new Error(await r.text()); } catch (err) { + if (err instanceof Error && err.name === "AbortError") { + // The user's cancel, not a failure: the dropped connection aborts the + // write on the device, which stays waiting in MoonBase. + ui.fail("Install canceled \u2014 the device is waiting in MoonBase."); + return; + } if (err instanceof Error && err.message.startsWith("error")) throw err; // A dead socket mid-upload (power cut, WiFi drop) is not a verdict: the watch // loop below sees where the device lands and retries the upload from there. + } finally { + ui.setUpload(null); } } diff --git a/test/python/test_flash_baud.py b/test/python/test_flash_baud.py index d1842b9a..7afa486f 100644 --- a/test/python/test_flash_baud.py +++ b/test/python/test_flash_baud.py @@ -41,7 +41,10 @@ def test_firmware_only_falls_back_to_lowest_sibling(): # No deviceModel known (a plain --firmware flash): take the lowest flashBaud among # models sharing the firmware, so an opt-down still protects an unidentified board. assert _catalog_flash_baud("esp32") == 460800 # LOLIN's 460800 protects - assert _catalog_flash_baud("esp32-eth") == 921600 # no sibling opt-down โ†’ fast + # esp32-eth gained the Olimex's opt-down (its USB bridge dies at 921600), so it now + # protects too; the S3 firmwares have no opted-down sibling and stay fast. + assert _catalog_flash_baud("esp32-eth") == 460800 + assert _catalog_flash_baud("esp32s3-n16r8") == 921600 # no sibling opt-down, fast def test_unknown_model_falls_through_to_firmware(): diff --git a/test/unit/core/unit_MoonBaseContract.cpp b/test/unit/core/unit_MoonBaseContract.cpp index e7bb5e9c..3e145fd2 100644 --- a/test/unit/core/unit_MoonBaseContract.cpp +++ b/test/unit/core/unit_MoonBaseContract.cpp @@ -1,13 +1,18 @@ // @module NetworkModule // @also FilesystemModule -// MoonBase reads the WiFi credentials with a bounded 1024-byte prefix read of -// /.config/NetworkModule.json (moonbase/main/moonbase_main.cpp loadCredentials): a tiny image -// has no JSON parser and no room for the whole file, which also carries every child module's -// config. That bound is a cross-image contract with NetworkModule's control registration order, -// and nothing else pins it: a control added ABOVE ssid/password would push them out of the -// prefix and silently break MoonBase's network join on every deployed 4 MB device. This test is -// the pin. +// MoonBase reads the WiFi credentials, the Ethernet wiring and the TX cap with a bounded +// 2048-byte prefix read of /.config/NetworkModule.json (moonbase/main/moonbase_main.cpp +// loadCredentials): a tiny image has no JSON parser and no room for the whole file, which also +// carries every child module's config. That bound is a cross-image contract with +// NetworkModule's control registration order, and nothing else pins it: a control added ABOVE +// these keys would push them out of the prefix and silently break MoonBase's network join (or +// its eth wiring) on every deployed 4 MB device. This test is the pin. +// +// The eth controls exist only on Ethernet-capable builds, so this desktop-run test pins the +// bound indirectly: every top-level Network control serializes BEFORE the first child module +// key ("0."), so end-of-top-level plus a stated worst case for the ESP32-only keys must +// sit inside the prefix. #include "doctest.h" #include "core/FilesystemModule.h" @@ -20,8 +25,8 @@ #include #include -// The credentials a provisioned device saves sit inside the first kilobyte of NetworkModule.json, where MoonBase's bounded prefix read finds them. -TEST_CASE("NetworkModule.json carries ssid and password within MoonBase's 1024-byte prefix read") { +// The keys MoonBase scrapes sit inside its 2048-byte prefix read of NetworkModule.json, with room for the ESP32-only eth block. +TEST_CASE("NetworkModule.json keeps MoonBase's scraped keys inside its 2048-byte prefix read") { char tmpRoot[256]; std::snprintf(tmpRoot, sizeof(tmpRoot), "/tmp/mm_moonbase_contract_%u", static_cast(mm::platform::millis())); @@ -48,14 +53,29 @@ TEST_CASE("NetworkModule.json carries ssid and password within MoonBase's 1024-b std::string content((std::istreambuf_iterator(f)), std::istreambuf_iterator()); f.close(); + constexpr size_t kPrefixRead = 2048; // moonbase_main.cpp loadCredentials buf size + // The ESP32-only additions to the top-level block that this desktop file cannot contain: + // 8 eth keys plus values (~260 bytes serialized) with margin. + constexpr size_t kEsp32OnlyBudget = 400; + const auto ssidEnd = content.find("\"ssid\":\"bench-ssid\""); const auto pwKey = content.find("\"password\":"); REQUIRE(ssidEnd != std::string::npos); REQUIRE(pwKey != std::string::npos); - // The whole password VALUE must fit too: key position + key + a worst-case 64-char - // passphrase escaped to at most twice its length stays under the bound. - CHECK(ssidEnd < 1024); - CHECK(pwKey + 12 + 2 * 64 + 2 < 1024); + CHECK(ssidEnd < kPrefixRead - kEsp32OnlyBudget); + // The whole password VALUE fits too: a worst-case 64-char passphrase escaped to twice + // its length. + CHECK(pwKey + 12 + 2 * 64 + 2 < kPrefixRead - kEsp32OnlyBudget); + + // Every top-level control precedes the first child-module key; with the ESP32-only budget + // on top, the whole scraped block stays inside the prefix. + const auto firstChild = content.find("\"0."); + if (firstChild != std::string::npos) { + CHECK(firstChild < kPrefixRead - kEsp32OnlyBudget); + } else { + // No children on this build: the whole file must fit with the budget to spare. + CHECK(content.size() < kPrefixRead - kEsp32OnlyBudget); + } scheduler.release(); std::filesystem::remove_all(tmpRoot);