refactor(trace): extract the trace-format transforms into their own package, and export traces from the backend

ARCHITECTURE.md

@@ -16,7 +16,10 @@ A devtools dashboard for end-to-end browser tests. Three test frameworks (Webdri
         │
         ▼
      [core]           framework-agnostic capture/reporting library
-        │
+        │                      │
+        │                      ▼
+        │                  [trace]  ◀── also used by backend
+        │                           pure transforms: events → trace.zip
         ▼ (WS frames typed by shared)
    [backend]          Fastify + WS gateway + baseline store + rerun spawner
         │
@@ -30,7 +33,7 @@ A separate piece, **`packages/script`**, is injected into the browser under test
 
 ## Packages
 
-The workspace is a pnpm monorepo. Two of the packages (`shared`, `core`) are workspace-internal — they're marked `"private": true` and never published; consumers bundle their code into their own `dist/`.
+The workspace is a pnpm monorepo. Three of the packages (`shared`, `trace`, `core`) are workspace-internal — they're marked `"private": true` and never published; consumers bundle their code into their own `dist/`.
 
 ### `packages/shared`
 
@@ -48,10 +51,29 @@ Contains the canonical definitions for:
 
 Imports from: nothing. Imported by: every other package.
 
+### `packages/trace`
+
+Trace-format transforms: captured events in, `trace.zip` out. Workspace-internal; inlined into each consumer at build time.
+
+Split out of `core` because **two layers need it and only one of them may import `core`**. Adapters build their own trace through `core`; the backend builds one on behalf of an adapter that cannot run Node-side trace code (the Python adapter), and §2.2 bars the backend from `core`. Everything here is a pure transform over `shared` types — no framework API, no driver, no capture, no I/O beyond writing the zip.
+
+Contains:
+
+- `writeTraceZip` / `buildTraceZip` — the zip writer and its resource model (`TraceZipResource`).
+- `buildActionEvents` — commands → the ordered before/after event stream the player replays, including the `(startTime, sequence, index)` ordering key.
+- `buildGroupPath` — the suite/test/step hierarchy a row is nested under.
+- `buildImageFrameSnapshots` / `upsertRichestSnapshot` / `buildDenseScreencast` / `thinScreencastFrames` — filmstrip and per-action frame snapshots.
+- `buildMutationsNdjson` / `reattributeDomAnchors` — the DOM mutation stream and its anchor repositioning.
+- `buildConsoleEvents`, `networkRequestToHar`, `buildSourceResources` / `sourceResourceName` / `callSourceToStack`, `generateTranscript`, `sha1Hex`.
+
+Imports from: `shared`. Imported by: `core` (which re-exports it, so adapters reach it unchanged) and `backend`.
+
 ### `packages/core`
 
 Framework-agnostic capture and reporting library. Workspace-internal; inlined into each adapter at build time.
 
+Adapter-side trace *orchestration and policy* stay here — `trace-finalizer` (what to write and when), `spec-trace-helpers` (slice boundaries and per-test slicing), `trace-retention` (`shouldRetainTrace`, which also governs screenshot and video retention). The transforms they call live in `trace`.
+
 Contains:
 
 - `SessionCapturerBase` — orchestrates per-session capture (console/stream patching, WS connection, command-id bookkeeping, upstream-send guard with `onUpstreamDrop` hook).
@@ -60,7 +82,7 @@ Contains:
 - `resolveAdapterOutputDir` — the dir-resolution helper that picks where screencast/trace files land (test-file dir → config dir → cwd, with a `node_modules/` skip).
 - Pure helpers: `assert-patcher`, `bidi` (`attachBidiHandlers`, `loadSeleniumSubmodule`, `arrayHeadersToObject`), `console` (`stripAnsi`, `detectLogLevel`, `createConsoleLogEntry`, `mapChromeBrowserLogs`, `chromeLogLevelToLogLevel`), `error` (`serializeError`, `errorMessage`), `finalize-screencast`, `net` (`isPortInUse`, `findFreePort`, `getRequestType`), `performance-capture` (`CAPTURE_PERFORMANCE_SCRIPT`, `applyPerformanceData`), `retry-tracker`, `script-loader` (`loadInjectableScript`, `pollUntilReady`), `stack` (`isUserCodeFrame`, `normalizeFilePath`, `getCallSourceFromStack`), `suite-helpers`, `test-discovery` (`findTestDefinitions`, `extractTestMetadata`), `uid` (`generateStableUid`, `deterministicUid`, `resetSignatureCounters`), `video-encoder` (`encodeToVideo`).
 
-Imports from: `shared`. Imported by: all three adapter packages.
+Imports from: `shared`, `trace`. Imported by: all three adapter packages.
 
 ### `packages/service` — WebdriverIO adapter
 
@@ -215,6 +237,7 @@ The repo has converged on a clear ownership story. When in doubt, the top-down d
 
 - A type, constant, enum, schema, or contract used by more than one package → **`shared`**.
 - Capture, parsing, normalization, sourcemap, UID, reporter, screencast, or WS-framing logic that doesn't depend on a specific framework's API → **`core`**.
+- A pure transform that turns captured events into trace-zip content → **`trace`**. The test is whether the backend would ever need it: it builds traces for adapters that can't, and it may not import `core`.
 - A specific framework's hook, driver patch, or runner integration → the matching **adapter** package. Adapter code calls `core` for the actual work and only owns the hook registration.
 - A backend HTTP route, WS handler, or rerun behavior → **`backend`**, with the contract added to `shared` first.
 - UI → **`app`**, consuming `shared` contracts only.
@@ -225,7 +248,7 @@ A few cross-cutting conventions follow from this layout:
 - Adapter packages don't import each other. Anything two adapters would both want lives in `core`.
 - Backend doesn't import adapter packages, and adapter packages don't import backend or app.
 - The script package is a leaf — adapters load its built bundle as a string and inject it; they don't import from it at runtime.
-- `shared` and `core` are private workspace packages. Consumers bundle them. The bundler config has to inline them (not externalize) or the published artifact won't resolve — see the build-config notes in `CLAUDE.md`.
+- `shared`, `trace`, and `core` are private workspace packages. Consumers bundle them. The bundler config has to inline them (not externalize) or the published artifact won't resolve — see the build-config notes in `CLAUDE.md`.
 
 ---
 

CLAUDE.md

@@ -10,7 +10,7 @@ Anyone working in the repo, human or AI agent, can use this as the source of tru
 
 A devtools dashboard for end-to-end browser tests. Three test frameworks (WebdriverIO, Nightwatch, Selenium) push the same normalized event stream through a single backend into a single Lit-based browser UI. The adapters are deliberately thin — they translate framework hooks into calls on a shared core capture/reporting library and own only the framework-specific glue.
 
-Package map and data flow are in [ARCHITECTURE.md](./ARCHITECTURE.md). The summary: `shared` for types and contracts, `core` for framework-agnostic capture, three adapters (`service`, `nightwatch-devtools`, `selenium-devtools`) for framework glue, `backend` for the server, `app` for the UI, `script` for the page-injected runtime.
+Package map and data flow are in [ARCHITECTURE.md](./ARCHITECTURE.md). The summary: `shared` for types and contracts, `trace` for the event→zip transforms, `core` for framework-agnostic capture, three adapters (`service`, `nightwatch-devtools`, `selenium-devtools`) for framework glue, `backend` for the server, `app` for the UI, `script` for the page-injected runtime.
 
 ---
 
@@ -48,6 +48,7 @@ Defined in root `tsconfig.json`:
 | `@wdio/selenium-devtools` / `*` | `packages/selenium-devtools/src/...` |
 | `@wdio/devtools-shared` / `*` | `packages/shared/src/...` |
 | `@wdio/devtools-core` / `*` | `packages/core/src/...` |
+| `@wdio/devtools-trace` / `*` | `packages/trace/src/...` |
 | `@wdio/elements` / `*` | `packages/elements/src/...` |
 
 These exist so imports stay short and grep-able. Long relative paths (`../../../components/…`) aren't used.
@@ -72,6 +73,14 @@ If the same logical change would land in two or more adapters, the logic belongs
 
 Some helpers are framework-agnostic by nature but used in only one adapter today (e.g. nightwatch's `parseNetworkFromPerfLogs` for CDP perf-log parsing, selenium's `detectRunner`/`captureLaunchCommand`). They stay in their adapter until a second consumer appears; at that point they move to core.
 
+### Trace-format transforms live in `trace`, one layer below `core`
+
+`packages/trace` holds the pure transforms that turn captured events into trace-zip content — the zip writer, action events, group paths, frame snapshots, mutations, HAR, sources, transcript. `core` keeps the adapter-side *orchestration and policy* that calls them: `trace-finalizer`, `spec-trace-helpers`, `trace-retention`.
+
+The split is not aesthetic. **`backend` may not import `core`** (it would pull framework-adapter logic into the server), but it does need to build a trace on behalf of an adapter that can't — the Python adapter ships no Node. `trace` is the layer both can reach, so it may import `shared` and nothing else; a single import of `core` from it re-creates the cycle the split exists to remove, and ESLint enforces that.
+
+The test for a new helper: *would the backend ever need this to build a zip?* If yes, `trace`. If it needs a driver, a framework hook, or a capture session, `core`.
+
 ### Adapters are thin and isolated
 
 Adapter packages own only:
@@ -96,13 +105,16 @@ No `any` crosses a package boundary. When a framework API forces a loosely-typed
 
 ### Workspace-internal packages stay bundled
 
-`packages/shared` and `packages/core` are `"private": true` and never published. Each consumer inlines their code into its own `dist/` at build time.
+`packages/shared`, `packages/trace` and `packages/core` are `"private": true` and never published. Each consumer inlines their code into its own `dist/` at build time.
 
-- Both deps are listed in `devDependencies` with `workspace:^`, never in `dependencies`. Vite and tsup both externalize anything in `dependencies` by default; `devDependencies` is what gets inlined.
-- Neither is added to a bundler's `external` config. Vite's `external` callback receives both the bare package name *and* the resolved absolute path (e.g. `/Users/.../packages/core/src/index.ts`); a check for only one form silently externalizes the other.
+- All three are listed in `devDependencies` with `workspace:^`, never in `dependencies`. Vite and tsup both externalize anything in `dependencies` by default; `devDependencies` is what gets inlined.
+- None of them is added to a bundler's `external` config. Vite's `external` callback receives both the bare package name *and* the resolved absolute path (e.g. `/Users/.../packages/core/src/index.ts`); a check for only one form silently externalizes the other.
+- That callback enumerates the private packages, so **adding a fourth one means editing it** — a package missing from the list falls through to the default and is externalized silently, producing a dist that dies at install with `ERR_MODULE_NOT_FOUND`. It is a `PRIVATE_WORKSPACE_PACKAGES` array rather than a chain of `||`s for exactly that reason. Adding a workspace package also means adding it to `pnpm-workspace.yaml`, whose `packages:` list is explicit rather than a glob.
 - The same callback receives bare relative imports (`./utils.js`, `../constants.js`). A check that allows only `./` will externalize `../`-style imports from subfolders and the dist crashes with `ERR_MODULE_NOT_FOUND` at install time.
 - `packages/service/vite.config.ts` is the canonical pattern for getting both right.
-- After any change to a bundler config or build script, `grep -nE "(from|require\()\s*['\"](@wdio/devtools-(core|shared)|.*/packages/(core|shared)/)" packages/<pkg>/dist/*.js` should return nothing. That's how you catch the absolute-path leak. Match on the `from`/`require(` prefix, not the bare package name: `LIBRARY_NAME = "@wdio/devtools-core"` (written into the trace's `context-options`) and `Symbol.for("@wdio/devtools-core/assert-patched")` are inlined string *values* that legitimately survive bundling, so a bare-name grep always reports a false leak.
+- After any change to a bundler config or build script, `grep -nE "(from|require\()\s*['\"](@wdio/devtools-(core|shared|trace)|.*/packages/(core|shared|trace)/)" packages/<pkg>/dist/*.js` should return nothing. That's how you catch the absolute-path leak. Match on the `from`/`require(` prefix, not the bare package name: `LIBRARY_NAME = "@wdio/devtools-core"` (written into the trace's `context-options`) and `Symbol.for("@wdio/devtools-core/assert-patched")` are inlined string *values* that legitimately survive bundling, so a bare-name grep always reports a false leak.
+
+- A **CJS-only dependency must be externalized, not inlined**, or esbuild rewrites its `require` into a shim that throws `Dynamic require of "fs" is not supported` the moment the module loads. Declaring it in `dependencies` is what externalizes it; that is why all three adapters — and now `backend` — list `yazl` there rather than in `devDependencies`. This is the opposite of the workspace-internal rule above, and for the same underlying reason: `dependencies` is externalized, `devDependencies` is inlined. Neither `pnpm build` nor `pnpm test` nor the leak grep notices — every one of them passes on a dist that dies on first import — so `packages/backend/tests/dist-bundling.test.ts` asserts the shim is absent.
 
 Bundlers in use: **vite** for `app`, `service`, `script`; **tsup** for `backend`, `nightwatch-devtools`, `selenium-devtools`.
 
@@ -190,13 +202,14 @@ A handful of tests need `@wdio/devtools-script` to be built first (the browser-i
 The decision tree from [ARCHITECTURE.md "Where things live"](./ARCHITECTURE.md#where-things-live) is the starting point. The general shape:
 
 - Shared concept → `shared`.
+- Pure transform producing trace-zip content → `trace`.
 - Framework-agnostic capture/reporting logic → `core`.
 - Framework-specific glue → the matching adapter.
 - Server route/WS handler → `backend` (contract in `shared` first).
 - UI → `app`.
 - Code that runs in the browser under test → `script`.
 
-When the right place is ambiguous (something between `shared` and `core`, or between `core` and an adapter), the question that resolves it is: *who else would want this?* If the answer is "any future adapter would," it's `core`. If "only the framework with X-specific API does," it's the adapter.
+When the right place is ambiguous (something between `shared` and `core`, or between `core` and an adapter), the question that resolves it is: *who else would want this?* If the answer is "any future adapter would," it's `core`. If "only the framework with X-specific API does," it's the adapter. If "the backend would, to build a zip for an adapter that can't," it's `trace`.
 
 ### While editing
 

eslint.config.cjs

@@ -281,7 +281,7 @@ module.exports = [
             {
               group: ['@wdio/devtools-core', '@wdio/devtools-core/*'],
               message:
-                'Backend must not depend on core (CLAUDE.md §2.2). core is framework-agnostic adapter logic; backend only needs shared contracts.'
+                'Backend must not depend on core (CLAUDE.md §2.2). core is framework-agnostic ADAPTER logic. For trace-format transforms use @wdio/devtools-trace, which sits below both; anything else belongs in shared.'
             }
           ]
         }
@@ -326,6 +326,11 @@ module.exports = [
               group: ['@wdio/devtools-core', '@wdio/devtools-core/*'],
               message:
                 'App must not import from core (CLAUDE.md §2.2). core is framework-agnostic adapter logic; the app receives normalized events over WS.'
+            },
+            {
+              group: ['@wdio/devtools-trace', '@wdio/devtools-trace/*'],
+              message:
+                'App must not import from trace (CLAUDE.md §2.2). trace WRITES the zip; the app reads one the backend has already parsed for it.'
             }
           ]
         }
@@ -374,5 +379,54 @@ module.exports = [
         }
       ]
     }
+  },
+
+  // CLAUDE.md §2.2 — trace sits below core and backend, and both import it.
+  // It holds pure transforms over shared types only: anything it pulled from
+  // core would re-create the dependency the split exists to remove.
+  {
+    files: ['packages/trace/**/*.{ts,tsx,js,mjs,cjs}'],
+    rules: {
+      'no-restricted-imports': [
+        'error',
+        {
+          patterns: [
+            {
+              group: ['@wdio/devtools-core', '@wdio/devtools-core/*'],
+              message:
+                'trace must not depend on core (CLAUDE.md §2.2). core imports trace; keeping the arrow one-way is what lets backend use trace at all.'
+            },
+            {
+              group: ['@wdio/devtools-backend', '@wdio/devtools-backend/*'],
+              message:
+                'trace must not depend on backend (CLAUDE.md §2.2). trace is the lower layer.'
+            },
+            {
+              group: ['@wdio/devtools-service', '@wdio/devtools-service/*'],
+              message:
+                'trace must not depend on any adapter (CLAUDE.md §2.2). Adapters import trace, not the other way around.'
+            },
+            {
+              group: [
+                '@wdio/nightwatch-devtools',
+                '@wdio/nightwatch-devtools/*'
+              ],
+              message:
+                'trace must not depend on any adapter (CLAUDE.md §2.2). Adapters import trace, not the other way around.'
+            },
+            {
+              group: ['@wdio/selenium-devtools', '@wdio/selenium-devtools/*'],
+              message:
+                'trace must not depend on any adapter (CLAUDE.md §2.2). Adapters import trace, not the other way around.'
+            },
+            {
+              group: ['@/*', '@components/*'],
+              message:
+                'trace must not depend on app (CLAUDE.md §2.2). trace is Node-side transform logic.'
+            }
+          ]
+        }
+      ]
+    }
   }
 ]

packages/backend/package.json

@@ -49,12 +49,14 @@
     "import-meta-resolve": "^4.2.0",
     "shell-quote": "^1.8.4",
     "tree-kill": "^1.2.2",
-    "ws": "^8.21.0"
+    "ws": "^8.21.0",
+    "yazl": "^3.3.1"
   },
   "devDependencies": {
     "@types/shell-quote": "^1.7.5",
     "@types/ws": "^8.18.1",
     "@wdio/devtools-shared": "workspace:^",
+    "@wdio/devtools-trace": "workspace:^",
     "nodemon": "^3.1.14",
     "tsup": "^8.5.1"
   }

packages/backend/src/baseline/types.ts

@@ -1,9 +1,11 @@
 import type {
   CommandLog,
   ConsoleLog,
+  Metadata,
   NetworkRequest,
   TestError,
-  TestStatus
+  TestStatus,
+  TraceMutation
 } from '@wdio/devtools-shared'
 
 // Backend storage uses the canonical shared types. The `*Like` aliases below
@@ -13,13 +15,10 @@ export type CommandLogLike = CommandLog
 export type ConsoleLogLike = ConsoleLog
 export type NetworkRequestLike = NetworkRequest
 
-// Mutations stay loose: the concrete shape (TraceMutation) lives in
-// packages/script (browser-side, depends on DOM types) and isn't safe to
-// import here.
-export interface MutationLike {
-  timestamp: number
-  [key: string]: unknown
-}
+// `TraceMutation` in shared is deliberately the Node-safe version of the
+// browser-side shape — string literals instead of DOM node types — so it flows
+// here without dragging the DOM lib into shared's compilation.
+export type MutationLike = TraceMutation
 
 export type NodeState = TestStatus
 export type NodeError = TestError
@@ -48,4 +47,11 @@ export interface ActiveRun {
   sources: Record<string, string>
   nodes: Map<string, TimeWindowNode>
   startedAt: number
+  /** Last `metadata` frame the worker sent. Preserve & Rerun never needed it;
+   *  a trace artifact does — it carries the session identity and capabilities
+   *  the viewer reads. */
+  metadata?: Metadata
+  /** Raw `logs` frames, the trace's transcript source. Only the JS adapters
+   *  send these, so this is routinely empty. */
+  traceLogs: string[]
 }

packages/backend/src/baseline/utils.ts

@@ -8,7 +8,8 @@ export function freshRun(): ActiveRun {
     mutations: [],
     sources: {},
     nodes: new Map(),
-    startedAt: Date.now()
+    startedAt: Date.now(),
+    traceLogs: []
   }
 }