perf(SDK-6463): prune excluded dirs from zip/md5 walks — minutes-long 'Creating tests.zip' stall on monorepos

[Bhargavi-BS]

SDK-6463 (follow-up) — minutes-long stall creating tests.zip on large monorepos

Customer (Sapiens, NX monorepo, ~120 exclude globs, home_directory: "./") reported very slow test-setup upload. Their log shows the time is NOT the upload (3s, small zip) — it's spent before it:

17:06:58 info: Creating tests.zip with files in ./          <- start
17:08:24 info: Uploading the tests to BrowserStack          <- ~86s later

plus a similar hidden cost earlier in the spec-md5 cache check.

Root cause (verified)

Three full-tree filesystem walks of the monorepo root, none of which prune excluded directories:

1. Zip walk — archiver.glob(pattern, { ignore }) (bin/helpers/archiver.js). archiver delegates to readdir-glob, whose ignore filters entries after the walker has already readdir+lstat-ed them. It therefore descends into node_modules, .git, dist, … visiting every file just to discard it. The library's separate skip option is the pruning mechanism — the CLI never used it.
2. Spec-md5 walk — checkSpecsMd5 → hashUtil uses readdir-glob the same way: a second identical full-tree walk.
3. Telemetry walk — utils.fetchFolderSize(node_modules) (runs.js) recursively stats the entire node_modules, blocking between archive and upload, only to report a folder size.

On the customer's Windows machine (per-file stat + AV scanning) with a real NX monorepo this is minutes per run.

Fix

• utils.getDirectorySkipPatterns(ignoreFiles) — reuse every ignore pattern ending in /** as a readdir-glob skip pattern; wired into both the archive glob and the md5 walk.
  Safety proof: skip prunes a directory whose relative path matches the pattern. A directory matching <x>/** is inside <x>, so every one of its descendants also matches <x>/** and was going to be ignored anyway — pruning cannot change which files are included. (Pruning engages one level below each excluded root: e.g. node_modules/** skips every child of node_modules, paying a single readdir of node_modules itself.)
• fetchFolderSize — cooperative 5s deadline; folder size is best-effort telemetry and must never stall a run.

Verification (real CLI code, synthetic 144k-file monorepo, customer's exact 130 exclude patterns)

Walk	Before	After	Output
archiveSpecs (tests.zip)	7.5s	0.1s	zip entry lists byte-identical
checkSpecsMd5	5.5s	0.0s	md5 unchanged (aeceb90f…) → upload-cache keys unaffected, no forced re-uploads

Direct readdir-glob measurement on the same tree: no-skip walk 3.04s / 140,883 entries visited vs skip 0.08s / 2,083 — the excluded trees are simply never entered.

Unit tests added for getDirectorySkipPatterns; suite: 675→677 passing, 0 new failures (16 pre-existing on master).

Scope

Separate from PR #1131 (config/a11y) and #1139 (a11y fail-guard): this addresses the customer's "test setup zip upload taking a lot of time" report independently.

🤖 Generated with Claude Code

[amaanbs]

Any downstream effects from this?

[Bhargavi-BS]

Verified the full flow — no functional downstream effects:

• fetchFolderSize has exactly one caller (runs.js → node_modules_size), and that value has exactly one sink: the dataToSend instrumentation object, merged into buildReportData and sent via sendUsageReport. It is telemetry-only — nothing branches on it (no upload decisions, no timeouts, no user-facing output).
• getDirectorySize is module-private (not exported), so there are no other consumers.

Behavior when the 5s deadline fires: the reported node_modules_size is partial (under-reported) for that run, with a debug log noting it. Worth noting the field was already best-effort — the existing catch returns 0 on any readdir/stat error, so consumers already tolerate inaccurate values. On repos where the walk finishes within 5s (the common case) the value is unchanged.

The trade: bounded, slightly-lossy telemetry vs. blocking the archive→upload pipeline for tens of seconds on large monorepos (the reported case). Happy to raise the cap or make it env-configurable if instrumentation accuracy is a concern.

[Bhargavi-BS]

Pushed an improvement on this in 609d50d that removes the concern entirely: the walk no longer blocks anything. fetchFolderSize is now started before the upload (not awaited) and awaited only after the upload completes, right before the value is consumed for dataToSend. In the common case the walk resolves while the upload is in flight, so it adds zero wall-clock and reports the full, accurate size. The 5s deadline stays as a backstop for pathological trees whose walk outlives the upload (worst case: partial size in the usage report, never a stalled run). fetchFolderSize never rejects, so the floating promise is safe.

[Bhargavi-BS]

Verified both modes, structurally and with real runs.

Structure — the await sits on the shared path before the modes ever diverge:

runs.js:286  walk starts (not awaited)      ─┐
runs.js:314  await nodeModulesSizePromise    ├─ shared path, both modes
runs.js:341  createBuild(...)               ─┘
runs.js:378  if (args.sync)  → pollBuildStatus     ← first sync/async divergence
runs.js:437  if (!args.sync) → exit message
runs.js:450  dataToSend.node_modules_size          ← plain resolved local, both modes

The value is awaited once, at a single point common to both modes, before createBuild — neither mode can observe an unresolved value, and neither mode's control flow changed. (fetchFolderSize never rejects, so the floating promise can't produce an unhandled rejection either.)

Empirical — real runs on a 144k-file synthetic monorepo (customer-style home_directory: "./" + their 130 exclude patterns), this branch @ 609d50d:

Mode	Zip creation	Upload	Build created	After build
async (default)	<1s	6s	✅ 5df4c606…	usage report sent, clean exit 0
--sync	<1s	4s	✅ c45b0044…	polling started ("BrowserStack machines are now setting up…") and followed the build ✓

For reference, published 1.36.12 on the same repo: 5s zip creation and 13s in the pre-zip md5 phase — this branch reduces both to ~0 (and on the reporting customer's Windows machine with a real NX tree, the zip phase alone was 86s).