Skip to content

fix(phpunit): scope file-only testsuites to declared files - #2517

Merged
chubes4 merged 1 commit into
mainfrom
fix/2516-phpunit-file-only-discovery
Sep 21, 2026
Merged

chubes4 merged 1 commit into
mainfrom
fix/2516-phpunit-file-only-discovery

Conversation

@chubes4

@chubes4 chubes4 commented Sep 21, 2026

Copy link
Copy Markdown
Collaborator

Problem

wp_codebox_phpunit_parse_config() (generated by phpunitConfigDiscoveryPhp() in packages/runtime-playground/src/phpunit-command-handlers.ts) defaults $directories to the whole test root and only overwrites it when a <testsuite> declares <directory> elements. A <testsuite> that declares only <file> entries never triggers that overwrite, so $directories silently stays at the whole-test-root default and discovery unions the declared files with a recursive scan of the entire tree.

Real PHPUnit does not do this: a <testsuite> with only <file> entries and no <directory> runs exactly those files and performs no directory-based discovery at all. That's the reference behaviour this PR restores.

The three cases

Config declares Before this PR After this PR
<directory> (with or without <file>) uses declared directories unchanged
only <file>, no <directory> falls back to whole-test-root scan, unioned with declared files fixed — runs only the declared files, no directory scan
neither whole-test-root default unchanged

Evidence

On Extra-Chill/extrachill-events, booking-alpha-concurrency declares exactly one file (InternalBookingHoldConcurrencyMySQLProof.php, 2 test methods) and no <directory>. A run of that suite reported 88 tests, 615 assertions — 44x the declared content. Three sibling suites (booking-mysql, booking-concurrency, booking-alpha) have the same shape; a fourth (managed-wordpress), which does declare a <directory>, scoped correctly and served as the control case that pointed at this discovery function specifically.

Downstream, the wide scan swept in a stub file three directories away from anything the failing suite's config named, causing Cannot redeclare function wp_get_ability() and fataling the entire suite before any of its actually-declared tests ran. That's tracked separately as Extra-Chill/extrachill-events#846 — this PR fixes the root cause it surfaced; the stub itself is fixed independently over there.

The fix

Two lines in phpunitConfigDiscoveryPhp():

if (!empty($config_dirs)) {
    $directories = $config_dirs;
    ...
} elseif (!empty($files)) {
    $directories = array();
}

$config_dirs and $files are already accumulated separately from <directory> and <file> elements across all matched testsuites. This only adds the missing branch: when no <directory> was declared but at least one <file> was, clear $directories instead of leaving it at the whole-tree default. Configs declaring <directory> (case 1) and configs declaring neither (case 3) take the same branches as before and are unaffected.

This is the shared discovery-config parser used by both the plugin PHPUnit path (wp_codebox_phpunit_parse_config) and the core PHPUnit path (core_pg_parse_phpunit_config) — both call sites get the fix since they share this generator function.

Tests

Added to tests/phpunit-discovery-only.test.ts (extended, following its existing execFileSync("php", ...) pattern already used in tests/phpunit-project-autoload.test.ts to execute the generated PHP directly rather than just type-checking the TS template):

  • case: only <file> declared runs exactly that file, no directory scan (the fix) — the regression test. Builds a tree with an unrelated *Test.php file sitting in the test root next to a file-only suite's declared file, and asserts discovery returns only the declared file. Without the fix in this PR, this fails with the unrelated file swept in (verified locally by reverting the fix and re-running).
  • case: <directory> declared stays scoped to it (unchanged) — control case, proves directory-declaring suites are unaffected.
  • case: neither <directory> nor <file> keeps the whole-test-root default (unchanged) — control case, proves the whole-tree default (used by configless/implicit discovery) is preserved.

This exercises the default discovery path (no --testsuite selection), which is the path the production bug actually lived in. It's distinct from the existing selected_testsuites coverage in tests/phpunit-project-autoload.test.ts, where $directories is unconditionally reset to array() before config parsing and was never affected by this bug — that's why that existing test passed both before and after this fix.

Ran the full existing PHPUnit-related test files (phpunit-discovery-only, phpunit-project-autoload, phpunit-result-paths, phpunit-runtime-failure-diagnostics, phpunit-runtime-rejection, phpunit-structured-evidence) plus npm run build — all pass.

Related

A <testsuite> with only <file> entries and no <directory> left
$directories at its whole-test-root default, so discovery unioned
the declared files with a recursive scan of the entire test tree.

Real PHPUnit runs exactly the declared files for a file-only suite.
Now $directories is only cleared to array() when the suite declared
files but no directories; directory-declaring and empty-declaration
suites are unaffected.

On extrachill-events, a suite declaring one file (2 test methods)
reported 88 tests / 615 assertions from this bug, and unrelated
sweep-ins fataled other suites with symbol redeclaration errors
(Extra-Chill/extrachill-events#846).

Fixes #2516
@chubes4
chubes4 merged commit a7d8d76 into main Sep 21, 2026
5 checks passed
chubes4 added a commit to Extra-Chill/extrachill-events that referenced this pull request Sep 21, 2026
…ry fix (#869)

Advances HOMEBOY_WP_CODEBOX_REF from 82e0216 to a7d8d76, the merge commit
of Automattic/wp-codebox#2517.

Before that fix, a PHPUnit testsuite declaring only <file> entries and no
<directory> had its discovery fall back to scanning the entire tests/ tree.
On this repo booking-alpha-concurrency declares one file containing two test
methods and was executing 88 tests, one of which trapped the PHP-WASM
interpreter and failed the whole gate.

Real PHPUnit runs exactly the declared files for a file-only suite, which is
what a7d8d76 restores.

Refs Automattic/wp-codebox#2516, #846
chubes4 added a commit to Extra-Chill/extrachill-events that referenced this pull request Sep 22, 2026
…ry fix

The redeclaration fatal (wp_get_ability + 3 duplicate classes) is fixed and
confirmed clear in two consecutive CI runs. The remaining failure is a
'php_wasm_unhandled_rejection: RuntimeError: unreachable' crash in
booking-alpha-concurrency — reproduced identically on both runs, unrelated
to anything this branch changed.

That crash is downstream of a bug already filed and fixed upstream:
Automattic/wp-codebox#2516 documents that wp_codebox_phpunit_parse_config()
scans the entire tests/ tree for any PHPUnit testsuite declaring only
<file> entries with no <directory>, instead of running only the declared
files. booking-mysql, booking-concurrency, booking-alpha, and
booking-alpha-concurrency all fit that shape, so each one has been silently
executing the whole 93-file tests/ tree instead of the 1-4 files it
declares. Automattic/wp-codebox#2517 fixes the discovery function directly
and merged to wp-codebox main as a7d8d7699f3d703b92fbd3e61bd124d13a7ad75d,
27 commits ahead of the currently pinned 82e02165. Advancing the pin scopes
these four suites down to their declared files, which is the most direct
available way to stop pulling unrelated, unvetted test files (and whatever
in them trips the wasm crash) into a runtime that was never supposed to
execute them.

Pin-only change, same pattern as #863's homeboy-extensions bump; no suite
XML or repository test code touched.

Refs #846.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

PHPUnit discovery scans the whole test root when a testsuite declares only <file> entries, silently sweeping in unrelated tests

1 participant