fix(phpunit): scope file-only testsuites to declared files - #2517
Merged
Merged
Conversation
A <testsuite> with only <file> entries and no <directory> left $directories at its whole-test-root default, so discovery unioned the declared files with a recursive scan of the entire test tree. Real PHPUnit runs exactly the declared files for a file-only suite. Now $directories is only cleared to array() when the suite declared files but no directories; directory-declaring and empty-declaration suites are unaffected. On extrachill-events, a suite declaring one file (2 test methods) reported 88 tests / 615 assertions from this bug, and unrelated sweep-ins fataled other suites with symbol redeclaration errors (Extra-Chill/extrachill-events#846). Fixes #2516
chubes4
added a commit
to Extra-Chill/extrachill-events
that referenced
this pull request
Sep 21, 2026
…ry fix (#869) Advances HOMEBOY_WP_CODEBOX_REF from 82e0216 to a7d8d76, the merge commit of Automattic/wp-codebox#2517. Before that fix, a PHPUnit testsuite declaring only <file> entries and no <directory> had its discovery fall back to scanning the entire tests/ tree. On this repo booking-alpha-concurrency declares one file containing two test methods and was executing 88 tests, one of which trapped the PHP-WASM interpreter and failed the whole gate. Real PHPUnit runs exactly the declared files for a file-only suite, which is what a7d8d76 restores. Refs Automattic/wp-codebox#2516, #846
This was referenced Sep 21, 2026
chubes4
added a commit
to Extra-Chill/extrachill-events
that referenced
this pull request
Sep 22, 2026
…ry fix The redeclaration fatal (wp_get_ability + 3 duplicate classes) is fixed and confirmed clear in two consecutive CI runs. The remaining failure is a 'php_wasm_unhandled_rejection: RuntimeError: unreachable' crash in booking-alpha-concurrency — reproduced identically on both runs, unrelated to anything this branch changed. That crash is downstream of a bug already filed and fixed upstream: Automattic/wp-codebox#2516 documents that wp_codebox_phpunit_parse_config() scans the entire tests/ tree for any PHPUnit testsuite declaring only <file> entries with no <directory>, instead of running only the declared files. booking-mysql, booking-concurrency, booking-alpha, and booking-alpha-concurrency all fit that shape, so each one has been silently executing the whole 93-file tests/ tree instead of the 1-4 files it declares. Automattic/wp-codebox#2517 fixes the discovery function directly and merged to wp-codebox main as a7d8d7699f3d703b92fbd3e61bd124d13a7ad75d, 27 commits ahead of the currently pinned 82e02165. Advancing the pin scopes these four suites down to their declared files, which is the most direct available way to stop pulling unrelated, unvetted test files (and whatever in them trips the wasm crash) into a runtime that was never supposed to execute them. Pin-only change, same pattern as #863's homeboy-extensions bump; no suite XML or repository test code touched. Refs #846.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Problem
wp_codebox_phpunit_parse_config()(generated byphpunitConfigDiscoveryPhp()inpackages/runtime-playground/src/phpunit-command-handlers.ts) defaults$directoriesto the whole test root and only overwrites it when a<testsuite>declares<directory>elements. A<testsuite>that declares only<file>entries never triggers that overwrite, so$directoriessilently stays at the whole-test-root default and discovery unions the declared files with a recursive scan of the entire tree.Real PHPUnit does not do this: a
<testsuite>with only<file>entries and no<directory>runs exactly those files and performs no directory-based discovery at all. That's the reference behaviour this PR restores.The three cases
<directory>(with or without<file>)<file>, no<directory>Evidence
On
Extra-Chill/extrachill-events,booking-alpha-concurrencydeclares exactly one file (InternalBookingHoldConcurrencyMySQLProof.php, 2 test methods) and no<directory>. A run of that suite reported 88 tests, 615 assertions — 44x the declared content. Three sibling suites (booking-mysql,booking-concurrency,booking-alpha) have the same shape; a fourth (managed-wordpress), which does declare a<directory>, scoped correctly and served as the control case that pointed at this discovery function specifically.Downstream, the wide scan swept in a stub file three directories away from anything the failing suite's config named, causing
Cannot redeclare function wp_get_ability()and fataling the entire suite before any of its actually-declared tests ran. That's tracked separately as Extra-Chill/extrachill-events#846 — this PR fixes the root cause it surfaced; the stub itself is fixed independently over there.The fix
Two lines in
phpunitConfigDiscoveryPhp():$config_dirsand$filesare already accumulated separately from<directory>and<file>elements across all matched testsuites. This only adds the missing branch: when no<directory>was declared but at least one<file>was, clear$directoriesinstead of leaving it at the whole-tree default. Configs declaring<directory>(case 1) and configs declaring neither (case 3) take the same branches as before and are unaffected.This is the shared discovery-config parser used by both the plugin PHPUnit path (
wp_codebox_phpunit_parse_config) and the core PHPUnit path (core_pg_parse_phpunit_config) — both call sites get the fix since they share this generator function.Tests
Added to
tests/phpunit-discovery-only.test.ts(extended, following its existingexecFileSync("php", ...)pattern already used intests/phpunit-project-autoload.test.tsto execute the generated PHP directly rather than just type-checking the TS template):case: only <file> declared runs exactly that file, no directory scan (the fix)— the regression test. Builds a tree with an unrelated*Test.phpfile sitting in the test root next to a file-only suite's declared file, and asserts discovery returns only the declared file. Without the fix in this PR, this fails with the unrelated file swept in (verified locally by reverting the fix and re-running).case: <directory> declared stays scoped to it (unchanged)— control case, proves directory-declaring suites are unaffected.case: neither <directory> nor <file> keeps the whole-test-root default (unchanged)— control case, proves the whole-tree default (used by configless/implicit discovery) is preserved.This exercises the default discovery path (no
--testsuiteselection), which is the path the production bug actually lived in. It's distinct from the existingselected_testsuitescoverage intests/phpunit-project-autoload.test.ts, where$directoriesis unconditionally reset toarray()before config parsing and was never affected by this bug — that's why that existing test passed both before and after this fix.Ran the full existing PHPUnit-related test files (
phpunit-discovery-only,phpunit-project-autoload,phpunit-result-paths,phpunit-runtime-failure-diagnostics,phpunit-runtime-rejection,phpunit-structured-evidence) plusnpm run build— all pass.Related