Harden Uncle Bob harness against false-green checks - #489
Conversation
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: 32d87685da
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
| never skip a layer silently. Every required layer must be an executable gate, | ||
| not merely a report: if it did not run, crashed, silently skipped required | ||
| inputs, or cannot enforce its stated constraint, treat the layer as failed. |
There was a problem hiding this comment.
Apply the hardening to the Harness+ prompts
When a user selects uncle-bob-coding-agent-harness-plus, ai-code--auto-test-harness-file-name routes them to the separate Harness+ or Harness+-diagnostics file, but both still contain the old report-only coverage and mutation guidance and omit the new fail-closed checks. Consequently, the selectable maintainability-enhanced harness remains vulnerable to the exact false-green behavior this change is intended to prevent; update both Harness+ prompt variants with the shared hardening clauses as well.
Useful? React with 👍 / 👎.
Summary
Follow up on #469 by selectively incorporating the highest-value hardening from the current
AmazingAng/old-coderskill without turning the bundled harness into a mirror of upstream.This PR focuses on failure modes where the gauntlet can look green without actually enforcing the intended constraint.
Changes
N/A,UNAVAILABLE, andSUBSTITUTEDlayers in EVIDENCEBoth bundled prompt variants are kept in sync; the diagnostics variant retains its existing diagnostics baseline/get_diagnostics constraint.
Why
Recent upstream
old-coderdogfooding found several important false-green cases: coverage commands that only printed percentages, manual mutation runners that could report kills without executing the intended mutant, and gauntlet orchestration that could stay green when a layer never ran. These are general harness lessons worth carrying into ai-code-interface.el.I intentionally did not vendor upstream's independent verifier protocol,
old-coder-api, demo-specific source-state machinery, or other ecosystem-specific additions. The goal is to cherry-pick durable harness principles rather than synchronize prompt text.Upstream reference point reviewed:
AmazingAng/old-coderthrough commita0eb529d393a1cb3ccc564e32b2104e7e75c7a29and the currentskills/old-coder/SKILL.md.Testing
Prompt-only change; no Elisp runtime behavior is modified. The two prompt variants were compared together to keep their shared harness contract aligned.