Skip to content

Harden Uncle Bob harness against false-green checks - #489

Merged
tninja merged 2 commits into
mainfrom
harden-uncle-bob-harness
Aug 23, 2026
Merged

Harden Uncle Bob harness against false-green checks#489
tninja merged 2 commits into
mainfrom
harden-uncle-bob-harness

Conversation

@tninja

@tninja tninja commented Aug 22, 2026

Copy link
Copy Markdown
Owner

Summary

Follow up on #469 by selectively incorporating the highest-value hardening from the current AmazingAng/old-coder skill without turning the bundled harness into a mirror of upstream.

This PR focuses on failure modes where the gauntlet can look green without actually enforcing the intended constraint.

Changes

  • require every gauntlet layer to be an executable gate, not merely a report
  • require coverage gates to fail non-zero when the requirement is missed
  • prefer established mutation tools; manual mutation must prove each mutant was applied and executed
  • make custom/home-grown checkers fail closed and require a known-bad negative control before trusting green
  • distinguish N/A, UNAVAILABLE, and SUBSTITUTED layers in EVIDENCE
  • require orchestration to verify that every declared required layer actually ran
  • clarify that answering a spec clarification question is not approval of the revised spec
  • persist the approved spec when practical and tighten isolated-working-tree guidance

Both bundled prompt variants are kept in sync; the diagnostics variant retains its existing diagnostics baseline/get_diagnostics constraint.

Why

Recent upstream old-coder dogfooding found several important false-green cases: coverage commands that only printed percentages, manual mutation runners that could report kills without executing the intended mutant, and gauntlet orchestration that could stay green when a layer never ran. These are general harness lessons worth carrying into ai-code-interface.el.

I intentionally did not vendor upstream's independent verifier protocol, old-coder-api, demo-specific source-state machinery, or other ecosystem-specific additions. The goal is to cherry-pick durable harness principles rather than synchronize prompt text.

Upstream reference point reviewed: AmazingAng/old-coder through commit a0eb529d393a1cb3ccc564e32b2104e7e75c7a29 and the current skills/old-coder/SKILL.md.

Testing

Prompt-only change; no Elisp runtime behavior is modified. The two prompt variants were compared together to keep their shared harness contract aligned.

@chatgpt-codex-connector chatgpt-codex-connector Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: 32d87685da

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

Comment on lines +67 to +69
never skip a layer silently. Every required layer must be an executable gate,
not merely a report: if it did not run, crashed, silently skipped required
inputs, or cannot enforce its stated constraint, treat the layer as failed.

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

P2 Badge Apply the hardening to the Harness+ prompts

When a user selects uncle-bob-coding-agent-harness-plus, ai-code--auto-test-harness-file-name routes them to the separate Harness+ or Harness+-diagnostics file, but both still contain the old report-only coverage and mutation guidance and omit the new fail-closed checks. Consequently, the selectable maintainability-enhanced harness remains vulnerable to the exact false-green behavior this change is intended to prevent; update both Harness+ prompt variants with the shared hardening clauses as well.

Useful? React with 👍 / 👎.

@tninja
tninja merged commit 06552f0 into main Aug 23, 2026
4 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant