feat(harness): fail a project Skill that points at a path it invented

[harlan-zw]

🔗 Linked issue

Related to #33

❓ Type of change

• 👌 Enhancement

📚 Description

#154 gave generate-project-skill sharper navigation rules, and I had no way to know whether an Agent follows them. A project Skill that names src/server/routes.ts reads exactly like one that names a real file. The Agent cannot check its own pointers, and neither could we.

The Harness already prepares the project source, so it knows every path the Agent could have seen. Anything else in an inline code span is invention, and promotion fails with the path named.

evals/ scores the same Skills on the direct path, where no check runs. Three cases, each run with the Skills loaded and without, so the delta is the value the Skills add rather than the model's baseline.

Case	With	Without	Δ
package-not-project	1.00	0.67	+0.33
project-skill-real-paths	1.00	0.75	+0.25
project-skill-rust	1.00	0.75	+0.25

Two runs per arm, Claude Code 2.1.278, $4.64. Not in CI: each run is a real Agent session.

The suite paid for itself on the first pass. Both Rust failures were a false positive I had written into the check: a run had reported that src/lib.rs is absent, which is a fact about the project, and the check called it an invented path. A line that denies a path now passes.

Loose ends:

• package-not-project is the weakest case. Baseline scored 0.67 there, so it is close to measuring the model.
• .claude-plugin/plugin.json exists so claude plugin eval can resolve skills/ as a target. It makes the repository loadable as a Claude Code plugin, which nothing else here relies on. Question: do you want that as a real distribution path, or should the manifest live somewhere that does not imply it?
• The check reads inline code spans, which is a heuristic. I biased every rule towards accepting, since a false rejection blocks a correct Skill. The unit tests fix the conservative cases I know about.

🤖 AI disclosure: Harlan Agent Kit modified this description. My AI open-source policy.

[harlan-github-agent]

🤖 MERGED

Harlan Agent Kit posted this automated review. It is not Harlan's personal review or approval. AI open source policy. Last updated: 2026-09-22 08:21 UTC.

GitHub merged this pull request.

• A prose mention of a bare file extension such as .md fails promotion as an invented path. Next: Reject tokens of the shape /^.[A-Za-z0-9]+$/ in pathCandidate, or require a non-dot character before the final extension dot when the token has no slash.
• A search command outside backtick fences or inline spans fails the mandatory search-command check. Next: Run searchCommand over the full SKILL.md text instead of only fenced backtick blocks plus inline spans, since widening the scan can only reduce false rejections.

46fb9903-be57-4d90-a5e7-2c9ee44562da