[Bug] memos_search: top-K contract, score calibration, and priority-dominated ranking suppress effective recall

[steven-ym-jiang]

[Bug] memos_search: top-K contract, score calibration, and priority-dominated ranking suppress effective recall

Component: apps/memos-local-plugin (bridge 2.0.16)
Reporter: hermes agent user, agentKind=hermes, profileId=default
Severity: High (silent false positives; agent may adopt wrong memory and stop searching)

---

Summary

A systematic recall benchmark (66 queries, methodology below) of memory.search
over a live local database found three reproducible issues that together suppress
effective recall despite the presence of retrievable content:

Dimension	Observed
Recall@1	14.6% (7/48 core queries hit the ground-truth item)
False-positive rate	8/8 (100%) — all 8 never-stored-topic queries returned a high-score result
Top-K contract	requested topK:50, observed exactly 1 result in every query
Score calibration	positive and negative queries both score ~1.05–1.32; minScore 0.45 filters nothing
Stability	100% deterministic, including deterministically wrong top-1

The embedding/indexing layer is healthy (semantic paraphrase variants hit);
the failure is in the output/ranking pipeline.

---

Issue A — Top-K contract: topK is ignored, output hard-capped at 1

Request memory.search { ..., topK: { tier1:50, tier2:50, tier3:50 } } returns exactly
one hit for every one of the 66 queries. Recall@3/@5 are not measurable on the
current bridge.

Expected: return up to K candidates so the consumer can validate content-level relevance.
Actual: the first (and often wrong) candidate is the only one ever surfaced.

Issue B — Score calibration: composite-score scale change (2.0.16) broke the relevance floor

After the 2.0.16 change to priority-weighted composite scores, relevant AND irrelevant
results cluster in the same band:

• core (positive) score range: 1.05–1.32
• negative (never-stored topic) score range: 1.05–1.13

minScore = 0.45 is therefore a no-op — the threshold has lost its meaning as a
relevance floor. Need upstream guidance on the new composite-score scale / relevance
semantics (or a re-calibrated threshold).

Issue C — Ranking: support × priority dominates, one episode hijacks unrelated queries

A single high-priority episode (ep_t64x2cdcc26p, a P1-A/B closure review) was returned
as top-1 for a large fraction of unrelated queries (dominant across 18/48 core misses),
and even for a same-domain competition query where the expected item existed
(competition C2: read_file offset limit ... expected po_9k7sfdrehzxe, top-1 was the
dominating episode, score 1.126).

This suggests semantic relevance is being overridden by priority/support weight in
ranking. Investigate whether priority weight should be bounded or applied only as a
tie-breaker.

---

Methodology (reproducible)

• 12 ground-truth items sampled from the live DB (traces×4, policies×3, world_model×2, skills×3, all with vectors)
• 4 query variants per item (verbatim / synonym / semantic-indirect / typo-noise) = 48 core queries
• 8 negative queries on topics never stored (false-positive probe)
• 4 same-domain competition queries + 2 stability queries × 3 runs
• Direct JSON-RPC over stdio to a fresh dist/bridge.cjs --agent=hermes --no-viewer
• Hit = top-1 returned refId matches the item's episode (trace→traces.episode_id, policy/wm→source_episodes_json; skills matched by snippet since IDs are redacted ***)

Full harness: memos-recall-benchmark skill (runner → raw_results.jsonl → finalize → metrics.json → report). Happy to attach artifacts or re-run on request.

Suggested investigation order

1. Top-K cap (biggest consumer-visible defect — blocks any downstream validation)
2. Score calibration (make minScore meaningful again or document new semantics)
3. Priority weight in ranking (bound the dominance)