Summary

Today we clarified why Signora's retrieval benchmark was showing low canonical Recall@K even when semantic search was returning useful evidence.

The important distinction is that Signora retrieves feedback atoms, which are meaning-sized chunks extracted from larger feedback items. A benchmark question can have many valid atoms that satisfy the same product topic, customer segment, severity, and text predicate. Exact citation recall only checks whether retrieval returned the specific pre-selected atom IDs, while predicate coverage checks whether retrieval returned any valid evidence matching the question's verified rule.

What I Ran

• Built the text-embedding-3-small index under dataset/index/openai_small.
• Ran Recall@20 and Recall@50 sweeps for OpenAI-small dense retrieval and the current hybrid retrieval baseline.
• Inspected the weakest dense retrieval cases for:
  • v2_q_015: high-severity offline task viewing
  • v2_q_027: high-severity support response time
  • v2_q_043: individual-customer Slack integration
• Updated scripts/evaluation/run_benchmark.py so normal benchmark output now reports Recall@20, Recall@50, Predicate coverage@20, and Predicate coverage@50.

Results

OpenAI-small default hybrid benchmark result:

• Canonical Recall@5: 0.092
• Canonical Recall@10: 0.109
• Canonical Recall@20: 0.109
• Canonical Recall@50: 0.109
• Predicate coverage@20: 0.338
• Predicate coverage@50: 0.176
• Precision@5: 0.953
• Abstention accuracy: 15/15

The dense-only diagnostic had similar canonical recall, but every test question had at least one predicate-matching result by @20.

Interpretation