Summary

I built a reusable error-analysis script for the benchmark results so we can understand why Canonical Recall is low instead of guessing.

The purpose of this experiment was not to choose a new embedding model. The model decision had already been made: text-embedding-3-small is still the practical production winner. The purpose here was to inspect failure modes and answer this question:

Is low Canonical Recall a real retrieval failure, or is the model often retrieving different-but-still-relevant evidence?

What I built

Added a new script:

scripts/benchmark/error_analysis.py

The script reads existing benchmark result JSON files, joins them back to the benchmark questions and indexed atom metadata, and writes a human-readable error-analysis report.

It does not call OpenAI or any other embedding provider. It works from already-generated evaluation results, so it is cheap and safe to run repeatedly.

What the script explains

For each benchmark question, the report shows:

• the question text
• Canonical Recall, Precision, MRR, and nDCG
• expected canonical atom IDs
• retrieved atom IDs
• which canonical atoms were missed
• which retrieved atoms were relevant but not canonical
• atom statements for both missed and retrieved evidence
• an interpretation category

The script classifies each question into one of these categories: