Date: 2026-06-26

Status: Local and OpenAI runs completed; Cohere, Google, and Voyage provider limitations documented

Pull request: #1 — Add reproducible embedding benchmark v2

Goal

Replace the previous embedding experiment with a reproducible benchmark that can fairly compare MiniLM, MPNet, OpenAI, Cohere, Google, and Voyage embeddings without treating good retrieval as a zero-score failure.

What was wrong with v1

The original benchmark had 130 questions, but most were repeats of only a few prompt templates. More importantly, its expected atom IDs were selected by corpus row number rather than by relevance to the question. A question about enterprise onboarding could therefore be graded against unrelated evidence about notifications or a support follow-up.

This explains the previous all-zero results. The corpus contains 8,897 atoms, while each question had only a few arbitrary expected IDs. A semantically correct result was almost guaranteed to miss those random IDs and be counted as incorrect. The v1 benchmark was retained for auditability but deprecated for model comparison.

Benchmark v2 design

Created a predicate-backed benchmark with:

• 59 unique retrieval questions across 20 supported product-feedback topics.
• 20 intentionally unsupported / no-answer questions.
• Explicit validation and test splits.
• Gold citations selected only when their stored corpus metadata satisfies the question's topic, customer-segment, or severity predicate.
• Preflight validation that stops the evaluation if an index has stale corpus data, wrong model identity, wrong vector dimensions, missing gold atom IDs, or gold records that violate a question filter.

The v2 benchmark measures both curated-citation recovery and broader predicate-backed relevance. This avoids falsely treating another relevant record as a failure merely because it was not one of a few canonical citations.

Framework implemented

Added scripts/benchmark/ with a shared model-adapter approach rather than one copy-pasted script per model:

• build_indexes.py builds self-describing, model-specific indexes from one fixed corpus and document recipe.
• models.py supports MiniLM, MPNet, OpenAI, Cohere, Google, and Voyage using the same document/query interface.