Summary

After the benchmark showed text-embedding-3-small as the practical hosted winner and MPNet as the strongest local baseline, I started turning the experiment into the actual retrieval pipeline. The goal was to stop treating the benchmark as a one-off experiment and make the chosen model usable in the product workflow.

Decision

Use text-embedding-3-small as the production default embedding model for Signora retrieval.

Keep all-mpnet-base-v2 as the local/offline fallback and baseline.

The reasoning is simple: text-embedding-3-small led the core retrieval metrics in the benchmark while being cheaper than text-embedding-3-large. MPNet remains important because it is the strongest local baseline and produced the strongest held-out local abstention result.

What changed in the code

I added a production retrieval model adapter:

• scripts/retrieval_models.py

This file defines explicit embedding profiles:

• openai_small -> text-embedding-3-small, production default
• openai_large -> text-embedding-3-large, hosted comparison
• mpnet -> all-mpnet-base-v2, local baseline
• minilm -> all-MiniLM-L6-v2, fast local baseline

I added the production index builder:

• scripts/build_retrieval_index.py

This builds dataset/index/ from the processed feedback atoms. By default, it uses text-embedding-3-small. It also writes a self-describing index_manifest.json recording the embedding profile, model name, backend, corpus hashes, document recipe, record count, embedding dimensions, and ranking defaults.

I added the production query script:

• scripts/query_retrieval.py

This loads the generated index, uses the same embedding model recorded in the manifest, embeds the user query, ranks atoms with dense semantic similarity plus keyword fusion, applies metadata filters, deduplicates results by feedback ID and statement, and returns citation-ready atom/feedback evidence.