Date: 06/26/2026
hour spent : 04 hours
To evaluate the impact of different embedding models on retrieval performance, we created a benchmark dataset containing 130 retrieval questions with their corresponding expected atomic items. We implemented eight embedding pipelines and generated eight evaluation files, one for each embedding model.
The objective was to determine whether changing only the embedding model—while keeping the retriever and retrieval pipeline unchanged—would result in measurable differences in retrieval performance. However, due to issues in the environment configuration and retriever setup, the experiment did not produce reliable results, preventing a fair comparison of the embedding models.
To establish a baseline for document retrieval performance, we evaluated multiple embedding models using a simple retrieval pipeline. The evaluation results for each embedding model were generated separately and stored in eight JSON files, each containing the embedding model information, evaluation status, retrieval metrics (where available), and diagnostic details.
During the investigation, we identified that the required environment configuration (.env) was not properly configured, preventing several embedding services from initializing correctly. Consequently, some embedding models were unable to generate or store vector embeddings, resulting in missing vector indexes and incomplete evaluations.
Although the Hugging Face-based embedding models successfully generated responses, the retrieved documents were consistently inaccurate and lacked semantic relevance to the input queries. This suggests that the retrieval pipeline—particularly the embedding generation, vector indexing, and retriever configuration—was not correctly established. As a result, the semantic retriever failed to return meaningful documents.
The generated evaluation files are summarized below.
| File Name | Embedding Model | Contents |
|---|---|---|
openai_large_benchmark.json |
text-embedding-3-large |
{ "model": "text-embedding-3-large", "slug": "openai_large", "evaluation_version": "retrieval_questions_v1 (130q + 21 abstain)", "last_updated": "2026-06-25", "status": "unavailable", "reason": "index not built — run embed_openai_large.py first", "metrics": null} |
openai_small_benchmark.json |
text-embedding-3-small |
{ "model": "text-embedding-3-small", "slug": "openai_small", "evaluation_version": "retrieval_questions_v1 (130q + 21 abstain)", "last_updated": "2026-06-25", "status": "unavailable", "reason": "index not built — run embed_openai_small.py first", "metrics": null} |
voyage_benchmark.json |
voyage-large-2 |
{ "model": "voyage-large-2", "slug": "voyage", "evaluation_version": "retrieval_questions_v1 (130q + 21 abstain)", "last_updated": "2026-06-25", "status": "unavailable", "reason": "index not built — run embed_voyage.py first", "metrics": null} |
cohere_benchmark.json |
embed-english-v3.0 |
{ "model": "embed-english-v3.0", "slug": "cohere", "evaluation_version": "retrieval_questions_v1 (130q + 21 abstain)", "last_updated": "2026-06-25", "status": "unavailable", "reason": "index not built — run embed_cohere.py first", "metrics": null} |
cohere_eval.json |
embed-english-v3.0 |
{ "model": "embed-english-v3.0", "overall": { "recall_at_8": 0.0, "precision_at_8": 0.0, "mrr": 0.0, "hit_rate_at_8": 0.0, "ndcg_at_8": 0.0, "questions_evaluated": 109 }, "by_difficulty": { "easy": 0.0, "medium": 0.0, "hard": 0.0 }, "abstention_correct": 21, "abstention_total": 21, "elapsed_seconds": 130} |
google_benchmark.json |
text-embedding-004 |
{ "model": "text-embedding-004", "slug": "google", "evaluation_version": "retrieval_questions_v1 (130q + 21 abstain)", "last_updated": "2026-06-25", "status": "unavailable", "reason": "index not built — run embed_google.py first", "metrics": null} |
minilm_benchmark.json |
all-MiniLM-L6-v2 |
{ "model": "all-MiniLM-L6-v2", "slug": "minilm", "evaluation_version": "retrieval_questions_v1 (130q + 21 abstain)", "last_updated": "2026-06-25", "status": "ok", "reason": null, "metrics": { "recall_at_5": 0.0, "recall_at_10": 0.0, "precision_at_5": 0.0, "citation_accuracy": 0.0, "unsupported_claim_rate": 1.0, "abstention_accuracy_num": 21, "abstention_accuracy_den": 21, "avg_latency_ms": 423.7602055051424, "mean_top1_semantic": 0.5587935699235409, "n_questions": 109, "ground_truth_status": "unverified_suggestion_pending_human_review" }} |
mpnet_benchmark.json |
all-mpnet-base-v2 |
{ "model": "all-mpnet-base-v2", "slug": "mpnet", "evaluation_version": "retrieval_questions_v1 (130q + 21 abstain)", "last_updated": "2026-06-25", "status": "ok", "reason": null, "metrics": { "recall_at_5": 0.0, "recall_at_10": 0.0, "precision_at_5": 0.0, "citation_accuracy": 0.0, "unsupported_claim_rate": 1.0, "abstention_accuracy_num": 21, "abstention_accuracy_den": 21, "avg_latency_ms": 476.6893467889648, "mean_top1_semantic": 0.5517800693665076, "n_questions": 109, "ground_truth_status": "unverified_suggestion_pending_human_review" }} |
The evaluation across all eight benchmark files indicates that the retrieval pipeline was not functioning as intended. Five embedding models could not be evaluated, while the remaining models completed execution but produced poor retrieval performance with inaccurate document retrieval.
Based on these observations, the reported results should not be considered representative of the actual capabilities of the evaluated embedding models. Instead, they indicate issues within the experimental setup, including the missing .env configuration, incomplete embedding generation, improper vector indexing, or incorrect retriever configuration.
Before conducting further benchmarking or comparing embedding models, the entire retrieval pipeline—including environment configuration, embedding generation, vector database indexing, and retriever initialization—should be rebuilt and validated to ensure that the system performs semantic retrieval correctly.
The embedding experiment produced mostly zero scores because the evaluation dataset was built incorrectly, not necessarily because the embedding models performed badly.
The main problem is that the benchmark questions were connected to random “correct answer” records from the dataset. For example, a question about enterprise onboarding might have been graded against unrelated records about notification settings or customer-support follow-ups.
Because the expected answers were random, even a good embedding model would retrieve sensible onboarding-related results and still be marked wrong. With nearly 9,000 possible records and only a few randomly selected “correct” records per question, getting zero matches was actually likely.
Other issues in the experiment: