Skip to content

Embedding models on real articles (#203)

52 openly licensed articles from backend/tests/fixtures/real_pages (13 each from the Go blog, Rust blog, Kubernetes blog and Wikipedia; licenses per article in manifest.json). Generated by backend/scripts/embedding_eval.py on 2026-10-21, CPU only.

Model Dim Max tokens Title to article MRR (R@1) Summary to article MRR (R@1) Same-source neighbours P@5 Docs/s Note
all-MiniLM-L6-v2 384 256 0.928 (0.87) 0.980 (0.96, n=50) 0.800 68.8 current default
bge-small-en-v1.5 384 512 0.952 (0.90) 0.990 (0.98, n=50) 0.788 26.3 same 384-d column
nomic-embed-text-v1.5 768 8192 0.710 (0.63) 0.786 (0.76, n=50) 0.750 0.6 768-d, 8k context
bge-m3 1024 8192 0.981 (0.96) 0.990 (0.98, n=50) 0.788 0.2 1024-d, 8k context, multilingual
  • Title to article: the page title is the query; all article bodies are the corpus.
  • Summary to article: an LLM-written brief of the article is the query (no shared title).
  • Neighbours P@5: share of each article's 5 nearest articles from the same source.
  • Docs/s: encoding speed of full article bodies on this machine's CPU.

What it shows

  • bge-small-en-v1.5 is the better default. It finds the right article more often than MiniLM (title MRR 0.952 vs 0.928; summary 0.990 vs 0.980) with the same 384-d vectors, so it fits the existing content_items.embedding column. Switching means re-embedding, not a migration. It is 2.6 times slower to encode, which is still fast (26 articles a second on a laptop CPU).
  • bge-m3 is the most accurate (title MRR 0.981) and multilingual, but at 0.2 articles a second on CPU (about 5 s per full article) it is too slow to embed every ingested item on a hosted CPU box. Worth revisiting only with a GPU or for non-English readers.
  • Nomic v1.5 did worst here (title MRR 0.710). The score is identical under transformers 4 and 5 (its remote code needs a small shim on 5, see the script), so this is the model on this data, not a loading bug. The planned 768-d Nomic cut-over (#244) is not supported by this evidence.
  • Same-source neighbours were 0.75 to 0.80 for every model: grouping by source doesn't separate them, so it says little about interest matching.

Limits

52 English technical articles from four sources is small, and titles share words with their articles, which favours every model. The summaries are a harder, independent paraphrase test. Full articles were encoded whole; long context models might do better on chunked text. Re-run on reader data before any switch, ideally with real queries from search logs.

Reproduce

From backend/:

python scripts/embedding_eval.py --out ../docs/eval/embedding-models.md

The summaries come from docs/eval/summary-models.jsonl (#225). Model weights download on first run (about 3 GB for all four).