RAG & Embeddings · entry 01/04

Embeddings

An embedding model turns text into a list of numbers arranged so that geometric closeness tracks similarity of meaning — the quiet trick under modern search.

Text becomes a point

An embedding model reads text and outputs a vector — 768, 1024, sometimes 3072 floating-point numbers. That vector is a coordinate in a space the model learned during training, arranged so that texts with similar meaning land near each other. "How do I reset my password" and "I'm locked out of my account" share almost no words, but their vectors sit close together, because the model was trained — typically by contrastive learning over millions of text pairs — to pull paraphrases near and push unrelated texts apart. Keyword search matches strings; embeddings match meaning. That single trick powers semantic search, recommendation, and the retrieval half of RAG.

Distance, in practice

Similarity is usually cosine similarity — the angle between two vectors, with their lengths ignored:

import numpy as np

def cosine(a, b):
    return np.dot(a, b) / (np.linalg.norm(a) * np.linalg.norm(b))

# 1.0 = same direction; near 0 = unrelated

Most current models ship vectors already normalized to unit length, at which point cosine collapses to a plain dot product. The absolute score means little — 0.83 is not "83% similar" — only the ranking matters, and every model has its own score distribution, so a threshold tuned on one model never transfers to another.

What geometry captures — and misses

Embeddings are excellent at topic, paraphrase, and "these two are about the same thing." They are bad at exactly what engineers assume is easy. Negation: "the drug reduced mortality" and "the drug did not reduce mortality" embed almost identically — same words, opposite claims. Numbers: raising a limit to 10 versus 100 barely moves the vector. Exact identifiers — part numbers, error codes, names — drown among a thousand near neighbors. Where exact strings decide correctness, pair vectors with keyword search instead of trusting geometry alone.

A different animal from chat models

Embedding models are not chat models with the lid off. They are separate, usually far smaller transformers trained on a different objective, and they generate nothing: text in, vector out, at a fraction of the price per token. That makes them useful well beyond RAG — cluster support tickets into themes, deduplicate near-identical documents, classify by training a tiny head on frozen vectors, recommend "more like this" by nearest neighbor. Any task shaped like "which texts are alike" is an embedding task before it is a generation task.

Failure mode

Mixing vectors from two models — or two versions of one model — in the same index. Vector spaces are not compatible across models: a query embedded with the new model scores garbage against documents embedded with the old one, and nothing errors; result quality just quietly craters. Upgrading the embedding model therefore means re-embedding every document you have. Price that in when you choose a model, not the week you want to switch.