AI Safety & Alignment · entry 05/09

Privacy & memorization

Models memorize slices of their training data and can be made to recite them — the extraction attacks, what actually mitigates, and why deletion is the hardest promise in machine learning.

Memorization is real and measurable

Language models generalize, and also memorize: repeated or distinctive training sequences — a phone number pasted into forums, a license header, a passage of a novel — can be reproduced verbatim under the right prompt. Extraction research made it concrete: adversarial prompting has pulled training text, personal details, and near-exact images out of deployed models, and memorization rises with model size, with duplication in the corpus, and with how much of the sequence an attacker can feed back as a prompt. This isn't a bug in one model; it's a property of high-capacity models fit to data — memorization that coexists with generalization rather than trading against it, surfacing on demand.

The quieter leaks

Verbatim recitation is the vivid case, not the whole threat. Membership inference asks a subtler question — was this person's record in the training set at all? — and sometimes answers it from output probabilities alone, which for a medical or financial corpus is itself a disclosure. And deployment leaks dwarf training leaks in practice: prompts and retrieved documents carrying data to a vendor, conversation logs retained for training, one user's pasted secrets surfacing in support tickets. A privacy review that only asks about weights has audited the smaller half, the pipeline is the rest.

What actually helps

Deduplicate training data — the single highest-leverage mitigation, since repetition drives memorization — and scrub high-risk identifiers upstream. Differential privacy offers real mathematical guarantees and real costs (utility drops as guarantees strengthen; frontier-scale DP pretraining remains impractical, though DP fine-tuning on sensitive data is deployed). Output filters catch exact regurgitation and miss paraphrase. And unlearning — removing one person's data after the fact — remains mostly an open problem: retraining without the data is the honest method and is priced accordingly, while approximate unlearning methods struggle to prove the influence is gone. "We'll delete it from the model" is, today, a promise to retrain, filter, or hope.

Failure mode

Testing privacy with a polite prompt. Teams ask the model "what do you know about Jane Doe," get a refusal, and file the risk closed — but refusal is a trained surface, and extraction attacks don't ask politely; they use completion prefixes, many samples, and the usual bag of tricks. If the data would be a breach when recited, the test is an adversarial extraction attempt under your red-team budget — or better, keeping it out of the corpus in the first place.