AI Infrastructure & MLOps · entry 04/05

The ML lifecycle

Data pipelines, versioning, deployment patterns, monitoring, retraining — the operational discipline that separates a model that demos from a model that survives production.

Everything is versioned or nothing is

A production model is a compound artifact: code, weights, training data, preprocessing, hyperparameters, and environment. Reproducing it — for debugging, audit, or rollback — requires pinning all of them, which is why mature teams version data and features with the same rigor as code, track experiments with full lineage, and can answer "what exactly is running in production and what trained it?" in one query. This sounds bureaucratic until the Tuesday the model misbehaves and the alternative is archaeology. Regulated industries increasingly require the answer in writing.

Deployment is a rehearsal, not a leap

Models earn production traffic in stages. Shadow mode: the new model runs on live inputs, predictions logged, nothing acted on — free evidence on real distribution. Canary: a small traffic slice, watched against the incumbent. Ramp: gradual rollout with automatic rollback wired to metrics, because model regressions are often silent — no exception, no 500, just worse decisions at scale. The test suite is different too: the eval set is the spec, so promotion gates are eval scores plus behavioral checks on known hard cases, not unit tests alone.

Production is a distribution, and it moves

The world drifts away from the training snapshot: user behavior shifts, upstream systems change formats, seasons turn, cameras move. Monitoring therefore watches three layers: system health (latency, errors), input drift (are today's inputs statistically like training?), and outcome quality (sampled human review, delayed labels, proxy metrics). Retraining cadence is an economic decision — scheduled, triggered by drift alarms, or continuous — and every retrain reruns the whole gauntlet above. The loop, not the model, is the asset.

Failure mode

The silent feedback loop. The model's own decisions shape the data it later trains on: the recommender only shows what it predicts you'll click, so clicks confirm it; the fraud model blocks the cases that would have taught it; the hiring filter never sees outcomes for those it rejected. Retrain naively and the system studies its own reflection, confident and increasingly blind. Break the loop deliberately — hold out randomized traffic, log counterfactuals, audit for it — because this one produces great metrics all the way down.