Applied & Classical ML · entry 01/06 · 3 min read

Tables still rule

Most machine learning in production operates on rows and columns, and on that data a gradient-boosted tree still routinely beats a neural network — for reasons worth understanding.

What most deployed ML actually is

Behind the headlines, the overwhelming majority of models in production consume tabular data: a row per customer, transaction, sensor reading or claim, and columns of mixed types — numbers, categories, dates, a few free-text fields someone regrets. Credit decisions, churn prediction, pricing, demand planning, fraud scoring, insurance underwriting, ad targeting. None of it involves a transformer, and the volume dwarfs everything else the field talks about.

Why trees win here

Gradient boosting over decision trees — the family behind XGBoost, LightGBM and CatBoost — is the production default, and was the clear benchmark winner through the early 2020s. That part is now genuinely contested: tabular foundation models, pretrained transformers that consume a whole small dataset in one forward pass, have started edging out tuned trees on small and mid-sized problems, while trees hold their lead on heavily categorical data and on larger tables. The reasons trees do so well are specific and worth knowing. Networks are biased toward smooth functions and are close to rotation-invariant — but in a table each column means something particular, so freely mixing columns destroys information a tree preserves, and real tabular relationships are irregular and threshold-shaped ("above 90 days overdue" behaves differently). Networks are also badly hurt by uninformative columns, which every real table has. Trees are invariant to monotonic rescaling, so skewed inputs need no normalization. XGBoost and LightGBM learn which way to send a missing value at each split, making missingness information rather than an error to impute away. And they train in minutes on a laptop and serve without a GPU, which decides more projects than any leaderboard.

The boosting idea in one paragraph

Fit a shallow tree. Look at what it got wrong. Fit a second shallow tree to those errors — formally, to the gradient of the loss, which for squared error is exactly the residual. Add it to the first, scaled down by a learning rate. Repeat until a held-out score stops improving, because boosting does overfit given enough rounds. Each tree is a weak model correcting the ensemble's current mistakes, and the sum is a strong one: the same measure the error, step toward less of it loop, walked in function space instead of weight space.

When to reach past them

Deep learning earns its place when the row contains something with structure inside it — long free text, an image, a sequence of events per entity — or when you want one model to serve many related tasks. The strong pattern is hybrid: use a network or an embedding model to turn the unstructured column into features, then hand those features to the tree with everything else.

Failure mode

Reaching for a neural network because the problem is important. Model class is rarely the constraint on a tabular project; the features and the label definition are. A team that spends its first month on architecture instead of on what the target actually means, and on whether each feature will exist at prediction time, ships something worse than a baseline that took an afternoon.