10  Tabular’s GPT moment

Tabular foundation models are consolidating a scattered field into one general-purpose engine.

Before GPT (Radford et al. 2018), approaches to natural language problems were scattered: statistical phrase-based systems for translation, rule-based pipelines like ELIZA for chatbots, n-gram language models for text and code completion, and hand-crafted pipelines for style transfer, each its own specialized model. Today, GPTs can do it all, and have evolved to something beyond.

Traditional machine learning is scattered

My approach to getting into machine learning was reading the book “Elements of Statistical Learning” (Hastie, Tibshirani, and Friedman 2009) cover to cover. This was a bit like visiting a zoo populated with machine learning algorithms where one was more exotic than the next: generalized linear models, decision trees, wavelets, Gaussian mixture models, neural networks, and many more. For many supervised tasks, the field has converged to boosted tree ensembles, but many tasks are still dominated by other algorithms: clustering is mostly k-means, time series forecasting often still runs on ARIMA and exponential smoothing, outlier detection on isolation forests, and survival analysis on Cox proportional hazards models. Tabular foundation models have the potential to unify the tabular landscape and are absorbing the space piece by piece. This works because they are general-purpose engines that can cover a broad range of tabular machine learning tasks as described in Chapter 10.

In the Application part, I covered applications of classification and regression like every other educational book: with easily accessible, polished datasets and rather simplistic tasks. But what happens beyond?

Performance suffers beyond friendly benchmarks

The strongest evidence in favor of TFMs comes from academic benchmarks such as TabArena (Erickson et al. 2025), which feature “friendly” data: independent and identically distributed (i.i.d.), small to midsize, containing mostly high-signal features, and so on. Reality rarely gives us such friendly tasks. What if we move beyond these friendly benchmarks to the more complex tasks and datasets? A lot of evidence is pointing towards foundation models not being there just yet:

  • Boosted tree ensembles still beat TFMs on datasets with many categorical features, larger binary classification tasks (roughly above 44,000 rows), and imbalanced multi-class problems (Neuralk-AI 2025).
  • Industrial and enterprise data is messy, contextual, and non-i.i.d., unlike the tidy datasets TFMs shine on (Keßler et al. 2026; Kim et al. 2026). On the enterprise benchmark EGI-Bench, a plain untuned random forest beats TabPFN-2.5 and TabICLv2, and the only TFM that holds up is ConTextTab, which handles strings natively; rankings are inconsistent with those on public benchmarks like TabArena (Erickson et al. 2025).
  • Another disadvantage compared to traditional machine learning: In-context learning estimates the posterior based on metrics during pre-training. Which may or may not align with your use case. In the insurance pricing article cited above (Ruppelt 2026), this was a problem.

These current caveats are real, but I see them as milestones rather than roadblocks, since TFMs, and especially the underlying prior-data fitted network idea, are general-purpose engines.

PFNs are general-purpose engines

TFMs and their PFN-foundation are general-purpose engines:

A pretrained tabular foundation model is also a general-purpose engine: it learns in-context, skipping the training step; the performance is very good out-of-the-box for many datasets; it outputs the entire predictive distribution, not just a point estimate; it deals with all kinds of features by default. Since TFMs are neural networks, we also get all the respective perks like fine-tuning, re-using embeddings, and flexibility in the architecture such as different encodings of input features.

Prior-data fitted networks are the conceptual basis on which we can pretrain models to solve whatever task we simulate or sample for the prior (Müller et al. 2022). The PFN-base is used to train regression and classification TFMs. But you could use that same engine to produce in-context models for Bayesian optimization or treatment effect estimators. Or you can add datasets/tasks to the prior to “fix” inductive biases of the TFMs produced by the PFN.

With both pretraining and model being very flexible, PFN-based TFMs are a potential foundation for all of tabular machine learning.

Patching the prior

Let’s say you develop a TFM and discover it fails when the target oscillates sine-wave-like along a feature, for example for sensor readings or periodic features. While your TFM is pretrained on the prior of millions of synthetic tasks, none of those contains such high-frequency oscillating patterns. As a fix, you patch the prior: Simulate datasets with oscillating patterns, add them to the pretraining, and with the next update, everyone using your model gets the capability of handling oscillating data. This scenario was not hypothetical: When Prior Labs updated TabPFN v2.6 to TabPFN v3, they patched the prior by adding sinusoidal activations (Grinsztajn et al. 2026). The prior is an entry point to improve the “capabilities” of a model, or, in other words, to instill inductive biases (see Chapter 5).

Other examples of “prior patching”: TabICL mixes tree-based structural causal models into its prior to instill the inductive biases of tree-based models (Qu et al. 2025). This all comes on top of PFNs already being able to adapt to inductive biases through in-context learning alone, as research suggests (Zheng et al. 2025).

If you can simulate it, you can pretrain for it

Regression and classification are just the first tasks to conquer. With the PFN-engine, it’s possible to cover a much wider range of tasks: As long as you can simulate or sample a suitable prior, you can pretrain a TFM from scratch. The same paradigm that produces TabPFN, TabICL, ConTextTab, and so on can produce other families of models. These are just some first examples of models pretrained for either different tasks or differently structured data including their model names:

Typically, adapting the TFM to a different task also requires changes in the architecture, at least in the target decoding.

But a dedicated model isn’t always necessary, because regression and classification can be enough. Tasks like time series forecasting as in Chapter 9, or survival analysis (Vargas Vieyra 2026) can be translated into regression or classification tasks. Also, TFMs can make great plug-and-play non-parametric estimators, such as for two-stage hurdle models or double machine learning (Chernozhukov et al. 2018).

Fine-tuning tabular foundation models

Pretraining new foundation models is more a thing for research labs, not something a small data science department would do if they have to build a churn model by next month. Another option to make a TFM work for a given task is to fine-tune an existing model with the new data, potentially with a different prediction head. This is possible because TFMs are neural networks at heart: You can make changes to the architecture, re-use embeddings, fine-tune, …

People fine-tune for quite different reasons:

  • Prior Labs fine-tuned TabPFN-3 on synthetic time-series data for probabilistic forecasting (Grinsztajn et al. 2026).
  • You can fine-tune proper scoring rule metrics if you are in a regression setup and want the model to optimize for an objective beyond the mean (Landsgesell, Knoll, and Wenzel 2026).
  • Real-TabPFN takes TabPFNv2 and continues its pre-training on a curated set of real tables from OpenML and Kaggle (Garg et al. 2025).
  • TabPFN was fine-tuned into FinPFN for regime-aware stock return prediction (Wang and Lera 2026).

Fine-tuning also addresses a problem of TFMs: TFMs are pretrained on specific metrics, which may be different from the metric in your current application.

TFMs are the first general-purpose technology I see to consolidate tabular machine learning. I would even go as far as calling TFMs the GPT-moment of tabular machine learning. We will have to see whether it’s “just” a unification, or whether this new paradigm amounts to something beyond, just like GPT did for the LLM and AI space.