1 Introduction
Tabular foundation models are here.
A “boring” spreadsheet can contain a scientific breakthrough based on years of research. A patient’s database entries may hold the keys to their diagnosis. A hedge fund’s edge may revolve around a single CSV file.
Many people think tables are boring, but I find them the most exciting data modality. They are also everywhere.
Tables are a structured mess
Tables are often called “structured”, because they often have clear, pre-defined formats, at least the tables that we use in machine learning and data science.
A table is a matrix of rows and columns, where each row is one instance of a single entity, and each column is one feature of that entity.
Within that structure, we may find very heterogeneous data, ranging from columns with pure text, another containing integers, a third one dates, and yet another mostly missing values. When we zoom out, we also find extreme heterogeneity between different tables: A log file can be a table with 2 columns, one for the timestamp, the other free text. Another table may contain participants of a study, and each column contains their expressions for genes, maybe thousands of genes, so a very different shape of table. In industry, a table may be millions of rows, containing a wild mix of columns with dates, free text, categories, small numbers, large numbers, …
Zooming out or zooming in, either way, it gets messy.
Foundation models conquered other data types
The current solution side for using such a wide variety of tables is as diverse as the tables themselves. Some approaches combine hundreds of decision trees. Other approaches are just linear models or one of the hundreds of ways to extend the linear model. Another approach tries to separate categories with support vectors (support vector machines). A wild zoo.
Looking beyond tabular, like image, audio, and text, a different picture emerges. For a long time, the situation was similar to tabular: to use machine learning with data from another modality, like images, you had a wide range of methods. This changed around 2012 when neural networks became deep learning, and all the other modalities switched slowly to that new foundation. In 2017, the transformer-based architecture (Vaswani et al. 2017) was introduced and kicked off another consolidation through enabling the training of foundation models. For language and beyond we now have very capable large language models. For vision, we have pretrained CLIP, DINOv2, and Segment Anything.
Foundation models are large models, pretrained on vast amounts of data that can be applied to a broad set of tasks.
While tabular matured during that time, it didn’t follow in the footsteps of other data modalities, but settled firmly on gradient-boosted tree ensembles. Don’t get me wrong, there have been countless, even successful, approaches to modeling tabular data with neural networks, but these have been added to the model zoo, and have not opened their own theme park.
Foundation models for tabular data
It took some time finding the right architecture and abstraction level for tabular transformation models, but a promising approach called prior-data fitted networks (PFNs) emerged (Müller et al. 2022) with the first PFN-based tabular foundation model called TabPFN (Hollmann et al. 2023). Since then, tabular foundation models have seen a flurry of research and new models. Just to name a few current foundation models, we have the TabPFN family by Prior Labs, TabICL by Inria, TabDPT by Layer 6, SAP-RPT-1-OSS by SAP, Seldon by Neuralk, Nexus by Fundamental, TabFM by Google, EXAONE-Tabular by LG, and more. And I suspect there is way more to come. The big question is whether tabular foundation models will just contribute to the zoo of approaches, or consolidate the field of tabular machine learning, as foundation models did for other data.
Tabular foundation models (TFMs) are large models that are pretrained to solve (predictive) tasks for tables.
I believe that foundation models will consolidate and transform the tabular space. More specifically, I believe that PFN-based ones with the following characteristics will:
- These models are transformer-based architectures with attention mechanisms.
- They are conceptually building on the idea of prior-data fitted networks.
- They are pretrained on mostly synthetic data (although real data can be used or fine-tuned on).
- Prediction is based on in-context learning: the classic training step is skipped; instead, training data is provided at prediction time.
Read the part Understanding Tabular Foundation Models to get an intuition about how these models work: a first look at using one (Chapter 2), prior-data fitted networks as the underlying idea (Chapter 3), how in-context learning turns that idea into predictions (Chapter 4), and how it all gets pretrained in the first place (Chapter 5). This is mostly based on the architecture of nanoTabPFN, an educational implementation of a TFM. We skip other approaches to foundation models such as hypernetworks, cross-table transfer models, and LLM-based approaches.
Some benchmarks show that TFMs outperform boosted tree ensembles. All while skipping hyperparameter tuning and model selection – the in-context learning kind of just works. For small datasets, it will also be faster than with other models. Read the part Applying Tabular Foundation Models to see some actual code examples and get a feel for how these new models work: classification (Chapter 6), regression (Chapter 7), quantile regression (Chapter 8), and time series forecasting (Chapter 9). However, this first part to that answer falls way too short and comes from a tabular zoo mindset.
I see foundation models not only competing with current machine learning algorithms, but becoming the literal foundation of tabular machine learning. Read Betting on Tabular Foundation Models to understand the broader perspective of how they fit into the tabular space and will, IMO, change it: tabular’s GPT moment (Chapter 10), what it means to inherit the LLM stack (Chapter 11), and where I think this is all heading (Chapter 12).