11 Inheriting the LLM stack
Tabular foundation models inherit the infrastructure, funding, and ecosystem built for large language models.
Context, capabilities, thinking mode … does some of the language around tabular foundation models sound familiar? It’s not a coincidence, as TFMs inherit the LLM stack.
Tabular finally gets a neural foundation
With TFMs being transformer-based neural networks, tabular gets a new base, which is much more aligned with the state-of-the-art of the other data modalities:
- Text: decoder-only transformers predicting the next token, which is the large language model.
- Images: vision transformers for recognition, and diffusion models with transformer backbones for generation.
- Speech: encoder-decoder transformers that map audio straight to text, as in Whisper.
- Video: the same recipe as images, with attention extended over the time axis.
- Protein structure: attention over residue pairs, as in AlphaFold.
Ok, tabular is finally more aligned with the other modalities, what’s the big deal? The big deal is a synergy between the modalities that has not been possible with boosted tree ensembles and other non-neural networks. Tabular becomes part of the innovation flywheel around the transformer architecture. To me, TFMs have been the last puzzle piece to connect tabular to all else. This interoperability and synchronization are not for free and will require years of research and development, but it’s happening.
TFMs inherit the LLM stack
In 2014, I trained my first deep learning model, a simple classifier for x-ray images. It was a pain, and my torturer was called TensorFlow. No shade against TensorFlow. Turns out I just trained the model in the wrong decade. Today, deep learning libraries have matured and something like PyTorch is a pleasure to use, compared to 2014.
Since then, GPUs have become more powerful, the transformer architecture has been invented, the deep learning ecosystem has grown (Hugging Face, Papers with Code, Weights & Biases), AI funding has exploded, and we are calling everything AI now. Deep learning had been going strong in the 2010s, but the huge boost happened in late 2022, when ChatGPT was released, and I guess that’s around the time people stopped calling it deep learning and are now calling it AI (again).
Researchers, engineers, financiers, and entrepreneurs are building out the LLM stack: AI compute centers, open source software, research all around transformers, capital, but also the talent and knowledge pool itself. LLM-based development is like this huge river, but the tabular ship is swimming on a slow trickling side stream. TFMs, in this analogy, are connecting this stream to the larger river, allowing tabular to benefit from everything happening in the AI space.
Tabular foundation models are not large language models, but are both based on transformer neural networks; that’s why they intersect on infrastructure and ecosystem.
Tabular becomes interoperable with other modalities
Multi-modality wasn’t impossible before, just clunky. At least for all non-neural approaches. For example, getting XGBoost to work on both tabular and text data requires either converting the text into tabular features, or feeding the tabular-based predictions into another model; none of that is learned end-to-end. In comparison, if both modalities can be modeled with a neural network, you can start thinking about multi-modal models: A unified approach that learns with multiple modalities end-to-end. Two footnotes here: Multi-modality does not require TFMs but the leverage comes from the base in neural networks, and it’s still hard work to find the right architecture to suit multiple modalities.
Since neural networks represent data as vectors (aka embeddings), we get something of a language in which we can represent things from images, text, and now also tabular. These embeddings may be re-used in other applications, be it in vector-based similarity search, as features for another model, or as input to a clustering algorithm.
An example of this is the TFM SAP-RPT-1-OSS, published as ConTextTab (Spinaci et al. 2025), which embeds the names of feature columns, the labels of categorical features, and all text features to enrich the table.
The infrastructure can now also be shared across modalities, as the tabular model runs on the same GPU and PyTorch installation as your text embedder or image embedder.
I’ve been in tabular machine learning for a long time, and have kind of internalized that working with tables typically means squeezing everything into table form, basically by featurizing any other modality: Turn images into features like color summaries, or use bag-of-words for text. Inelegant and lossy. I for one am looking forward to approaches that handle all kinds of data modalities using the right inductive biases and in a unified way.
The rise of the tabular foundation company
Boosted tree ensembles have dominated the tabular machine learning space since the mid-2010s, when XGBoost (T. Chen and Guestrin 2016) started topping Kaggle leaderboards. Despite their success, I have yet to see an XGBoost startup, or a CatBoost unit in a tech company. For TFMs, this story is different. Very different. Multiple startups are pretraining their own foundation models: Prior Labs (Germany), Neuralk-AI (France), Fundamental (US), Layer 6 (Canada), and Synthefy (US). Tech companies like Google, SAP, and LG are also competing in this space with their own models.
In the LLM space, we have a similar incorporation, with different providers developing their own models, from Anthropic’s Claude to Mistral’s Vibe and OpenAI’s ChatGPT. It’s only a matter of time until tabular foundation companies switch their model versioning from the easy-to-understand numbering to the unpredictable system common in the LLM space. Let’s see when we’ll get TabPFN Gauss (Extra) or TabICL Smaragd 5.12.
Many of these labs and companies have a semi-open source approach, where the inference code is published under a permissive license, while model weights get a non-commercial license, and the prior and the pretraining code remain closed.
Closed-source, GPUs, and checkpoints
The LLM stack also brings some complications to tabular.
While the TFM’s open source situation is much better than the LLM’s, it’s more closed than traditional machine learning. The machine learning community has a strong open source and sharing culture: you’ll find code repositories with friendly licenses openly published on GitHub and other platforms; research is often uploaded to arxiv.org making it available for free to everyone; people freely share their knowledge (counting myself and this book here as well). Looking to other disciplines, such an open culture is not a given, and something to cherish. Closing up the space would be a deviation from that culture.
Inheriting the LLM stack would also make the space more restrictive in other ways. For example, I don’t have a GPU at home. I am what they call “GPU-poor”. For tabular machine learning, this has never been an issue, as I could happily run all the CPU-based, traditional machine learning algorithms on my M1 MacBook Air. Technically, you can run TFMs on a CPU, and it’s fine with small datasets, but you’ll quickly find yourself looking for a GPU.
Last but not least, TFMs change how we manage models. Once TFMs become your central engine for making predictions, you have to think about how you update models, as it could come with unintended consequences in terms of performance. Even if a new version of a TFM seems better on paper, you have no guarantee it will be better on any or all of your particular tasks. Meaning you have to test and monitor. This happens in the LLM space: Researchers found that GPT-4 regressed from March to June 2023 on various tasks and instruction-following (L. Chen, Zaharia, and Zou 2023). For closed source models, an additional risk is model discontinuation. Again, an example from the LLM space: OpenAI replaced GPT-4o with GPT-5 in August 2025, which caused a backlash (OpenAI 2026).
Good or bad, I believe that the LLM inheritance will shape tabular’s future roadmap, which we zoom out and take a look at in the next chapter.