12 Roadmap and Challenges
Tabular foundation models may well become the foundation for tabular machine learning and data science, while becoming agentic.
Sometimes a new paradigm comes along and changes how things are done: Shipping containers standardized how we shipped goods and changed logistics forever; Large language models changed how we model language, with repercussions far beyond. Tabular foundation models might do the same to the tabular space.
Foundation models are eating tabular machine learning
Prior-data fitted networks are the universal engine that allows training TFMs well beyond classification and regression, as described in Chapter 10. Clustering, survival analysis, anomaly detection, OMICS data, missing value imputation, Bayesian optimization, causal inference, feature importance, time-series forecasting, … no task is safe. A direction I have not mentioned so far is relational data. The current approach to, say, churn prediction, is to join the customer table with the contracts table, the billing and payments table, the usage or activity log, and the support-ticket table. Table joins across relational databases for machine learning typically require making decisions about how to aggregate data from 1 to n relationships. Specifically pretrained TFMs, also called relational foundation models, can work attention across tables with the foreign keys. Combined with the growing capability for multi-modality, TFMs really creep into all the niches.
It will be, and in many ways is, way more convenient to use a TFM instead of traditional machine learning, because of the lack of training, hyperparameter tuning, and so on. The list of conveniences is steadily growing.
The largest technical barrier I see is still scaling to larger datasets and the slow inference. The main pain point is the attention mechanism: TabICLv2, for example, scales as \(O(n^2 + nm^2)\) for a table with \(n\) rows and \(m\) columns (Qu et al. 2026), and doubling the data can more than double the runtime. So, maybe large datasets will remain safe, at least for a while. However, even there, there are still many knobs to turn: Labs are still investing heavily in Research and Development, and architectures are still evolving. Also, there are already some solutions: Context engineering, KV caching, and distillation into gradient-boosted trees or multi-layer perceptrons for fast inference. A counter point to distillation: If you have to distill, isn’t that just traditional machine learning with extra steps? However, it still supports, in my opinion, a TFM-centric machine learning approach: the teacher model is always the TFM, and you can engineer very efficient distillation approaches.
Building platforms around foundation models
TFMs are just functions that take as input training and test data, and output the prediction. Yet, that’s not what you’ll find when you look for them, as the trend is to build entire ecosystems around them.
To me, the model itself is a focal point, regardless of whether it’s a traditional model or a foundation model. It’s the point where all our task-relevant data converges and gets projected into our target space. Before the focal point, we converge: decide on the data, specify the task, engineer features, clean and merge data, and more. After the focal point, we diverge: compute performance metrics, interpret the predictions, quantify uncertainty, monitor, study fairness, and more. Both this convergent pre-processing and the divergent post-processing are being added to the tabular foundation ecosystem. The foundation models are not staying isolated model artifacts to be called for when needed, but rather are built into platforms.
You don’t need TFMs for building such ecosystems. Libraries like scikit-learn (Pedregosa et al. 2011) have been adding things like interpretability and calibration methods. Even libraries such as XGBoost (Chen and Guestrin 2016) come with built-in plotting functions and explainability tools. But while it’s possible to build ecosystems for any prediction function, TFMs are a better candidate because of their generality as mentioned in the section before. Also, most TFMs spit out the entire distribution, which makes it much easier to chain things like uncertainty quantification.
The direction everything is moving in: TFMs are becoming the engine of larger, batteries-included platforms.
Large language models are eating tabular machine learning?
In Chapter 11, I argued that TFMs are inheriting the LLM stack. This is the bottom-up perspective of how LLMs and tabular machine learning interact. And while LLMs themselves don’t come with the right inductive biases to model tabular data, they encroach into the tabular machine learning space from a different plane of abstraction: LLM-powered automation approaches (called agentic AI in its current state) are automating processes in tabular machine learning.
Like many others, I’ve been caught by surprise by how quickly LLM-based approaches to coding improved and suddenly just worked. This immediately translates to doing tabular machine learning tasks, as you can simply ask your agentic AI of choice to write a Python script to build a time series forecasting model or whatever for you. The plain agentic AIs for coding are already capable enough to change the data scientist’s job forever in a top-down fashion. This begs the question of how much anyone should care about TFMs, or whether this is now completely abstracted away. Will it matter whether that underlying tool is XGBoost or TabPFN? Or is that abstracted away; a “choice” by the LLM?
There is even an argument to be made that agentic AI is taking away the benefits of tabular foundation models, since one of the points in favor of TFMs is that they reduce the grunt work of tuning and model selection. However, I never thought this to be the strongest argument in favor of TFMs. I’m positive about the new tech since PFNs and TFMs are general purpose engines with the potential to build a new foundation for tabular machine learning.
LLM-powered automations like agentic AI are reshaping the data science field, including tabular machine learning. Processes, tools, and products will rearrange around that new reality, but this will take years. How the development of TFMs will merge with the LLM-powered automations remains to be seen. If TFMs keep eating into the tabular space, beating other approaches in terms of performance and convenience, they may well wedge into the new LLM stack.
TFMs require a new modeling mindset
I have been in tabular my entire career, from a classic education as a statistician, to data science jobs in industry, a PhD in machine learning interpretability and writing books. During those years, I actively pulled in projects involving other modalities, like images or text, but it has never been my focus and I always found it to be a very different world, as it has also been more shaped by computer scientists than statisticians. Mindset and methods of tabular machine learning have always been distinct from the other modalities, especially from my statistician perspective. Coming from a kitchen sink of support vector machines, random forests, and generalized additive models, I found it hard to switch gears and fully immerse into the world of neural networks. TFMs have this additional complexity as they deviate from the traditional train and predict in favor of a pretrain-and-in-context-learning paradigm. This certainly requires a bit of getting used to and I see a lot of hesitation and blocks in data scientists I’ve talked to. The vibe feels similar to when I was talking to statisticians about machine learning in the 2010s. So one part of the adoption is also about the modeling mindset of the data scientist or machine learner working with these models. If you want to learn more about these different modeling cultures, you can read my short book “Modeling Mindsets”.
However, some encouraging words to everyone working in tabular: There is another side, a skillset that elevates beyond these trends in automation and all-inclusive trends started by LLMs and TFMs. Tabular is a very special modality – at least it has a special place in my heart. Hiding behind these tables are complexities that go beyond building some types of functions that map one column to another. To be a great data scientist means understanding causality, missing data mechanisms, data-generating processes, distributions, data collection, sources of uncertainty, and much more. And then there is the human factor: Convince Bob from marketing to finally release the data, sit down with the nurse to learn how the data was collected with all the ins and outs, figure out with the finance team how a forecast model needs to be shaped to integrate into their current reporting rhythm.
While my software developer friends complain, I couldn’t be happier about coding agents. I no longer have to wrestle with the exact syntax for reshaping a dataframe from wide to long with pandas.melt(), remembering which argument is id_vars and which is value_vars, but operate on the abstraction level I love: “reshape this so each row is one measurement.” It’s a good time to have great taste in machine learning and Rick-Rubin your way through the tasks. TFMs too help that trend. I’m happy if I no longer have to think about which parameters to tune for an SVM, but instead can focus on how to measure the model’s success, or have more time figuring out what is actually needed.
I’m betting on foundation models
I’ve been aware of TFMs for a while now, but it was in the beginning of 2026 when I decided to go all-in. Personally, I’m betting with my attention (pun intended) that TFMs will become the new foundation of tabular machine learning and data science more broadly. It’s possible I’m wrong, but the current trajectory looks very promising. The people building and investing in these models and pushing the frontier are making this possible with real money, talent, research, and code behind it. My bet is on TFMs expanding their reach, becoming better and better, both in performance and speed.
Where do you place your bet?