import seaborn as sns
from sklearn.model_selection import train_test_split
from tabicl import TabICLClassifier
penguins = sns.load_dataset("penguins")
X = penguins.drop(columns="species")
y = penguins["species"]
X_train, X_test, y_train, y_test = train_test_split(X, y)
clf = TabICLClassifier()
clf.fit(X_train, y_train)
y_pred = clf.predict(X_test)2 First look
Tabular foundation models use the same .fit() and .predict() interface as any other model. Underneath, they work in a completely different way.
Before we go into all the complex stuff like architecture, in-context learning, and pretraining, let’s see how the use of a tabular foundation model differs from traditional machine learning.
Looks like traditional machine learning
Let’s start simple. The following code snippet shows a classic case of classification, more particularly, we want to classify the penguin species based on their bodily measurements (Horst, Hill, and Gorman 2020; Gorman, Williams, and Fraser 2014). After splitting the data into training and test data, we use TabICL, a TFM, to classify the data:
This looks like … business-as-usual!? While TabICL is one of these novel TFMs, it seems to work the same as any other machine learning algorithm. We could swap out two lines and get a Random Forest instead:
- Line 3:
from sklearn.ensemble import RandomForestClassifier - Line 10:
clf = RandomForestClassifier()
The reason why applying TabICL looks the same as applying the good old Random Forest is a design-choice by the developers. The developers of TabICL and those of other TFMs build upon the scikit-learn API, which standardizes machine learning calls with functions like .fit() and .predict(). But underneath, TFMs work differently from traditional machine learning.
Things don’t add up in the code above
The .fit doesn’t do much: For example, running the code, the “.predict” is suspiciously slow compared to the “.fit” call. For traditional machine learning, the .fit does the heavy lifting: It’s where XGBoost grows hundreds or even thousands of trees, or a multi-layered feed-forward neural network adapts weights in thousands of batches of data. If .predict were a theater performance, then .fit would be all the rehearsals beforehand. But TabICL is just getting dressed, not rehearsing this particular piece. In other words: calling .fit for TabICL only loads the model weights, and pre-processes the data.
The model is pretrained: Wait, what weights is the model actually loading, if no specific training happens? While no training happens on the particular dataset, the weights stem from a pretraining process, involving millions of synthetic datasets.
No task-specific training: These pretrained weights don’t change in our code example above. Neither in .fit, nor in .predict.
.predict does the heavy lifting: Making predictions is surprisingly slow, compared to the .fit call, because that’s where the main computations are done. The TFM performs in-context learning: Provided with both training and test data at inference time, the model may attend to training data points to make predictions.
No more hyperparameter tuning: You may have noticed that the code snippet above skips hyperparameter tuning. You may have attributed this to the author’s laziness. You would have been right, but with TFMs, my laziness and reality have finally converged. TFMs don’t need hyperparameter-tuning.
Missing data imputation: The penguins dataset has 11 rows with missing values. TabICL bit it down without complaints. While in this case, it’s just a missing data imputation wrapper, pretraining enables TFMs to learn principled handling of missing values.
Many more gimmicks: TFMs also work with regression, where they predict the entire predictive distribution, meaning you get things like quantile regression and uncertainty quantification for free. You can use these models for time series forecasting as well. They extrapolate well, and are quite “capable” in many more ways we will explore in this book.
So, while TFMs carry forward the .fit and .predict tradition, it’s a different paradigm.