6  Classification

Tabular foundation models classify data in-context, without a training loop or hyperparameter search. They compete with gradient-boosted trees without any tuning at all.

Let’s start gently: A multi-class classification example with tabular foundation models, in this case, TabICL (Qu et al. 2026).

Installing TabICL

Well, not much to say here. You can install TabICL like most Python packages from PyPI. The examples in this book use version 2.1.1.

pip install tabicl==2.1.1

We classify the Land Mines dataset

We use the Land Mines dataset (Yilmaz, Kahraman, and Söyler 2018) from the UCI Machine Learning Repository. The task is to classify the mine type based on magnetic sensors, with five classes (M=1 for “no mine”, M=2–M=5 for different mine types; I shifted everything to 0–4 so XGBoost does not throw an error later). It’s 338 rows, 3 features, no missing data. The features are the induced voltage V; the height of the sensor above the ground H; and the soil type S (six categories).

Let’s load the data from the UCI Repository:

from ucimlrepo import fetch_ucirepo

land_mines = fetch_ucirepo(id=763)
X = land_mines.data.features.copy()
X["S"] = X["S"].astype(str).astype("category")
y = land_mines.data.targets["M"] - 1
y.value_counts().sort_index().to_frame().T

M	0	1	2	3	4
count	71	70	66	66	65

The five classes are roughly balanced.

Predict in-context with TabICL

Time to “train” and “predict” the data with TabICL:

from sklearn.model_selection import train_test_split
from tabicl import TabICLClassifier

X_train, X_test, y_train, y_test = train_test_split(
    X, y, test_size=0.25, random_state=1, stratify=y
)

clf = TabICLClassifier(random_state=1)
clf.fit(X_train, y_train)

y_proba_tficl = clf.predict_proba(X_test)
y_pred_tficl = clf.classes_[y_proba_tficl.argmax(axis=1)]

The first call to .fit() does a few things:

• If you use TabICL for the first time, it downloads the model weights from HuggingFace.
• Once downloaded, the weights of the pretrained TabICL model are loaded into memory.
• Pre-processing of inputs, like encoding labels and converting features to numeric data.

The cool thing is: We don’t need any hyperparameter tuning for TabICL, since it does in-context learning in calls to .predict or predict_proba.

Don’t be fooled by the syntax

Although .predict only takes X_test as input, behind the scenes also X_train and y_train get sent to the model. Even y_test will be part of the input, although represented by a mean-imputed vector based on y_train.

Let’s get an idea of the model’s performance.

import pandas as pd
from sklearn.metrics import classification_report

pd.DataFrame(
    classification_report(y_test, y_pred_tficl, output_dict=True)
).T.round(3)

	precision	recall	f1-score	support
0	0.857	1.000	0.923	18.000
1	0.947	1.000	0.973	18.000
2	0.688	0.688	0.688	16.000
3	0.929	0.765	0.839	17.000
4	0.600	0.562	0.581	16.000
accuracy	0.812	0.812	0.812	0.812
macro avg	0.804	0.803	0.801	85.000
weighted avg	0.810	0.812	0.808	85.000

A constant predictor on five roughly-balanced classes would reach only about 20% accuracy, so we are clearly doing better. Much better. And this without any effort regarding hyperparameter tuning or model selection. However, there is a price to pay, and that’s compute.

Fit is faster than predict

With TFMs, .fit() becomes the cheap call, and .predict() the expensive one. This is a reversal from traditional machine learning, where training is the most expensive step, especially when it involves hyperparameter tuning. Let’s have a look at how bad it is:

from time import perf_counter

start = perf_counter()
clf.fit(X_train, y_train)
fit_time = perf_counter() - start

start = perf_counter()
clf.predict_proba(X_test)
predict_time = perf_counter() - start

print(f"fit:     {fit_time:.3f}s")
print(f"predict: {predict_time:.3f}s")

fit:     0.176s
predict: 0.558s

For our small dataset, predict takes roughly three times as long as fit.

Levers for faster prediction

Since prediction is the expensive call, it’s worth knowing what you can pull: pick a faster model, such as TabICLv2 (Qu et al. 2026); run on GPU instead of CPU, worth roughly 30x; cache the training representations via key-value (KV) caching, so repeated predictions skip redundant attention; shrink the context by subsampling or selecting training rows; use fewer than the default 8 estimators; distill the model into a lightweight tree ensemble for deployment; and drop features, since runtime grows quadratically with the feature count.

TabICL beats XGBoost

Let’s compare TabICL to a boosted tree ensemble. For this, we use random search to tune the XGBoost (Chen and Guestrin 2016) hyperparameters that Probst, Boulesteix, and Bischl (2019) found most worth tuning, leaving aside the choice of booster, since we stick with trees: learning_rate, min_child_weight, colsample_bylevel, colsample_bytree, and subsample:

from scipy.stats import loguniform, uniform
from sklearn.model_selection import RandomizedSearchCV
from xgboost import XGBClassifier

param_distributions = {
    "learning_rate": loguniform(1e-2, 3e-1),
    "min_child_weight": loguniform(1.0, 20.0),
    "colsample_bylevel": uniform(0.6, 0.4),
    "colsample_bytree": uniform(0.6, 0.4),
    "subsample": uniform(0.6, 0.4),
}

xgb_search = RandomizedSearchCV(
    XGBClassifier(n_estimators=500, enable_categorical=True, random_state=1, n_jobs=1),
    param_distributions,
    n_iter=100,
    scoring="roc_auc_ovr",
    random_state=1,
    n_jobs=1,
)
xgb_search.fit(X_train, y_train)

y_pred_xgb = xgb_search.predict(X_test)
y_proba_xgb = xgb_search.predict_proba(X_test)

And now, for the comparison:

from sklearn.metrics import accuracy_score, f1_score, roc_auc_score

def score(y_pred, y_proba):
    return {
        "Accuracy": accuracy_score(y_test, y_pred),
        "ROC AUC":  roc_auc_score(y_test, y_proba, multi_class="ovr"),
        "F1":       f1_score(y_test, y_pred, average="macro"),
    }

pd.DataFrame({
    "TabICL":        score(y_pred_tficl, y_proba_tficl),
    "Tuned XGBoost": score(y_pred_xgb, y_proba_xgb),
}).round(3)

	TabICL	Tuned XGBoost
Accuracy	0.812	0.541
ROC AUC	0.959	0.831
F1	0.801	0.517

In all three metrics, TabICL beats XGBoost by far. Don’t expect this to be the case with every dataset, but on many benchmarks, TFMs tend to beat boosted tree ensembles.

Chen, Tianqi, and Carlos Guestrin. 2016. “XGBoost: A Scalable Tree Boosting System.” In Proceedings of the 22nd ACM SIGKDD International Conference on Knowledge Discovery and Data Mining, 785–94. ACM. https://doi.org/10.1145/2939672.2939785.

Probst, Philipp, Anne-Laure Boulesteix, and Bernd Bischl. 2019. “Tunability: Importance of Hyperparameters of Machine Learning Algorithms.” Journal of Machine Learning Research 20 (53): 1–32. https://doi.org/10.48550/arXiv.1802.09596.

Qu, Jingang, David Holzmüller, Gaël Varoquaux, and Marine Le Morvan. 2026. “TabICLv2: A Better, Faster, Scalable, and Open Tabular Foundation Model.” arXiv Preprint arXiv:2602.11139. https://doi.org/10.48550/arXiv.2602.11139.

Yilmaz, Cemal, Hamdi Tolga Kahraman, and Salih Söyler. 2018. “Passive Mine Detection and Classification Method Based on Hybrid Model.” IEEE Access 6: 47870–88. https://doi.org/10.1109/ACCESS.2018.2866538.