5 Classification – Tabular Foundation Models

Installing TabICL

Well, not much to say here. You can install TabICL like most Python packages from PyPI. The examples in this book use version 2.1.1.

pip install tabicl==2.1.1

We classify the Land Mines dataset

We use the Land Mines dataset (Yilmaz, Kahraman, and Söyler 2018) from the UCI Machine Learning Repository. The task is to classify the mine type based on magnetic sensors, with five classes (M=1 for “no mine”, M=2–M=5 for different mine types; I shifted everything to 0–4 so XGBoost does not throw an error later). It’s 338 rows, 3 features, no missing data. The features are the induced voltage V; the height of the sensor above the ground H; and the soil type S (six categories).

Let’s load the data from the UCI Repository:

from ucimlrepo import fetch_ucirepo

land_mines = fetch_ucirepo(id=763)
X = land_mines.data.features.copy()
X["S"] = X["S"].astype(str).astype("category")
y = land_mines.data.targets["M"] - 1
y.value_counts().sort_index().to_frame().T

M	0	1	2	3	4
count	71	70	66	66	65

The five classes are roughly balanced.

Predict in-context with TabICL

Time to “train” and “predict” the data with TabICL:

from sklearn.model_selection import train_test_split
from tabicl import TabICLClassifier

X_train, X_test, y_train, y_test = train_test_split(
    X, y, test_size=0.25, random_state=1, stratify=y
)

clf = TabICLClassifier(random_state=1)
clf.fit(X_train, y_train)

y_proba_tficl = clf.predict_proba(X_test)
y_pred_tficl = clf.classes_[y_proba_tficl.argmax(axis=1)]

The first call to .fit() does a few things:

If you use TabICL for the first time, it downloads the model weights from HuggingFace.
Once downloaded, the weights of the pretrained TabICL model are loaded into memory.
Pre-processing of inputs, like encoding labels and converting features to numeric data.

The cool thing is: We don’t need any hyperparameter tuning for TabICL, since it does in-context learning in calls to .predict or predict_proba.

Don’t be fooled by the syntax

Although .predict only takes X_test as input, behind the scenes also X_train and y_train get sent to the model. Even y_test will be part of the input, although represented by a mean-imputed vector based on y_train.

Let’s get an idea of the model’s performance.

import pandas as pd
from sklearn.metrics import classification_report

pd.DataFrame(
    classification_report(y_test, y_pred_tficl, output_dict=True)
).T.round(3)

	precision	recall	f1-score	support
0	0.857	1.000	0.923	18.000
1	0.947	1.000	0.973	18.000
2	0.688	0.688	0.688	16.000
3	0.929	0.765	0.839	17.000
4	0.600	0.562	0.581	16.000
accuracy	0.812	0.812	0.812	0.812
macro avg	0.804	0.803	0.801	85.000
weighted avg	0.810	0.812	0.808	85.000

A constant predictor on five roughly-balanced classes would reach only about 20% accuracy, so we are clearly doing better. Much better. And this without any effort regarding hyperparameter tuning or model selection. However, there is a price to pay, and that’s compute.

Fit is faster than predict

With tabular foundation models, .fit() becomes the cheap call, and .predict() the expensive one. This is a reversal from traditional machine learning, where training is the most expensive step, especially when it involves hyperparameter tuning. Let’s have a look at how bad it is:

from time import perf_counter

start = perf_counter()
clf.fit(X_train, y_train)
fit_time = perf_counter() - start

start = perf_counter()
clf.predict_proba(X_test)
predict_time = perf_counter() - start

print(f"fit:     {fit_time:.3f}s")
print(f"predict: {predict_time:.3f}s")

fit:     0.186s
predict: 0.525s

For our small dataset, predict takes roughly three times as long as fit.

TabICL beats XGBoost

Let’s compare TabICL to a boosted tree ensemble. For this, we use random search to tune the five XGBoost (Chen and Guestrin 2016) hyperparameters that matter most according to Probst, Boulesteix, and Bischl (2019): learning_rate, max_depth, min_child_weight, subsample, and colsample_bytree:

from scipy.stats import loguniform, randint, uniform
from sklearn.model_selection import RandomizedSearchCV
from xgboost import XGBClassifier

param_distributions = {
    "learning_rate": loguniform(1e-2, 3e-1),
    "max_depth": randint(3, 11),
    "min_child_weight": loguniform(1.0, 20.0),
    "subsample": uniform(0.6, 0.4),
    "colsample_bytree": uniform(0.6, 0.4),
}

xgb_search = RandomizedSearchCV(
    XGBClassifier(n_estimators=500, enable_categorical=True, random_state=1, n_jobs=1),
    param_distributions,
    n_iter=100,
    scoring="roc_auc_ovr",
    random_state=1,
    n_jobs=1,
)
xgb_search.fit(X_train, y_train)

y_pred_xgb = xgb_search.predict(X_test)
y_proba_xgb = xgb_search.predict_proba(X_test)

And now, for the comparison:

from sklearn.metrics import accuracy_score, f1_score, roc_auc_score

def score(y_pred, y_proba):
    return {
        "Accuracy": accuracy_score(y_test, y_pred),
        "ROC AUC":  roc_auc_score(y_test, y_proba, multi_class="ovr"),
        "F1":       f1_score(y_test, y_pred, average="macro"),
    }

pd.DataFrame({
    "TabICL":        score(y_pred_tficl, y_proba_tficl),
    "Tuned XGBoost": score(y_pred_xgb, y_proba_xgb),
}).round(3)

	TabICL	Tuned XGBoost
Accuracy	0.812	0.541
ROC AUC	0.959	0.833
F1	0.801	0.524

In all three metrics, TabICL beats XGBoost by far. Don’t expect this to be the case with every dataset, but on many benchmarks, tabular foundation models tend to beat boosted tree ensembles.