38 lines
1.7 KiB
Org Mode
38 lines
1.7 KiB
Org Mode
* impl/paper-tabpfn
|
||
|
||
[[id:151d5686-6f40-4158-a59a-b0be94cdc969][impl/research]]
|
||
|
||
*TabPFN: A Transformer That Solves Small Tabular Classification in a Second*
|
||
Hollmann et al. — ICLR 2023 — arXiv:2207.01848
|
||
|
||
** Problem
|
||
|
||
Small tabular datasets demand expensive hyperparameter search and still often lose to boosted trees. TabPFN asks: can you eliminate tuning entirely while matching AutoML?
|
||
|
||
** Core Method
|
||
|
||
A Transformer pre-trained *offline* on millions of synthetic datasets sampled from structural causal models. At inference, the full training set is passed as context — no gradient updates. The model receives =(X_train, y_train, X_test)= as one sequence and outputs predictions in a single forward pass (in-context learning).
|
||
|
||
** Evaluation
|
||
|
||
- 18 OpenML-CC18 datasets + 67 small numerical OpenML datasets
|
||
- Up to 1,000 training points, 100 features, 10 classes
|
||
- Compared against AutoML systems (Auto-sklearn), XGBoost, random forests
|
||
|
||
** Key Results
|
||
|
||
- Outperforms boosted trees on small datasets; matches top AutoML
|
||
- 230× faster than AutoML baselines; 5,700× with GPU
|
||
- No hyperparameter tuning required
|
||
|
||
** Limitations
|
||
|
||
- v1: numerical features only, no missing values, max ~1,000 training samples
|
||
- TabPFN v2 (2025, arXiv:2502.17361) lifts most constraints — handles larger datasets, mixed types, missing values
|
||
|
||
** Relevance to ROLL
|
||
|
||
- [[id:8f59b736-04ea-4d11-9195-30d125a127f8][impl/paper-beyond-rebalancing]] identifies TabPFN as the best-performing classifier on imbalanced tabular data without rebalancing — it is the current bar to beat
|
||
- ROLL's niche (optimizing TPR at a specific FPR threshold) is orthogonal: TabPFN uses no custom loss or ROC objective
|
||
- If evaluating on small KEEL datasets (≤1,000 samples), TabPFN is the strongest baseline to include
|