roam/implresearch.org

• impl/research
  • Key Papers
  • Competing Strategies
  • Dataset Coverage vs Literature
    • Well Covered by ROLL
    • Gaps vs Literature (datasets in papers ROLL doesn't have)
    • Non-Standard or Unusual in ROLL
  • Recommendations for Baseline Strengthening
  • Noisy Labels Literature
    • Competing Method Concepts
    • Noisy Label Datasets (real-world, from the literature)
    • Surveys
    • Robust Loss Functions
    • Label Correction / Transition Matrix
    • Sample Selection
    • Semi-Supervised / DivideMix Family
    • AUC / Ranking Objectives and Noise Robustness
    • BibTeX Entries (staged here; move to thesis.bib when writing sec:noisy-labels)
  • Paper Subnodes

impl/research

Survey of academic literature on class imbalance in deep learning, relevant to ROLL's thesis positioning.

Key Papers

Paper	Venue	Node
CLIMB (arXiv:2505.17451)	NeurIPS 2025	—
impl/paper-beyond-rebalancing	2024	detailed node
Simplifying NN Training Under Class Imbalance (arXiv:2312.02517)	2023	—
Investigating Group DRO (arXiv:2303.02505)	2023	—
impl/paper-tabpfn	ICLR 2023	detailed node
Survey on Imbalanced Learning (Springer 2024)	Springer AI Review	—
Rethinking Class Imbalance (arXiv:2305.03900)	2023	—

Competing Strategies

Methods the literature benchmarks against (relevant as ROLL baselines):

• Resampling: SMOTE, ADASYN, CSMOUTE, BorderlineSMOTE, ROSE
• Cost-sensitive: class weighting, focal loss, asymmetric loss
• Ensemble: BalancedBagging, EasyEnsemble, RUSBoost, BalancedRandomForest
• Threshold moving: post-hoc calibration on decision threshold
• DL-specific: LDAM-DRW, M2m, MiSLAS, BBN (mostly image long-tail)
• Tabular DL baselines: XGBoost, LightGBM, CatBoost, MLP, ResNet, FT-Transformer, impl/paper-tabpfn
• CLIMB finding: ensembles dominate; naive rebalancing (SMOTE alone) often underperforms

Metrics used: AUC-ROC, G-Mean, F1, Precision/Recall. AUC and G-Mean are the standard for imbalanced eval. ROLL's TPR-at-FPR framing is non-standard but more practically useful — position this as an advantage.

Dataset Coverage vs Literature

Well Covered by ROLL

• All glass variants (glass0–6) — standard KEEL
• Yeast3, ecoli-0-1_vs_5, wisconsin, cleveland, pima, haberman, iris0, vowel0, vehicle2, page-blocks, new-thyroid1, led7digit
• Adult, Forest Cover, Bank Marketing (medium tabular)
• Credit Card Fraud (~285K, IR 577:1) — common in fraud literature

Gaps vs Literature (datasets in papers ROLL doesn't have)

Dataset	IR	Samples	Appears In
Abalone9-18	~130	731	Beyond Rebalancing, CLIMB
Annthyroid	7.2	6916	Beyond Rebalancing, many UCI surveys
Satellite	22	6435	Beyond Rebalancing
Segment	6	2310	Beyond Rebalancing
Yeast4/5/6	8–33	~1484	Beyond Rebalancing, CLIMB
Ecoli4	15.8	336	Beyond Rebalancing
KC1/KC2/PC1/CM1 (software)	5–13	415–1783	Beyond Rebalancing
Pen-local/Pen-global	9–671	7291	Beyond Rebalancing

Non-Standard or Unusual in ROLL

• Higgs: ROLL samples 500K balanced (50/50) — not a standard imbalanced benchmark; physics ML context
• Home Credit: Kaggle competition dataset; rare in academic imbalance papers
• CIFAR-10 binary (class 1 vs rest, IR ~9): DL imbalance papers use long-tail formulation instead — results not directly comparable to LDAM/MiSLAS tables

Recommendations for Baseline Strengthening

Priority additions (available in KEEL, low effort):

1. Yeast4, Yeast5, Yeast6 — stress-test high IR range
2. Annthyroid — one of the most cited UCI imbalanced datasets
3. Abalone9-18 — extreme IR (130:1), covers the hard regime
4. Ecoli4 — rounds out ecoli coverage at IR 15.8

Lower priority (useful if sweeping many baselines):

1. Satellite, Segment, Pen-local — common in full KEEL sweeps
2. KC1/PC1 — software metrics datasets; different domain from biology/finance

Noisy Labels Literature

Sources found 2026-07-13 for thesis sec:noisy-labels (section 2.5 in related work chapter). Poisoning finding: label-poisoning (true positives duplicated with flipped label) hurts bce-weighted more than roll. ROLL's distributional/threshold objective is structurally more robust to class-conditional label noise without any explicit noise-handling mechanism.

Competing Method Concepts

Four families per Li & Zhu survey (arXiv:2404.04159). All require knowing or estimating something about the noise; ROLL requires none of it.

1. Robust Loss Functions — change the loss so mislabeled samples cannot dominate the gradient.

• Key insight: symmetry condition (Ghosh 2017): sum_k l(f(x),k)=C => noise-tolerant. CE fails this; MAE satisfies it.
• MAE: provably tolerant, very slow to converge.
• GCE / L_q (Zhang & Sabuncu 2018): interpolates MAE<->CE via q. Tune for tradeoff.
• Asymmetric losses (Zhou 2021): saturate on suspected noisy samples; different margins per class direction. Suited to class-conditional noise.
• Label smoothing: a soft regularization variant; does not fully satisfy symmetry condition but reduces overconfidence on noisy labels.
• Requires: approximate noise rate to choose hyperparameters.

2. Noise Transition Matrix / Label Correction — model the corruption explicitly.

• Estimate T where T[i,j] = P(observed label j | true label i).
• Forward correction (Patrini 2017): multiply model output by T^-1 before the loss.
• Requires: a clean anchor set or structural assumptions about T. Hard for one-sided asymmetric noise (our poisoning: only positives corrupted) without an anchor.
• Representative methods: GLC, RoG, CAN, Noise Adaptation Layer.

3. Sample Selection — identify and train only on likely-clean samples.

• Memorization effect: DNNs fit clean patterns first, noisy labels late in training. Small-loss samples early on are likely clean.
• Co-teaching (Han 2018): two nets cross-select small-loss samples each epoch.
• MentorNet, JoCoR, TopoFilter, MORPH are variants.
• Requires: two models; breaks down at high noise or with imbalance (minority samples often have high loss even when clean, so they get incorrectly filtered out).

4. Semi-Supervised Learning (SSL)-Based — treat noisy samples as unlabeled, not wrong.

• Divide training into clean (labeled) and noisy (unlabeled) subsets, then run SSL.
• DivideMix: fits a mixture model to per-sample loss to decide clean/noisy split, then applies MixMatch. The dominant method on standard benchmarks as of 2024.
• Other representatives: SELFIE, Jo-SRC, UNICON, TCL, SoftMatch.
• Requires: reliable clean/noisy split; the mixture model can fail under imbalance because minority-class loss distributions overlap with noisy majority distributions.

Common thread: all four families are corrective — they assume or estimate noise structure. ROLL's robustness is structural (distributional objective), not corrective.

Benchmark gap (confirmed by Li & Zhu 2024): the entire field evaluates on multi-class image datasets with synthetic symmetric/asymmetric noise or a handful of real web-scraped image datasets (Clothing1M ~38.5% noise, Food-101N ~18.4%, ANIMAL-10N ~8%, WebVision ~20%, CIFAR-10N/100N human-reannotated). No tabular data, no binary imbalanced setting, no one-sided minority-class corruption anywhere in the standard benchmark suite.

Noisy Label Datasets (real-world, from the literature)

Dataset	Size	Noise rate	Domain
Clothing1M	~1M	~38.5%	clothing images (web)
Food-101N	~310K	~18.4%	food images (web)
ANIMAL-10N	55K	~8%	animal images
WebVision	2.5M	~20%	general web images
CIFAR-10N / CIFAR-100N	50K	varies	CIFAR re-annotated by humans

All are image datasets. No standard noisy-label tabular benchmark exists.

Surveys

Key	Venue	arXiv / DOI	Notes
song2022survey	IEEE TNNLS 2022/23	10.1109/TNNLS.2022.3152527	Main survey; 62 methods in 5 categories
survey2025label	Systems Sci & Control Eng 2025	10.1080/21642583.2025.2488120	Deep learning focus; 5-type taxonomy
li2024noisy	arXiv:2404.04159 (Apr 2024)	—	Li & Zhu; 4-category taxonomy; confirmed no binary/imbalanced coverage

Robust Loss Functions

Key	Venue	Notes
ghosh2017robust	AAAI 2017, arXiv:1712.09482	Symmetry condition: sum_k l(f(x),k)=C => noise-tolerant; CE fails; MAE passes
zhang2018generalized	NeurIPS 2018	GCE: L_q interpolates MAE<->CE via q; q->0 = MAE, q->1 = CE
zhou2021asymmetric	ICML 2021, PMLR v139	Asymmetric loss; different margins per class; suited to class-conditional noise
feng2020can	IJCAI 2020	Can CE be robust? — only under very restricted conditions
symmetrization2025	arXiv:2605.20347	Symmetrize any loss via additive complement (2025)

Label Correction / Transition Matrix

Key	Venue	Notes
patrini2017making	CVPR 2017	Estimate T[i,j]=P(y_tilde=j	y=i); correct logits by T-1. Canonical forward-correction
cannings2020classification	Biometrika 2020, arXiv:1805.11505	kNN/SVM robust to imperfect labels; LDA not unless class priors equal

Sample Selection

Key	Venue	Notes
han2018coteaching	NeurIPS 2018	Co-teaching: two nets select small-loss samples for each other per epoch

Semi-Supervised / DivideMix Family

Key	Venue	Notes
dividemix	ICLR 2020	Dominant SSL-based method; GMM splits clean/noisy, then MixMatch

AUC / Ranking Objectives and Noise Robustness

Key	Venue	Notes
xie2024wsauc	IEEE TPAMI 2024, arXiv:2305.14258	WSAUC/rpAUC; unifies weak supervision for AUC; rpAUC removes high-loss pairs — more robust under noisy labels. ADDED to thesis.bib
auc2025retrieval	arXiv:2510.00137	AUC-driven learning for neural retrieval; robustness framing

BibTeX Entries (staged here; move to thesis.bib when writing sec:noisy-labels)

@article{song2022survey,
  title={Learning From Noisy Labels With Deep Neural Networks: A Survey},
  author={Song, Hwanjun and Kim, Minseok and Park, Dongkwan and Shin, Yooju and Lee, Jae-Gil},
  journal={IEEE Transactions on Neural Networks and Learning Systems},
  volume={34},
  number={11},
  pages={8135--8153},
  year={2023},
  doi={10.1109/TNNLS.2022.3152527}
}

@article{li2024noisy,
  title={Noisy Label Processing for Classification: A Survey},
  author={Li, Mengting and Zhu, Chuang},
  journal={arXiv preprint arXiv:2404.04159},
  year={2024}
}

@inproceedings{ghosh2017robust,
  title={Robust Loss Functions under Label Noise for Deep Neural Networks},
  author={Ghosh, Aritra and Kumar, Himanshu and Sastry, P. S.},
  booktitle={Proceedings of the Thirty-First AAAI Conference on Artificial Intelligence},
  pages={1919--1925},
  year={2017}
}

@inproceedings{patrini2017making,
  title={Making Deep Neural Networks Robust to Label Noise: A Loss Correction Approach},
  author={Patrini, Giorgio and Rozza, Alessandro and Krishna Menon, Aditya and Nock, Richard and Qu, Lizhen},
  booktitle={Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition},
  pages={1944--1952},
  year={2017}
}

@inproceedings{zhang2018generalized,
  title={Generalized Cross Entropy Loss for Training Deep Neural Networks with Noisy Labels},
  author={Zhang, Zhilu and Sabuncu, Mert R.},
  booktitle={Advances in Neural Information Processing Systems},
  volume={31},
  year={2018}
}

@inproceedings{han2018coteaching,
  title={Co-teaching: Robust Training of Deep Neural Networks with Extremely Noisy Labels},
  author={Han, Bo and Yao, Quanming and Yu, Xingrui and Niu, Gang and Xu, Miao and Hu, Weihua and Tsang, Ivor W. and Sugiyama, Masashi},
  booktitle={Advances in Neural Information Processing Systems},
  volume={31},
  year={2018}
}

@inproceedings{zhou2021asymmetric,
  title={Asymmetric Loss Functions for Learning with Noisy Labels},
  author={Zhou, Xiong and Liu, Xianming and Jiang, Junjun and Gao, Xin and Ji, Xiangyang},
  booktitle={Proceedings of the 38th International Conference on Machine Learning},
  pages={12846--12856},
  year={2021},
  volume={139},
  series={Proceedings of Machine Learning Research},
  publisher={PMLR}
}

@inproceedings{feng2020can,
  title={Can Cross Entropy Loss Be Robust to Label Noise?},
  author={Feng, Lei and Shu, Senlin and Lin, Zhuoyi and Lv, Fengmei and Li, Li and An, Bo},
  booktitle={Proceedings of the Twenty-Ninth International Joint Conference on Artificial Intelligence},
  pages={2206--2212},
  year={2020}
}

@article{cannings2020classification,
  title={Classification with imperfect training labels},
  author={Cannings, Timothy I. and Fan, Yingying and Samworth, Richard J.},
  journal={Biometrika},
  volume={107},
  number={2},
  pages={311--330},
  year={2020},
  doi={10.1093/biomet/asaa011}
}

@article{symmetrization2025,
  title={Symmetrization of Loss Functions for Robust Training of Neural Networks
         in the Presence of Noisy Labels},
  journal={arXiv preprint arXiv:2605.20347},
  year={2025}
}

@article{auc2025retrieval,
  title={Optimizing What Matters: {AUC}-Driven Learning for Robust Neural Retrieval},
  journal={arXiv preprint arXiv:2510.00137},
  year={2025}
}

Paper Subnodes

• impl/paper-tabpfn — TabPFN: in-context learning for small tabular classification (ICLR 2023)
• impl/paper-beyond-rebalancing — benchmark of 12 classifiers under imbalance, no rebalancing (2024)