roam/implresearch.org at 2a872c725b915d9cc3b9fe9b9cce6b8a7b6b2b9c

aner/roam

Fork 0

Files

T

aner 2a872c725b backup: 2026-08-29 06:59

2026-08-29 06:59:35 +03:00

16 KiB

Raw Blame History

impl/research

impl/research

Survey of academic literature on class imbalance in deep learning, relevant to ROLL's thesis positioning.

Key Papers

Paper	Venue	Node
CLIMB (arXiv:2505.17451)	NeurIPS 2025	—
impl/paper-beyond-rebalancing	2024	detailed node
Simplifying NN Training Under Class Imbalance (arXiv:2312.02517)	2023	—
Investigating Group DRO (arXiv:2303.02505)	2023	—
impl/paper-tabpfn	ICLR 2023	detailed node
Survey on Imbalanced Learning (Springer 2024)	Springer AI Review	—
Rethinking Class Imbalance (arXiv:2305.03900)	2023	—

Competing Strategies

Methods the literature benchmarks against (relevant as ROLL baselines):

Resampling: SMOTE, ADASYN, CSMOUTE, BorderlineSMOTE, ROSE
Cost-sensitive: class weighting, focal loss, asymmetric loss
Ensemble: BalancedBagging, EasyEnsemble, RUSBoost, BalancedRandomForest
Threshold moving: post-hoc calibration on decision threshold
DL-specific: LDAM-DRW, M2m, MiSLAS, BBN (mostly image long-tail)
Tabular DL baselines: XGBoost, LightGBM, CatBoost, MLP, ResNet, FT-Transformer, impl/paper-tabpfn
CLIMB finding: ensembles dominate; naive rebalancing (SMOTE alone) often underperforms

Metrics used: AUC-ROC, G-Mean, F1, Precision/Recall. AUC and G-Mean are the standard for imbalanced eval. ROLL's TPR-at-FPR framing is non-standard but more practically useful — position this as an advantage.

Dataset Coverage vs Literature

Well Covered by ROLL

All glass variants (glass0–6) — standard KEEL
Yeast3, ecoli-0-1_vs_5, wisconsin, cleveland, pima, haberman, iris0, vowel0, vehicle2, page-blocks, new-thyroid1, led7digit
Adult, Forest Cover, Bank Marketing (medium tabular)
Credit Card Fraud (~285K, IR 577:1) — common in fraud literature

Gaps vs Literature (datasets in papers ROLL doesn't have)

Dataset	IR	Samples	Appears In
Abalone9-18	~130	731	Beyond Rebalancing, CLIMB
Annthyroid	7.2	6916	Beyond Rebalancing, many UCI surveys
Satellite	22	6435	Beyond Rebalancing
Segment	6	2310	Beyond Rebalancing
Yeast4/5/6	8–33	~1484	Beyond Rebalancing, CLIMB
Ecoli4	15.8	336	Beyond Rebalancing
KC1/KC2/PC1/CM1 (software)	5–13	415–1783	Beyond Rebalancing
Pen-local/Pen-global	9–671	7291	Beyond Rebalancing

Non-Standard or Unusual in ROLL

Higgs: ROLL samples 500K balanced (50/50) — not a standard imbalanced benchmark; physics ML context
Home Credit: Kaggle competition dataset; rare in academic imbalance papers
CIFAR-10 binary (class 1 vs rest, IR ~9): DL imbalance papers use long-tail formulation instead — results not directly comparable to LDAM/MiSLAS tables

Recommendations for Baseline Strengthening

Priority additions (available in KEEL, low effort):

Yeast4, Yeast5, Yeast6 — stress-test high IR range
Annthyroid — one of the most cited UCI imbalanced datasets
Abalone9-18 — extreme IR (130:1), covers the hard regime
Ecoli4 — rounds out ecoli coverage at IR 15.8

Lower priority (useful if sweeping many baselines):

Satellite, Segment, Pen-local — common in full KEEL sweeps
KC1/PC1 — software metrics datasets; different domain from biology/finance

Noisy Labels Literature

Sources found 2026-07-13 for thesis sec:noisy-labels (section 2.5 in related work chapter). Poisoning finding: label-poisoning (true positives duplicated with flipped label) hurts bce-weighted more than roll. ROLL's distributional/threshold objective is structurally more robust to class-conditional label noise without any explicit noise-handling mechanism.

Competing Method Concepts

Four families per Li & Zhu survey (arXiv:2404.04159). All require knowing or estimating something about the noise; ROLL requires none of it.

1. Robust Loss Functions — change the loss so mislabeled samples cannot dominate the gradient.

Key insight: symmetry condition (Ghosh 2017): sum_k l(f(x),k)=C => noise-tolerant. CE fails this; MAE satisfies it.
MAE: provably tolerant, very slow to converge.
GCE / L_q (Zhang & Sabuncu 2018): interpolates MAE<->CE via q. Tune for tradeoff.
Asymmetric losses (Zhou 2021): saturate on suspected noisy samples; different margins per class direction. Suited to class-conditional noise.
Label smoothing: a soft regularization variant; does not fully satisfy symmetry condition but reduces overconfidence on noisy labels.
Requires: approximate noise rate to choose hyperparameters.

2. Noise Transition Matrix / Label Correction — model the corruption explicitly.

Estimate T where T[i,j] = P(observed label j | true label i).
Forward correction (Patrini 2017): multiply model output by T^-1 before the loss.
Requires: a clean anchor set or structural assumptions about T. Hard for one-sided asymmetric noise (our poisoning: only positives corrupted) without an anchor.
Representative methods: GLC, RoG, CAN, Noise Adaptation Layer.

3. Sample Selection — identify and train only on likely-clean samples.

Memorization effect: DNNs fit clean patterns first, noisy labels late in training. Small-loss samples early on are likely clean.
Co-teaching (Han 2018): two nets cross-select small-loss samples each epoch.
MentorNet, JoCoR, TopoFilter, MORPH are variants.
Requires: two models; breaks down at high noise or with imbalance (minority samples often have high loss even when clean, so they get incorrectly filtered out).

4. Semi-Supervised Learning (SSL)-Based — treat noisy samples as unlabeled, not wrong.

Divide training into clean (labeled) and noisy (unlabeled) subsets, then run SSL.
DivideMix: fits a mixture model to per-sample loss to decide clean/noisy split, then applies MixMatch. The dominant method on standard benchmarks as of 2024.
Other representatives: SELFIE, Jo-SRC, UNICON, TCL, SoftMatch.
Requires: reliable clean/noisy split; the mixture model can fail under imbalance because minority-class loss distributions overlap with noisy majority distributions.

Common thread: all four families are corrective — they assume or estimate noise structure. ROLL's robustness is structural (distributional objective), not corrective.

Benchmark gap (confirmed by Li & Zhu 2024): the entire field evaluates on multi-class image datasets with synthetic symmetric/asymmetric noise or a handful of real web-scraped image datasets (Clothing1M ~38.5% noise, Food-101N ~18.4%, ANIMAL-10N ~8%, WebVision ~20%, CIFAR-10N/100N human-reannotated). No tabular data, no binary imbalanced setting, no one-sided minority-class corruption anywhere in the standard benchmark suite.

Noisy Label Datasets (real-world, from the literature)

Dataset	Size	Noise rate	Domain
Clothing1M	~1M	~38.5%	clothing images (web)
Food-101N	~310K	~18.4%	food images (web)
ANIMAL-10N	55K	~8%	animal images
WebVision	2.5M	~20%	general web images
CIFAR-10N / CIFAR-100N	50K	varies	CIFAR re-annotated by humans

All are image datasets. No standard noisy-label tabular benchmark exists.

Surveys

Key	Venue	arXiv / DOI	Notes
song2022survey	IEEE TNNLS 2022/23	10.1109/TNNLS.2022.3152527	Main survey; 62 methods in 5 categories
survey2025label	Systems Sci & Control Eng 2025	10.1080/21642583.2025.2488120	Deep learning focus; 5-type taxonomy
li2024noisy	arXiv:2404.04159 (Apr 2024)	—	Li & Zhu; 4-category taxonomy; confirmed no binary/imbalanced coverage

Robust Loss Functions

Key	Venue	Notes
ghosh2017robust	AAAI 2017, arXiv:1712.09482	Symmetry condition: sum_k l(f(x),k)=C => noise-tolerant; CE fails; MAE passes
zhang2018generalized	NeurIPS 2018	GCE: L_q interpolates MAE<->CE via q; q->0 = MAE, q->1 = CE
zhou2021asymmetric	ICML 2021, PMLR v139	Asymmetric loss; different margins per class; suited to class-conditional noise
feng2020can	IJCAI 2020	Can CE be robust? — only under very restricted conditions
symmetrization2025	arXiv:2605.20347	Symmetrize any loss via additive complement (2025)

Label Correction / Transition Matrix

Key	Venue	Notes
patrini2017making	CVPR 2017	Estimate T[i,j]=P(y_tilde=j	y=i); correct logits by T^-1. Canonical forward-correction
cannings2020classification	Biometrika 2020, arXiv:1805.11505	kNN/SVM robust to imperfect labels; LDA not unless class priors equal

Sample Selection

Key	Venue	Notes
han2018coteaching	NeurIPS 2018	Co-teaching: two nets select small-loss samples for each other per epoch

Semi-Supervised / DivideMix Family

Key	Venue	Notes
dividemix	ICLR 2020	Dominant SSL-based method; GMM splits clean/noisy, then MixMatch

AUC / Ranking Objectives and Noise Robustness

Key	Venue	Notes
xie2024wsauc	IEEE TPAMI 2024, arXiv:2305.14258	WSAUC/rpAUC; unifies weak supervision for AUC; rpAUC removes high-loss pairs — more robust under noisy labels. ADDED to thesis.bib
auc2025retrieval	arXiv:2510.00137	AUC-driven learning for neural retrieval; robustness framing

BibTeX Entries (staged here; move to thesis.bib when writing sec:noisy-labels)

@article{song2022survey,
  title={Learning From Noisy Labels With Deep Neural Networks: A Survey},
  author={Song, Hwanjun and Kim, Minseok and Park, Dongkwan and Shin, Yooju and Lee, Jae-Gil},
  journal={IEEE Transactions on Neural Networks and Learning Systems},
  volume={34},
  number={11},
  pages={8135--8153},
  year={2023},
  doi={10.1109/TNNLS.2022.3152527}
}

@article{li2024noisy,
  title={Noisy Label Processing for Classification: A Survey},
  author={Li, Mengting and Zhu, Chuang},
  journal={arXiv preprint arXiv:2404.04159},
  year={2024}
}

@inproceedings{ghosh2017robust,
  title={Robust Loss Functions under Label Noise for Deep Neural Networks},
  author={Ghosh, Aritra and Kumar, Himanshu and Sastry, P. S.},
  booktitle={Proceedings of the Thirty-First AAAI Conference on Artificial Intelligence},
  pages={1919--1925},
  year={2017}
}

@inproceedings{patrini2017making,
  title={Making Deep Neural Networks Robust to Label Noise: A Loss Correction Approach},
  author={Patrini, Giorgio and Rozza, Alessandro and Krishna Menon, Aditya and Nock, Richard and Qu, Lizhen},
  booktitle={Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition},
  pages={1944--1952},
  year={2017}
}

@inproceedings{zhang2018generalized,
  title={Generalized Cross Entropy Loss for Training Deep Neural Networks with Noisy Labels},
  author={Zhang, Zhilu and Sabuncu, Mert R.},
  booktitle={Advances in Neural Information Processing Systems},
  volume={31},
  year={2018}
}

@inproceedings{han2018coteaching,
  title={Co-teaching: Robust Training of Deep Neural Networks with Extremely Noisy Labels},
  author={Han, Bo and Yao, Quanming and Yu, Xingrui and Niu, Gang and Xu, Miao and Hu, Weihua and Tsang, Ivor W. and Sugiyama, Masashi},
  booktitle={Advances in Neural Information Processing Systems},
  volume={31},
  year={2018}
}

@inproceedings{zhou2021asymmetric,
  title={Asymmetric Loss Functions for Learning with Noisy Labels},
  author={Zhou, Xiong and Liu, Xianming and Jiang, Junjun and Gao, Xin and Ji, Xiangyang},
  booktitle={Proceedings of the 38th International Conference on Machine Learning},
  pages={12846--12856},
  year={2021},
  volume={139},
  series={Proceedings of Machine Learning Research},
  publisher={PMLR}
}

@inproceedings{feng2020can,
  title={Can Cross Entropy Loss Be Robust to Label Noise?},
  author={Feng, Lei and Shu, Senlin and Lin, Zhuoyi and Lv, Fengmei and Li, Li and An, Bo},
  booktitle={Proceedings of the Twenty-Ninth International Joint Conference on Artificial Intelligence},
  pages={2206--2212},
  year={2020}
}

@article{cannings2020classification,
  title={Classification with imperfect training labels},
  author={Cannings, Timothy I. and Fan, Yingying and Samworth, Richard J.},
  journal={Biometrika},
  volume={107},
  number={2},
  pages={311--330},
  year={2020},
  doi={10.1093/biomet/asaa011}
}

@article{symmetrization2025,
  title={Symmetrization of Loss Functions for Robust Training of Neural Networks
         in the Presence of Noisy Labels},
  journal={arXiv preprint arXiv:2605.20347},
  year={2025}
}

@article{auc2025retrieval,
  title={Optimizing What Matters: {AUC}-Driven Learning for Robust Neural Retrieval},
  journal={arXiv preprint arXiv:2510.00137},
  year={2025}
}

NIH ChestX-ray14 Experiment — Citations

Approach: frozen pretrained DenseNet-121 (TorchXRayVision) as feature extractor → 1024-dim embeddings stored to disk → train MLP classifier using existing infrastructure. Justified by:

TorchXRayVision (Cohen et al., MIDL 2022, arXiv:2111.00595) — open library with pretrained DenseNet-121 weights trained on NIH ChestX-ray14 + CheXpert + MIMIC-CXR. Explicitly designed for feature extraction / transfer learning. pip-installable. GitHub: https://github.com/mlmed/torchxrayvision Key quote: "pre-trained models can also be used as features extractors for semi-supervised training or transfer learning."
Transfer learning with chest X-rays for ER patient classification (Scientific Reports 2020, https://www.nature.com/articles/s41598-020-78060-4) — uses frozen CheXNet output as features for downstream classification. Directly precedents our approach.

BibTeX:

@inproceedings{cohen2022torchxrayvision,
  title={{TorchXRayVision}: A library of chest {X}-ray datasets and models},
  author={Cohen, Joseph Paul and Viviano, Joseph D. and Bertin, Paul and Morrison,
          Paul and Torabian, Parisa and Guarrera, Matteo and Lungren, Matthew P. and
          Chaudhari, Akshay and Brooks, Rupert and Hashir, Mohammad and Bertrand, Hadrien},
  booktitle={Medical Imaging with Deep Learning},
  year={2022},
  url={https://arxiv.org/abs/2111.00595}
}

@article{bolourani2021transfer,
  title={Transfer learning with chest {X}-rays for {ER} patient classification},
  author={Bolourani, Sina and Brennan, Michael and Wang, Ping and Panjeton, Gregg and
          Naaraayan, Arun and Mukherjee, Vikramjit and Bhargava, Mohit and Schreiber,
          Theodore L. and Bhartia, Anubhav},
  journal={Scientific Reports},
  volume={11},
  pages={21906},
  year={2021},
  doi={10.1038/s41598-020-78060-4}
}

Practical notes:

Dataset: 42GB, download to beacon-wsl (916GB free). Mac at 95% capacity, insufficient.
Feature extraction: one GPU pass after food101n finishes (~3.5 days from 2026-07-28)
Embeddings: ~450MB (112K × 1024 × float32), preload into RAM like tabular data
Binary task: one pathology vs rest (e.g. Effusion — highest prevalence ~12%, IR ~7)
Label noise: NLP-extracted from radiology reports, estimated ~20% noise rate

Paper Subnodes

impl/paper-tabpfn — TabPFN: in-context learning for small tabular classification (ICLR 2023)
impl/paper-beyond-rebalancing — benchmark of 12 classifiers under imbalance, no rebalancing (2024)

16 KiB Raw Blame History Unescape Escape