283 lines
13 KiB
Org Mode
283 lines
13 KiB
Org Mode
* impl/research
|
||
|
||
Survey of academic literature on class imbalance in deep learning, relevant to ROLL's thesis positioning.
|
||
|
||
** Key Papers
|
||
|
||
| Paper | Venue | Node |
|
||
|-------|-------|------|
|
||
| CLIMB (arXiv:2505.17451) | NeurIPS 2025 | — |
|
||
| [[id:8f59b736-04ea-4d11-9195-30d125a127f8][impl/paper-beyond-rebalancing]] | 2024 | detailed node |
|
||
| Simplifying NN Training Under Class Imbalance (arXiv:2312.02517) | 2023 | — |
|
||
| Investigating Group DRO (arXiv:2303.02505) | 2023 | — |
|
||
| [[id:bf0fc08a-e806-48df-b188-7a2c4c41c693][impl/paper-tabpfn]] | ICLR 2023 | detailed node |
|
||
| Survey on Imbalanced Learning (Springer 2024) | Springer AI Review | — |
|
||
| Rethinking Class Imbalance (arXiv:2305.03900) | 2023 | — |
|
||
|
||
** Competing Strategies
|
||
|
||
Methods the literature benchmarks against (relevant as ROLL baselines):
|
||
|
||
- *Resampling*: SMOTE, ADASYN, CSMOUTE, BorderlineSMOTE, ROSE
|
||
- *Cost-sensitive*: class weighting, focal loss, asymmetric loss
|
||
- *Ensemble*: BalancedBagging, EasyEnsemble, RUSBoost, BalancedRandomForest
|
||
- *Threshold moving*: post-hoc calibration on decision threshold
|
||
- *DL-specific*: LDAM-DRW, M2m, MiSLAS, BBN (mostly image long-tail)
|
||
- *Tabular DL baselines*: XGBoost, LightGBM, CatBoost, MLP, ResNet, FT-Transformer, [[id:bf0fc08a-e806-48df-b188-7a2c4c41c693][impl/paper-tabpfn]]
|
||
- *CLIMB finding*: ensembles dominate; naive rebalancing (SMOTE alone) often underperforms
|
||
|
||
Metrics used: AUC-ROC, G-Mean, F1, Precision/Recall. AUC and G-Mean are the standard for imbalanced eval.
|
||
ROLL's TPR-at-FPR framing is non-standard but more practically useful — position this as an advantage.
|
||
|
||
** Dataset Coverage vs Literature
|
||
|
||
*** Well Covered by ROLL
|
||
- All glass variants (glass0–6) — standard KEEL
|
||
- Yeast3, ecoli-0-1_vs_5, wisconsin, cleveland, pima, haberman, iris0, vowel0, vehicle2, page-blocks, new-thyroid1, led7digit
|
||
- Adult, Forest Cover, Bank Marketing (medium tabular)
|
||
- Credit Card Fraud (~285K, IR 577:1) — common in fraud literature
|
||
|
||
*** Gaps vs Literature (datasets in papers ROLL doesn't have)
|
||
|
||
| Dataset | IR | Samples | Appears In |
|
||
|---------|----|---------|------------|
|
||
| Abalone9-18 | ~130 | 731 | [[id:8f59b736-04ea-4d11-9195-30d125a127f8][Beyond Rebalancing]], CLIMB |
|
||
| Annthyroid | 7.2 | 6916 | [[id:8f59b736-04ea-4d11-9195-30d125a127f8][Beyond Rebalancing]], many UCI surveys |
|
||
| Satellite | 22 | 6435 | [[id:8f59b736-04ea-4d11-9195-30d125a127f8][Beyond Rebalancing]] |
|
||
| Segment | 6 | 2310 | [[id:8f59b736-04ea-4d11-9195-30d125a127f8][Beyond Rebalancing]] |
|
||
| Yeast4/5/6 | 8–33 | ~1484 | [[id:8f59b736-04ea-4d11-9195-30d125a127f8][Beyond Rebalancing]], CLIMB |
|
||
| Ecoli4 | 15.8 | 336 | [[id:8f59b736-04ea-4d11-9195-30d125a127f8][Beyond Rebalancing]] |
|
||
| KC1/KC2/PC1/CM1 (software) | 5–13 | 415–1783 | [[id:8f59b736-04ea-4d11-9195-30d125a127f8][Beyond Rebalancing]] |
|
||
| Pen-local/Pen-global | 9–671 | 7291 | [[id:8f59b736-04ea-4d11-9195-30d125a127f8][Beyond Rebalancing]] |
|
||
|
||
*** Non-Standard or Unusual in ROLL
|
||
- *Higgs*: ROLL samples 500K balanced (50/50) — not a standard imbalanced benchmark; physics ML context
|
||
- *Home Credit*: Kaggle competition dataset; rare in academic imbalance papers
|
||
- *CIFAR-10 binary* (class 1 vs rest, IR ~9): DL imbalance papers use long-tail formulation instead — results not directly comparable to LDAM/MiSLAS tables
|
||
|
||
** Recommendations for Baseline Strengthening
|
||
|
||
Priority additions (available in KEEL, low effort):
|
||
1. Yeast4, Yeast5, Yeast6 — stress-test high IR range
|
||
2. Annthyroid — one of the most cited UCI imbalanced datasets
|
||
3. Abalone9-18 — extreme IR (130:1), covers the hard regime
|
||
4. Ecoli4 — rounds out ecoli coverage at IR 15.8
|
||
|
||
Lower priority (useful if sweeping many baselines):
|
||
5. Satellite, Segment, Pen-local — common in full KEEL sweeps
|
||
6. KC1/PC1 — software metrics datasets; different domain from biology/finance
|
||
|
||
** Noisy Labels Literature
|
||
|
||
Sources found 2026-07-13 for thesis sec:noisy-labels (section 2.5 in related work chapter).
|
||
Poisoning finding: label-poisoning (true positives duplicated with flipped label) hurts
|
||
bce-weighted more than roll. ROLL's distributional/threshold objective is structurally
|
||
more robust to class-conditional label noise without any explicit noise-handling mechanism.
|
||
|
||
*** Competing Method Concepts
|
||
|
||
Four families per Li & Zhu survey (arXiv:2404.04159). All require knowing or estimating
|
||
something about the noise; ROLL requires none of it.
|
||
|
||
*1. Robust Loss Functions* — change the loss so mislabeled samples cannot dominate the gradient.
|
||
- Key insight: symmetry condition (Ghosh 2017): sum_k l(f(x),k)=C => noise-tolerant.
|
||
CE fails this; MAE satisfies it.
|
||
- MAE: provably tolerant, very slow to converge.
|
||
- GCE / L_q (Zhang & Sabuncu 2018): interpolates MAE<->CE via q. Tune for tradeoff.
|
||
- Asymmetric losses (Zhou 2021): saturate on suspected noisy samples; different margins
|
||
per class direction. Suited to class-conditional noise.
|
||
- Label smoothing: a soft regularization variant; does not fully satisfy symmetry condition
|
||
but reduces overconfidence on noisy labels.
|
||
- Requires: approximate noise rate to choose hyperparameters.
|
||
|
||
*2. Noise Transition Matrix / Label Correction* — model the corruption explicitly.
|
||
- Estimate T where T[i,j] = P(observed label j | true label i).
|
||
- Forward correction (Patrini 2017): multiply model output by T^{-1} before the loss.
|
||
- Requires: a clean anchor set or structural assumptions about T. Hard for one-sided
|
||
asymmetric noise (our poisoning: only positives corrupted) without an anchor.
|
||
- Representative methods: GLC, RoG, CAN, Noise Adaptation Layer.
|
||
|
||
*3. Sample Selection* — identify and train only on likely-clean samples.
|
||
- Memorization effect: DNNs fit clean patterns first, noisy labels late in training.
|
||
Small-loss samples early on are likely clean.
|
||
- Co-teaching (Han 2018): two nets cross-select small-loss samples each epoch.
|
||
- MentorNet, JoCoR, TopoFilter, MORPH are variants.
|
||
- Requires: two models; breaks down at high noise or with imbalance (minority samples
|
||
often have high loss even when clean, so they get incorrectly filtered out).
|
||
|
||
*4. Semi-Supervised Learning (SSL)-Based* — treat noisy samples as unlabeled, not wrong.
|
||
- Divide training into clean (labeled) and noisy (unlabeled) subsets, then run SSL.
|
||
- DivideMix: fits a mixture model to per-sample loss to decide clean/noisy split,
|
||
then applies MixMatch. The dominant method on standard benchmarks as of 2024.
|
||
- Other representatives: SELFIE, Jo-SRC, UNICON, TCL, SoftMatch.
|
||
- Requires: reliable clean/noisy split; the mixture model can fail under imbalance
|
||
because minority-class loss distributions overlap with noisy majority distributions.
|
||
|
||
*Common thread*: all four families are corrective — they assume or estimate noise
|
||
structure. ROLL's robustness is structural (distributional objective), not corrective.
|
||
|
||
*Benchmark gap (confirmed by Li & Zhu 2024)*: the entire field evaluates on multi-class
|
||
image datasets with synthetic symmetric/asymmetric noise or a handful of real web-scraped
|
||
image datasets (Clothing1M ~38.5% noise, Food-101N ~18.4%, ANIMAL-10N ~8%,
|
||
WebVision ~20%, CIFAR-10N/100N human-reannotated). No tabular data, no binary
|
||
imbalanced setting, no one-sided minority-class corruption anywhere in the standard
|
||
benchmark suite.
|
||
|
||
*** Noisy Label Datasets (real-world, from the literature)
|
||
|
||
| Dataset | Size | Noise rate | Domain |
|
||
|---------|------|------------|--------|
|
||
| Clothing1M | ~1M | ~38.5% | clothing images (web) |
|
||
| Food-101N | ~310K | ~18.4% | food images (web) |
|
||
| ANIMAL-10N | 55K | ~8% | animal images |
|
||
| WebVision | 2.5M | ~20% | general web images |
|
||
| CIFAR-10N / CIFAR-100N | 50K | varies | CIFAR re-annotated by humans |
|
||
|
||
All are image datasets. No standard noisy-label tabular benchmark exists.
|
||
|
||
*** Surveys
|
||
|
||
| Key | Venue | arXiv / DOI | Notes |
|
||
|-----|-------|-------------|-------|
|
||
| song2022survey | IEEE TNNLS 2022/23 | 10.1109/TNNLS.2022.3152527 | Main survey; 62 methods in 5 categories |
|
||
| survey2025label | Systems Sci & Control Eng 2025 | 10.1080/21642583.2025.2488120 | Deep learning focus; 5-type taxonomy |
|
||
| li2024noisy | arXiv:2404.04159 (Apr 2024) | — | Li & Zhu; 4-category taxonomy; confirmed no binary/imbalanced coverage |
|
||
|
||
*** Robust Loss Functions
|
||
|
||
| Key | Venue | Notes |
|
||
|-----|-------|-------|
|
||
| ghosh2017robust | AAAI 2017, arXiv:1712.09482 | Symmetry condition: sum_k l(f(x),k)=C => noise-tolerant; CE fails; MAE passes |
|
||
| zhang2018generalized | NeurIPS 2018 | GCE: L_q interpolates MAE<->CE via q; q->0 = MAE, q->1 = CE |
|
||
| zhou2021asymmetric | ICML 2021, PMLR v139 | Asymmetric loss; different margins per class; suited to class-conditional noise |
|
||
| feng2020can | IJCAI 2020 | Can CE be robust? — only under very restricted conditions |
|
||
| symmetrization2025 | arXiv:2605.20347 | Symmetrize any loss via additive complement (2025) |
|
||
|
||
*** Label Correction / Transition Matrix
|
||
|
||
| Key | Venue | Notes |
|
||
|-----|-------|-------|
|
||
| patrini2017making | CVPR 2017 | Estimate T[i,j]=P(y_tilde=j|y=i); correct logits by T^{-1}. Canonical forward-correction |
|
||
| cannings2020classification | Biometrika 2020, arXiv:1805.11505 | kNN/SVM robust to imperfect labels; LDA not unless class priors equal |
|
||
|
||
*** Sample Selection
|
||
|
||
| Key | Venue | Notes |
|
||
|-----|-------|-------|
|
||
| han2018coteaching | NeurIPS 2018 | Co-teaching: two nets select small-loss samples for each other per epoch |
|
||
|
||
*** Semi-Supervised / DivideMix Family
|
||
|
||
| Key | Venue | Notes |
|
||
|-----|-------|-------|
|
||
| dividemix | ICLR 2020 | Dominant SSL-based method; GMM splits clean/noisy, then MixMatch |
|
||
|
||
*** AUC / Ranking Objectives and Noise Robustness
|
||
|
||
| Key | Venue | Notes |
|
||
|-----|-------|-------|
|
||
| xie2024wsauc | IEEE TPAMI 2024, arXiv:2305.14258 | WSAUC/rpAUC; unifies weak supervision for AUC; rpAUC removes high-loss pairs — more robust under noisy labels. ADDED to thesis.bib |
|
||
| auc2025retrieval | arXiv:2510.00137 | AUC-driven learning for neural retrieval; robustness framing |
|
||
|
||
*** BibTeX Entries (staged here; move to thesis.bib when writing sec:noisy-labels)
|
||
|
||
#+begin_src bibtex
|
||
@article{song2022survey,
|
||
title={Learning From Noisy Labels With Deep Neural Networks: A Survey},
|
||
author={Song, Hwanjun and Kim, Minseok and Park, Dongkwan and Shin, Yooju and Lee, Jae-Gil},
|
||
journal={IEEE Transactions on Neural Networks and Learning Systems},
|
||
volume={34},
|
||
number={11},
|
||
pages={8135--8153},
|
||
year={2023},
|
||
doi={10.1109/TNNLS.2022.3152527}
|
||
}
|
||
|
||
@article{li2024noisy,
|
||
title={Noisy Label Processing for Classification: A Survey},
|
||
author={Li, Mengting and Zhu, Chuang},
|
||
journal={arXiv preprint arXiv:2404.04159},
|
||
year={2024}
|
||
}
|
||
|
||
@inproceedings{ghosh2017robust,
|
||
title={Robust Loss Functions under Label Noise for Deep Neural Networks},
|
||
author={Ghosh, Aritra and Kumar, Himanshu and Sastry, P. S.},
|
||
booktitle={Proceedings of the Thirty-First AAAI Conference on Artificial Intelligence},
|
||
pages={1919--1925},
|
||
year={2017}
|
||
}
|
||
|
||
@inproceedings{patrini2017making,
|
||
title={Making Deep Neural Networks Robust to Label Noise: A Loss Correction Approach},
|
||
author={Patrini, Giorgio and Rozza, Alessandro and Krishna Menon, Aditya and Nock, Richard and Qu, Lizhen},
|
||
booktitle={Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition},
|
||
pages={1944--1952},
|
||
year={2017}
|
||
}
|
||
|
||
@inproceedings{zhang2018generalized,
|
||
title={Generalized Cross Entropy Loss for Training Deep Neural Networks with Noisy Labels},
|
||
author={Zhang, Zhilu and Sabuncu, Mert R.},
|
||
booktitle={Advances in Neural Information Processing Systems},
|
||
volume={31},
|
||
year={2018}
|
||
}
|
||
|
||
@inproceedings{han2018coteaching,
|
||
title={Co-teaching: Robust Training of Deep Neural Networks with Extremely Noisy Labels},
|
||
author={Han, Bo and Yao, Quanming and Yu, Xingrui and Niu, Gang and Xu, Miao and Hu, Weihua and Tsang, Ivor W. and Sugiyama, Masashi},
|
||
booktitle={Advances in Neural Information Processing Systems},
|
||
volume={31},
|
||
year={2018}
|
||
}
|
||
|
||
@inproceedings{zhou2021asymmetric,
|
||
title={Asymmetric Loss Functions for Learning with Noisy Labels},
|
||
author={Zhou, Xiong and Liu, Xianming and Jiang, Junjun and Gao, Xin and Ji, Xiangyang},
|
||
booktitle={Proceedings of the 38th International Conference on Machine Learning},
|
||
pages={12846--12856},
|
||
year={2021},
|
||
volume={139},
|
||
series={Proceedings of Machine Learning Research},
|
||
publisher={PMLR}
|
||
}
|
||
|
||
@inproceedings{feng2020can,
|
||
title={Can Cross Entropy Loss Be Robust to Label Noise?},
|
||
author={Feng, Lei and Shu, Senlin and Lin, Zhuoyi and Lv, Fengmei and Li, Li and An, Bo},
|
||
booktitle={Proceedings of the Twenty-Ninth International Joint Conference on Artificial Intelligence},
|
||
pages={2206--2212},
|
||
year={2020}
|
||
}
|
||
|
||
@article{cannings2020classification,
|
||
title={Classification with imperfect training labels},
|
||
author={Cannings, Timothy I. and Fan, Yingying and Samworth, Richard J.},
|
||
journal={Biometrika},
|
||
volume={107},
|
||
number={2},
|
||
pages={311--330},
|
||
year={2020},
|
||
doi={10.1093/biomet/asaa011}
|
||
}
|
||
|
||
@article{symmetrization2025,
|
||
title={Symmetrization of Loss Functions for Robust Training of Neural Networks
|
||
in the Presence of Noisy Labels},
|
||
journal={arXiv preprint arXiv:2605.20347},
|
||
year={2025}
|
||
}
|
||
|
||
@article{auc2025retrieval,
|
||
title={Optimizing What Matters: {AUC}-Driven Learning for Robust Neural Retrieval},
|
||
journal={arXiv preprint arXiv:2510.00137},
|
||
year={2025}
|
||
}
|
||
#+end_src
|
||
|
||
** Paper Subnodes
|
||
|
||
- [[id:bf0fc08a-e806-48df-b188-7a2c4c41c693][impl/paper-tabpfn]] — TabPFN: in-context learning for small tabular classification (ICLR 2023)
|
||
- [[id:8f59b736-04ea-4d11-9195-30d125a127f8][impl/paper-beyond-rebalancing]] — benchmark of 12 classifiers under imbalance, no rebalancing (2024)
|