3.7 KiB
impl/todos
- Look into previous results success/failures — understand failure modes of datasets where ROLL performs poorly (glass1, pima, glass6 from 2026-07-08 KEEL run; see impl/experiments for full results)
- Run credit card fraud experiment
- Track 1: controlled noise injection on KEEL datasets
- CheXpert experiment
- NIH ChestX-ray14 experiment
- ANIMAL-10N experiment
- Food-101N experiment
- Clothing1M experiment
- WebVision experiment
TODO Look into previous results success/failures — understand failure modes of datasets where ROLL performs poorly (glass1, pima, glass6 from 2026-07-08 KEEL run; see impl/experiments for full results)
TODO Run credit card fraud experiment
Experiment file already exists: experiments/large/experiment-creditcard.py.
Data at ~/.data/creditcard/creditcard.csv (downloaded via Kaggle CLI).
- Run all 7 configs (roll-aoc, bce-weighted, mae, gce-0.7, libauc-auroc, focal-loss, asymmetric-loss)
- Tabular, ~285K rows, naturally imbalanced (~0.17% positive) — fast to run
- Supports the AUC-ROLL noisy dataset story (real-world messiness, severe imbalance)
TODO Track 1: controlled noise injection on KEEL datasets
Take 3–5 KEEL datasets with varying IR and inject label noise at 10%, 20%, 30% on minority class. Goal: reproduce the CIFAR-10N noise-level ablation on imbalanced tabular data.
- Choose datasets spanning a range of IR (e.g. low/medium/high imbalance)
- Write noise injection utility (flip minority labels randomly at given rate)
- Run all 7 configs per dataset × noise level
- Expected result: ROLL-AUC advantage grows with noise level (mirrors CIFAR finding)
- Key for thesis: validates the noise-sensitivity story on tabular domain
TODO CheXpert experiment
~224K chest X-ray images, label noise from automated NLP extraction of radiology reports. Severe class imbalance. Clinically relevant domain. Dataset: https://stanfordmlgroup.github.io/competitions/chexpert/ (requires registration)
- Binary classification: one pathology vs rest (e.g. pleural effusion, highest prevalence)
- Likely needs pretrained ResNet backbone (images are 224×224)
- Medium-large compute cost; run on remote Mac or cluster
- Priority: lower than credit card fraud and Track 1; run after those confirm the story
TODO NIH ChestX-ray14 experiment
~112K chest X-ray images, label noise from NLP extraction of radiology reports. Similar story to CheXpert but smaller; could substitute if CheXpert access is delayed. Dataset: https://nihcc.app.box.com/v/ChestXray-NIHCC
- Same setup considerations as CheXpert
- Priority: same as CheXpert — run one or the other, not necessarily both
TODO ANIMAL-10N experiment
Dataset loader written: Animal10NDataset in src/datasets.py.
Data must be manually downloaded from https://nihalsid.github.io/animal-10n/ and extracted to ~/.data/animal10n/ (train/ and test/ subdirs).
Next steps:
- Download the dataset on the remote Mac
- Write
experiments/other/experiment-animal10n.py(all 7 configs: roll-aoc, bce-weighted, mae, gce-0.7, libauc-auroc, focal-loss, asymmetric-loss) - ConvNet needs adjusted Linear layer: 64×64 images → after 3 MaxPool → 8×8 spatial → 64*8*8=4096 flatten (vs 1024 for CIFAR)
- positive_class=0 (cat vs rest), IR ~9, same setup as CIFAR-10N
TODO Food-101N experiment
~310K training images, ~18.4% web noise. Binary: one class vs rest. Dataset: https://kuanghuei.github.io/Food-101N-Dataset/ (requires Google account for download)
- Write
Food101NDatasetinsrc/datasets.pyusing torchvision.datasets.ImageFolder - Images are 224×224; may need ResNet backbone or resize to 64×64 for ConvNet
- Lower priority than ANIMAL-10N (larger, more infra work)
TODO Clothing1M experiment
~1M images, ~38.5% web noise. Binary: one category vs rest. Dataset: https://github.com/Cysu/noisy_label (requires request to authors)
- Very large; likely needs ResNet pretrained backbone
- Lowest priority of the noisy-label datasets
TODO WebVision experiment
2.5M images, ~20% web noise. Very large.
- Likely out of scope unless we get access to a GPU cluster
- Keep as a long-term stretch goal