5.9 KiB
impl/todos
- Look into previous results success/failures — understand failure modes of datasets where ROLL performs poorly (glass1, pima, glass6 from 2026-07-08 KEEL run; see impl/experiments for full results)
- Fill out ANIMAL-10N download form and download to Mac
- Run libauc-auroc on credit card fraud (CPU)
- Re-run credit card fraud weighted baselines
- Re-run all KEEL experiments with weighted baselines
- ACTIVE Food-101N experiment
- Resume NIH ChestX-ray14 experiment
- CIFAR-100N experiment
- ISIC 2020 Melanoma experiment
- ACTIVE KEEL label noise ablation
- CheXpert experiment
- ANIMAL-10N experiment
- Noisy Ostracods experiment
- Clothing1M experiment
- WebVision experiment
DONE Look into previous results success/failures — understand failure modes of datasets where ROLL performs poorly (glass1, pima, glass6 from 2026-07-08 KEEL run; see impl/experiments for full results)
TODO Fill out ANIMAL-10N download form and download to Mac
Form: https://forms.gle/8mbmbNgDFQ2rA1fLA
After receiving download link, run on Mac:
./scripts/run_remote.sh experiments/other/experiment-animal10n.py --detach
(data goes to ~/.data/animal10n/ on the Mac)
DONE Run libauc-auroc on credit card fraud (CPU)
Completed. Results in results-final/creditcard/.
DONE Re-run credit card fraud weighted baselines
Completed. All 8 configs present in results-final/creditcard/.
DONE Re-run all KEEL experiments with weighted baselines
Completed as of 2026-07-27. All 12 datasets × 8 configs in results-final/keel/.
_base.py confirmed to pass pos_weight=imbalance_ratio to gce-0.7, mae, focal-loss, asymmetric-loss.
ACTIVE Food-101N experiment
Running on beacon-wsl (as of 2026-07-28). experiment-food101n-rerun-weighted.py — 6 configs × 3 episodes.
6/18 episodes done as of 2026-07-28 (~3 days remaining).
Writes to results-final/food101n/.
Tail: ssh aner@192.168.1.200 'tail -f ~/roll-impl/run.log'
Fetch: ./scripts/fetch_results_beacon.sh
TODO Resume NIH ChestX-ray14 experiment
Crashed 2026-07-31 ~20:51 on Mac during libauc-auroc config.
Error: ValueError: All episodes have NaN/inf predictions — cannot build ROC curve
- Crash in
src/summary.pyROC generation after training, not during training itself - The 6 non-libauc configs (roll-aoc, roll-tpr90, bce-weighted, gce-0.7, mae, focal-loss, asymmetric-loss) may have completed — check
results-final/chestxray14/before resuming - Fix: diagnose why libauc produces NaN/inf outputs; likely learning rate or optimizer issue with PESG on MPS/CPU
- Run script:
experiments/other/experiment-chestxray14.py— writes directly toresults-final/chestxray14/ - Resume via:
./scripts/run_remote.sh experiments/other/experiment-chestxray14.py --detach(after fixing the libauc NaN issue or skipping libauc-auroc)
TODO CIFAR-100N experiment
Same paper and infrastructure as CIFAR-10N (Wei et al. 2022, arXiv:2110.12088). Download: ~160MB — same CIFAR-100 images + noise label file, no new image download needed. Key difference from CIFAR-10N: noise rate is 40.2% (vs 9–40% for CIFAR-10N) — harder noise regime. Setup: binarize same way as CIFAR-10N (one class vs rest), reuse ConvNet architecture. Run on Mac (trivial compute, same as CIFAR-10N). Priority: HIGH — near-zero effort, extends noise ablation to a harder regime.
TODO ISIC 2020 Melanoma experiment
Binary dermoscopy classification: malignant (584 images, 1.8%) vs benign (32,542 images, 98.2%). IR ≈ 56:1. Label noise: real inter-dermatologist disagreement and clinical ambiguity (not annotator error). Download: ~2–4GB via Kaggle pre-resized 224×224 JPEG dataset (nischaydnk/isic-2020-jpg-224x224-resized). Architecture: pretrained CNN backbone (same pattern as ChestX-ray14 — frozen backbone + linear head or fine-tune). Run on beacon-wsl after Food-101N + ChestX-ray14 complete. Priority: MEDIUM-HIGH — strong clinical story, extreme imbalance + real noise, feasible size. Note: only 584 positive examples — conclusions will be limited; mention in thesis.
ACTIVE KEEL label noise ablation
Running 5%, 10%, 20% label noise on all 12 KEEL datasets (10 episodes each) on Mac.
Script: experiments/keel/run_all_noise_levels.sh (launched 2026-08-01)
Results: results-final/keel-noise-05/, results-final/keel-noise-10/, results-final/keel-noise-20/
(date-stamped subdirs per dataset, e.g. keel-noise-05/glass0/YYYY-MM-DD-HH-MM/)
0% baseline: results-final/keel/ (existing, 5 episodes)
Fetch when done: ./scripts/fetch_results.sh
TODO CheXpert experiment
~224K chest X-ray images, label noise from automated NLP extraction of radiology reports. Severe class imbalance. Clinically relevant domain. Dataset: https://stanfordmlgroup.github.io/competitions/chexpert/ (requires registration)
- Binary classification: one pathology vs rest (e.g. pleural effusion, highest prevalence)
- Likely needs pretrained ResNet backbone (images are 224×224)
- Medium-large compute cost; run on remote Mac or cluster
- Priority: lower than credit card fraud and Track 1; run after those confirm the story
TODO ANIMAL-10N experiment
Dataset loader written: Animal10NDataset in src/datasets.py.
Experiment script written: experiments/other/experiment-animal10n.py.
Data must be downloaded after filling form (see todo above).
- ConvNet with image_size=64 (4096 linear), positive_class=0 (cat), IR ~9
- MPS-aware: sequential_episodes=True, libauc forced to CPU
TODO Noisy Ostracods experiment
NeurIPS 2024 D&B track (arXiv:2412.02313). ~71k micro-crustacean specimens, specialist-annotated. IR = 22,429 (extremely imbalanced), ~5.6% real expert-level label noise. Download: estimated ~7–15GB JPEG (476×476px avg), hosted on SharePoint via GitHub repo. GitHub: https://github.com/H-Jamieu/Noisy_ostracods Binarization needed: multi-class (78 genera), would need to pick one vs rest. Caveat: paper reports existing noise-robust methods fail to beat CE — a strong ROLL result would be notable. Priority: LOW — more setup friction (SharePoint, not Kaggle), binarization decision needed, uncertain payoff.
TODO Clothing1M experiment
~1M images, ~38.5% web noise. Binary: one category vs rest. Dataset: https://github.com/Cysu/noisy_label (requires request to authors)
- Very large; likely needs ResNet pretrained backbone
- Lowest priority of the noisy-label datasets
TODO WebVision experiment
2.5M images, ~20% web noise. Very large.
- Likely out of scope unless we get access to a GPU cluster
- Keep as a long-term stretch goal