roam/impltodos.org

:PROPERTIES:
:ID: fdc18323-e9c2-4fc6-ace5-065035d30c51
:END:

#+title: impl/todos
#+filetags: :project: :todo:

* DONE Look into previous results success/failures — understand failure modes of datasets where ROLL performs poorly (glass1, pima, glass6 from 2026-07-08 KEEL run; see [[id:a53cbe84-cd8d-45c2-a8cf-34ab520a3ea5][impl/experiments]] for full results)

* DONE Fill out ANIMAL-10N download form and download to Mac
Form: https://forms.gle/8mbmbNgDFQ2rA1fLA
After receiving download link, run on Mac:
=./scripts/run_remote.sh experiments/other/experiment-animal10n.py --detach=
(data goes to =~/.data/animal10n/= on the Mac)

* DONE Run libauc-auroc on credit card fraud (CPU)
Completed. Results in =results-final/creditcard/=.

* DONE Re-run credit card fraud weighted baselines
Completed. All 8 configs present in =results-final/creditcard/=.

* DONE Re-run all KEEL experiments with weighted baselines
Completed as of 2026-07-27. All 12 datasets × 8 configs in =results-final/keel/=.
=_base.py= confirmed to pass =pos_weight=imbalance_ratio= to gce-0.7, mae, focal-loss, asymmetric-loss.

* DONE Food-101N experiment
Running on beacon-wsl (as of 2026-07-28). =experiment-food101n-rerun-weighted.py= — 6 configs × 3 episodes.
6/18 episodes done as of 2026-07-28 (~3 days remaining).
Writes to =results-final/food101n/=.
Tail: =ssh aner@192.168.1.200 'tail -f ~/roll-impl/run.log'=
Fetch: =./scripts/fetch_results_beacon.sh=

* DONE Resume NIH ChestX-ray14 experiment
Crashed 2026-07-31 ~20:51 on Mac during =libauc-auroc= config.
Error: =ValueError: All episodes have NaN/inf predictions — cannot build ROC curve=
- Crash in =src/summary.py= ROC generation after training, not during training itself
- The 6 non-libauc configs (roll-aoc, roll-tpr90, bce-weighted, gce-0.7, mae, focal-loss, asymmetric-loss) may have completed — check =results-final/chestxray14/= before resuming
- Fix: diagnose why libauc produces NaN/inf outputs; likely learning rate or optimizer issue with PESG on MPS/CPU
- Run script: =experiments/other/experiment-chestxray14.py= — writes directly to =results-final/chestxray14/=
- Resume via: =./scripts/run_remote.sh experiments/other/experiment-chestxray14.py --detach=
  (after fixing the libauc NaN issue or skipping libauc-auroc)

* DONE CIFAR-100N experiment
Same paper and infrastructure as CIFAR-10N (Wei et al. 2022, arXiv:2110.12088).
Download: ~160MB — same CIFAR-100 images + noise label file, no new image download needed.
Key difference from CIFAR-10N: noise rate is 40.2% (vs 9–40% for CIFAR-10N) — harder noise regime.
Setup: binarize same way as CIFAR-10N (one class vs rest), reuse ConvNet architecture.
Run on Mac (trivial compute, same as CIFAR-10N).
Priority: HIGH — near-zero effort, extends noise ablation to a harder regime.

* DONE ISIC 2020 Melanoma experiment
Binary dermoscopy classification: malignant (584 images, 1.8%) vs benign (32,542 images, 98.2%). IR ≈ 56:1.
Label noise: real inter-dermatologist disagreement and clinical ambiguity (not annotator error).
Download: ~2–4GB via Kaggle pre-resized 224×224 JPEG dataset (nischaydnk/isic-2020-jpg-224x224-resized).
Architecture: pretrained CNN backbone (same pattern as ChestX-ray14 — frozen backbone + linear head or fine-tune).
Run on beacon-wsl after Food-101N + ChestX-ray14 complete.
Priority: MEDIUM-HIGH — strong clinical story, extreme imbalance + real noise, feasible size.
Note: only 584 positive examples — conclusions will be limited; mention in thesis.

* DONE KEEL label noise ablation
Running 5%, 10%, 20% label noise on all 12 KEEL datasets (10 episodes each) on Mac.
Script: =experiments/keel/run_all_noise_levels.sh= (launched 2026-08-01)
Results: =results-final/keel-noise-05/=, =results-final/keel-noise-10/=, =results-final/keel-noise-20/=
(date-stamped subdirs per dataset, e.g. =keel-noise-05/glass0/YYYY-MM-DD-HH-MM/=)
0% baseline: =results-final/keel/= (existing, 5 episodes)
Fetch when done: =./scripts/fetch_results.sh=

* DONE CheXpert experiment
~224K chest X-ray images, label noise from automated NLP extraction of radiology reports.
Severe class imbalance. Clinically relevant domain.
Dataset: https://stanfordmlgroup.github.io/competitions/chexpert/ (requires registration)
- Binary classification: one pathology vs rest (e.g. pleural effusion, highest prevalence)
- Likely needs pretrained ResNet backbone (images are 224×224)
- Medium-large compute cost; run on remote Mac or cluster
- Priority: lower than credit card fraud and Track 1; run after those confirm the story

* DONE ANIMAL-10N experiment
Dataset loader written: =Animal10NDataset= in =src/datasets.py=.
Experiment script written: =experiments/other/experiment-animal10n.py=.
Data must be downloaded after filling form (see todo above).
- ConvNet with image_size=64 (4096 linear), positive_class=0 (cat), IR ~9
- MPS-aware: sequential_episodes=True, libauc forced to CPU

* DONE Noisy Ostracods experiment
NeurIPS 2024 D&B track (arXiv:2412.02313). ~71k micro-crustacean specimens, specialist-annotated.
IR = 22,429 (extremely imbalanced), ~5.6% real expert-level label noise.
Download: estimated ~7–15GB JPEG (476×476px avg), hosted on SharePoint via GitHub repo.
GitHub: https://github.com/H-Jamieu/Noisy_ostracods
Binarization needed: multi-class (78 genera), would need to pick one vs rest.
Caveat: paper reports existing noise-robust methods fail to beat CE — a strong ROLL result would be notable.
Priority: LOW — more setup friction (SharePoint, not Kaggle), binarization decision needed, uncertain payoff.

* DONE Clothing1M experiment
~1M images, ~38.5% web noise. Binary: one category vs rest.
Dataset: https://github.com/Cysu/noisy_label (requires request to authors)
- Very large; likely needs ResNet pretrained backbone
- Lowest priority of the noisy-label datasets

* DONE WebVision experiment
2.5M images, ~20% web noise. Very large.
- Likely out of scope unless we get access to a GPU cluster
- Keep as a long-term stretch goal