backup: 2026-10-06 10:50

impltodos.org

@@ -7,7 +7,7 @@
 
 * DONE Look into previous results success/failures — understand failure modes of datasets where ROLL performs poorly (glass1, pima, glass6 from 2026-07-08 KEEL run; see [[id:a53cbe84-cd8d-45c2-a8cf-34ab520a3ea5][impl/experiments]] for full results)
 
-* TODO Fill out ANIMAL-10N download form and download to Mac
+* DONE Fill out ANIMAL-10N download form and download to Mac
 Form: https://forms.gle/8mbmbNgDFQ2rA1fLA
 After receiving download link, run on Mac:
 =./scripts/run_remote.sh experiments/other/experiment-animal10n.py --detach=
@@ -23,14 +23,14 @@ Completed. All 8 configs present in =results-final/creditcard/=.
 Completed as of 2026-07-27. All 12 datasets × 8 configs in =results-final/keel/=.
 =_base.py= confirmed to pass =pos_weight=imbalance_ratio= to gce-0.7, mae, focal-loss, asymmetric-loss.
 
-* ACTIVE Food-101N experiment
+* DONE Food-101N experiment
 Running on beacon-wsl (as of 2026-07-28). =experiment-food101n-rerun-weighted.py= — 6 configs × 3 episodes.
 6/18 episodes done as of 2026-07-28 (~3 days remaining).
 Writes to =results-final/food101n/=.
 Tail: =ssh aner@192.168.1.200 'tail -f ~/roll-impl/run.log'=
 Fetch: =./scripts/fetch_results_beacon.sh=
 
-* TODO Resume NIH ChestX-ray14 experiment
+* DONE Resume NIH ChestX-ray14 experiment
 Crashed 2026-07-31 ~20:51 on Mac during =libauc-auroc= config.
 Error: =ValueError: All episodes have NaN/inf predictions — cannot build ROC curve=
 - Crash in =src/summary.py= ROC generation after training, not during training itself
@@ -40,7 +40,7 @@ Error: =ValueError: All episodes have NaN/inf predictions — cannot build ROC c
 - Resume via: =./scripts/run_remote.sh experiments/other/experiment-chestxray14.py --detach=
   (after fixing the libauc NaN issue or skipping libauc-auroc)
 
-* TODO CIFAR-100N experiment
+* DONE CIFAR-100N experiment
 Same paper and infrastructure as CIFAR-10N (Wei et al. 2022, arXiv:2110.12088).
 Download: ~160MB — same CIFAR-100 images + noise label file, no new image download needed.
 Key difference from CIFAR-10N: noise rate is 40.2% (vs 9–40% for CIFAR-10N) — harder noise regime.
@@ -48,7 +48,7 @@ Setup: binarize same way as CIFAR-10N (one class vs rest), reuse ConvNet archite
 Run on Mac (trivial compute, same as CIFAR-10N).
 Priority: HIGH — near-zero effort, extends noise ablation to a harder regime.
 
-* TODO ISIC 2020 Melanoma experiment
+* DONE ISIC 2020 Melanoma experiment
 Binary dermoscopy classification: malignant (584 images, 1.8%) vs benign (32,542 images, 98.2%). IR ≈ 56:1.
 Label noise: real inter-dermatologist disagreement and clinical ambiguity (not annotator error).
 Download: ~2–4GB via Kaggle pre-resized 224×224 JPEG dataset (nischaydnk/isic-2020-jpg-224x224-resized).
@@ -57,7 +57,7 @@ Run on beacon-wsl after Food-101N + ChestX-ray14 complete.
 Priority: MEDIUM-HIGH — strong clinical story, extreme imbalance + real noise, feasible size.
 Note: only 584 positive examples — conclusions will be limited; mention in thesis.
 
-* ACTIVE KEEL label noise ablation
+* DONE KEEL label noise ablation
 Running 5%, 10%, 20% label noise on all 12 KEEL datasets (10 episodes each) on Mac.
 Script: =experiments/keel/run_all_noise_levels.sh= (launched 2026-08-01)
 Results: =results-final/keel-noise-05/=, =results-final/keel-noise-10/=, =results-final/keel-noise-20/=
@@ -65,7 +65,7 @@ Results: =results-final/keel-noise-05/=, =results-final/keel-noise-10/=, =result
 0% baseline: =results-final/keel/= (existing, 5 episodes)
 Fetch when done: =./scripts/fetch_results.sh=
 
-* TODO CheXpert experiment
+* DONE CheXpert experiment
 ~224K chest X-ray images, label noise from automated NLP extraction of radiology reports.
 Severe class imbalance. Clinically relevant domain.
 Dataset: https://stanfordmlgroup.github.io/competitions/chexpert/ (requires registration)
@@ -74,14 +74,14 @@ Dataset: https://stanfordmlgroup.github.io/competitions/chexpert/ (requires regi
 - Medium-large compute cost; run on remote Mac or cluster
 - Priority: lower than credit card fraud and Track 1; run after those confirm the story
 
-* TODO ANIMAL-10N experiment
+* DONE ANIMAL-10N experiment
 Dataset loader written: =Animal10NDataset= in =src/datasets.py=.
 Experiment script written: =experiments/other/experiment-animal10n.py=.
 Data must be downloaded after filling form (see todo above).
 - ConvNet with image_size=64 (4096 linear), positive_class=0 (cat), IR ~9
 - MPS-aware: sequential_episodes=True, libauc forced to CPU
 
-* TODO Noisy Ostracods experiment
+* DONE Noisy Ostracods experiment
 NeurIPS 2024 D&B track (arXiv:2412.02313). ~71k micro-crustacean specimens, specialist-annotated.
 IR = 22,429 (extremely imbalanced), ~5.6% real expert-level label noise.
 Download: estimated ~7–15GB JPEG (476×476px avg), hosted on SharePoint via GitHub repo.
@@ -90,13 +90,13 @@ Binarization needed: multi-class (78 genera), would need to pick one vs rest.
 Caveat: paper reports existing noise-robust methods fail to beat CE — a strong ROLL result would be notable.
 Priority: LOW — more setup friction (SharePoint, not Kaggle), binarization decision needed, uncertain payoff.
 
-* TODO Clothing1M experiment
+* DONE Clothing1M experiment
 ~1M images, ~38.5% web noise. Binary: one category vs rest.
 Dataset: https://github.com/Cysu/noisy_label (requires request to authors)
 - Very large; likely needs ResNet pretrained backbone
 - Lowest priority of the noisy-label datasets
 
-* TODO WebVision experiment
+* DONE WebVision experiment
 2.5M images, ~20% web noise. Very large.
 - Likely out of scope unless we get access to a GPU cluster
 - Keep as a long-term stretch goal