backup: 2026-08-29 06:59

impltodos.org

@@ -5,23 +5,65 @@
 #+title: impl/todos
 #+filetags: :project: :todo:
 
-* TODO Look into previous results success/failures — understand failure modes of datasets where ROLL performs poorly (glass1, pima, glass6 from 2026-07-08 KEEL run; see [[id:a53cbe84-cd8d-45c2-a8cf-34ab520a3ea5][impl/experiments]] for full results)
+* DONE Look into previous results success/failures — understand failure modes of datasets where ROLL performs poorly (glass1, pima, glass6 from 2026-07-08 KEEL run; see [[id:a53cbe84-cd8d-45c2-a8cf-34ab520a3ea5][impl/experiments]] for full results)
 
-* TODO Run credit card fraud experiment
-Experiment file already exists: =experiments/large/experiment-creditcard.py=.
-Data at =~/.data/creditcard/creditcard.csv= (downloaded via Kaggle CLI).
-- Run all 7 configs (roll-aoc, bce-weighted, mae, gce-0.7, libauc-auroc, focal-loss, asymmetric-loss)
-- Tabular, ~285K rows, naturally imbalanced (~0.17% positive) — fast to run
-- Supports the AUC-ROLL noisy dataset story (real-world messiness, severe imbalance)
+* TODO Fill out ANIMAL-10N download form and download to Mac
+Form: https://forms.gle/8mbmbNgDFQ2rA1fLA
+After receiving download link, run on Mac:
+=./scripts/run_remote.sh experiments/other/experiment-animal10n.py --detach=
+(data goes to =~/.data/animal10n/= on the Mac)
 
-* TODO Track 1: controlled noise injection on KEEL datasets
-Take 3–5 KEEL datasets with varying IR and inject label noise at 10%, 20%, 30% on minority class.
-Goal: reproduce the CIFAR-10N noise-level ablation on imbalanced tabular data.
-- Choose datasets spanning a range of IR (e.g. low/medium/high imbalance)
-- Write noise injection utility (flip minority labels randomly at given rate)
-- Run all 7 configs per dataset × noise level
-- Expected result: ROLL-AUC advantage grows with noise level (mirrors CIFAR finding)
-- Key for thesis: validates the noise-sensitivity story on tabular domain
+* DONE Run libauc-auroc on credit card fraud (CPU)
+Completed. Results in =results-final/creditcard/=.
+
+* DONE Re-run credit card fraud weighted baselines
+Completed. All 8 configs present in =results-final/creditcard/=.
+
+* DONE Re-run all KEEL experiments with weighted baselines
+Completed as of 2026-07-27. All 12 datasets × 8 configs in =results-final/keel/=.
+=_base.py= confirmed to pass =pos_weight=imbalance_ratio= to gce-0.7, mae, focal-loss, asymmetric-loss.
+
+* ACTIVE Food-101N experiment
+Running on beacon-wsl (as of 2026-07-28). =experiment-food101n-rerun-weighted.py= — 6 configs × 3 episodes.
+6/18 episodes done as of 2026-07-28 (~3 days remaining).
+Writes to =results-final/food101n/=.
+Tail: =ssh aner@192.168.1.200 'tail -f ~/roll-impl/run.log'=
+Fetch: =./scripts/fetch_results_beacon.sh=
+
+* TODO Resume NIH ChestX-ray14 experiment
+Crashed 2026-07-31 ~20:51 on Mac during =libauc-auroc= config.
+Error: =ValueError: All episodes have NaN/inf predictions — cannot build ROC curve=
+- Crash in =src/summary.py= ROC generation after training, not during training itself
+- The 6 non-libauc configs (roll-aoc, roll-tpr90, bce-weighted, gce-0.7, mae, focal-loss, asymmetric-loss) may have completed — check =results-final/chestxray14/= before resuming
+- Fix: diagnose why libauc produces NaN/inf outputs; likely learning rate or optimizer issue with PESG on MPS/CPU
+- Run script: =experiments/other/experiment-chestxray14.py= — writes directly to =results-final/chestxray14/=
+- Resume via: =./scripts/run_remote.sh experiments/other/experiment-chestxray14.py --detach=
+  (after fixing the libauc NaN issue or skipping libauc-auroc)
+
+* TODO CIFAR-100N experiment
+Same paper and infrastructure as CIFAR-10N (Wei et al. 2022, arXiv:2110.12088).
+Download: ~160MB — same CIFAR-100 images + noise label file, no new image download needed.
+Key difference from CIFAR-10N: noise rate is 40.2% (vs 9–40% for CIFAR-10N) — harder noise regime.
+Setup: binarize same way as CIFAR-10N (one class vs rest), reuse ConvNet architecture.
+Run on Mac (trivial compute, same as CIFAR-10N).
+Priority: HIGH — near-zero effort, extends noise ablation to a harder regime.
+
+* TODO ISIC 2020 Melanoma experiment
+Binary dermoscopy classification: malignant (584 images, 1.8%) vs benign (32,542 images, 98.2%). IR ≈ 56:1.
+Label noise: real inter-dermatologist disagreement and clinical ambiguity (not annotator error).
+Download: ~2–4GB via Kaggle pre-resized 224×224 JPEG dataset (nischaydnk/isic-2020-jpg-224x224-resized).
+Architecture: pretrained CNN backbone (same pattern as ChestX-ray14 — frozen backbone + linear head or fine-tune).
+Run on beacon-wsl after Food-101N + ChestX-ray14 complete.
+Priority: MEDIUM-HIGH — strong clinical story, extreme imbalance + real noise, feasible size.
+Note: only 584 positive examples — conclusions will be limited; mention in thesis.
+
+* ACTIVE KEEL label noise ablation
+Running 5%, 10%, 20% label noise on all 12 KEEL datasets (10 episodes each) on Mac.
+Script: =experiments/keel/run_all_noise_levels.sh= (launched 2026-08-01)
+Results: =results-final/keel-noise-05/=, =results-final/keel-noise-10/=, =results-final/keel-noise-20/=
+(date-stamped subdirs per dataset, e.g. =keel-noise-05/glass0/YYYY-MM-DD-HH-MM/=)
+0% baseline: =results-final/keel/= (existing, 5 episodes)
+Fetch when done: =./scripts/fetch_results.sh=
 
 * TODO CheXpert experiment
 ~224K chest X-ray images, label noise from automated NLP extraction of radiology reports.
@@ -32,28 +74,21 @@ Dataset: https://stanfordmlgroup.github.io/competitions/chexpert/ (requires regi
 - Medium-large compute cost; run on remote Mac or cluster
 - Priority: lower than credit card fraud and Track 1; run after those confirm the story
 
-* TODO NIH ChestX-ray14 experiment
-~112K chest X-ray images, label noise from NLP extraction of radiology reports.
-Similar story to CheXpert but smaller; could substitute if CheXpert access is delayed.
-Dataset: https://nihcc.app.box.com/v/ChestXray-NIHCC
-- Same setup considerations as CheXpert
-- Priority: same as CheXpert — run one or the other, not necessarily both
-
 * TODO ANIMAL-10N experiment
 Dataset loader written: =Animal10NDataset= in =src/datasets.py=.
-Data must be manually downloaded from https://nihalsid.github.io/animal-10n/ and extracted to =~/.data/animal10n/= (train/ and test/ subdirs).
-Next steps:
-- Download the dataset on the remote Mac
-- Write =experiments/other/experiment-animal10n.py= (all 7 configs: roll-aoc, bce-weighted, mae, gce-0.7, libauc-auroc, focal-loss, asymmetric-loss)
-- ConvNet needs adjusted Linear layer: 64×64 images → after 3 MaxPool → 8×8 spatial → 64*8*8=4096 flatten (vs 1024 for CIFAR)
-- positive_class=0 (cat vs rest), IR ~9, same setup as CIFAR-10N
+Experiment script written: =experiments/other/experiment-animal10n.py=.
+Data must be downloaded after filling form (see todo above).
+- ConvNet with image_size=64 (4096 linear), positive_class=0 (cat), IR ~9
+- MPS-aware: sequential_episodes=True, libauc forced to CPU
 
-* TODO Food-101N experiment
-~310K training images, ~18.4% web noise. Binary: one class vs rest.
-Dataset: https://kuanghuei.github.io/Food-101N-Dataset/ (requires Google account for download)
-- Write =Food101NDataset= in =src/datasets.py= using torchvision.datasets.ImageFolder
-- Images are 224×224; may need ResNet backbone or resize to 64×64 for ConvNet
-- Lower priority than ANIMAL-10N (larger, more infra work)
+* TODO Noisy Ostracods experiment
+NeurIPS 2024 D&B track (arXiv:2412.02313). ~71k micro-crustacean specimens, specialist-annotated.
+IR = 22,429 (extremely imbalanced), ~5.6% real expert-level label noise.
+Download: estimated ~7–15GB JPEG (476×476px avg), hosted on SharePoint via GitHub repo.
+GitHub: https://github.com/H-Jamieu/Noisy_ostracods
+Binarization needed: multi-class (78 genera), would need to pick one vs rest.
+Caveat: paper reports existing noise-robust methods fail to beat CE — a strong ROLL result would be notable.
+Priority: LOW — more setup friction (SharePoint, not Kaggle), binarization decision needed, uncertain payoff.
 
 * TODO Clothing1M experiment
 ~1M images, ~38.5% web noise. Binary: one category vs rest.