backup: 2026-07-25 14:39

impltodos.org

@@ -0,0 +1,67 @@
+:PROPERTIES:
+:ID: fdc18323-e9c2-4fc6-ace5-065035d30c51
+:END:
+
+#+title: impl/todos
+#+filetags: :project: :todo:
+
+* TODO Look into previous results success/failures — understand failure modes of datasets where ROLL performs poorly (glass1, pima, glass6 from 2026-07-08 KEEL run; see [[id:a53cbe84-cd8d-45c2-a8cf-34ab520a3ea5][impl/experiments]] for full results)
+
+* TODO Run credit card fraud experiment
+Experiment file already exists: =experiments/large/experiment-creditcard.py=.
+Data at =~/.data/creditcard/creditcard.csv= (downloaded via Kaggle CLI).
+- Run all 7 configs (roll-aoc, bce-weighted, mae, gce-0.7, libauc-auroc, focal-loss, asymmetric-loss)
+- Tabular, ~285K rows, naturally imbalanced (~0.17% positive) — fast to run
+- Supports the AUC-ROLL noisy dataset story (real-world messiness, severe imbalance)
+
+* TODO Track 1: controlled noise injection on KEEL datasets
+Take 3–5 KEEL datasets with varying IR and inject label noise at 10%, 20%, 30% on minority class.
+Goal: reproduce the CIFAR-10N noise-level ablation on imbalanced tabular data.
+- Choose datasets spanning a range of IR (e.g. low/medium/high imbalance)
+- Write noise injection utility (flip minority labels randomly at given rate)
+- Run all 7 configs per dataset × noise level
+- Expected result: ROLL-AUC advantage grows with noise level (mirrors CIFAR finding)
+- Key for thesis: validates the noise-sensitivity story on tabular domain
+
+* TODO CheXpert experiment
+~224K chest X-ray images, label noise from automated NLP extraction of radiology reports.
+Severe class imbalance. Clinically relevant domain.
+Dataset: https://stanfordmlgroup.github.io/competitions/chexpert/ (requires registration)
+- Binary classification: one pathology vs rest (e.g. pleural effusion, highest prevalence)
+- Likely needs pretrained ResNet backbone (images are 224×224)
+- Medium-large compute cost; run on remote Mac or cluster
+- Priority: lower than credit card fraud and Track 1; run after those confirm the story
+
+* TODO NIH ChestX-ray14 experiment
+~112K chest X-ray images, label noise from NLP extraction of radiology reports.
+Similar story to CheXpert but smaller; could substitute if CheXpert access is delayed.
+Dataset: https://nihcc.app.box.com/v/ChestXray-NIHCC
+- Same setup considerations as CheXpert
+- Priority: same as CheXpert — run one or the other, not necessarily both
+
+* TODO ANIMAL-10N experiment
+Dataset loader written: =Animal10NDataset= in =src/datasets.py=.
+Data must be manually downloaded from https://nihalsid.github.io/animal-10n/ and extracted to =~/.data/animal10n/= (train/ and test/ subdirs).
+Next steps:
+- Download the dataset on the remote Mac
+- Write =experiments/other/experiment-animal10n.py= (all 7 configs: roll-aoc, bce-weighted, mae, gce-0.7, libauc-auroc, focal-loss, asymmetric-loss)
+- ConvNet needs adjusted Linear layer: 64×64 images → after 3 MaxPool → 8×8 spatial → 64*8*8=4096 flatten (vs 1024 for CIFAR)
+- positive_class=0 (cat vs rest), IR ~9, same setup as CIFAR-10N
+
+* TODO Food-101N experiment
+~310K training images, ~18.4% web noise. Binary: one class vs rest.
+Dataset: https://kuanghuei.github.io/Food-101N-Dataset/ (requires Google account for download)
+- Write =Food101NDataset= in =src/datasets.py= using torchvision.datasets.ImageFolder
+- Images are 224×224; may need ResNet backbone or resize to 64×64 for ConvNet
+- Lower priority than ANIMAL-10N (larger, more infra work)
+
+* TODO Clothing1M experiment
+~1M images, ~38.5% web noise. Binary: one category vs rest.
+Dataset: https://github.com/Cysu/noisy_label (requires request to authors)
+- Very large; likely needs ResNet pretrained backbone
+- Lowest priority of the noisy-label datasets
+
+* TODO WebVision experiment
+2.5M images, ~20% web noise. Very large.
+- Likely out of scope unless we get access to a GPU cluster
+- Keep as a long-term stretch goal