:PROPERTIES:
:ID: fdc18323-e9c2-4fc6-ace5-065035d30c51
:END:

#+title: impl/todos
#+filetags: :project: :todo:

* TODO Look into previous results success/failures — understand failure modes of datasets where ROLL performs poorly (glass1, pima, glass6 from 2026-07-08 KEEL run; see [[id:a53cbe84-cd8d-45c2-a8cf-34ab520a3ea5][impl/experiments]] for full results)

* TODO Run credit card fraud experiment
Experiment file already exists: =experiments/large/experiment-creditcard.py=.
Data at =~/.data/creditcard/creditcard.csv= (downloaded via Kaggle CLI).
- Run all 7 configs (roll-aoc, bce-weighted, mae, gce-0.7, libauc-auroc, focal-loss, asymmetric-loss)
- Tabular, ~285K rows, naturally imbalanced (~0.17% positive) — fast to run
- Supports the AUC-ROLL noisy dataset story (real-world messiness, severe imbalance)

* TODO Track 1: controlled noise injection on KEEL datasets
Take 3–5 KEEL datasets with varying IR and inject label noise at 10%, 20%, 30% on minority class.
Goal: reproduce the CIFAR-10N noise-level ablation on imbalanced tabular data.
- Choose datasets spanning a range of IR (e.g. low/medium/high imbalance)
- Write noise injection utility (flip minority labels randomly at given rate)
- Run all 7 configs per dataset × noise level
- Expected result: ROLL-AUC advantage grows with noise level (mirrors CIFAR finding)
- Key for thesis: validates the noise-sensitivity story on tabular domain

* TODO CheXpert experiment
~224K chest X-ray images, label noise from automated NLP extraction of radiology reports.
Severe class imbalance. Clinically relevant domain.
Dataset: https://stanfordmlgroup.github.io/competitions/chexpert/ (requires registration)
- Binary classification: one pathology vs rest (e.g. pleural effusion, highest prevalence)
- Likely needs pretrained ResNet backbone (images are 224×224)
- Medium-large compute cost; run on remote Mac or cluster
- Priority: lower than credit card fraud and Track 1; run after those confirm the story

* TODO NIH ChestX-ray14 experiment
~112K chest X-ray images, label noise from NLP extraction of radiology reports.
Similar story to CheXpert but smaller; could substitute if CheXpert access is delayed.
Dataset: https://nihcc.app.box.com/v/ChestXray-NIHCC
- Same setup considerations as CheXpert
- Priority: same as CheXpert — run one or the other, not necessarily both

* TODO ANIMAL-10N experiment
Dataset loader written: =Animal10NDataset= in =src/datasets.py=.
Data must be manually downloaded from https://nihalsid.github.io/animal-10n/ and extracted to =~/.data/animal10n/= (train/ and test/ subdirs).
Next steps:
- Download the dataset on the remote Mac
- Write =experiments/other/experiment-animal10n.py= (all 7 configs: roll-aoc, bce-weighted, mae, gce-0.7, libauc-auroc, focal-loss, asymmetric-loss)
- ConvNet needs adjusted Linear layer: 64×64 images → after 3 MaxPool → 8×8 spatial → 64*8*8=4096 flatten (vs 1024 for CIFAR)
- positive_class=0 (cat vs rest), IR ~9, same setup as CIFAR-10N

* TODO Food-101N experiment
~310K training images, ~18.4% web noise. Binary: one class vs rest.
Dataset: https://kuanghuei.github.io/Food-101N-Dataset/ (requires Google account for download)
- Write =Food101NDataset= in =src/datasets.py= using torchvision.datasets.ImageFolder
- Images are 224×224; may need ResNet backbone or resize to 64×64 for ConvNet
- Lower priority than ANIMAL-10N (larger, more infra work)

* TODO Clothing1M experiment
~1M images, ~38.5% web noise. Binary: one category vs rest.
Dataset: https://github.com/Cysu/noisy_label (requires request to authors)
- Very large; likely needs ResNet pretrained backbone
- Lowest priority of the noisy-label datasets

* TODO WebVision experiment
2.5M images, ~20% web noise. Very large.
- Likely out of scope unless we get access to a GPU cluster
- Keep as a long-term stretch goal