68 lines
3.7 KiB
Org Mode
68 lines
3.7 KiB
Org Mode
:PROPERTIES:
|
||
:ID: fdc18323-e9c2-4fc6-ace5-065035d30c51
|
||
:END:
|
||
|
||
#+title: impl/todos
|
||
#+filetags: :project: :todo:
|
||
|
||
* TODO Look into previous results success/failures — understand failure modes of datasets where ROLL performs poorly (glass1, pima, glass6 from 2026-07-08 KEEL run; see [[id:a53cbe84-cd8d-45c2-a8cf-34ab520a3ea5][impl/experiments]] for full results)
|
||
|
||
* TODO Run credit card fraud experiment
|
||
Experiment file already exists: =experiments/large/experiment-creditcard.py=.
|
||
Data at =~/.data/creditcard/creditcard.csv= (downloaded via Kaggle CLI).
|
||
- Run all 7 configs (roll-aoc, bce-weighted, mae, gce-0.7, libauc-auroc, focal-loss, asymmetric-loss)
|
||
- Tabular, ~285K rows, naturally imbalanced (~0.17% positive) — fast to run
|
||
- Supports the AUC-ROLL noisy dataset story (real-world messiness, severe imbalance)
|
||
|
||
* TODO Track 1: controlled noise injection on KEEL datasets
|
||
Take 3–5 KEEL datasets with varying IR and inject label noise at 10%, 20%, 30% on minority class.
|
||
Goal: reproduce the CIFAR-10N noise-level ablation on imbalanced tabular data.
|
||
- Choose datasets spanning a range of IR (e.g. low/medium/high imbalance)
|
||
- Write noise injection utility (flip minority labels randomly at given rate)
|
||
- Run all 7 configs per dataset × noise level
|
||
- Expected result: ROLL-AUC advantage grows with noise level (mirrors CIFAR finding)
|
||
- Key for thesis: validates the noise-sensitivity story on tabular domain
|
||
|
||
* TODO CheXpert experiment
|
||
~224K chest X-ray images, label noise from automated NLP extraction of radiology reports.
|
||
Severe class imbalance. Clinically relevant domain.
|
||
Dataset: https://stanfordmlgroup.github.io/competitions/chexpert/ (requires registration)
|
||
- Binary classification: one pathology vs rest (e.g. pleural effusion, highest prevalence)
|
||
- Likely needs pretrained ResNet backbone (images are 224×224)
|
||
- Medium-large compute cost; run on remote Mac or cluster
|
||
- Priority: lower than credit card fraud and Track 1; run after those confirm the story
|
||
|
||
* TODO NIH ChestX-ray14 experiment
|
||
~112K chest X-ray images, label noise from NLP extraction of radiology reports.
|
||
Similar story to CheXpert but smaller; could substitute if CheXpert access is delayed.
|
||
Dataset: https://nihcc.app.box.com/v/ChestXray-NIHCC
|
||
- Same setup considerations as CheXpert
|
||
- Priority: same as CheXpert — run one or the other, not necessarily both
|
||
|
||
* TODO ANIMAL-10N experiment
|
||
Dataset loader written: =Animal10NDataset= in =src/datasets.py=.
|
||
Data must be manually downloaded from https://nihalsid.github.io/animal-10n/ and extracted to =~/.data/animal10n/= (train/ and test/ subdirs).
|
||
Next steps:
|
||
- Download the dataset on the remote Mac
|
||
- Write =experiments/other/experiment-animal10n.py= (all 7 configs: roll-aoc, bce-weighted, mae, gce-0.7, libauc-auroc, focal-loss, asymmetric-loss)
|
||
- ConvNet needs adjusted Linear layer: 64×64 images → after 3 MaxPool → 8×8 spatial → 64*8*8=4096 flatten (vs 1024 for CIFAR)
|
||
- positive_class=0 (cat vs rest), IR ~9, same setup as CIFAR-10N
|
||
|
||
* TODO Food-101N experiment
|
||
~310K training images, ~18.4% web noise. Binary: one class vs rest.
|
||
Dataset: https://kuanghuei.github.io/Food-101N-Dataset/ (requires Google account for download)
|
||
- Write =Food101NDataset= in =src/datasets.py= using torchvision.datasets.ImageFolder
|
||
- Images are 224×224; may need ResNet backbone or resize to 64×64 for ConvNet
|
||
- Lower priority than ANIMAL-10N (larger, more infra work)
|
||
|
||
* TODO Clothing1M experiment
|
||
~1M images, ~38.5% web noise. Binary: one category vs rest.
|
||
Dataset: https://github.com/Cysu/noisy_label (requires request to authors)
|
||
- Very large; likely needs ResNet pretrained backbone
|
||
- Lowest priority of the noisy-label datasets
|
||
|
||
* TODO WebVision experiment
|
||
2.5M images, ~20% web noise. Very large.
|
||
- Likely out of scope unless we get access to a GPU cluster
|
||
- Keep as a long-term stretch goal
|