roam/thesisexperiments.org

thesis/experiments

• Data Sources
• ChestXray14 Setup (IMPORTANT)
• Key Results
  • ChestXray14 (primary new result — real-world organic noise)
  • KEEL Noise Experiment (primary thesis result)
  • CIFAR-10N (image noise)
  • Large Datasets
• Gotchas
• Section Structure

Chapter 4 of the BGU MSc thesis on ROLL. Parent: thesis

Data Sources

• results-final/keel/ — 12 KEEL datasets × 8 methods × 5 episodes, clean labels
• results-final/keel-noise-05/10/20/ — same 12 datasets × 8 methods × 10 episodes, asymmetric noise
• results-final/cifar10n/ — 3 noise levels (clean/aggre/worse) × 7 methods × 3 episodes; no roll-tpr90
• results-final/bank-marketing/, creditcard/, homecredit/ — 5 episodes each, all methods
• results-final/chestxray14/ — 5 episodes × 7 methods (libauc missing due to bug); image dataset

ChestXray14 Setup (IMPORTANT)

• Dataset: NIH ChestX-ray14 (~112K frontal X-rays, 30,805 patients)
• Task: Effusion (pleural effusion) vs. rest; IR ~7.5:1, ~12% prevalence
• Label noise: ~20% organic — labels from NLP extraction of radiology reports (NOT injected)

• Feature extraction: DenseNet-121 pretrained on NIH via TorchXRayVision (densenet121-res224-nih)

  • Frozen; global avg pool of penultimate feature map → 1024-dim embeddings
  • Script: scripts/extract_chestxray14_features.py
• Classifier: 1024→512→128→1 MLP with Dropout(0.3)
• n=5 episodes; LibAUC excluded (PESG optimizer instability on this task)
• Experiment: experiments/other/experiment-chestxray14.py

Key Results

ChestXray14 (primary new result — real-world organic noise)

Method	AUC	Std
roll-aoc	0.839	0.001	← BEST, clear win
gce-0.7	0.829	0.004
roll-tpr	0.828	0.002
mae	0.810	0.001
focal	0.798	0.009
bce-w	0.788	0.008
asym.	0.729	0.007
libauc	—

KEEL Noise Experiment (primary thesis result)

Mean AUC across all 12 KEEL datasets at increasing asymmetric label-noise rates:

Method	0%	5%	10%	20%	Drop 0→20
roll-aoc	0.880	0.871	0.862	0.847	−0.033
bce-w	0.872	0.869	0.862	0.843	−0.029
focal	0.879	0.868	0.857	0.842	−0.037
asym.	0.892	0.872	0.870	0.833	−0.059
gce-0.7	0.886	0.870	0.865	0.831	−0.056
libauc	0.882	0.859	0.854	0.809	−0.073
mae	0.875	0.878	0.866	0.804	−0.071
roll-tpr	0.843	0.786	0.771	0.773	−0.070

ROLL-AOC is best at 20% noise despite being mid-pack at 0%. GCE/MAE degrade most.

CIFAR-10N (image noise)

Noise	roll-aoc	gce	mae	libauc	bce	focal	asym
clean	0.980	0.976	0.976	0.979	0.977	0.977	0.972
aggre	0.934	0.932	0.930	0.921	0.924	0.912	0.910
worse	0.821	0.774	0.809	0.752	0.738	0.734	0.720

Large Datasets

• Credit Card Fraud: roll-tpr90 best (0.974), roll-aoc close (0.968)
• Home Credit: roll-aoc best (0.712), libauc very close (0.711)
• Bank Marketing: ROLL FAILS — roll-aoc 0.511, roll-tpr90 0.438; GCE best (0.746). Open question.

Gotchas

• "Label-noise experiment" is NOT a poisoning attack — it's asymmetric noise injection (flip pos→neg)
• keel-noise runs use 10 episodes; clean keel uses 5
• libauc-auroc missing 2 episodes for glass2 in clean KEEL (n=58 not 60)
• cifar10n has no roll-tpr90 variant; table excludes it
• chestxray14 libauc missing entirely — PESG optimizer instability; needs rerun
• Bank Marketing failure cause unknown

Section Structure

Section	Label	Status
Datasets (incl. ChestXray14)	sec:experiments:data_sets	Done
Performance Measures	sec:experiments:performance_measures	Done
Configurations	sec:experiments:configurations	Done
KEEL Results	sec:results:keel	Done
CIFAR-10N Results	sec:results:cifar10n	Done
Large Datasets Results	sec:results:large	Done
ChestXray14 Results	sec:results:chestxray14	Done
Label-Noise Robustness on KEEL	sec:experiments:noise	Done
Ablation Study	sec:experiments:ablation	Stub
Sensitivity Analysis	sec:experiments:sensitivity	Stub
Summary	sec:experiments:summary	TODO