:PROPERTIES:
:ID: 59f84838-71a9-4aab-a494-8e6bae2080fb
:END:

#+title: thesis/experiments
#+filetags: :project: :knowledge: :experiments:

Chapter 4 of the BGU MSc thesis on ROLL. Parent: [[id:6294e2be-6189-4473-b363-a1dd9a75ff9b][thesis]]

** Data Sources

- =results-final/keel/= — 12 KEEL datasets × 8 methods × 5 episodes, clean labels
- =results-final/keel-noise-05/10/20/= — same 12 datasets × 8 methods × 10 episodes, asymmetric noise
- =results-final/cifar10n/= — 3 noise levels (clean/aggre/worse) × 7 methods × 3 episodes; no roll-tpr90
- =results-final/bank-marketing/=, =creditcard/=, =homecredit/= — 5 episodes each, all methods
- =results-final/chestxray14/= — 5 episodes × 7 methods (libauc missing due to bug); image dataset

** ChestXray14 Setup (IMPORTANT)

- Dataset: NIH ChestX-ray14 (~112K frontal X-rays, 30,805 patients)
- Task: Effusion (pleural effusion) vs. rest; IR ~7.5:1, ~12% prevalence
- Label noise: ~20% organic — labels from NLP extraction of radiology reports (NOT injected)
- Feature extraction: DenseNet-121 pretrained on NIH via TorchXRayVision (=densenet121-res224-nih=)
  - Frozen; global avg pool of penultimate feature map → 1024-dim embeddings
  - Script: =scripts/extract_chestxray14_features.py=
- Classifier: 1024→512→128→1 MLP with Dropout(0.3)
- n=5 episodes; LibAUC excluded (PESG optimizer instability on this task)
- Experiment: =experiments/other/experiment-chestxray14.py=

** Key Results

*** ChestXray14 (primary new result — real-world organic noise)
| Method   | AUC   | Std   |
|----------|-------|-------|
| roll-aoc | 0.839 | 0.001 | ← BEST, clear win
| gce-0.7  | 0.829 | 0.004 |
| roll-tpr | 0.828 | 0.002 |
| mae      | 0.810 | 0.001 |
| focal    | 0.798 | 0.009 |
| bce-w    | 0.788 | 0.008 |
| asym.    | 0.729 | 0.007 |
| libauc   | ---   |       |

*** KEEL Noise Experiment (primary thesis result)
Mean AUC across all 12 KEEL datasets at increasing asymmetric label-noise rates:

| Method    | 0%    | 5%    | 10%   | 20%   | Drop 0→20 |
|-----------|-------|-------|-------|-------|-----------|
| roll-aoc  | 0.880 | 0.871 | 0.862 | 0.847 | −0.033    |
| bce-w     | 0.872 | 0.869 | 0.862 | 0.843 | −0.029    |
| focal     | 0.879 | 0.868 | 0.857 | 0.842 | −0.037    |
| asym.     | 0.892 | 0.872 | 0.870 | 0.833 | −0.059    |
| gce-0.7   | 0.886 | 0.870 | 0.865 | 0.831 | −0.056    |
| libauc    | 0.882 | 0.859 | 0.854 | 0.809 | −0.073    |
| mae       | 0.875 | 0.878 | 0.866 | 0.804 | −0.071    |
| roll-tpr  | 0.843 | 0.786 | 0.771 | 0.773 | −0.070    |

*ROLL-AOC is best at 20% noise* despite being mid-pack at 0%. GCE/MAE degrade most.

*** CIFAR-10N (image noise)
| Noise  | roll-aoc | gce   | mae   | libauc | bce   | focal | asym  |
|--------|----------|-------|-------|--------|-------|-------|-------|
| clean  | 0.980    | 0.976 | 0.976 | 0.979  | 0.977 | 0.977 | 0.972 |
| aggre  | 0.934    | 0.932 | 0.930 | 0.921  | 0.924 | 0.912 | 0.910 |
| worse  | 0.821    | 0.774 | 0.809 | 0.752  | 0.738 | 0.734 | 0.720 |

*** Large Datasets
- Credit Card Fraud: roll-tpr90 best (0.974), roll-aoc close (0.968)
- Home Credit: roll-aoc best (0.712), libauc very close (0.711)
- Bank Marketing: ROLL FAILS — roll-aoc 0.511, roll-tpr90 0.438; GCE best (0.746). Open question.

** Gotchas

- "Label-noise experiment" is NOT a poisoning attack — it's asymmetric noise injection (flip pos→neg)
- keel-noise runs use 10 episodes; clean keel uses 5
- libauc-auroc missing 2 episodes for glass2 in clean KEEL (n=58 not 60)
- cifar10n has no roll-tpr90 variant; table excludes it
- chestxray14 libauc missing entirely — PESG optimizer instability; needs rerun
- Bank Marketing failure cause unknown

** Section Structure

| Section | Label | Status |
|---------|-------|--------|
| Datasets (incl. ChestXray14) | sec:experiments:data_sets | Done |
| Performance Measures | sec:experiments:performance_measures | Done |
| Configurations | sec:experiments:configurations | Done |
| KEEL Results | sec:results:keel | Done |
| CIFAR-10N Results | sec:results:cifar10n | Done |
| Large Datasets Results | sec:results:large | Done |
| ChestXray14 Results | sec:results:chestxray14 | Done |
| Label-Noise Robustness on KEEL | sec:experiments:noise | Done |
| Ablation Study | sec:experiments:ablation | Stub |
| Sensitivity Analysis | sec:experiments:sensitivity | Stub |
| Summary | sec:experiments:summary | TODO |