backup: 2026-10-05 09:48

thesisexperiments_1.org

@@ -5,20 +5,13 @@
 #+title: thesis/experiments
 #+filetags: :project: :knowledge: :experiments:
 
-:PROPERTIES:
-:ID: 59f84838-71a9-4aab-a494-8e6bea2080fb
-:END:
-
-#+title: thesis/experiments
-#+filetags: :project: :knowledge: :experiments:
-
 Chapter 4 of the BGU MSc thesis on ROLL. Parent: [[id:6294e2be-6189-4473-b363-a1dd9a75fb9b][thesis]]
 
 ** Data Sources
 
 - =results-final/keel/= — 12 KEEL datasets × 8 methods × 5 episodes, clean labels
 - =results-final/keel-noise-05/10/20/= — same 12 datasets × 8 methods × 10 episodes, asymmetric noise
-- =results-final/cifar10n/= — 3 noise levels (clean/aggre/worse) × 7 methods × 3 episodes; no roll-tpr90
+- =results-final/cifar10n/= — 3 noise levels × 7 methods × 3 episodes; no roll-tpr90
 - =results-final/bank-marketing/=, =creditcard/=, =homecredit/= — 5 episodes each, all methods
 - Each directory contains per-dataset =auc.csv= files (already aggregated from pkl)
 
@@ -38,16 +31,21 @@ Mean AUC across all 12 KEEL datasets at increasing asymmetric label-noise rates:
 | mae       | 0.875 | 0.878 | 0.866 | 0.804 | −0.071    |
 | roll-tpr  | 0.843 | 0.786 | 0.771 | 0.773 | −0.070    |
 
-*ROLL-AOC is best at 20% noise* despite being mid-pack at 0%. GCE/MAE (designed for noise) degrade most.
+*ROLL-AOC is best at 20% noise* despite being mid-pack at 0%.
 
-*** CIFAR-10N (image noise)
-| Noise  | roll-aoc | gce   | mae   | libauc | bce   | focal | asym  |
-|--------|----------|-------|-------|--------|-------|-------|-------|
-| clean  | 0.980    | 0.976 | 0.976 | 0.979  | 0.977 | 0.977 | 0.972 |
-| aggre  | 0.934    | 0.932 | 0.930 | 0.921  | 0.924 | 0.912 | 0.910 |
-| worse  | 0.821    | 0.774 | 0.809 | 0.752  | 0.738 | 0.734 | 0.720 |
+*** CIFAR-10N (image noise) — updated 2026-09-02
+Results from =results-final/cifar10n/auc.csv= (n=3 episodes, all 7 methods).
+Regenerated via =scripts/regen_cifar10n_summaries.py= after fixing cuda→cpu map_location.
 
-ROLL-AOC best at all three noise levels on CIFAR-10N.
+| Noise | roll-aoc | libauc | gce   | mae   | asym  | bce   | focal |
+|-------|----------|--------|-------|-------|-------|-------|-------|
+| clean | 0.980    | 0.979  | 0.976 | 0.974 | 0.973 | 0.975 | 0.973 |
+| aggre | 0.932    | 0.931  | 0.930 | 0.929 | 0.927 | 0.927 | 0.922 |
+| worse | 0.819    | 0.810  | 0.811 | 0.810 | 0.810 | 0.806 | 0.809 |
+
+ROLL-AOC best at all noise levels. At aggre, leads LibAUC by 0.001 and GCE by 0.002.
+At worse, all baselines cluster 0.806–0.811; ROLL leads GCE by 0.008.
+Degradation (clean→worse): ROLL-AOC −0.161 (least); others −0.163 to −0.169.
 
 *** Large Datasets
 - Credit Card Fraud: roll-tpr90 best (0.974), roll-aoc close (0.968); both beat all baselines
@@ -60,6 +58,7 @@ ROLL-AOC best at all three noise levels on CIFAR-10N.
 - keel-noise runs use 10 episodes; clean keel uses 5. Mixing is fine for averages.
 - libauc-auroc missing 2 episodes for glass2 in clean KEEL (n=58 not 60) — known gap
 - cifar10n has no roll-tpr90 variant; table excludes it
+- =scripts/regen_cifar10n_summaries.py= originally only remapped mps→cpu, not cuda→cpu. Fixed 2026-09-02.
 - Bank Marketing failure cause unknown; do not claim ROLL is universally superior on large tabular data
 
 ** Section Structure
@@ -70,7 +69,7 @@ ROLL-AOC best at all three noise levels on CIFAR-10N.
 | Performance Measures | sec:experiments:performance_measures | Done |
 | Configurations | sec:experiments:configurations | Done |
 | KEEL Results | sec:results:keel | Done — table filled |
-| CIFAR-10N Results | sec:results:cifar10n | Done — table filled |
+| CIFAR-10N Results | sec:results:cifar10n | Done — updated 2026-09-02, all 7 methods |
 | Large Datasets Results | sec:results:large | Done — table filled |
 | Label-Noise Robustness on KEEL | sec:experiments:noise | Done — table added |
 | Ablation Study | sec:experiments:ablation | Stub — future work |