backup: 2026-08-29 06:59

2026-08-29 06:59:35 +03:00
parent 07112f1a4c
commit 2a872c725b
19 changed files with 823 additions and 136 deletions
@@ -0,0 +1,78 @@
+:PROPERTIES:
+:ID: c41756b5-fd5f-456e-ab2b-18f27db7d61d
+:END:
+
+#+title: thesis/experiments
+#+filetags: :project: :knowledge: :experiments:
+
+:PROPERTIES:
+:ID: 59f84838-71a9-4aab-a494-8e6bea2080fb
+:END:
+
+#+title: thesis/experiments
+#+filetags: :project: :knowledge: :experiments:
+
+Chapter 4 of the BGU MSc thesis on ROLL. Parent: [[id:6294e2be-6189-4473-b363-a1dd9a75fb9b][thesis]]
+
+** Data Sources
+
+- =results-final/keel/= — 12 KEEL datasets × 8 methods × 5 episodes, clean labels
+- =results-final/keel-noise-05/10/20/= — same 12 datasets × 8 methods × 10 episodes, asymmetric noise
+- =results-final/cifar10n/= — 3 noise levels (clean/aggre/worse) × 7 methods × 3 episodes; no roll-tpr90
+- =results-final/bank-marketing/=, =creditcard/=, =homecredit/= — 5 episodes each, all methods
+- Each directory contains per-dataset =auc.csv= files (already aggregated from pkl)
+
+** Key Results
+
+*** KEEL Noise Experiment (primary thesis result)
+Mean AUC across all 12 KEEL datasets at increasing asymmetric label-noise rates:
+
+| Method    | 0%    | 5%    | 10%   | 20%   | Drop 0→20 |
+|-----------|-------|-------|-------|-------|-----------|
+| roll-aoc  | 0.880 | 0.871 | 0.862 | 0.847 | −0.033    |
+| bce-w     | 0.872 | 0.869 | 0.862 | 0.843 | −0.029    |
+| focal     | 0.879 | 0.868 | 0.857 | 0.842 | −0.037    |
+| asym.     | 0.892 | 0.872 | 0.870 | 0.833 | −0.059    |
+| gce-0.7   | 0.886 | 0.870 | 0.865 | 0.831 | −0.056    |
+| libauc    | 0.882 | 0.859 | 0.854 | 0.809 | −0.073    |
+| mae       | 0.875 | 0.878 | 0.866 | 0.804 | −0.071    |
+| roll-tpr  | 0.843 | 0.786 | 0.771 | 0.773 | −0.070    |
+
+*ROLL-AOC is best at 20% noise* despite being mid-pack at 0%. GCE/MAE (designed for noise) degrade most.
+
+*** CIFAR-10N (image noise)
+| Noise  | roll-aoc | gce   | mae   | libauc | bce   | focal | asym  |
+|--------|----------|-------|-------|--------|-------|-------|-------|
+| clean  | 0.980    | 0.976 | 0.976 | 0.979  | 0.977 | 0.977 | 0.972 |
+| aggre  | 0.934    | 0.932 | 0.930 | 0.921  | 0.924 | 0.912 | 0.910 |
+| worse  | 0.821    | 0.774 | 0.809 | 0.752  | 0.738 | 0.734 | 0.720 |
+
+ROLL-AOC best at all three noise levels on CIFAR-10N.
+
+*** Large Datasets
+- Credit Card Fraud: roll-tpr90 best (0.974), roll-aoc close (0.968); both beat all baselines
+- Home Credit: roll-aoc best (0.712), libauc very close (0.711)
+- Bank Marketing: ROLL FAILS — roll-aoc 0.511, roll-tpr90 0.438; GCE best (0.746). Open question.
+
+** Gotchas
+
+- "Label-noise experiment" is NOT a poisoning attack — it's asymmetric noise injection (flip pos→neg)
+- keel-noise runs use 10 episodes; clean keel uses 5. Mixing is fine for averages.
+- libauc-auroc missing 2 episodes for glass2 in clean KEEL (n=58 not 60) — known gap
+- cifar10n has no roll-tpr90 variant; table excludes it
+- Bank Marketing failure cause unknown; do not claim ROLL is universally superior on large tabular data
+
+** Section Structure
+
+| Section | Label | Status |
+|---------|-------|--------|
+| Datasets | sec:experiments:data_sets | Done |
+| Performance Measures | sec:experiments:performance_measures | Done |
+| Configurations | sec:experiments:configurations | Done |
+| KEEL Results | sec:results:keel | Done — table filled |
+| CIFAR-10N Results | sec:results:cifar10n | Done — table filled |
+| Large Datasets Results | sec:results:large | Done — table filled |
+| Label-Noise Robustness on KEEL | sec:experiments:noise | Done — table added |
+| Ablation Study | sec:experiments:ablation | Stub — future work |
+| Sensitivity Analysis | sec:experiments:sensitivity | Stub — future work |
+| Summary | sec:experiments:summary | TODO — key points noted in comments |