78 lines
3.9 KiB
Org Mode
78 lines
3.9 KiB
Org Mode
:PROPERTIES:
|
||
:ID: c41756b5-fd5f-456e-ab2b-18f27db7d61d
|
||
:END:
|
||
|
||
#+title: thesis/experiments
|
||
#+filetags: :project: :knowledge: :experiments:
|
||
|
||
Chapter 4 of the BGU MSc thesis on ROLL. Parent: [[id:6294e2be-6189-4473-b363-a1dd9a75fb9b][thesis]]
|
||
|
||
** Data Sources
|
||
|
||
- =results-final/keel/= — 12 KEEL datasets × 8 methods × 5 episodes, clean labels
|
||
- =results-final/keel-noise-05/10/20/= — same 12 datasets × 8 methods × 10 episodes, asymmetric noise
|
||
- =results-final/cifar10n/= — 3 noise levels × 7 methods × 3 episodes; no roll-tpr90
|
||
- =results-final/bank-marketing/=, =creditcard/=, =homecredit/= — 5 episodes each, all methods
|
||
- Each directory contains per-dataset =auc.csv= files (already aggregated from pkl)
|
||
|
||
** Key Results
|
||
|
||
*** KEEL Noise Experiment (primary thesis result)
|
||
Mean AUC across all 12 KEEL datasets at increasing asymmetric label-noise rates:
|
||
|
||
| Method | 0% | 5% | 10% | 20% | Drop 0→20 |
|
||
|-----------|-------|-------|-------|-------|-----------|
|
||
| roll-aoc | 0.880 | 0.871 | 0.862 | 0.847 | −0.033 |
|
||
| bce-w | 0.872 | 0.869 | 0.862 | 0.843 | −0.029 |
|
||
| focal | 0.879 | 0.868 | 0.857 | 0.842 | −0.037 |
|
||
| asym. | 0.892 | 0.872 | 0.870 | 0.833 | −0.059 |
|
||
| gce-0.7 | 0.886 | 0.870 | 0.865 | 0.831 | −0.056 |
|
||
| libauc | 0.882 | 0.859 | 0.854 | 0.809 | −0.073 |
|
||
| mae | 0.875 | 0.878 | 0.866 | 0.804 | −0.071 |
|
||
| roll-tpr | 0.843 | 0.786 | 0.771 | 0.773 | −0.070 |
|
||
|
||
*ROLL-AOC is best at 20% noise* despite being mid-pack at 0%.
|
||
|
||
*** CIFAR-10N (image noise) — updated 2026-09-02
|
||
Results from =results-final/cifar10n/auc.csv= (n=3 episodes, all 7 methods).
|
||
Regenerated via =scripts/regen_cifar10n_summaries.py= after fixing cuda→cpu map_location.
|
||
|
||
| Noise | roll-aoc | libauc | gce | mae | asym | bce | focal |
|
||
|-------|----------|--------|-------|-------|-------|-------|-------|
|
||
| clean | 0.980 | 0.979 | 0.976 | 0.974 | 0.973 | 0.975 | 0.973 |
|
||
| aggre | 0.932 | 0.931 | 0.930 | 0.929 | 0.927 | 0.927 | 0.922 |
|
||
| worse | 0.819 | 0.810 | 0.811 | 0.810 | 0.810 | 0.806 | 0.809 |
|
||
|
||
ROLL-AOC best at all noise levels. At aggre, leads LibAUC by 0.001 and GCE by 0.002.
|
||
At worse, all baselines cluster 0.806–0.811; ROLL leads GCE by 0.008.
|
||
Degradation (clean→worse): ROLL-AOC −0.161 (least); others −0.163 to −0.169.
|
||
|
||
*** Large Datasets
|
||
- Credit Card Fraud: roll-tpr90 best (0.974), roll-aoc close (0.968); both beat all baselines
|
||
- Home Credit: roll-aoc best (0.712), libauc very close (0.711)
|
||
- Bank Marketing: ROLL FAILS — roll-aoc 0.511, roll-tpr90 0.438; GCE best (0.746). Open question.
|
||
|
||
** Gotchas
|
||
|
||
- "Label-noise experiment" is NOT a poisoning attack — it's asymmetric noise injection (flip pos→neg)
|
||
- keel-noise runs use 10 episodes; clean keel uses 5. Mixing is fine for averages.
|
||
- libauc-auroc missing 2 episodes for glass2 in clean KEEL (n=58 not 60) — known gap
|
||
- cifar10n has no roll-tpr90 variant; table excludes it
|
||
- =scripts/regen_cifar10n_summaries.py= originally only remapped mps→cpu, not cuda→cpu. Fixed 2026-09-02.
|
||
- Bank Marketing failure cause unknown; do not claim ROLL is universally superior on large tabular data
|
||
|
||
** Section Structure
|
||
|
||
| Section | Label | Status |
|
||
|---------|-------|--------|
|
||
| Datasets | sec:experiments:data_sets | Done |
|
||
| Performance Measures | sec:experiments:performance_measures | Done |
|
||
| Configurations | sec:experiments:configurations | Done |
|
||
| KEEL Results | sec:results:keel | Done — table filled |
|
||
| CIFAR-10N Results | sec:results:cifar10n | Done — updated 2026-09-02, all 7 methods |
|
||
| Large Datasets Results | sec:results:large | Done — table filled |
|
||
| Label-Noise Robustness on KEEL | sec:experiments:noise | Done — table added |
|
||
| Ablation Study | sec:experiments:ablation | Stub — future work |
|
||
| Sensitivity Analysis | sec:experiments:sensitivity | Stub — future work |
|
||
| Summary | sec:experiments:summary | TODO — key points noted in comments |
|