roam/thesisexperiments_1.org

thesis/experiments

• Data Sources
• Key Results
  • KEEL Noise Experiment (primary thesis result)
  • CIFAR-10N (image noise) — updated 2026-09-02
  • Large Datasets
• Gotchas
• Section Structure

Chapter 4 of the BGU MSc thesis on ROLL. Parent: thesis

Data Sources

• results-final/keel/ — 12 KEEL datasets × 8 methods × 5 episodes, clean labels
• results-final/keel-noise-05/10/20/ — same 12 datasets × 8 methods × 10 episodes, asymmetric noise
• results-final/cifar10n/ — 3 noise levels × 7 methods × 3 episodes; no roll-tpr90
• results-final/bank-marketing/, creditcard/, homecredit/ — 5 episodes each, all methods
• Each directory contains per-dataset auc.csv files (already aggregated from pkl)

Key Results

KEEL Noise Experiment (primary thesis result)

Mean AUC across all 12 KEEL datasets at increasing asymmetric label-noise rates:

Method	0%	5%	10%	20%	Drop 0→20
roll-aoc	0.880	0.871	0.862	0.847	−0.033
bce-w	0.872	0.869	0.862	0.843	−0.029
focal	0.879	0.868	0.857	0.842	−0.037
asym.	0.892	0.872	0.870	0.833	−0.059
gce-0.7	0.886	0.870	0.865	0.831	−0.056
libauc	0.882	0.859	0.854	0.809	−0.073
mae	0.875	0.878	0.866	0.804	−0.071
roll-tpr	0.843	0.786	0.771	0.773	−0.070

ROLL-AOC is best at 20% noise despite being mid-pack at 0%.

CIFAR-10N (image noise) — updated 2026-09-02

Results from results-final/cifar10n/auc.csv (n=3 episodes, all 7 methods). Regenerated via scripts/regen_cifar10n_summaries.py after fixing cuda→cpu map_location.

Noise	roll-aoc	libauc	gce	mae	asym	bce	focal
clean	0.980	0.979	0.976	0.974	0.973	0.975	0.973
aggre	0.932	0.931	0.930	0.929	0.927	0.927	0.922
worse	0.819	0.810	0.811	0.810	0.810	0.806	0.809

ROLL-AOC best at all noise levels. At aggre, leads LibAUC by 0.001 and GCE by 0.002. At worse, all baselines cluster 0.806–0.811; ROLL leads GCE by 0.008. Degradation (clean→worse): ROLL-AOC −0.161 (least); others −0.163 to −0.169.

Large Datasets

• Credit Card Fraud: roll-tpr90 best (0.974), roll-aoc close (0.968); both beat all baselines
• Home Credit: roll-aoc best (0.712), libauc very close (0.711)
• Bank Marketing: ROLL FAILS — roll-aoc 0.511, roll-tpr90 0.438; GCE best (0.746). Open question.

Gotchas

• "Label-noise experiment" is NOT a poisoning attack — it's asymmetric noise injection (flip pos→neg)
• keel-noise runs use 10 episodes; clean keel uses 5. Mixing is fine for averages.
• libauc-auroc missing 2 episodes for glass2 in clean KEEL (n=58 not 60) — known gap
• cifar10n has no roll-tpr90 variant; table excludes it
• scripts/regen_cifar10n_summaries.py originally only remapped mps→cpu, not cuda→cpu. Fixed 2026-09-02.
• Bank Marketing failure cause unknown; do not claim ROLL is universally superior on large tabular data

Section Structure

Section	Label	Status
Datasets	sec:experiments:data_sets	Done
Performance Measures	sec:experiments:performance_measures	Done
Configurations	sec:experiments:configurations	Done
KEEL Results	sec:results:keel	Done — table filled
CIFAR-10N Results	sec:results:cifar10n	Done — updated 2026-09-02, all 7 methods
Large Datasets Results	sec:results:large	Done — table filled
Label-Noise Robustness on KEEL	sec:experiments:noise	Done — table added
Ablation Study	sec:experiments:ablation	Stub — future work
Sensitivity Analysis	sec:experiments:sensitivity	Stub — future work
Summary	sec:experiments:summary	TODO — key points noted in comments