roam/thesisexperiments_1.org

thesis/experiments thesis/experiments

• Data Sources
• Key Results
  • KEEL Noise Experiment (primary thesis result)
  • CIFAR-10N (image noise)
  • Large Datasets
• Gotchas
• Section Structure

Chapter 4 of the BGU MSc thesis on ROLL. Parent: thesis

Data Sources

• results-final/keel/ — 12 KEEL datasets × 8 methods × 5 episodes, clean labels
• results-final/keel-noise-05/10/20/ — same 12 datasets × 8 methods × 10 episodes, asymmetric noise
• results-final/cifar10n/ — 3 noise levels (clean/aggre/worse) × 7 methods × 3 episodes; no roll-tpr90
• results-final/bank-marketing/, creditcard/, homecredit/ — 5 episodes each, all methods
• Each directory contains per-dataset auc.csv files (already aggregated from pkl)

Key Results

KEEL Noise Experiment (primary thesis result)

Mean AUC across all 12 KEEL datasets at increasing asymmetric label-noise rates:

Method	0%	5%	10%	20%	Drop 0→20
roll-aoc	0.880	0.871	0.862	0.847	−0.033
bce-w	0.872	0.869	0.862	0.843	−0.029
focal	0.879	0.868	0.857	0.842	−0.037
asym.	0.892	0.872	0.870	0.833	−0.059
gce-0.7	0.886	0.870	0.865	0.831	−0.056
libauc	0.882	0.859	0.854	0.809	−0.073
mae	0.875	0.878	0.866	0.804	−0.071
roll-tpr	0.843	0.786	0.771	0.773	−0.070

ROLL-AOC is best at 20% noise despite being mid-pack at 0%. GCE/MAE (designed for noise) degrade most.

CIFAR-10N (image noise)

Noise	roll-aoc	gce	mae	libauc	bce	focal	asym
clean	0.980	0.976	0.976	0.979	0.977	0.977	0.972
aggre	0.934	0.932	0.930	0.921	0.924	0.912	0.910
worse	0.821	0.774	0.809	0.752	0.738	0.734	0.720

ROLL-AOC best at all three noise levels on CIFAR-10N.

Large Datasets

• Credit Card Fraud: roll-tpr90 best (0.974), roll-aoc close (0.968); both beat all baselines
• Home Credit: roll-aoc best (0.712), libauc very close (0.711)
• Bank Marketing: ROLL FAILS — roll-aoc 0.511, roll-tpr90 0.438; GCE best (0.746). Open question.

Gotchas

• "Label-noise experiment" is NOT a poisoning attack — it's asymmetric noise injection (flip pos→neg)
• keel-noise runs use 10 episodes; clean keel uses 5. Mixing is fine for averages.
• libauc-auroc missing 2 episodes for glass2 in clean KEEL (n=58 not 60) — known gap
• cifar10n has no roll-tpr90 variant; table excludes it
• Bank Marketing failure cause unknown; do not claim ROLL is universally superior on large tabular data

Section Structure

Section	Label	Status
Datasets	sec:experiments:data_sets	Done
Performance Measures	sec:experiments:performance_measures	Done
Configurations	sec:experiments:configurations	Done
KEEL Results	sec:results:keel	Done — table filled
CIFAR-10N Results	sec:results:cifar10n	Done — table filled
Large Datasets Results	sec:results:large	Done — table filled
Label-Noise Robustness on KEEL	sec:experiments:noise	Done — table added
Ablation Study	sec:experiments:ablation	Stub — future work
Sensitivity Analysis	sec:experiments:sensitivity	Stub — future work
Summary	sec:experiments:summary	TODO — key points noted in comments