impl/experiments

impl/experiments

impl/experiments

Baseline Config Suite (current)

8 configs run on every dataset. All share StepLR(step_size=50, γ=0.5).

Config	Loss	Optimizer	lr	Notes
roll-aoc	kernelized_roll_aoc()	RMSprop	1e-3	wd=1e-5; KernelScheduler(32,20)
roll-tpr90	kernelized_roll_tpr(tpr=0.9)	RMSprop	1e-3	wd=1e-5; KernelScheduler(32,20)
bce-weighted	BCEWithLogitsLoss	Adam	1e-3	pos_weight=imbalance_ratio
gce-0.7	gce_loss(q=0.7)	Adam	1e-3	wd=1e-5
mae	mae_loss	Adam	1e-3	wd=1e-5
focal-loss	focal_loss(γ=2.0)	Adam	1e-3	wd=1e-5
asymmetric-loss	asymmetric_loss	Adam	1e-3	wd=1e-5; γ_pos=0,γ_neg=4,clip=0.05
libauc-auroc	AUCMLoss (Yuan ICCV 2021)	PESG	0.01	must be paired with PESG; CPU-only on MPS

RMSprop chosen for ROLL (not Adam) because KDE gradients are non-stationary — KernelScheduler anneals bandwidth during training, shifting the loss landscape. Adam's first-moment momentum accumulates stale directional bias from earlier kernel widths; RMSprop adapts step magnitude only, making it more reactive to the current gradient without directional memory.

LibAUC/PESG: AUC is a pairwise (non-decomposable) metric; Yuan et al. reformulate as minimax with auxiliary scalars a≈E[f(x⁺)], b≈E[f(x⁻)], α (dual), enabling per-mini-batch updates. PESG updates all four variable sets jointly; loss module owns a,b,α so optimizer and loss are tightly coupled (hence opt_factory pattern in our code).

KEEL Noise Experiment — Setup & Config Fix

Setup: 12 datasets × 3 noise levels (5%, 10%, 20%) × 10 episodes × 8 configs. Label noise injected via make_noisy_splitter (random label flip on training set only). Results in results-final/keel-noise-{05,10,20}/<dataset>/. Runner: experiments/keel/run_all_noise_levels.sh → run_all_noisy.sh → run_noisy.py. CPU forced (6.6× faster than MPS for KEEL batch sizes); MLflow unset.

Config fairness bug (2026-08-28): roll-aoc was using max_iters=1500 / patience=100 vs baselines max_iters=5000 / patience=500. This is a structural training-budget disadvantage baked into _base.py (roll_config vs robust_config/bce_config). Fix: experiments/keel/run_noisy_fair_roll.py reruns roll-aoc only with fair settings; results swapped into results-final via results-fair-roll/. Same bug applies to noiseless keel (results-final/keel/, 5 episodes) — still needs rerun.

KEEL Noise Experiment — Results (2026-08-28, fair roll-aoc config)

Mean test AUC averaged across all 12 datasets, by noise level:

Config	5%	10%	20%	mean
roll-aoc	0.8713	0.8623	0.8471	0.8602
asymmetric-loss	0.8719	0.8695	0.8330	0.8581
bce-weighted	0.8686	0.8622	0.8434	0.8581
focal-loss	0.8679	0.8573	0.8419	0.8557
gce-0.7	0.8704	0.8650	0.8305	0.8553
mae	0.8776	0.8663	0.8035	0.8491
libauc-auroc	0.8593	0.8544	0.8093	0.8410
roll-tpr90	0.7859	0.7705	0.7728	0.7764

Per-dataset at 20% noise: roll-aoc wins 7/12 datasets. roll-aoc is the most noise-robust: degrades only −0.024 from 5%→20%; mae drops −0.074, libauc-auroc −0.050, asymmetric-loss −0.039.

Key findings for thesis:

roll-aoc ranks #1 overall (mean 0.8602) once given a fair training budget.
Noise robustness is roll-aoc's clearest advantage — the gap widens at 20% noise.
mae is competitive at low noise but not noise-robust; not a strong baseline at 20%.
libauc-auroc consistently underperforms the simpler baselines (bce-weighted, focal).
asymmetric-loss is roll-aoc's closest competitor across all conditions.
roll-tpr90 is a poor AUC-maximiser (expected — it optimises TPR@FPR=0.1, not AUC).

CIFAR-10N Experiment (2026-07-19 → 2026-07-20)

Run dir: results/cifar10n/2026-07-19-06-52 Configs: roll-aoc, bce-weighted, mae, gce-0.7, libauc-auroc × 3 episodes × 3 noise types = 45 runs. Noise types: clean (original CIFAR-10 labels), aggre (human aggregated, ~moderate noise), worse (single worst-annotator labels, ~heavy noise). Architecture: ConvNet (3 conv blocks + 2 FC, trained from scratch). Positive class: automobile (class 1) vs rest, IR ~9. LibAUC ran on CPU (MPS unsupported); all others on MPS. PESG lr=1e-3 (not 0.1 — from-scratch training explodes at 0.1).

Results (mean test AUC over 3 episodes):

Config	clean	aggre	worse
roll-aoc	0.9800	0.9336	0.8200
bce-weighted	0.9772	0.9267	0.8044
gce-0.7	0.9776	0.9289	0.7790
libauc-auroc	0.9777	0.9228	0.7690
mae	0.6954	0.7756	0.5979

Key findings:

roll-aoc is the top performer at every noise level, with the gap widening under heavy noise.
Under worse noise, roll-aoc beats bce-weighted by 1.6pp and libauc-auroc by 5.1pp.
gce-0.7 and libauc-auroc degrade more steeply than roll-aoc under increasing noise.
mae is weak throughout; satisfies the symmetry condition theoretically but fails in practice on this task.

Summaries: results/cifar10n/2026-07-19-06-52/auc.csv (45 rows), roc-clean.html, roc-aggre.html, roc-worse.html. Regen script: scripts/regen_cifar10n_summaries.py — loads MPS-saved pkls via default_restore_location patch (maps mps→cpu).

Poisoning Experiment (2026-07-13 → 2026-07-15)

Setup: for each KEEL dataset, train positives are duplicated N times with label flipped to 0 (false). Originals keep their true label. Val/test splits untouched. Code: LabelPoisonedDataset + PoisonedSplitter in src/experiment.py. Runner: experiments/keel/run_poisoned.py <dataset> <n_duplicates> [--no-mp], results go to poison/<dataset>/. Batch runner: experiments/keel/run_all_poisoned.sh [n_duplicates] (default 3). Configs run (poisoned): roll-aoc, bce-weighted, mae, gce-0.7, libauc-auroc. Episodes: 15.

Key findings (2026-07-15, full 12-dataset run):

roll-aoc is statistically significantly better than libauc-auroc on glass1, pima, vehicle2 (one-sided paired t-test and Wilcoxon, p < 0.05).
libauc-auroc is numerically unstable under poisoning on some datasets: produces NaN/Inf predictions on many episodes. glass2: only 2/15 valid episodes; vowel0: only 1/15 valid episode. These are silently filtered in write_auc_csv (src/summary.py:344) — the auc.csv will have far fewer rows than expected for libauc on those datasets.
mae and gce-0.7 perform poorly under poisoning across all datasets.
roll-aoc and libauc-auroc are competitive on easy datasets (glass6, new-thyroid1, vowel0, wisconsin) where both reach ~0.99 AUC.

Large Datasets

Experiment	Dataset	Status	Notes
experiment-higgs	HiggsDataset	OK	500k samples, ~50/50 balance
experiment-creditcard	CreditCardFraud	MISSING	`/home/aner/.data/creditcard/creditcard.csv`
experiment-homecredit	HomeCreditDataset	MISSING	`/home/aner/.data/homecredit/`

KEEL Run 2026-07-08 — Setup

Results folders: /home/aner/projects/research-projects/roll/impl/results/*/2026-07-08-*/

Changes from 2026-07-07 run:

Bandwidth: ISJ → Silverman's rule (ISJ crashed on small/degenerate batches mid-training)
Kernel scheduler: KernelScheduler(16.0, decay_every=20) (old: 100.0/500)
LR: roll 1e-1, bce 1e-3 (was both 1e-3)
Weight decay: roll 0.1, bce 1e-3 (was none)
Dropout: 0.1 (was 0.0)
Batch size: 256 (was 128; tried 32 but too small for balanced sampling)
Patience: 100 (was 500)
Episodes: 5 (was 1)
FPR target changed fpr-0.05 → fpr-0.40, then FPR variants dropped entirely

Scoring: mean(fpr) + std(fpr) across 5 episodes (penalises variance and instability; lower is better). Note: mean−std was tried first but incorrectly rewarded instability — corrected to mean+std.

run_all.sh

Runs KEEL datasets via shell script in experiments/keel/. On failure, continues to next dataset.

8.7 KiB Raw Blame History Unescape Escape

impl/experiments