impl/experiments

impl/experiments

impl/experiments

Current Config Set (as of 2026-07-08)

roll — kernelized_roll_tpr(0.95)
roll+bce — kernelized_roll_tpr(0.95, bce_weight=0.5, bce_pos_weight=imbalance_ratio)
bce — plain BCEWithLogitsLoss
bce-weighted — BCE with pos_weight = num_false/num_true

FPR variants (roll-fpr-0.40, roll-fpr-0.40+bce) removed — less consistent, added complexity for marginal gain. Target metric: FPR@TPR=0.95 (lower is better), written to tpr_summary.csv.

CIFAR-10N Experiment (2026-07-19 → 2026-07-20)

Run dir: results/cifar10n/2026-07-19-06-52 Configs: roll-aoc, bce-weighted, mae, gce-0.7, libauc-auroc × 3 episodes × 3 noise types = 45 runs. Noise types: clean (original CIFAR-10 labels), aggre (human aggregated, ~moderate noise), worse (single worst-annotator labels, ~heavy noise). Architecture: ConvNet (3 conv blocks + 2 FC, trained from scratch). Positive class: automobile (class 1) vs rest, IR ~9. LibAUC ran on CPU (MPS unsupported); all others on MPS. PESG lr=1e-3 (not 0.1 — from-scratch training explodes at 0.1).

Results (mean test AUC over 3 episodes):

Config	clean	aggre	worse
roll-aoc	0.9800	0.9336	0.8200
bce-weighted	0.9772	0.9267	0.8044
gce-0.7	0.9776	0.9289	0.7790
libauc-auroc	0.9777	0.9228	0.7690
mae	0.6954	0.7756	0.5979

Key findings:

roll-aoc is the top performer at every noise level, with the gap widening under heavy noise.
Under worse noise, roll-aoc beats bce-weighted by 1.6pp and libauc-auroc by 5.1pp.
gce-0.7 and libauc-auroc degrade more steeply than roll-aoc under increasing noise.
mae is weak throughout; satisfies the symmetry condition theoretically but fails in practice on this task.

Summaries: results/cifar10n/2026-07-19-06-52/auc.csv (45 rows), roc-clean.html, roc-aggre.html, roc-worse.html. Regen script: scripts/regen_cifar10n_summaries.py — loads MPS-saved pkls via default_restore_location patch (maps mps→cpu).

Poisoning Experiment (2026-07-13 → 2026-07-15)

Setup: for each KEEL dataset, train positives are duplicated N times with label flipped to 0 (false). Originals keep their true label. Val/test splits untouched. Code: LabelPoisonedDataset + PoisonedSplitter in src/experiment.py. Runner: experiments/keel/run_poisoned.py <dataset> <n_duplicates> [--no-mp], results go to poison/<dataset>/. Batch runner: experiments/keel/run_all_poisoned.sh [n_duplicates] (default 3). Configs run (poisoned): roll-aoc, bce-weighted, mae, gce-0.7, libauc-auroc. Episodes: 15.

Key findings (2026-07-15, full 12-dataset run):

roll-aoc is statistically significantly better than libauc-auroc on glass1, pima, vehicle2 (one-sided paired t-test and Wilcoxon, p < 0.05).
libauc-auroc is numerically unstable under poisoning on some datasets: produces NaN/Inf predictions on many episodes. glass2: only 2/15 valid episodes; vowel0: only 1/15 valid episode. These are silently filtered in write_auc_csv (src/summary.py:344) — the auc.csv will have far fewer rows than expected for libauc on those datasets.
mae and gce-0.7 perform poorly under poisoning across all datasets.
roll-aoc and libauc-auroc are competitive on easy datasets (glass6, new-thyroid1, vowel0, wisconsin) where both reach ~0.99 AUC.

KEEL Run 2026-07-08 — Setup

Results folders: /home/aner/projects/research-projects/roll/impl/results/*/2026-07-08-*/

Changes from 2026-07-07 run:

Bandwidth: ISJ → Silverman's rule (ISJ crashed on small/degenerate batches mid-training)
Kernel scheduler: KernelScheduler(16.0, decay_every=20) (old: 100.0/500)
LR: roll 1e-1, bce 1e-3 (was both 1e-3)
Weight decay: roll 0.1, bce 1e-3 (was none)
Dropout: 0.1 (was 0.0)
Batch size: 256 (was 128; tried 32 but too small for balanced sampling)
Patience: 100 (was 500)
Episodes: 5 (was 1)
FPR target changed fpr-0.05 → fpr-0.40, then FPR variants dropped entirely

Scoring: mean(fpr) + std(fpr) across 5 episodes (penalises variance and instability; lower is better). Note: mean−std was tried first but incorrectly rewarded instability — corrected to mean+std.

KEEL Run 2026-07-08 — Results (mean + std, FPR@TPR=0.95)

Dataset	roll	roll+bce	bce	bce-weighted	Winner
glass0	0.568	—	0.677	0.706	roll
glass1	0.809	0.776	0.744	0.676	bce-weighted
glass2	0.702	0.642	0.793	0.683	roll+bce
glass6	0.081	0.060	0.042	0.054	bce
haberman	0.910	0.945	1.015	0.940	roll
iris0	0.000	0.000	0.000	0.600	roll/bce/roll+bce (tied)
new-thyroid1	0.077	0.048	0.057	0.055	roll+bce
pima	0.710	0.744	0.627	0.681	bce
vehicle2	0.148	0.097	0.163	0.108	roll+bce
vowel0	0.160	0.095	0.059	0.066	bce
wisconsin	0.011	0.007	0.008	0.011	roll+bce
yeast3	0.398	0.539	0.451	0.552	roll

Win count: roll+bce: 4, bce: 3, roll: 3, bce-weighted: 1

KEEL Run 2026-07-08 — Findings

roll+bce is the most consistent ROLL variant: wins glass2, new-thyroid1, vehicle2, wisconsin.
roll alone wins where it has a large mean advantage that outweighs variance penalty (haberman, glass0, yeast3).
bce wins on glass6, pima, vowel0; bce-weighted wins glass1 but its iris0 score is inflated by one catastrophic episode.
No single method dominates. ROLL adds clear value on haberman, glass0, yeast3 (hard/imbalanced datasets).
High episode variance on haberman, yeast3, glass1 — small test sets make results noisy.

Previous Run — KEEL Results (FPR@TPR=0.95, 2026-07-07, single episode, raw mean)

Dataset	roll	roll+bce	roll-fpr+bce	bce	bce-weighted	Best
glass0	0.878	0.490	0.673	0.857	0.449	bce-weighted
glass1	0.872	0.830	0.702	0.872	0.681	bce-weighted
haberman	0.870	1.0	1.0	0.922	1.0	roll (barely)
pima	0.649	0.714	0.673	0.649	0.595	bce-weighted
vehicle2	0.095	0.032	0.032	0.058	0.058	roll+bce
vowel0	0.141	0.0	0.0	0.0	0.004	tied
wisconsin	0.964	0.0	0.0	0.0	0.0	tied

Config: 1 episode, ISJ bandwidth, no weight decay/dropout, lr=1e-3 for all, gamma=100/decay=500.

Large Datasets

Experiment	Dataset	Status	Notes
experiment-higgs	HiggsDataset	OK	500k samples, ~50/50 balance
experiment-creditcard	CreditCardFraud	MISSING	`/home/aner/.data/creditcard/creditcard.csv`
experiment-homecredit	HomeCreditDataset	MISSING	`/home/aner/.data/homecredit/`

run_all.sh

Runs KEEL datasets via shell script in experiments/keel/. On failure, continues to next dataset.

7.6 KiB Raw Blame History Unescape Escape

impl/experiments