:PROPERTIES:
:ID: a53cbe84-cd8d-45c2-a8cf-34ab520a3ea5
:END:

#+title: impl/experiments
#+filetags: :project: :knowledge: :experiments:

* impl/experiments

** Baseline Config Suite (current)

8 configs run on every dataset. All share StepLR(step_size=50, γ=0.5).

| Config            | Loss                        | Optimizer | lr    | Notes                          |
|-------------------+-----------------------------+-----------+-------+--------------------------------|
| roll-aoc          | kernelized_roll_aoc()       | RMSprop   | 1e-3  | wd=1e-5; KernelScheduler(32,20)|
| roll-tpr90        | kernelized_roll_tpr(tpr=0.9)| RMSprop   | 1e-3  | wd=1e-5; KernelScheduler(32,20)|
| bce-weighted      | BCEWithLogitsLoss           | Adam      | 1e-3  | pos_weight=imbalance_ratio     |
| gce-0.7           | gce_loss(q=0.7)             | Adam      | 1e-3  | wd=1e-5                        |
| mae               | mae_loss                    | Adam      | 1e-3  | wd=1e-5                        |
| focal-loss        | focal_loss(γ=2.0)           | Adam      | 1e-3  | wd=1e-5                        |
| asymmetric-loss   | asymmetric_loss             | Adam      | 1e-3  | wd=1e-5; γ_pos=0,γ_neg=4,clip=0.05 |
| libauc-auroc      | AUCMLoss (Yuan ICCV 2021)   | PESG      | 0.01  | must be paired with PESG; CPU-only on MPS |

RMSprop chosen for ROLL (not Adam) because KDE gradients are non-stationary — KernelScheduler
anneals bandwidth during training, shifting the loss landscape. Adam's first-moment momentum
accumulates stale directional bias from earlier kernel widths; RMSprop adapts step magnitude
only, making it more reactive to the current gradient without directional memory.

LibAUC/PESG: AUC is a pairwise (non-decomposable) metric; Yuan et al. reformulate as minimax
with auxiliary scalars a≈E[f(x⁺)], b≈E[f(x⁻)], α (dual), enabling per-mini-batch updates.
PESG updates all four variable sets jointly; loss module owns a,b,α so optimizer and loss
are tightly coupled (hence opt_factory pattern in our code).

** KEEL Noise Experiment — Setup & Config Fix

*Setup:* 12 datasets × 3 noise levels (5%, 10%, 20%) × 10 episodes × 8 configs.
Label noise injected via =make_noisy_splitter= (random label flip on training set only).
Results in =results-final/keel-noise-{05,10,20}/<dataset>/=.
Runner: =experiments/keel/run_all_noise_levels.sh= → =run_all_noisy.sh= → =run_noisy.py=.
CPU forced (6.6× faster than MPS for KEEL batch sizes); MLflow unset.

*Config fairness bug (2026-08-28):* roll-aoc was using =max_iters=1500= / =patience=100=
vs baselines =max_iters=5000= / =patience=500=. This is a structural training-budget
disadvantage baked into =_base.py= (roll_config vs robust_config/bce_config).
Fix: =experiments/keel/run_noisy_fair_roll.py= reruns roll-aoc only with fair settings;
results swapped into =results-final= via =results-fair-roll/=.
*Same bug applies to noiseless keel (results-final/keel/, 5 episodes) — still needs rerun.*

** KEEL Noise Experiment — Results (2026-08-28, fair roll-aoc config)

Mean test AUC averaged across all 12 datasets, by noise level:

| Config          |   5%   |  10%   |  20%   |  mean  |
|-----------------+--------+--------+--------+--------|
| *roll-aoc*      | 0.8713 | 0.8623 | *0.8471* | *0.8602* |
| asymmetric-loss | 0.8719 | 0.8695 | 0.8330 | 0.8581 |
| bce-weighted    | 0.8686 | 0.8622 | 0.8434 | 0.8581 |
| focal-loss      | 0.8679 | 0.8573 | 0.8419 | 0.8557 |
| gce-0.7         | 0.8704 | 0.8650 | 0.8305 | 0.8553 |
| mae             | 0.8776 | 0.8663 | 0.8035 | 0.8491 |
| libauc-auroc    | 0.8593 | 0.8544 | 0.8093 | 0.8410 |
| roll-tpr90      | 0.7859 | 0.7705 | 0.7728 | 0.7764 |

Per-dataset at 20% noise: roll-aoc wins 7/12 datasets.
roll-aoc is the most noise-robust: degrades only −0.024 from 5%→20%;
mae drops −0.074, libauc-auroc −0.050, asymmetric-loss −0.039.

*Key findings for thesis:*
- roll-aoc ranks #1 overall (mean 0.8602) once given a fair training budget.
- Noise robustness is roll-aoc's clearest advantage — the gap widens at 20% noise.
- mae is competitive at low noise but not noise-robust; not a strong baseline at 20%.
- libauc-auroc consistently underperforms the simpler baselines (bce-weighted, focal).
- asymmetric-loss is roll-aoc's closest competitor across all conditions.
- roll-tpr90 is a poor AUC-maximiser (expected — it optimises TPR@FPR=0.1, not AUC).

** CIFAR-10N Experiment (2026-07-19 → 2026-07-20)

Run dir: =results/cifar10n/2026-07-19-06-52=
Configs: =roll-aoc=, =bce-weighted=, =mae=, =gce-0.7=, =libauc-auroc= × 3 episodes × 3 noise types = 45 runs.
Noise types: =clean= (original CIFAR-10 labels), =aggre= (human aggregated, ~moderate noise), =worse= (single worst-annotator labels, ~heavy noise).
Architecture: ConvNet (3 conv blocks + 2 FC, trained from scratch). Positive class: automobile (class 1) vs rest, IR ~9.
LibAUC ran on CPU (MPS unsupported); all others on MPS. PESG lr=1e-3 (not 0.1 — from-scratch training explodes at 0.1).

*Results (mean test AUC over 3 episodes):*

| Config       | clean  | aggre  | worse  |
|--------------+--------+--------+--------|
| roll-aoc     | 0.9800 | 0.9336 | 0.8200 |
| bce-weighted | 0.9772 | 0.9267 | 0.8044 |
| gce-0.7      | 0.9776 | 0.9289 | 0.7790 |
| libauc-auroc | 0.9777 | 0.9228 | 0.7690 |
| mae          | 0.6954 | 0.7756 | 0.5979 |

*Key findings:*
- =roll-aoc= is the top performer at every noise level, with the gap widening under heavy noise.
- Under =worse= noise, =roll-aoc= beats =bce-weighted= by 1.6pp and =libauc-auroc= by 5.1pp.
- =gce-0.7= and =libauc-auroc= degrade more steeply than =roll-aoc= under increasing noise.
- =mae= is weak throughout; satisfies the symmetry condition theoretically but fails in practice on this task.

Summaries: =results/cifar10n/2026-07-19-06-52/auc.csv= (45 rows), =roc-clean.html=, =roc-aggre.html=, =roc-worse.html=.
Regen script: =scripts/regen_cifar10n_summaries.py= — loads MPS-saved pkls via =default_restore_location= patch (maps mps→cpu).

** Poisoning Experiment (2026-07-13 → 2026-07-15)

Setup: for each KEEL dataset, train positives are duplicated N times with label flipped to 0 (false).
Originals keep their true label. Val/test splits untouched.
Code: =LabelPoisonedDataset= + =PoisonedSplitter= in =src/experiment.py=.
Runner: =experiments/keel/run_poisoned.py <dataset> <n_duplicates> [--no-mp]=, results go to =poison/<dataset>/=.
Batch runner: =experiments/keel/run_all_poisoned.sh [n_duplicates]= (default 3).
Configs run (poisoned): =roll-aoc=, =bce-weighted=, =mae=, =gce-0.7=, =libauc-auroc=.
Episodes: 15.

*Key findings (2026-07-15, full 12-dataset run):*
- =roll-aoc= is statistically significantly better than =libauc-auroc= on glass1, pima, vehicle2 (one-sided paired t-test and Wilcoxon, p < 0.05).
- =libauc-auroc= is numerically unstable under poisoning on some datasets: produces NaN/Inf predictions on many episodes. glass2: only 2/15 valid episodes; vowel0: only 1/15 valid episode. These are silently filtered in =write_auc_csv= (=src/summary.py:344=) — the auc.csv will have far fewer rows than expected for libauc on those datasets.
- =mae= and =gce-0.7= perform poorly under poisoning across all datasets.
- =roll-aoc= and =libauc-auroc= are competitive on easy datasets (glass6, new-thyroid1, vowel0, wisconsin) where both reach ~0.99 AUC.

** Large Datasets

| Experiment            | Dataset           | Status  | Notes                                        |
|-----------------------+-------------------+---------+----------------------------------------------|
| experiment-higgs      | HiggsDataset      | OK      | 500k samples, ~50/50 balance                 |
| experiment-creditcard | CreditCardFraud   | MISSING | =/home/aner/.data/creditcard/creditcard.csv= |
| experiment-homecredit | HomeCreditDataset | MISSING | =/home/aner/.data/homecredit/=               |

** KEEL Run 2026-07-08 — Setup

Results folders: =/home/aner/projects/research-projects/roll/impl/results/*/2026-07-08-*/=

Changes from 2026-07-07 run:
- Bandwidth: ISJ → *Silverman's rule* (ISJ crashed on small/degenerate batches mid-training)
- Kernel scheduler: =KernelScheduler(16.0, decay_every=20)= (old: 100.0/500)
- LR: roll =1e-1=, bce =1e-3= (was both =1e-3=)
- Weight decay: roll =0.1=, bce =1e-3= (was none)
- Dropout: =0.1= (was =0.0=)
- Batch size: =256= (was 128; tried 32 but too small for balanced sampling)
- Patience: =100= (was =500=)
- Episodes: =5= (was =1=)
- FPR target changed =fpr-0.05= → =fpr-0.40=, then FPR variants dropped entirely

Scoring: =mean(fpr) + std(fpr)= across 5 episodes (penalises variance and instability; lower is better).
Note: mean−std was tried first but incorrectly rewarded instability — corrected to mean+std.

** run_all.sh

Runs KEEL datasets via shell script in =experiments/keel/=.
On failure, continues to next dataset.