150 lines
8.7 KiB
Org Mode
150 lines
8.7 KiB
Org Mode
:PROPERTIES:
|
||
:ID: a53cbe84-cd8d-45c2-a8cf-34ab520a3ea5
|
||
:END:
|
||
|
||
#+title: impl/experiments
|
||
#+filetags: :project: :knowledge: :experiments:
|
||
|
||
* impl/experiments
|
||
|
||
** Baseline Config Suite (current)
|
||
|
||
8 configs run on every dataset. All share StepLR(step_size=50, γ=0.5).
|
||
|
||
| Config | Loss | Optimizer | lr | Notes |
|
||
|-------------------+-----------------------------+-----------+-------+--------------------------------|
|
||
| roll-aoc | kernelized_roll_aoc() | RMSprop | 1e-3 | wd=1e-5; KernelScheduler(32,20)|
|
||
| roll-tpr90 | kernelized_roll_tpr(tpr=0.9)| RMSprop | 1e-3 | wd=1e-5; KernelScheduler(32,20)|
|
||
| bce-weighted | BCEWithLogitsLoss | Adam | 1e-3 | pos_weight=imbalance_ratio |
|
||
| gce-0.7 | gce_loss(q=0.7) | Adam | 1e-3 | wd=1e-5 |
|
||
| mae | mae_loss | Adam | 1e-3 | wd=1e-5 |
|
||
| focal-loss | focal_loss(γ=2.0) | Adam | 1e-3 | wd=1e-5 |
|
||
| asymmetric-loss | asymmetric_loss | Adam | 1e-3 | wd=1e-5; γ_pos=0,γ_neg=4,clip=0.05 |
|
||
| libauc-auroc | AUCMLoss (Yuan ICCV 2021) | PESG | 0.01 | must be paired with PESG; CPU-only on MPS |
|
||
|
||
RMSprop chosen for ROLL (not Adam) because KDE gradients are non-stationary — KernelScheduler
|
||
anneals bandwidth during training, shifting the loss landscape. Adam's first-moment momentum
|
||
accumulates stale directional bias from earlier kernel widths; RMSprop adapts step magnitude
|
||
only, making it more reactive to the current gradient without directional memory.
|
||
|
||
LibAUC/PESG: AUC is a pairwise (non-decomposable) metric; Yuan et al. reformulate as minimax
|
||
with auxiliary scalars a≈E[f(x⁺)], b≈E[f(x⁻)], α (dual), enabling per-mini-batch updates.
|
||
PESG updates all four variable sets jointly; loss module owns a,b,α so optimizer and loss
|
||
are tightly coupled (hence opt_factory pattern in our code).
|
||
|
||
** KEEL Noise Experiment — Setup & Config Fix
|
||
|
||
*Setup:* 12 datasets × 3 noise levels (5%, 10%, 20%) × 10 episodes × 8 configs.
|
||
Label noise injected via =make_noisy_splitter= (random label flip on training set only).
|
||
Results in =results-final/keel-noise-{05,10,20}/<dataset>/=.
|
||
Runner: =experiments/keel/run_all_noise_levels.sh= → =run_all_noisy.sh= → =run_noisy.py=.
|
||
CPU forced (6.6× faster than MPS for KEEL batch sizes); MLflow unset.
|
||
|
||
*Config fairness bug (2026-08-28):* roll-aoc was using =max_iters=1500= / =patience=100=
|
||
vs baselines =max_iters=5000= / =patience=500=. This is a structural training-budget
|
||
disadvantage baked into =_base.py= (roll_config vs robust_config/bce_config).
|
||
Fix: =experiments/keel/run_noisy_fair_roll.py= reruns roll-aoc only with fair settings;
|
||
results swapped into =results-final= via =results-fair-roll/=.
|
||
*Same bug applies to noiseless keel (results-final/keel/, 5 episodes) — still needs rerun.*
|
||
|
||
** KEEL Noise Experiment — Results (2026-08-28, fair roll-aoc config)
|
||
|
||
Mean test AUC averaged across all 12 datasets, by noise level:
|
||
|
||
| Config | 5% | 10% | 20% | mean |
|
||
|-----------------+--------+--------+--------+--------|
|
||
| *roll-aoc* | 0.8713 | 0.8623 | *0.8471* | *0.8602* |
|
||
| asymmetric-loss | 0.8719 | 0.8695 | 0.8330 | 0.8581 |
|
||
| bce-weighted | 0.8686 | 0.8622 | 0.8434 | 0.8581 |
|
||
| focal-loss | 0.8679 | 0.8573 | 0.8419 | 0.8557 |
|
||
| gce-0.7 | 0.8704 | 0.8650 | 0.8305 | 0.8553 |
|
||
| mae | 0.8776 | 0.8663 | 0.8035 | 0.8491 |
|
||
| libauc-auroc | 0.8593 | 0.8544 | 0.8093 | 0.8410 |
|
||
| roll-tpr90 | 0.7859 | 0.7705 | 0.7728 | 0.7764 |
|
||
|
||
Per-dataset at 20% noise: roll-aoc wins 7/12 datasets.
|
||
roll-aoc is the most noise-robust: degrades only −0.024 from 5%→20%;
|
||
mae drops −0.074, libauc-auroc −0.050, asymmetric-loss −0.039.
|
||
|
||
*Key findings for thesis:*
|
||
- roll-aoc ranks #1 overall (mean 0.8602) once given a fair training budget.
|
||
- Noise robustness is roll-aoc's clearest advantage — the gap widens at 20% noise.
|
||
- mae is competitive at low noise but not noise-robust; not a strong baseline at 20%.
|
||
- libauc-auroc consistently underperforms the simpler baselines (bce-weighted, focal).
|
||
- asymmetric-loss is roll-aoc's closest competitor across all conditions.
|
||
- roll-tpr90 is a poor AUC-maximiser (expected — it optimises TPR@FPR=0.1, not AUC).
|
||
|
||
** CIFAR-10N Experiment (2026-07-19 → 2026-07-20)
|
||
|
||
Run dir: =results/cifar10n/2026-07-19-06-52=
|
||
Configs: =roll-aoc=, =bce-weighted=, =mae=, =gce-0.7=, =libauc-auroc= × 3 episodes × 3 noise types = 45 runs.
|
||
Noise types: =clean= (original CIFAR-10 labels), =aggre= (human aggregated, ~moderate noise), =worse= (single worst-annotator labels, ~heavy noise).
|
||
Architecture: ConvNet (3 conv blocks + 2 FC, trained from scratch). Positive class: automobile (class 1) vs rest, IR ~9.
|
||
LibAUC ran on CPU (MPS unsupported); all others on MPS. PESG lr=1e-3 (not 0.1 — from-scratch training explodes at 0.1).
|
||
|
||
*Results (mean test AUC over 3 episodes):*
|
||
|
||
| Config | clean | aggre | worse |
|
||
|--------------+--------+--------+--------|
|
||
| roll-aoc | 0.9800 | 0.9336 | 0.8200 |
|
||
| bce-weighted | 0.9772 | 0.9267 | 0.8044 |
|
||
| gce-0.7 | 0.9776 | 0.9289 | 0.7790 |
|
||
| libauc-auroc | 0.9777 | 0.9228 | 0.7690 |
|
||
| mae | 0.6954 | 0.7756 | 0.5979 |
|
||
|
||
*Key findings:*
|
||
- =roll-aoc= is the top performer at every noise level, with the gap widening under heavy noise.
|
||
- Under =worse= noise, =roll-aoc= beats =bce-weighted= by 1.6pp and =libauc-auroc= by 5.1pp.
|
||
- =gce-0.7= and =libauc-auroc= degrade more steeply than =roll-aoc= under increasing noise.
|
||
- =mae= is weak throughout; satisfies the symmetry condition theoretically but fails in practice on this task.
|
||
|
||
Summaries: =results/cifar10n/2026-07-19-06-52/auc.csv= (45 rows), =roc-clean.html=, =roc-aggre.html=, =roc-worse.html=.
|
||
Regen script: =scripts/regen_cifar10n_summaries.py= — loads MPS-saved pkls via =default_restore_location= patch (maps mps→cpu).
|
||
|
||
** Poisoning Experiment (2026-07-13 → 2026-07-15)
|
||
|
||
Setup: for each KEEL dataset, train positives are duplicated N times with label flipped to 0 (false).
|
||
Originals keep their true label. Val/test splits untouched.
|
||
Code: =LabelPoisonedDataset= + =PoisonedSplitter= in =src/experiment.py=.
|
||
Runner: =experiments/keel/run_poisoned.py <dataset> <n_duplicates> [--no-mp]=, results go to =poison/<dataset>/=.
|
||
Batch runner: =experiments/keel/run_all_poisoned.sh [n_duplicates]= (default 3).
|
||
Configs run (poisoned): =roll-aoc=, =bce-weighted=, =mae=, =gce-0.7=, =libauc-auroc=.
|
||
Episodes: 15.
|
||
|
||
*Key findings (2026-07-15, full 12-dataset run):*
|
||
- =roll-aoc= is statistically significantly better than =libauc-auroc= on glass1, pima, vehicle2 (one-sided paired t-test and Wilcoxon, p < 0.05).
|
||
- =libauc-auroc= is numerically unstable under poisoning on some datasets: produces NaN/Inf predictions on many episodes. glass2: only 2/15 valid episodes; vowel0: only 1/15 valid episode. These are silently filtered in =write_auc_csv= (=src/summary.py:344=) — the auc.csv will have far fewer rows than expected for libauc on those datasets.
|
||
- =mae= and =gce-0.7= perform poorly under poisoning across all datasets.
|
||
- =roll-aoc= and =libauc-auroc= are competitive on easy datasets (glass6, new-thyroid1, vowel0, wisconsin) where both reach ~0.99 AUC.
|
||
|
||
** Large Datasets
|
||
|
||
| Experiment | Dataset | Status | Notes |
|
||
|-----------------------+-------------------+---------+----------------------------------------------|
|
||
| experiment-higgs | HiggsDataset | OK | 500k samples, ~50/50 balance |
|
||
| experiment-creditcard | CreditCardFraud | MISSING | =/home/aner/.data/creditcard/creditcard.csv= |
|
||
| experiment-homecredit | HomeCreditDataset | MISSING | =/home/aner/.data/homecredit/= |
|
||
|
||
** KEEL Run 2026-07-08 — Setup
|
||
|
||
Results folders: =/home/aner/projects/research-projects/roll/impl/results/*/2026-07-08-*/=
|
||
|
||
Changes from 2026-07-07 run:
|
||
- Bandwidth: ISJ → *Silverman's rule* (ISJ crashed on small/degenerate batches mid-training)
|
||
- Kernel scheduler: =KernelScheduler(16.0, decay_every=20)= (old: 100.0/500)
|
||
- LR: roll =1e-1=, bce =1e-3= (was both =1e-3=)
|
||
- Weight decay: roll =0.1=, bce =1e-3= (was none)
|
||
- Dropout: =0.1= (was =0.0=)
|
||
- Batch size: =256= (was 128; tried 32 but too small for balanced sampling)
|
||
- Patience: =100= (was =500=)
|
||
- Episodes: =5= (was =1=)
|
||
- FPR target changed =fpr-0.05= → =fpr-0.40=, then FPR variants dropped entirely
|
||
|
||
Scoring: =mean(fpr) + std(fpr)= across 5 episodes (penalises variance and instability; lower is better).
|
||
Note: mean−std was tried first but incorrectly rewarded instability — corrected to mean+std.
|
||
|
||
** run_all.sh
|
||
|
||
Runs KEEL datasets via shell script in =experiments/keel/=.
|
||
On failure, continues to next dataset.
|