backup: 2026-08-29 06:59
This commit is contained in:
+72
-56
@@ -7,15 +7,72 @@
|
||||
|
||||
* impl/experiments
|
||||
|
||||
** Current Config Set (as of 2026-07-08)
|
||||
** Baseline Config Suite (current)
|
||||
|
||||
- =roll= — =kernelized_roll_tpr(0.95)=
|
||||
- =roll+bce= — =kernelized_roll_tpr(0.95, bce_weight=0.5, bce_pos_weight=imbalance_ratio)=
|
||||
- =bce= — plain =BCEWithLogitsLoss=
|
||||
- =bce-weighted= — BCE with =pos_weight = num_false/num_true=
|
||||
8 configs run on every dataset. All share StepLR(step_size=50, γ=0.5).
|
||||
|
||||
FPR variants (=roll-fpr-0.40=, =roll-fpr-0.40+bce=) removed — less consistent, added complexity for marginal gain.
|
||||
Target metric: FPR@TPR=0.95 (lower is better), written to =tpr_summary.csv=.
|
||||
| Config | Loss | Optimizer | lr | Notes |
|
||||
|-------------------+-----------------------------+-----------+-------+--------------------------------|
|
||||
| roll-aoc | kernelized_roll_aoc() | RMSprop | 1e-3 | wd=1e-5; KernelScheduler(32,20)|
|
||||
| roll-tpr90 | kernelized_roll_tpr(tpr=0.9)| RMSprop | 1e-3 | wd=1e-5; KernelScheduler(32,20)|
|
||||
| bce-weighted | BCEWithLogitsLoss | Adam | 1e-3 | pos_weight=imbalance_ratio |
|
||||
| gce-0.7 | gce_loss(q=0.7) | Adam | 1e-3 | wd=1e-5 |
|
||||
| mae | mae_loss | Adam | 1e-3 | wd=1e-5 |
|
||||
| focal-loss | focal_loss(γ=2.0) | Adam | 1e-3 | wd=1e-5 |
|
||||
| asymmetric-loss | asymmetric_loss | Adam | 1e-3 | wd=1e-5; γ_pos=0,γ_neg=4,clip=0.05 |
|
||||
| libauc-auroc | AUCMLoss (Yuan ICCV 2021) | PESG | 0.01 | must be paired with PESG; CPU-only on MPS |
|
||||
|
||||
RMSprop chosen for ROLL (not Adam) because KDE gradients are non-stationary — KernelScheduler
|
||||
anneals bandwidth during training, shifting the loss landscape. Adam's first-moment momentum
|
||||
accumulates stale directional bias from earlier kernel widths; RMSprop adapts step magnitude
|
||||
only, making it more reactive to the current gradient without directional memory.
|
||||
|
||||
LibAUC/PESG: AUC is a pairwise (non-decomposable) metric; Yuan et al. reformulate as minimax
|
||||
with auxiliary scalars a≈E[f(x⁺)], b≈E[f(x⁻)], α (dual), enabling per-mini-batch updates.
|
||||
PESG updates all four variable sets jointly; loss module owns a,b,α so optimizer and loss
|
||||
are tightly coupled (hence opt_factory pattern in our code).
|
||||
|
||||
** KEEL Noise Experiment — Setup & Config Fix
|
||||
|
||||
*Setup:* 12 datasets × 3 noise levels (5%, 10%, 20%) × 10 episodes × 8 configs.
|
||||
Label noise injected via =make_noisy_splitter= (random label flip on training set only).
|
||||
Results in =results-final/keel-noise-{05,10,20}/<dataset>/=.
|
||||
Runner: =experiments/keel/run_all_noise_levels.sh= → =run_all_noisy.sh= → =run_noisy.py=.
|
||||
CPU forced (6.6× faster than MPS for KEEL batch sizes); MLflow unset.
|
||||
|
||||
*Config fairness bug (2026-08-28):* roll-aoc was using =max_iters=1500= / =patience=100=
|
||||
vs baselines =max_iters=5000= / =patience=500=. This is a structural training-budget
|
||||
disadvantage baked into =_base.py= (roll_config vs robust_config/bce_config).
|
||||
Fix: =experiments/keel/run_noisy_fair_roll.py= reruns roll-aoc only with fair settings;
|
||||
results swapped into =results-final= via =results-fair-roll/=.
|
||||
*Same bug applies to noiseless keel (results-final/keel/, 5 episodes) — still needs rerun.*
|
||||
|
||||
** KEEL Noise Experiment — Results (2026-08-28, fair roll-aoc config)
|
||||
|
||||
Mean test AUC averaged across all 12 datasets, by noise level:
|
||||
|
||||
| Config | 5% | 10% | 20% | mean |
|
||||
|-----------------+--------+--------+--------+--------|
|
||||
| *roll-aoc* | 0.8713 | 0.8623 | *0.8471* | *0.8602* |
|
||||
| asymmetric-loss | 0.8719 | 0.8695 | 0.8330 | 0.8581 |
|
||||
| bce-weighted | 0.8686 | 0.8622 | 0.8434 | 0.8581 |
|
||||
| focal-loss | 0.8679 | 0.8573 | 0.8419 | 0.8557 |
|
||||
| gce-0.7 | 0.8704 | 0.8650 | 0.8305 | 0.8553 |
|
||||
| mae | 0.8776 | 0.8663 | 0.8035 | 0.8491 |
|
||||
| libauc-auroc | 0.8593 | 0.8544 | 0.8093 | 0.8410 |
|
||||
| roll-tpr90 | 0.7859 | 0.7705 | 0.7728 | 0.7764 |
|
||||
|
||||
Per-dataset at 20% noise: roll-aoc wins 7/12 datasets.
|
||||
roll-aoc is the most noise-robust: degrades only −0.024 from 5%→20%;
|
||||
mae drops −0.074, libauc-auroc −0.050, asymmetric-loss −0.039.
|
||||
|
||||
*Key findings for thesis:*
|
||||
- roll-aoc ranks #1 overall (mean 0.8602) once given a fair training budget.
|
||||
- Noise robustness is roll-aoc's clearest advantage — the gap widens at 20% noise.
|
||||
- mae is competitive at low noise but not noise-robust; not a strong baseline at 20%.
|
||||
- libauc-auroc consistently underperforms the simpler baselines (bce-weighted, focal).
|
||||
- asymmetric-loss is roll-aoc's closest competitor across all conditions.
|
||||
- roll-tpr90 is a poor AUC-maximiser (expected — it optimises TPR@FPR=0.1, not AUC).
|
||||
|
||||
** CIFAR-10N Experiment (2026-07-19 → 2026-07-20)
|
||||
|
||||
@@ -60,6 +117,14 @@ Episodes: 15.
|
||||
- =mae= and =gce-0.7= perform poorly under poisoning across all datasets.
|
||||
- =roll-aoc= and =libauc-auroc= are competitive on easy datasets (glass6, new-thyroid1, vowel0, wisconsin) where both reach ~0.99 AUC.
|
||||
|
||||
** Large Datasets
|
||||
|
||||
| Experiment | Dataset | Status | Notes |
|
||||
|-----------------------+-------------------+---------+----------------------------------------------|
|
||||
| experiment-higgs | HiggsDataset | OK | 500k samples, ~50/50 balance |
|
||||
| experiment-creditcard | CreditCardFraud | MISSING | =/home/aner/.data/creditcard/creditcard.csv= |
|
||||
| experiment-homecredit | HomeCreditDataset | MISSING | =/home/aner/.data/homecredit/= |
|
||||
|
||||
** KEEL Run 2026-07-08 — Setup
|
||||
|
||||
Results folders: =/home/aner/projects/research-projects/roll/impl/results/*/2026-07-08-*/=
|
||||
@@ -78,55 +143,6 @@ Changes from 2026-07-07 run:
|
||||
Scoring: =mean(fpr) + std(fpr)= across 5 episodes (penalises variance and instability; lower is better).
|
||||
Note: mean−std was tried first but incorrectly rewarded instability — corrected to mean+std.
|
||||
|
||||
** KEEL Run 2026-07-08 — Results (mean + std, FPR@TPR=0.95)
|
||||
|
||||
| Dataset | roll | roll+bce | bce | bce-weighted | Winner |
|
||||
|--------------+-------+----------+-------+--------------+--------------|
|
||||
| glass0 | 0.568 | — | 0.677 | 0.706 | roll |
|
||||
| glass1 | 0.809 | 0.776 | 0.744 | 0.676 | bce-weighted |
|
||||
| glass2 | 0.702 | 0.642 | 0.793 | 0.683 | roll+bce |
|
||||
| glass6 | 0.081 | 0.060 | 0.042 | 0.054 | bce |
|
||||
| haberman | 0.910 | 0.945 | 1.015 | 0.940 | roll |
|
||||
| iris0 | 0.000 | 0.000 | 0.000 | 0.600 | roll/bce/roll+bce (tied) |
|
||||
| new-thyroid1 | 0.077 | 0.048 | 0.057 | 0.055 | roll+bce |
|
||||
| pima | 0.710 | 0.744 | 0.627 | 0.681 | bce |
|
||||
| vehicle2 | 0.148 | 0.097 | 0.163 | 0.108 | roll+bce |
|
||||
| vowel0 | 0.160 | 0.095 | 0.059 | 0.066 | bce |
|
||||
| wisconsin | 0.011 | 0.007 | 0.008 | 0.011 | roll+bce |
|
||||
| yeast3 | 0.398 | 0.539 | 0.451 | 0.552 | roll |
|
||||
|
||||
Win count: roll+bce: 4, bce: 3, roll: 3, bce-weighted: 1
|
||||
|
||||
** KEEL Run 2026-07-08 — Findings
|
||||
|
||||
- =roll+bce= is the most consistent ROLL variant: wins glass2, new-thyroid1, vehicle2, wisconsin.
|
||||
- =roll= alone wins where it has a large mean advantage that outweighs variance penalty (haberman, glass0, yeast3).
|
||||
- =bce= wins on glass6, pima, vowel0; =bce-weighted= wins glass1 but its iris0 score is inflated by one catastrophic episode.
|
||||
- No single method dominates. ROLL adds clear value on haberman, glass0, yeast3 (hard/imbalanced datasets).
|
||||
- High episode variance on haberman, yeast3, glass1 — small test sets make results noisy.
|
||||
|
||||
** Previous Run — KEEL Results (FPR@TPR=0.95, 2026-07-07, single episode, raw mean)
|
||||
|
||||
| Dataset | roll | roll+bce | roll-fpr+bce | bce | bce-weighted | Best |
|
||||
|-----------+-------+----------+--------------+-------+--------------+---------------|
|
||||
| glass0 | 0.878 | 0.490 | 0.673 | 0.857 | 0.449 | bce-weighted |
|
||||
| glass1 | 0.872 | 0.830 | 0.702 | 0.872 | 0.681 | bce-weighted |
|
||||
| haberman | 0.870 | 1.0 | 1.0 | 0.922 | 1.0 | roll (barely) |
|
||||
| pima | 0.649 | 0.714 | 0.673 | 0.649 | 0.595 | bce-weighted |
|
||||
| vehicle2 | 0.095 | 0.032 | 0.032 | 0.058 | 0.058 | roll+bce |
|
||||
| vowel0 | 0.141 | 0.0 | 0.0 | 0.0 | 0.004 | tied |
|
||||
| wisconsin | 0.964 | 0.0 | 0.0 | 0.0 | 0.0 | tied |
|
||||
|
||||
Config: 1 episode, ISJ bandwidth, no weight decay/dropout, lr=1e-3 for all, gamma=100/decay=500.
|
||||
|
||||
** Large Datasets
|
||||
|
||||
| Experiment | Dataset | Status | Notes |
|
||||
|-----------------------+-------------------+---------+----------------------------------------------|
|
||||
| experiment-higgs | HiggsDataset | OK | 500k samples, ~50/50 balance |
|
||||
| experiment-creditcard | CreditCardFraud | MISSING | =/home/aner/.data/creditcard/creditcard.csv= |
|
||||
| experiment-homecredit | HomeCreditDataset | MISSING | =/home/aner/.data/homecredit/= |
|
||||
|
||||
** run_all.sh
|
||||
|
||||
Runs KEEL datasets via shell script in =experiments/keel/=.
|
||||
|
||||
Reference in New Issue
Block a user