134 lines
7.6 KiB
Org Mode
134 lines
7.6 KiB
Org Mode
:PROPERTIES:
|
||
:ID: a53cbe84-cd8d-45c2-a8cf-34ab520a3ea5
|
||
:END:
|
||
|
||
#+title: impl/experiments
|
||
#+filetags: :project: :knowledge: :experiments:
|
||
|
||
* impl/experiments
|
||
|
||
** Current Config Set (as of 2026-07-08)
|
||
|
||
- =roll= — =kernelized_roll_tpr(0.95)=
|
||
- =roll+bce= — =kernelized_roll_tpr(0.95, bce_weight=0.5, bce_pos_weight=imbalance_ratio)=
|
||
- =bce= — plain =BCEWithLogitsLoss=
|
||
- =bce-weighted= — BCE with =pos_weight = num_false/num_true=
|
||
|
||
FPR variants (=roll-fpr-0.40=, =roll-fpr-0.40+bce=) removed — less consistent, added complexity for marginal gain.
|
||
Target metric: FPR@TPR=0.95 (lower is better), written to =tpr_summary.csv=.
|
||
|
||
** CIFAR-10N Experiment (2026-07-19 → 2026-07-20)
|
||
|
||
Run dir: =results/cifar10n/2026-07-19-06-52=
|
||
Configs: =roll-aoc=, =bce-weighted=, =mae=, =gce-0.7=, =libauc-auroc= × 3 episodes × 3 noise types = 45 runs.
|
||
Noise types: =clean= (original CIFAR-10 labels), =aggre= (human aggregated, ~moderate noise), =worse= (single worst-annotator labels, ~heavy noise).
|
||
Architecture: ConvNet (3 conv blocks + 2 FC, trained from scratch). Positive class: automobile (class 1) vs rest, IR ~9.
|
||
LibAUC ran on CPU (MPS unsupported); all others on MPS. PESG lr=1e-3 (not 0.1 — from-scratch training explodes at 0.1).
|
||
|
||
*Results (mean test AUC over 3 episodes):*
|
||
|
||
| Config | clean | aggre | worse |
|
||
|--------------+--------+--------+--------|
|
||
| roll-aoc | 0.9800 | 0.9336 | 0.8200 |
|
||
| bce-weighted | 0.9772 | 0.9267 | 0.8044 |
|
||
| gce-0.7 | 0.9776 | 0.9289 | 0.7790 |
|
||
| libauc-auroc | 0.9777 | 0.9228 | 0.7690 |
|
||
| mae | 0.6954 | 0.7756 | 0.5979 |
|
||
|
||
*Key findings:*
|
||
- =roll-aoc= is the top performer at every noise level, with the gap widening under heavy noise.
|
||
- Under =worse= noise, =roll-aoc= beats =bce-weighted= by 1.6pp and =libauc-auroc= by 5.1pp.
|
||
- =gce-0.7= and =libauc-auroc= degrade more steeply than =roll-aoc= under increasing noise.
|
||
- =mae= is weak throughout; satisfies the symmetry condition theoretically but fails in practice on this task.
|
||
|
||
Summaries: =results/cifar10n/2026-07-19-06-52/auc.csv= (45 rows), =roc-clean.html=, =roc-aggre.html=, =roc-worse.html=.
|
||
Regen script: =scripts/regen_cifar10n_summaries.py= — loads MPS-saved pkls via =default_restore_location= patch (maps mps→cpu).
|
||
|
||
** Poisoning Experiment (2026-07-13 → 2026-07-15)
|
||
|
||
Setup: for each KEEL dataset, train positives are duplicated N times with label flipped to 0 (false).
|
||
Originals keep their true label. Val/test splits untouched.
|
||
Code: =LabelPoisonedDataset= + =PoisonedSplitter= in =src/experiment.py=.
|
||
Runner: =experiments/keel/run_poisoned.py <dataset> <n_duplicates> [--no-mp]=, results go to =poison/<dataset>/=.
|
||
Batch runner: =experiments/keel/run_all_poisoned.sh [n_duplicates]= (default 3).
|
||
Configs run (poisoned): =roll-aoc=, =bce-weighted=, =mae=, =gce-0.7=, =libauc-auroc=.
|
||
Episodes: 15.
|
||
|
||
*Key findings (2026-07-15, full 12-dataset run):*
|
||
- =roll-aoc= is statistically significantly better than =libauc-auroc= on glass1, pima, vehicle2 (one-sided paired t-test and Wilcoxon, p < 0.05).
|
||
- =libauc-auroc= is numerically unstable under poisoning on some datasets: produces NaN/Inf predictions on many episodes. glass2: only 2/15 valid episodes; vowel0: only 1/15 valid episode. These are silently filtered in =write_auc_csv= (=src/summary.py:344=) — the auc.csv will have far fewer rows than expected for libauc on those datasets.
|
||
- =mae= and =gce-0.7= perform poorly under poisoning across all datasets.
|
||
- =roll-aoc= and =libauc-auroc= are competitive on easy datasets (glass6, new-thyroid1, vowel0, wisconsin) where both reach ~0.99 AUC.
|
||
|
||
** KEEL Run 2026-07-08 — Setup
|
||
|
||
Results folders: =/home/aner/projects/research-projects/roll/impl/results/*/2026-07-08-*/=
|
||
|
||
Changes from 2026-07-07 run:
|
||
- Bandwidth: ISJ → *Silverman's rule* (ISJ crashed on small/degenerate batches mid-training)
|
||
- Kernel scheduler: =KernelScheduler(16.0, decay_every=20)= (old: 100.0/500)
|
||
- LR: roll =1e-1=, bce =1e-3= (was both =1e-3=)
|
||
- Weight decay: roll =0.1=, bce =1e-3= (was none)
|
||
- Dropout: =0.1= (was =0.0=)
|
||
- Batch size: =256= (was 128; tried 32 but too small for balanced sampling)
|
||
- Patience: =100= (was =500=)
|
||
- Episodes: =5= (was =1=)
|
||
- FPR target changed =fpr-0.05= → =fpr-0.40=, then FPR variants dropped entirely
|
||
|
||
Scoring: =mean(fpr) + std(fpr)= across 5 episodes (penalises variance and instability; lower is better).
|
||
Note: mean−std was tried first but incorrectly rewarded instability — corrected to mean+std.
|
||
|
||
** KEEL Run 2026-07-08 — Results (mean + std, FPR@TPR=0.95)
|
||
|
||
| Dataset | roll | roll+bce | bce | bce-weighted | Winner |
|
||
|--------------+-------+----------+-------+--------------+--------------|
|
||
| glass0 | 0.568 | — | 0.677 | 0.706 | roll |
|
||
| glass1 | 0.809 | 0.776 | 0.744 | 0.676 | bce-weighted |
|
||
| glass2 | 0.702 | 0.642 | 0.793 | 0.683 | roll+bce |
|
||
| glass6 | 0.081 | 0.060 | 0.042 | 0.054 | bce |
|
||
| haberman | 0.910 | 0.945 | 1.015 | 0.940 | roll |
|
||
| iris0 | 0.000 | 0.000 | 0.000 | 0.600 | roll/bce/roll+bce (tied) |
|
||
| new-thyroid1 | 0.077 | 0.048 | 0.057 | 0.055 | roll+bce |
|
||
| pima | 0.710 | 0.744 | 0.627 | 0.681 | bce |
|
||
| vehicle2 | 0.148 | 0.097 | 0.163 | 0.108 | roll+bce |
|
||
| vowel0 | 0.160 | 0.095 | 0.059 | 0.066 | bce |
|
||
| wisconsin | 0.011 | 0.007 | 0.008 | 0.011 | roll+bce |
|
||
| yeast3 | 0.398 | 0.539 | 0.451 | 0.552 | roll |
|
||
|
||
Win count: roll+bce: 4, bce: 3, roll: 3, bce-weighted: 1
|
||
|
||
** KEEL Run 2026-07-08 — Findings
|
||
|
||
- =roll+bce= is the most consistent ROLL variant: wins glass2, new-thyroid1, vehicle2, wisconsin.
|
||
- =roll= alone wins where it has a large mean advantage that outweighs variance penalty (haberman, glass0, yeast3).
|
||
- =bce= wins on glass6, pima, vowel0; =bce-weighted= wins glass1 but its iris0 score is inflated by one catastrophic episode.
|
||
- No single method dominates. ROLL adds clear value on haberman, glass0, yeast3 (hard/imbalanced datasets).
|
||
- High episode variance on haberman, yeast3, glass1 — small test sets make results noisy.
|
||
|
||
** Previous Run — KEEL Results (FPR@TPR=0.95, 2026-07-07, single episode, raw mean)
|
||
|
||
| Dataset | roll | roll+bce | roll-fpr+bce | bce | bce-weighted | Best |
|
||
|-----------+-------+----------+--------------+-------+--------------+---------------|
|
||
| glass0 | 0.878 | 0.490 | 0.673 | 0.857 | 0.449 | bce-weighted |
|
||
| glass1 | 0.872 | 0.830 | 0.702 | 0.872 | 0.681 | bce-weighted |
|
||
| haberman | 0.870 | 1.0 | 1.0 | 0.922 | 1.0 | roll (barely) |
|
||
| pima | 0.649 | 0.714 | 0.673 | 0.649 | 0.595 | bce-weighted |
|
||
| vehicle2 | 0.095 | 0.032 | 0.032 | 0.058 | 0.058 | roll+bce |
|
||
| vowel0 | 0.141 | 0.0 | 0.0 | 0.0 | 0.004 | tied |
|
||
| wisconsin | 0.964 | 0.0 | 0.0 | 0.0 | 0.0 | tied |
|
||
|
||
Config: 1 episode, ISJ bandwidth, no weight decay/dropout, lr=1e-3 for all, gamma=100/decay=500.
|
||
|
||
** Large Datasets
|
||
|
||
| Experiment | Dataset | Status | Notes |
|
||
|-----------------------+-------------------+---------+----------------------------------------------|
|
||
| experiment-higgs | HiggsDataset | OK | 500k samples, ~50/50 balance |
|
||
| experiment-creditcard | CreditCardFraud | MISSING | =/home/aner/.data/creditcard/creditcard.csv= |
|
||
| experiment-homecredit | HomeCreditDataset | MISSING | =/home/aner/.data/homecredit/= |
|
||
|
||
** run_all.sh
|
||
|
||
Runs KEEL datasets via shell script in =experiments/keel/=.
|
||
On failure, continues to next dataset.
|