:PROPERTIES:
:ID: a53cbe84-cd8d-45c2-a8cf-34ab520a3ea5
:END:

#+title: impl/experiments
#+filetags: :project: :knowledge: :experiments:

* impl/experiments

** Current Config Set (as of 2026-07-08)

- =roll= — =kernelized_roll_tpr(0.95)=
- =roll+bce= — =kernelized_roll_tpr(0.95, bce_weight=0.5, bce_pos_weight=imbalance_ratio)=
- =bce= — plain =BCEWithLogitsLoss=
- =bce-weighted= — BCE with =pos_weight = num_false/num_true=

FPR variants (=roll-fpr-0.40=, =roll-fpr-0.40+bce=) removed — less consistent, added complexity for marginal gain.
Target metric: FPR@TPR=0.95 (lower is better), written to =tpr_summary.csv=.

** CIFAR-10N Experiment (2026-07-19 → 2026-07-20)

Run dir: =results/cifar10n/2026-07-19-06-52=
Configs: =roll-aoc=, =bce-weighted=, =mae=, =gce-0.7=, =libauc-auroc= × 3 episodes × 3 noise types = 45 runs.
Noise types: =clean= (original CIFAR-10 labels), =aggre= (human aggregated, ~moderate noise), =worse= (single worst-annotator labels, ~heavy noise).
Architecture: ConvNet (3 conv blocks + 2 FC, trained from scratch). Positive class: automobile (class 1) vs rest, IR ~9.
LibAUC ran on CPU (MPS unsupported); all others on MPS. PESG lr=1e-3 (not 0.1 — from-scratch training explodes at 0.1).

*Results (mean test AUC over 3 episodes):*

| Config       | clean  | aggre  | worse  |
|--------------+--------+--------+--------|
| roll-aoc     | 0.9800 | 0.9336 | 0.8200 |
| bce-weighted | 0.9772 | 0.9267 | 0.8044 |
| gce-0.7      | 0.9776 | 0.9289 | 0.7790 |
| libauc-auroc | 0.9777 | 0.9228 | 0.7690 |
| mae          | 0.6954 | 0.7756 | 0.5979 |

*Key findings:*
- =roll-aoc= is the top performer at every noise level, with the gap widening under heavy noise.
- Under =worse= noise, =roll-aoc= beats =bce-weighted= by 1.6pp and =libauc-auroc= by 5.1pp.
- =gce-0.7= and =libauc-auroc= degrade more steeply than =roll-aoc= under increasing noise.
- =mae= is weak throughout; satisfies the symmetry condition theoretically but fails in practice on this task.

Summaries: =results/cifar10n/2026-07-19-06-52/auc.csv= (45 rows), =roc-clean.html=, =roc-aggre.html=, =roc-worse.html=.
Regen script: =scripts/regen_cifar10n_summaries.py= — loads MPS-saved pkls via =default_restore_location= patch (maps mps→cpu).

** Poisoning Experiment (2026-07-13 → 2026-07-15)

Setup: for each KEEL dataset, train positives are duplicated N times with label flipped to 0 (false).
Originals keep their true label. Val/test splits untouched.
Code: =LabelPoisonedDataset= + =PoisonedSplitter= in =src/experiment.py=.
Runner: =experiments/keel/run_poisoned.py <dataset> <n_duplicates> [--no-mp]=, results go to =poison/<dataset>/=.
Batch runner: =experiments/keel/run_all_poisoned.sh [n_duplicates]= (default 3).
Configs run (poisoned): =roll-aoc=, =bce-weighted=, =mae=, =gce-0.7=, =libauc-auroc=.
Episodes: 15.

*Key findings (2026-07-15, full 12-dataset run):*
- =roll-aoc= is statistically significantly better than =libauc-auroc= on glass1, pima, vehicle2 (one-sided paired t-test and Wilcoxon, p < 0.05).
- =libauc-auroc= is numerically unstable under poisoning on some datasets: produces NaN/Inf predictions on many episodes. glass2: only 2/15 valid episodes; vowel0: only 1/15 valid episode. These are silently filtered in =write_auc_csv= (=src/summary.py:344=) — the auc.csv will have far fewer rows than expected for libauc on those datasets.
- =mae= and =gce-0.7= perform poorly under poisoning across all datasets.
- =roll-aoc= and =libauc-auroc= are competitive on easy datasets (glass6, new-thyroid1, vowel0, wisconsin) where both reach ~0.99 AUC.

** KEEL Run 2026-07-08 — Setup

Results folders: =/home/aner/projects/research-projects/roll/impl/results/*/2026-07-08-*/=

Changes from 2026-07-07 run:
- Bandwidth: ISJ → *Silverman's rule* (ISJ crashed on small/degenerate batches mid-training)
- Kernel scheduler: =KernelScheduler(16.0, decay_every=20)= (old: 100.0/500)
- LR: roll =1e-1=, bce =1e-3= (was both =1e-3=)
- Weight decay: roll =0.1=, bce =1e-3= (was none)
- Dropout: =0.1= (was =0.0=)
- Batch size: =256= (was 128; tried 32 but too small for balanced sampling)
- Patience: =100= (was =500=)
- Episodes: =5= (was =1=)
- FPR target changed =fpr-0.05= → =fpr-0.40=, then FPR variants dropped entirely

Scoring: =mean(fpr) + std(fpr)= across 5 episodes (penalises variance and instability; lower is better).
Note: mean−std was tried first but incorrectly rewarded instability — corrected to mean+std.

** KEEL Run 2026-07-08 — Results (mean + std, FPR@TPR=0.95)

| Dataset      | roll  | roll+bce | bce   | bce-weighted | Winner       |
|--------------+-------+----------+-------+--------------+--------------|
| glass0       | 0.568 | —        | 0.677 | 0.706        | roll         |
| glass1       | 0.809 | 0.776    | 0.744 | 0.676        | bce-weighted |
| glass2       | 0.702 | 0.642    | 0.793 | 0.683        | roll+bce     |
| glass6       | 0.081 | 0.060    | 0.042 | 0.054        | bce          |
| haberman     | 0.910 | 0.945    | 1.015 | 0.940        | roll         |
| iris0        | 0.000 | 0.000    | 0.000 | 0.600        | roll/bce/roll+bce (tied) |
| new-thyroid1 | 0.077 | 0.048    | 0.057 | 0.055        | roll+bce     |
| pima         | 0.710 | 0.744    | 0.627 | 0.681        | bce          |
| vehicle2     | 0.148 | 0.097    | 0.163 | 0.108        | roll+bce     |
| vowel0       | 0.160 | 0.095    | 0.059 | 0.066        | bce          |
| wisconsin    | 0.011 | 0.007    | 0.008 | 0.011        | roll+bce     |
| yeast3       | 0.398 | 0.539    | 0.451 | 0.552        | roll         |

Win count: roll+bce: 4, bce: 3, roll: 3, bce-weighted: 1

** KEEL Run 2026-07-08 — Findings

- =roll+bce= is the most consistent ROLL variant: wins glass2, new-thyroid1, vehicle2, wisconsin.
- =roll= alone wins where it has a large mean advantage that outweighs variance penalty (haberman, glass0, yeast3).
- =bce= wins on glass6, pima, vowel0; =bce-weighted= wins glass1 but its iris0 score is inflated by one catastrophic episode.
- No single method dominates. ROLL adds clear value on haberman, glass0, yeast3 (hard/imbalanced datasets).
- High episode variance on haberman, yeast3, glass1 — small test sets make results noisy.

** Previous Run — KEEL Results (FPR@TPR=0.95, 2026-07-07, single episode, raw mean)

| Dataset   | roll  | roll+bce | roll-fpr+bce | bce   | bce-weighted | Best          |
|-----------+-------+----------+--------------+-------+--------------+---------------|
| glass0    | 0.878 | 0.490    | 0.673        | 0.857 | 0.449        | bce-weighted  |
| glass1    | 0.872 | 0.830    | 0.702        | 0.872 | 0.681        | bce-weighted  |
| haberman  | 0.870 | 1.0      | 1.0          | 0.922 | 1.0          | roll (barely) |
| pima      | 0.649 | 0.714    | 0.673        | 0.649 | 0.595        | bce-weighted  |
| vehicle2  | 0.095 | 0.032    | 0.032        | 0.058 | 0.058        | roll+bce      |
| vowel0    | 0.141 | 0.0      | 0.0          | 0.0   | 0.004        | tied          |
| wisconsin | 0.964 | 0.0      | 0.0          | 0.0   | 0.0          | tied          |

Config: 1 episode, ISJ bandwidth, no weight decay/dropout, lr=1e-3 for all, gamma=100/decay=500.

** Large Datasets

| Experiment            | Dataset           | Status  | Notes                                        |
|-----------------------+-------------------+---------+----------------------------------------------|
| experiment-higgs      | HiggsDataset      | OK      | 500k samples, ~50/50 balance                 |
| experiment-creditcard | CreditCardFraud   | MISSING | =/home/aner/.data/creditcard/creditcard.csv= |
| experiment-homecredit | HomeCreditDataset | MISSING | =/home/aner/.data/homecredit/=               |

** run_all.sh

Runs KEEL datasets via shell script in =experiments/keel/=.
On failure, continues to next dataset.