impl/work-history/recent

2026-07-24 Thu

Investigated stalled cifar10n remote run: 5 workers from 6:45AM were still alive after 15.5h with 0 results; previous session's ps check had missed them, making the run appear dead
Diagnosed OOM: memory at ~46MB free at 7h mark with 5 concurrent MPS workers on 16GB; new run with 3 workers also OOM'd after 8.5h (26GB compressed pages, 5B decompressions)
Fixed root cause: added sequential_episodes flag to run_configurations and _perform_multiple_episodes in src/experiment.py — when True, bypasses the subprocess-parallel path and runs episodes one at a time in-process on MPS; cifar10n now uses this
Killed 3-worker subprocess run on remote; restarted cifar10n with sequential_episodes=True — now running single-process, one episode at a time
Added kaggle to flake.nix buildInputs; set up ~/.config/kaggle/kaggle.json (user: anerzakobar); downloaded creditcard.csv (144MB) to ~/.data/creditcard/
Rewrote experiments/large/experiment-creditcard.py: replaced old ad-hoc FPR configs with full baseline suite matching KEEL/_base.py — roll-aoc, roll-tpr90, bce-weighted, libauc-auroc, gce-0.7, mae, focal-loss, asymmetric-loss; StepLR on all configs; N_EPISODES=5
Launched creditcard experiment locally in background (5 episodes × 8 configs, CPU multiprocessing)
Reviewed KEEL AUC results table: libauc-auroc wins 6/12 datasets (mean 0.890); asymmetric-loss strong second (0.887); roll-aoc 6th (0.872); roll-tpr90 last (0.842, dragged by new-thyroid1 instability)