roam/implwork-historyrecent.org

impl/work-history/recent

• 2026-08-28 Thu
• 2026-08-01 Fri
• 2026-07-24 Thu
  • Archives

2026-08-28 Thu

• Returned from vacation; checked both remote machines for run status
• beacon-wsl (food101n experiment-food101n-rerun-weighted.py): found crashed during gce-0.7 ep1 ~2 weeks ago — root cause: MLFLOW_TRACKING_URI was set pointing to mlflow.zakobar.com, server went down mid-training causing MlflowException (502s then DNS failure); bce-weighted (3/3) and mae (3/3) were complete, gce-0.7 ep0 saved, ep1 dir empty
• Fixed MLflow crash vector: (1) unset MLFLOW_TRACKING_URI at top of resume script; (2) added try/except in _MLFlowRun.track to suppress per-metric exceptions without crashing training
• Created experiment-food101n-rerun-weighted-resume.py with per-episode skip logic (checks test-res.pkl existence); launched detached on beacon-wsl; confirmed GPU active (RTX 3050), gce-0.7 ep1+2 and focal-loss ep0 completed, focal-loss ep1 in progress as of last check (~35h remaining)
• mac (keel noise run from Aug 26): fully completed — all 12 datasets × 3 noise levels × 8 configs × 10 episodes done
• Fetched keel noise results from mac; flattened results-final/keel-noise-{05,10,20}/ structure (init_experiment had inserted timestamp subdirs; removed them, kept latest Aug-26 run per dataset, discarded stale Aug-01 partial dirs)
• Discovered roll-aoc config disadvantage in KEEL runs: max_iters=1500 and patience=100 vs baselines max_iters=5000 / patience=500 (3.3× fewer epochs, 5× less patience); confirmed no override in run_noisy.py or run_all_noisy.sh
• Created experiments/keel/run_noisy_fair_roll.py: reruns roll-aoc only across all 12 datasets × 3 noise levels × 10 episodes with fair config (max_iters=5000, patience=500); has skip logic; writes to results-fair-roll/keel-noise-{05,10,20}/<dataset>/roll-aoc/; launched detached on mac (CPU, no MLflow)
• Plan: when results-fair-roll is done, swap roll-aoc/ dirs into results-final/keel-noise-*/ to replace unfair results

2026-08-01 Fri

• Checked experiment status: beacon-wsl running food101n (5/21 episodes done, bce-weighted ep3 in progress); Mac idle — chestxray14 had crashed Jul 31 20:51
• Diagnosed chestxray14 crash: ValueError: All episodes have NaN/inf predictions — cannot build ROC curve during ROC summary generation after libauc-auroc training; non-libauc configs may have completed; logged as TODO with resume notes
• Set up KEEL label noise ablation (5%, 10%, 20%) × 12 datasets × 10 episodes → results-final/keel-noise-{05,10,20}/; created experiments/keel/run_all_noise_levels.sh
• Added --results-dir, --cpu, --no-mp flags to run_noisy.py and run_all_noisy.sh; added force_cpu param to run_keel_experiment in _base.py
• Benchmarked MPS vs CPU for KEEL-sized batches (72 samples, 9 features, 3-layer MLP): CPU is 6.6x faster — MPS kernel dispatch overhead dominates at tiny batch sizes
• Discovered sequential_episodes=True is dead code for CPU + is_mp=True path: flag only gates the MPS subprocess branch; CPU always uses mp.Pool(n_episodes) when is_mp=True, regardless of the flag
• Discovered: MLflow was connected to remote server in the original MPS run, adding per-epoch network I/O that dominated wall time (~1.6s/epoch vs 54ms without)
• Changed gen_criteria logging from logging.info to logging.debug (reduces StreamHandler flush overhead per epoch; per-epoch losses still captured in debug.log)
• Settled on --cpu --no-mp + unset MLFLOW_TRACKING_URI for all KEEL noisy runs; hardcoded in run_all_noise_levels.sh
• Identified residual per-epoch overhead: ~54ms (ExperimentDataLoader shuffle+argwhere per epoch, pd.DataFrame construction ×2 in gen_criteria, extra loss forward in _get_loss) vs 0.5ms raw compute
• Launched final KEEL noise ablation on Mac: CPU, no-mp, no MLflow; estimated ~21h for all 3 noise levels

2026-07-24 Thu

• Investigated stalled cifar10n remote run: 5 workers from 6:45AM were still alive after 15.5h with 0 results; previous session's ps check had missed them, making the run appear dead
• Diagnosed OOM: memory at ~46MB free at 7h mark with 5 concurrent MPS workers on 16GB; new run with 3 workers also OOM'd after 8.5h (26GB compressed pages, 5B decompressions)
• Fixed root cause: added sequential_episodes flag to run_configurations and _perform_multiple_episodes in src/experiment.py — when True, bypasses the subprocess-parallel path and runs episodes one at a time in-process on MPS; cifar10n now uses this
• Killed 3-worker subprocess run on remote; restarted cifar10n with sequential_episodes=True — now running single-process, one episode at a time
• Added kaggle to flake.nix buildInputs; set up ~/.config/kaggle/kaggle.json (user: anerzakobar); downloaded creditcard.csv (144MB) to ~/.data/creditcard/
• Rewrote experiments/large/experiment-creditcard.py: replaced old ad-hoc FPR configs with full baseline suite matching KEEL/_base.py — roll-aoc, roll-tpr90, bce-weighted, libauc-auroc, gce-0.7, mae, focal-loss, asymmetric-loss; StepLR on all configs; N_EPISODES=5
• Launched creditcard experiment locally in background (5 episodes × 8 configs, CPU multiprocessing)
• Reviewed KEEL AUC results table: libauc-auroc wins 6/12 datasets (mean 0.890); asymmetric-loss strong second (0.887); roll-aoc 6th (0.872); roll-tpr90 last (0.842, dragged by new-thyroid1 instability)

Archives