46 lines
5.4 KiB
Org Mode
46 lines
5.4 KiB
Org Mode
:PROPERTIES:
|
||
:ID: 863ca60e-6a3c-45d2-86fd-fb4ed386e31b
|
||
:END:
|
||
|
||
#+title: impl/work-history/recent
|
||
#+filetags: :project: :work-history:
|
||
|
||
* 2026-08-28 Thu
|
||
|
||
- Returned from vacation; checked both remote machines for run status
|
||
- beacon-wsl (food101n =experiment-food101n-rerun-weighted.py=): found crashed during gce-0.7 ep1 ~2 weeks ago — root cause: =MLFLOW_TRACKING_URI= was set pointing to =mlflow.zakobar.com=, server went down mid-training causing =MlflowException= (502s then DNS failure); bce-weighted (3/3) and mae (3/3) were complete, gce-0.7 ep0 saved, ep1 dir empty
|
||
- Fixed MLflow crash vector: (1) unset =MLFLOW_TRACKING_URI= at top of resume script; (2) added try/except in =_MLFlowRun.track= to suppress per-metric exceptions without crashing training
|
||
- Created =experiment-food101n-rerun-weighted-resume.py= with per-episode skip logic (checks =test-res.pkl= existence); launched detached on beacon-wsl; confirmed GPU active (RTX 3050), gce-0.7 ep1+2 and focal-loss ep0 completed, focal-loss ep1 in progress as of last check (~35h remaining)
|
||
- mac (keel noise run from Aug 26): fully completed — all 12 datasets × 3 noise levels × 8 configs × 10 episodes done
|
||
- Fetched keel noise results from mac; flattened =results-final/keel-noise-{05,10,20}/= structure (=init_experiment= had inserted timestamp subdirs; removed them, kept latest Aug-26 run per dataset, discarded stale Aug-01 partial dirs)
|
||
- Discovered roll-aoc config disadvantage in KEEL runs: =max_iters=1500= and =patience=100= vs baselines =max_iters=5000= / =patience=500= (3.3× fewer epochs, 5× less patience); confirmed no override in =run_noisy.py= or =run_all_noisy.sh=
|
||
- Created =experiments/keel/run_noisy_fair_roll.py=: reruns roll-aoc only across all 12 datasets × 3 noise levels × 10 episodes with fair config (=max_iters=5000=, =patience=500=); has skip logic; writes to =results-fair-roll/keel-noise-{05,10,20}/<dataset>/roll-aoc/=; launched detached on mac (CPU, no MLflow)
|
||
- Plan: when =results-fair-roll= is done, swap =roll-aoc/= dirs into =results-final/keel-noise-*/= to replace unfair results
|
||
|
||
* 2026-08-01 Fri
|
||
|
||
- Checked experiment status: beacon-wsl running food101n (5/21 episodes done, bce-weighted ep3 in progress); Mac idle — chestxray14 had crashed Jul 31 20:51
|
||
- Diagnosed chestxray14 crash: =ValueError: All episodes have NaN/inf predictions — cannot build ROC curve= during ROC summary generation after libauc-auroc training; non-libauc configs may have completed; logged as TODO with resume notes
|
||
- Set up KEEL label noise ablation (5%, 10%, 20%) × 12 datasets × 10 episodes → =results-final/keel-noise-{05,10,20}/=; created =experiments/keel/run_all_noise_levels.sh=
|
||
- Added =--results-dir=, =--cpu=, =--no-mp= flags to =run_noisy.py= and =run_all_noisy.sh=; added =force_cpu= param to =run_keel_experiment= in =_base.py=
|
||
- Benchmarked MPS vs CPU for KEEL-sized batches (72 samples, 9 features, 3-layer MLP): CPU is 6.6x faster — MPS kernel dispatch overhead dominates at tiny batch sizes
|
||
- Discovered =sequential_episodes=True= is dead code for CPU + =is_mp=True= path: flag only gates the MPS subprocess branch; CPU always uses =mp.Pool(n_episodes)= when =is_mp=True=, regardless of the flag
|
||
- Discovered: MLflow was connected to remote server in the original MPS run, adding per-epoch network I/O that dominated wall time (~1.6s/epoch vs 54ms without)
|
||
- Changed =gen_criteria= logging from =logging.info= to =logging.debug= (reduces StreamHandler flush overhead per epoch; per-epoch losses still captured in debug.log)
|
||
- Settled on =--cpu --no-mp= + =unset MLFLOW_TRACKING_URI= for all KEEL noisy runs; hardcoded in =run_all_noise_levels.sh=
|
||
- Identified residual per-epoch overhead: ~54ms (ExperimentDataLoader shuffle+argwhere per epoch, =pd.DataFrame= construction ×2 in =gen_criteria=, extra loss forward in =_get_loss=) vs 0.5ms raw compute
|
||
- Launched final KEEL noise ablation on Mac: CPU, no-mp, no MLflow; estimated ~21h for all 3 noise levels
|
||
|
||
* 2026-07-24 Thu
|
||
|
||
- Investigated stalled cifar10n remote run: 5 workers from 6:45AM were still alive after 15.5h with 0 results; previous session's ps check had missed them, making the run appear dead
|
||
- Diagnosed OOM: memory at ~46MB free at 7h mark with 5 concurrent MPS workers on 16GB; new run with 3 workers also OOM'd after 8.5h (26GB compressed pages, 5B decompressions)
|
||
- Fixed root cause: added =sequential_episodes= flag to =run_configurations= and =_perform_multiple_episodes= in =src/experiment.py= — when True, bypasses the subprocess-parallel path and runs episodes one at a time in-process on MPS; cifar10n now uses this
|
||
- Killed 3-worker subprocess run on remote; restarted cifar10n with =sequential_episodes=True= — now running single-process, one episode at a time
|
||
- Added =kaggle= to =flake.nix= buildInputs; set up =~/.config/kaggle/kaggle.json= (user: anerzakobar); downloaded creditcard.csv (144MB) to =~/.data/creditcard/=
|
||
- Rewrote =experiments/large/experiment-creditcard.py=: replaced old ad-hoc FPR configs with full baseline suite matching KEEL/_base.py — roll-aoc, roll-tpr90, bce-weighted, libauc-auroc, gce-0.7, mae, focal-loss, asymmetric-loss; StepLR on all configs; N_EPISODES=5
|
||
- Launched creditcard experiment locally in background (5 episodes × 8 configs, CPU multiprocessing)
|
||
- Reviewed KEEL AUC results table: libauc-auroc wins 6/12 datasets (mean 0.890); asymmetric-loss strong second (0.887); roll-aoc 6th (0.872); roll-tpr90 last (0.842, dragged by new-thyroid1 instability)
|
||
|
||
** Archives
|