aner/roam
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

backup: 2026-08-29 06:59

Browse Source
This commit is contained in:
Aner Zakobar
2026-08-29 06:59:35 +03:00
parent 07112f1a4c
commit 2a872c725b
19 changed files with 823 additions and 136 deletions
Download Patch File Download Diff File Expand all files Collapse all files
implwork-historyrecent.org
+26
View File
@@ -5,6 +5,32 @@
#+title: impl/work-history/recent
#+filetags: :project: :work-history:
* 2026-08-28 Thu
- Returned from vacation; checked both remote machines for run status
- beacon-wsl (food101n =experiment-food101n-rerun-weighted.py=): found crashed during gce-0.7 ep1 ~2 weeks ago — root cause: =MLFLOW_TRACKING_URI= was set pointing to =mlflow.zakobar.com=, server went down mid-training causing =MlflowException= (502s then DNS failure); bce-weighted (3/3) and mae (3/3) were complete, gce-0.7 ep0 saved, ep1 dir empty
- Fixed MLflow crash vector: (1) unset =MLFLOW_TRACKING_URI= at top of resume script; (2) added try/except in =_MLFlowRun.track= to suppress per-metric exceptions without crashing training
- Created =experiment-food101n-rerun-weighted-resume.py= with per-episode skip logic (checks =test-res.pkl= existence); launched detached on beacon-wsl; confirmed GPU active (RTX 3050), gce-0.7 ep1+2 and focal-loss ep0 completed, focal-loss ep1 in progress as of last check (~35h remaining)
- mac (keel noise run from Aug 26): fully completed — all 12 datasets × 3 noise levels × 8 configs × 10 episodes done
- Fetched keel noise results from mac; flattened =results-final/keel-noise-{05,10,20}/= structure (=init_experiment= had inserted timestamp subdirs; removed them, kept latest Aug-26 run per dataset, discarded stale Aug-01 partial dirs)
- Discovered roll-aoc config disadvantage in KEEL runs: =max_iters=1500= and =patience=100= vs baselines =max_iters=5000= / =patience=500= (3.3× fewer epochs, 5× less patience); confirmed no override in =run_noisy.py= or =run_all_noisy.sh=
- Created =experiments/keel/run_noisy_fair_roll.py=: reruns roll-aoc only across all 12 datasets × 3 noise levels × 10 episodes with fair config (=max_iters=5000=, =patience=500=); has skip logic; writes to =results-fair-roll/keel-noise-{05,10,20}/<dataset>/roll-aoc/=; launched detached on mac (CPU, no MLflow)
- Plan: when =results-fair-roll= is done, swap =roll-aoc/= dirs into =results-final/keel-noise-*/= to replace unfair results
* 2026-08-01 Fri
- Checked experiment status: beacon-wsl running food101n (5/21 episodes done, bce-weighted ep3 in progress); Mac idle — chestxray14 had crashed Jul 31 20:51
- Diagnosed chestxray14 crash: =ValueError: All episodes have NaN/inf predictions — cannot build ROC curve= during ROC summary generation after libauc-auroc training; non-libauc configs may have completed; logged as TODO with resume notes
- Set up KEEL label noise ablation (5%, 10%, 20%) × 12 datasets × 10 episodes → =results-final/keel-noise-{05,10,20}/=; created =experiments/keel/run_all_noise_levels.sh=
- Added =--results-dir=, =--cpu=, =--no-mp= flags to =run_noisy.py= and =run_all_noisy.sh=; added =force_cpu= param to =run_keel_experiment= in =_base.py=
- Benchmarked MPS vs CPU for KEEL-sized batches (72 samples, 9 features, 3-layer MLP): CPU is 6.6x faster — MPS kernel dispatch overhead dominates at tiny batch sizes
- Discovered =sequential_episodes=True= is dead code for CPU + =is_mp=True= path: flag only gates the MPS subprocess branch; CPU always uses =mp.Pool(n_episodes)= when =is_mp=True=, regardless of the flag
- Discovered: MLflow was connected to remote server in the original MPS run, adding per-epoch network I/O that dominated wall time (~1.6s/epoch vs 54ms without)
- Changed =gen_criteria= logging from =logging.info= to =logging.debug= (reduces StreamHandler flush overhead per epoch; per-epoch losses still captured in debug.log)
- Settled on =--cpu --no-mp= + =unset MLFLOW_TRACKING_URI= for all KEEL noisy runs; hardcoded in =run_all_noise_levels.sh=
- Identified residual per-epoch overhead: ~54ms (ExperimentDataLoader shuffle+argwhere per epoch, =pd.DataFrame= construction ×2 in =gen_criteria=, extra loss forward in =_get_loss=) vs 0.5ms raw compute
- Launched final KEEL noise ablation on Mac: CPU, no-mp, no MLflow; estimated ~21h for all 3 noise levels
* 2026-07-24 Thu
- Investigated stalled cifar10n remote run: 5 workers from 6:45AM were still alive after 15.5h with 0 results; previous session's ps check had missed them, making the run appear dead