backup: 2026-08-29 06:59

impl.org

@@ -12,14 +12,14 @@ ROLL (Rate Optimized Likelyhood-based Loss) — PyTorch research project impleme
 
 - =src/roll.py= — Loss function implementations (Normal/Beta/Kernelized ROLL)
 - =src/experiment.py= — Training loop, evaluation infra, =run_configurations()= entry point
-- =src/datasets.py= — Dataset loaders incl. =Cifar10NDataset=, =ImbalancedCifar10Dataset=
-- =src/networks.py= — =KeelNet= MLP architecture + =ConvNet= (moved from experiment scripts for subprocess pickle compatibility)
+- =src/datasets.py= — Dataset loaders incl. =Cifar10NDataset=, =Animal10NDataset=, =Food101NDataset=
+- =src/networks.py= — =KeelNet= MLP + =ConvNet(image_size=32|64)= (parameterized for 32×32 or 64×64)
 - =src/utils.py= — Logging, =init_experiment=, =get_device()= (CUDA/MPS/CPU)
 - =src/_episode_worker.py= — Subprocess entry point for parallel MPS episode execution
 - =experiments/keel/= — KEEL experiment scripts; =_base.py= shared runner
-- =experiments/other/= — CIFAR-10, CIFAR-10N, imbalanced CIFAR-10, adult, gaussian, etc.
-- =experiments/large/= — Higgs, credit card, home credit
-- =scripts/= — Remote runner: =setup_remote.sh=, =run_remote.sh=, =fetch_results.sh=, =tail_remote.sh=
+- =experiments/other/= — CIFAR-10, CIFAR-10N, ANIMAL-10N, adult, gaussian, etc.
+- =experiments/large/= — Higgs, credit card, home credit, Food-101N
+- =scripts/= — Remote runners: =run_remote.sh= (Mac), =run_remote_beacon.sh= (beacon-wsl)
 
 ** Conventions
 
@@ -31,37 +31,48 @@ ROLL (Rate Optimized Likelyhood-based Loss) — PyTorch research project impleme
 - =get_device()= auto-selects CUDA → Apple MPS → CPU
 - *Two-tier results strategy*: =results/= holds ephemeral date-stamped run dirs (deletable); =results-final/= holds keeper results for the thesis — one flat copy per config per dataset, no date subfolders. Both are in =.gitignore=.
 - Resume scripts (=experiment-*-resume.py=) write directly into =results-final/<dataset>/= to land in the right place immediately. Delete after run completes.
+- ConvNet for 64×64 images: use =partial(ConvNet, image_size=64)= as net_creator (gives 4096-feature linear layer)
+- *KEEL runs*: always use =--cpu --no-mp= + =unset MLFLOW_TRACKING_URI= — CPU is 6.6x faster than MPS for tiny batches; mp.Pool spawn overhead exceeds episode runtime; MLflow adds per-epoch I/O
 
 ** Gotchas
 
 - Dataset paths injected as env vars by Nix shell hook (=$keel_wisconsin_dir=, etc.) — must use =nix develop= on Linux
-- =_calc_moments()= in roll.py is unused and has a variable typo (=array= vs =arr=)
 - CIFAR-10 binary: class 1 (automobile) vs rest; natural IR ~9
 - MPS tensors cannot cross multiprocessing process boundaries — =run_configurations()= auto-disables =is_mp= when =device.type == 'mps'=
-- MPS concurrency sweet spot: N=3 independent subprocesses for cifar10n on 16GB M1 Pro; N=5 caused OOM after ~15h.
-- cifar10n on remote now runs with =sequential_episodes=True= (one episode at a time in-process) to avoid MPS unified memory exhaustion
-- Subprocess workers (=_episode_worker.py=) write stdout+stderr to per-episode =worker.log= — 0 bytes is normal for successful runs (only WARNING+ logged)
-- Kaggle CLI: available via =nix run nixpkgs#kaggle= or inside =nix develop= (added to =flake.nix= buildInputs). Credentials at =~/.config/kaggle/kaggle.json= (username: anerzakobar). creditcard.csv downloaded to =~/.data/creditcard/creditcard.csv=.
-- Large datasets (creditcard, homecredit, higgs) are manual downloads; =credit_card_fraud_dir= / =home_credit_dir= env vars set by shell hook to =~/.data/...=
+- cifar10n on remote runs with =sequential_episodes=True= to avoid MPS unified memory exhaustion
+- LibAUC does not support MPS — force =device=torch.device('cpu')= for libauc-auroc on Mac
+- On beacon-wsl, first =nix develop= takes ~10 min (downloads all store paths incl. CUDA PyTorch)
+- ANIMAL-10N download requires filling Google Form: https://forms.gle/8mbmbNgDFQ2rA1fLA
+- =sequential_episodes=True= is dead code for CPU + =is_mp=True=: the flag only gates the MPS subprocess branch; CPU always spawns =mp.Pool(n_episodes)= when =is_mp=True=
+- Per-epoch Python overhead in KEEL: ~54ms (ExperimentDataLoader shuffle+argwhere, =pd.DataFrame= ×2 in =gen_criteria=, extra loss forward in =_get_loss=) vs 0.5ms raw compute — not worth optimising unless runtime becomes critical
+- =gen_criteria= logs per-epoch loss at DEBUG level (changed from INFO 2026-08-01) — visible in =debug.log= per run dir, not console
+- MLflow: if =MLFLOW_TRACKING_URI= is set and points to a live server, per-epoch =log_metric= calls add significant overhead — always =unset MLFLOW_TRACKING_URI= for KEEL
 
-** Remote Machine
+** Remote Machines
 
-- Host: =chenzakobar@192.168.1.190= (Mac OS, Python 3.13 via Homebrew)
-- SSH key: =~/.ssh/roll_remote= — dedicated ed25519 key, NOT the YubiKey/GPG key
-- Remote project dir: =~/roll-impl=; venv: =~/roll-venv=; env vars: =~/roll-env.sh=
-- Use =--detach= flag with =run_remote.sh= for long experiments (survives sleep/disconnect)
-- Kill a detached run: =ssh -i ~/.ssh/roll_remote -o IdentitiesOnly=yes chenzakobar@192.168.1.190 'pkill -f <script-name>'=
+- *Mac* — =chenzakobar@192.168.1.190= (macOS, Python 3.13 venv at =~/roll-venv=, Apple MPS)
+  - SSH key: =~/.ssh/roll_remote= (dedicated ed25519, NOT YubiKey)
+  - Runner: =scripts/run_remote.sh= + =scripts/fetch_results.sh=
+  - Kill detached run: =ssh ... 'pkill -f <script-name>'=
+- *beacon-wsl* — =aner@192.168.1.200= (NixOS-WSL, RTX 3050 6GB VRAM, CUDA)
+  - SSH key: YubiKey GPG key via agent (default SSH config)
+  - Runner: =scripts/run_remote_beacon.sh= + =scripts/fetch_results_beacon.sh=
+  - Uses =nix develop --command python= (no venv; flake has =cudaSupport=true=)
+  - Kaggle creds copied to =~/.config/kaggle/kaggle.json=
 
 ** Key Files
 
 - [[file:src/roll.py][src/roll.py]] — Core loss: =KernelizedROLLoss= custom autograd Function with KDE backward pass
 - [[file:src/experiment.py][src/experiment.py]] — =ExperimentConfiguration= dataclass, =Criteriorator= ABC, =run_configurations()=
 - [[file:experiments/keel/_base.py][experiments/keel/_base.py]] — shared KEEL runner; config suite: roll-aoc, roll-tpr90, bce-weighted, libauc-auroc, gce-0.7, mae, focal-loss, asymmetric-loss
+- [[file:experiments/keel/run_all_noise_levels.sh][experiments/keel/run_all_noise_levels.sh]] — runs 5%/10%/20% noise × all 12 datasets × 10 episodes; hardcodes =--cpu --no-mp=, unsets MLflow
 - [[file:experiments/other/experiment-cifar10n.py][experiments/other/experiment-cifar10n.py]] — CIFAR-10N (noisy labels: clean/aggre/worse), sequential_episodes=True
-- [[file:experiments/large/experiment-creditcard.py][experiments/large/experiment-creditcard.py]] — Credit card fraud dataset
-- [[file:scripts/run_remote.sh][scripts/run_remote.sh]] — rsync + run on remote Mac; =--detach= for sleep-safe long runs; excludes =results-final/= from push, includes it in fetch
-- [[file:flake.nix][flake.nix]] — Nix env; kaggle + sshpass in buildInputs
-- =results-final/cifar10n/= — Keeper results: clean/aggre/worse × 7 configs × 3 episodes. As of 2026-07-25: clean+aggre complete; worse-roll-aoc/bce-weighted/mae complete; worse-gce-0.7/focal-loss/asymmetric-loss/libauc-auroc running on remote.
+- [[file:experiments/other/experiment-animal10n.py][experiments/other/experiment-animal10n.py]] — ANIMAL-10N (human noise ~8%), runs on Mac MPS
+- [[file:experiments/large/experiment-food101n.py][experiments/large/experiment-food101n.py]] — Food-101N (web noise ~18%, 7GB), runs on beacon-wsl CUDA
+- [[file:scripts/run_remote.sh][scripts/run_remote.sh]] — rsync + run on remote Mac; =--detach= for sleep-safe long runs
+- [[file:scripts/run_remote_beacon.sh][scripts/run_remote_beacon.sh]] — rsync + run on beacon-wsl via =nix develop=
+- [[file:flake.nix][flake.nix]] — Nix env; kaggle + sshpass in buildInputs; =cudaSupport=true=
+- =results-final/cifar10n/= — Complete: clean/aggre/worse × 7 configs × 3 episodes (as of 2026-07-26)
 
 ** Subnodes