85 lines
6.6 KiB
Org Mode
85 lines
6.6 KiB
Org Mode
:PROPERTIES:
|
||
:ID: b362dad3-19b7-486a-8d26-128a83643924
|
||
:END:
|
||
|
||
#+title: impl
|
||
|
||
* impl
|
||
|
||
ROLL (Rate Optimized Likelyhood-based Loss) — PyTorch research project implementing custom loss functions for binary classification using kernel density estimation (KDE) to optimize TPR at target FPR thresholds. Targets imbalanced classification problems.
|
||
|
||
** Architecture
|
||
|
||
- =src/roll.py= — Loss function implementations (Normal/Beta/Kernelized ROLL)
|
||
- =src/experiment.py= — Training loop, evaluation infra, =run_configurations()= entry point
|
||
- =src/datasets.py= — Dataset loaders incl. =Cifar10NDataset=, =Animal10NDataset=, =Food101NDataset=
|
||
- =src/networks.py= — =KeelNet= MLP + =ConvNet(image_size=32|64)= (parameterized for 32×32 or 64×64)
|
||
- =src/utils.py= — Logging, =init_experiment=, =get_device()= (CUDA/MPS/CPU)
|
||
- =src/_episode_worker.py= — Subprocess entry point for parallel MPS episode execution
|
||
- =experiments/keel/= — KEEL experiment scripts; =_base.py= shared runner
|
||
- =experiments/other/= — CIFAR-10, CIFAR-10N, ANIMAL-10N, adult, gaussian, etc.
|
||
- =experiments/large/= — Higgs, credit card, home credit, Food-101N
|
||
- =scripts/= — Remote runners: =run_remote.sh= (Mac), =run_remote_beacon.sh= (beacon-wsl)
|
||
|
||
** Conventions
|
||
|
||
- All hyperparameters in Python dataclasses (=ExperimentConfiguration=), no CLI parsing
|
||
- Experiments follow =experiment-*.py= naming; all call =run_configurations()=
|
||
- KEEL experiments share =experiments/keel/_base.py= runner; individual files just call it
|
||
- All datasets expose: =__getitem__=, =__len__=, =.x=, =.y= attributes
|
||
- Episode-based eval: N independent train runs per config, results aggregated
|
||
- =get_device()= auto-selects CUDA → Apple MPS → CPU
|
||
- *Two-tier results strategy*: =results/= holds ephemeral date-stamped run dirs (deletable); =results-final/= holds keeper results for the thesis — one flat copy per config per dataset, no date subfolders. Both are in =.gitignore=.
|
||
- Resume scripts (=experiment-*-resume.py=) write directly into =results-final/<dataset>/= to land in the right place immediately. Delete after run completes.
|
||
- ConvNet for 64×64 images: use =partial(ConvNet, image_size=64)= as net_creator (gives 4096-feature linear layer)
|
||
- *KEEL runs*: always use =--cpu --no-mp= + =unset MLFLOW_TRACKING_URI= — CPU is 6.6x faster than MPS for tiny batches; mp.Pool spawn overhead exceeds episode runtime; MLflow adds per-epoch I/O
|
||
|
||
** Gotchas
|
||
|
||
- Dataset paths injected as env vars by Nix shell hook (=$keel_wisconsin_dir=, etc.) — must use =nix develop= on Linux
|
||
- CIFAR-10 binary: class 1 (automobile) vs rest; natural IR ~9
|
||
- MPS tensors cannot cross multiprocessing process boundaries — =run_configurations()= auto-disables =is_mp= when =device.type == 'mps'=
|
||
- cifar10n on remote runs with =sequential_episodes=True= to avoid MPS unified memory exhaustion
|
||
- LibAUC does not support MPS — force =device=torch.device('cpu')= for libauc-auroc on Mac
|
||
- On beacon-wsl, first =nix develop= takes ~10 min (downloads all store paths incl. CUDA PyTorch)
|
||
- ANIMAL-10N download requires filling Google Form: https://forms.gle/8mbmbNgDFQ2rA1fLA
|
||
- =sequential_episodes=True= is dead code for CPU + =is_mp=True=: the flag only gates the MPS subprocess branch; CPU always spawns =mp.Pool(n_episodes)= when =is_mp=True=
|
||
- Per-epoch Python overhead in KEEL: ~54ms (ExperimentDataLoader shuffle+argwhere, =pd.DataFrame= ×2 in =gen_criteria=, extra loss forward in =_get_loss=) vs 0.5ms raw compute — not worth optimising unless runtime becomes critical
|
||
- =gen_criteria= logs per-epoch loss at DEBUG level (changed from INFO 2026-08-01) — visible in =debug.log= per run dir, not console
|
||
- MLflow: if =MLFLOW_TRACKING_URI= is set and points to a live server, per-epoch =log_metric= calls add significant overhead — always =unset MLFLOW_TRACKING_URI= for KEEL
|
||
|
||
** Remote Machines
|
||
|
||
- *Mac* — =chenzakobar@192.168.1.190= (macOS, Python 3.13 venv at =~/roll-venv=, Apple MPS)
|
||
- SSH key: =~/.ssh/roll_remote= (dedicated ed25519, NOT YubiKey)
|
||
- Runner: =scripts/run_remote.sh= + =scripts/fetch_results.sh=
|
||
- Kill detached run: =ssh ... 'pkill -f <script-name>'=
|
||
- *beacon-wsl* — =aner@192.168.1.200= (NixOS-WSL, RTX 3050 6GB VRAM, CUDA)
|
||
- SSH key: YubiKey GPG key via agent (default SSH config)
|
||
- Runner: =scripts/run_remote_beacon.sh= + =scripts/fetch_results_beacon.sh=
|
||
- Uses =nix develop --command python= (no venv; flake has =cudaSupport=true=)
|
||
- Kaggle creds copied to =~/.config/kaggle/kaggle.json=
|
||
|
||
** Key Files
|
||
|
||
- [[file:src/roll.py][src/roll.py]] — Core loss: =KernelizedROLLoss= custom autograd Function with KDE backward pass
|
||
- [[file:src/experiment.py][src/experiment.py]] — =ExperimentConfiguration= dataclass, =Criteriorator= ABC, =run_configurations()=
|
||
- [[file:experiments/keel/_base.py][experiments/keel/_base.py]] — shared KEEL runner; config suite: roll-aoc, roll-tpr90, bce-weighted, libauc-auroc, gce-0.7, mae, focal-loss, asymmetric-loss
|
||
- [[file:experiments/keel/run_all_noise_levels.sh][experiments/keel/run_all_noise_levels.sh]] — runs 5%/10%/20% noise × all 12 datasets × 10 episodes; hardcodes =--cpu --no-mp=, unsets MLflow
|
||
- [[file:experiments/other/experiment-cifar10n.py][experiments/other/experiment-cifar10n.py]] — CIFAR-10N (noisy labels: clean/aggre/worse), sequential_episodes=True
|
||
- [[file:experiments/other/experiment-animal10n.py][experiments/other/experiment-animal10n.py]] — ANIMAL-10N (human noise ~8%), runs on Mac MPS
|
||
- [[file:experiments/large/experiment-food101n.py][experiments/large/experiment-food101n.py]] — Food-101N (web noise ~18%, 7GB), runs on beacon-wsl CUDA
|
||
- [[file:scripts/run_remote.sh][scripts/run_remote.sh]] — rsync + run on remote Mac; =--detach= for sleep-safe long runs
|
||
- [[file:scripts/run_remote_beacon.sh][scripts/run_remote_beacon.sh]] — rsync + run on beacon-wsl via =nix develop=
|
||
- [[file:flake.nix][flake.nix]] — Nix env; kaggle + sshpass in buildInputs; =cudaSupport=true=
|
||
- =results-final/cifar10n/= — Complete: clean/aggre/worse × 7 configs × 3 episodes (as of 2026-07-26)
|
||
|
||
** Subnodes
|
||
|
||
- [[id:001430d5-e1e7-4e72-baf6-17399bfd6447][impl/loss-functions]] — Loss variants, KDE internals, gradient computation
|
||
- [[id:a53cbe84-cd8d-45c2-a8cf-34ab520a3ea5][impl/experiments]] — Experiment structure, training flow, metrics, output layout
|
||
- [[id:b8a9886a-d349-43e5-a745-817a148c1fd8][impl/datasets]] — Dataset catalog, KEEL list, eval metrics
|
||
- [[id:151d5686-6f40-4158-a59a-b0be94cdc969][impl/research]] — Literature survey: competing methods, dataset gaps, key papers
|
||
- [[id:fdc18323-e9c2-4fc6-ace5-065035d30c51][impl/todos]] — Project TODOs
|
||
- [[id:863ca60e-6a3c-45d2-86fd-fb4ed386e31b][impl/work-history/recent]] — Daily work log
|