74 lines
5.6 KiB
Org Mode
74 lines
5.6 KiB
Org Mode
:PROPERTIES:
|
||
:ID: b362dad3-19b7-486a-8d26-128a83643924
|
||
:END:
|
||
|
||
#+title: impl
|
||
|
||
* impl
|
||
|
||
ROLL (Rate Optimized Likelyhood-based Loss) — PyTorch research project implementing custom loss functions for binary classification using kernel density estimation (KDE) to optimize TPR at target FPR thresholds. Targets imbalanced classification problems.
|
||
|
||
** Architecture
|
||
|
||
- =src/roll.py= — Loss function implementations (Normal/Beta/Kernelized ROLL)
|
||
- =src/experiment.py= — Training loop, evaluation infra, =run_configurations()= entry point
|
||
- =src/datasets.py= — Dataset loaders incl. =Cifar10NDataset=, =ImbalancedCifar10Dataset=
|
||
- =src/networks.py= — =KeelNet= MLP architecture + =ConvNet= (moved from experiment scripts for subprocess pickle compatibility)
|
||
- =src/utils.py= — Logging, =init_experiment=, =get_device()= (CUDA/MPS/CPU)
|
||
- =src/_episode_worker.py= — Subprocess entry point for parallel MPS episode execution
|
||
- =experiments/keel/= — KEEL experiment scripts; =_base.py= shared runner
|
||
- =experiments/other/= — CIFAR-10, CIFAR-10N, imbalanced CIFAR-10, adult, gaussian, etc.
|
||
- =experiments/large/= — Higgs, credit card, home credit
|
||
- =scripts/= — Remote runner: =setup_remote.sh=, =run_remote.sh=, =fetch_results.sh=, =tail_remote.sh=
|
||
|
||
** Conventions
|
||
|
||
- All hyperparameters in Python dataclasses (=ExperimentConfiguration=), no CLI parsing
|
||
- Experiments follow =experiment-*.py= naming; all call =run_configurations()=
|
||
- KEEL experiments share =experiments/keel/_base.py= runner; individual files just call it
|
||
- All datasets expose: =__getitem__=, =__len__=, =.x=, =.y= attributes
|
||
- Episode-based eval: N independent train runs per config, results aggregated
|
||
- =get_device()= auto-selects CUDA → Apple MPS → CPU
|
||
- *Two-tier results strategy*: =results/= holds ephemeral date-stamped run dirs (deletable); =results-final/= holds keeper results for the thesis — one flat copy per config per dataset, no date subfolders. Both are in =.gitignore=.
|
||
- Resume scripts (=experiment-*-resume.py=) write directly into =results-final/<dataset>/= to land in the right place immediately. Delete after run completes.
|
||
|
||
** Gotchas
|
||
|
||
- Dataset paths injected as env vars by Nix shell hook (=$keel_wisconsin_dir=, etc.) — must use =nix develop= on Linux
|
||
- =_calc_moments()= in roll.py is unused and has a variable typo (=array= vs =arr=)
|
||
- CIFAR-10 binary: class 1 (automobile) vs rest; natural IR ~9
|
||
- MPS tensors cannot cross multiprocessing process boundaries — =run_configurations()= auto-disables =is_mp= when =device.type == 'mps'=
|
||
- MPS concurrency sweet spot: N=3 independent subprocesses for cifar10n on 16GB M1 Pro; N=5 caused OOM after ~15h.
|
||
- cifar10n on remote now runs with =sequential_episodes=True= (one episode at a time in-process) to avoid MPS unified memory exhaustion
|
||
- Subprocess workers (=_episode_worker.py=) write stdout+stderr to per-episode =worker.log= — 0 bytes is normal for successful runs (only WARNING+ logged)
|
||
- Kaggle CLI: available via =nix run nixpkgs#kaggle= or inside =nix develop= (added to =flake.nix= buildInputs). Credentials at =~/.config/kaggle/kaggle.json= (username: anerzakobar). creditcard.csv downloaded to =~/.data/creditcard/creditcard.csv=.
|
||
- Large datasets (creditcard, homecredit, higgs) are manual downloads; =credit_card_fraud_dir= / =home_credit_dir= env vars set by shell hook to =~/.data/...=
|
||
|
||
** Remote Machine
|
||
|
||
- Host: =chenzakobar@192.168.1.190= (Mac OS, Python 3.13 via Homebrew)
|
||
- SSH key: =~/.ssh/roll_remote= — dedicated ed25519 key, NOT the YubiKey/GPG key
|
||
- Remote project dir: =~/roll-impl=; venv: =~/roll-venv=; env vars: =~/roll-env.sh=
|
||
- Use =--detach= flag with =run_remote.sh= for long experiments (survives sleep/disconnect)
|
||
- Kill a detached run: =ssh -i ~/.ssh/roll_remote -o IdentitiesOnly=yes chenzakobar@192.168.1.190 'pkill -f <script-name>'=
|
||
|
||
** Key Files
|
||
|
||
- [[file:src/roll.py][src/roll.py]] — Core loss: =KernelizedROLLoss= custom autograd Function with KDE backward pass
|
||
- [[file:src/experiment.py][src/experiment.py]] — =ExperimentConfiguration= dataclass, =Criteriorator= ABC, =run_configurations()=
|
||
- [[file:experiments/keel/_base.py][experiments/keel/_base.py]] — shared KEEL runner; config suite: roll-aoc, roll-tpr90, bce-weighted, libauc-auroc, gce-0.7, mae, focal-loss, asymmetric-loss
|
||
- [[file:experiments/other/experiment-cifar10n.py][experiments/other/experiment-cifar10n.py]] — CIFAR-10N (noisy labels: clean/aggre/worse), sequential_episodes=True
|
||
- [[file:experiments/large/experiment-creditcard.py][experiments/large/experiment-creditcard.py]] — Credit card fraud dataset
|
||
- [[file:scripts/run_remote.sh][scripts/run_remote.sh]] — rsync + run on remote Mac; =--detach= for sleep-safe long runs; excludes =results-final/= from push, includes it in fetch
|
||
- [[file:flake.nix][flake.nix]] — Nix env; kaggle + sshpass in buildInputs
|
||
- =results-final/cifar10n/= — Keeper results: clean/aggre/worse × 7 configs × 3 episodes. As of 2026-07-25: clean+aggre complete; worse-roll-aoc/bce-weighted/mae complete; worse-gce-0.7/focal-loss/asymmetric-loss/libauc-auroc running on remote.
|
||
|
||
** Subnodes
|
||
|
||
- [[id:001430d5-e1e7-4e72-baf6-17399bfd6447][impl/loss-functions]] — Loss variants, KDE internals, gradient computation
|
||
- [[id:a53cbe84-cd8d-45c2-a8cf-34ab520a3ea5][impl/experiments]] — Experiment structure, training flow, metrics, output layout
|
||
- [[id:b8a9886a-d349-43e5-a745-817a148c1fd8][impl/datasets]] — Dataset catalog, KEEL list, eval metrics
|
||
- [[id:151d5686-6f40-4158-a59a-b0be94cdc969][impl/research]] — Literature survey: competing methods, dataset gaps, key papers
|
||
- [[id:fdc18323-e9c2-4fc6-ace5-065035d30c51][impl/todos]] — Project TODOs
|
||
- [[id:863ca60e-6a3c-45d2-86fd-fb4ed386e31b][impl/work-history/recent]] — Daily work log
|