backup: 2026-07-25 14:39
This commit is contained in:
@@ -1,3 +1,9 @@
|
||||
:PROPERTIES:
|
||||
:ID: b362dad3-19b7-486a-8d26-128a83643924
|
||||
:END:
|
||||
|
||||
#+title: impl
|
||||
|
||||
* impl
|
||||
|
||||
ROLL (Rate Optimized Likelyhood-based Loss) — PyTorch research project implementing custom loss functions for binary classification using kernel density estimation (KDE) to optimize TPR at target FPR thresholds. Targets imbalanced classification problems.
|
||||
@@ -6,11 +12,14 @@ ROLL (Rate Optimized Likelyhood-based Loss) — PyTorch research project impleme
|
||||
|
||||
- =src/roll.py= — Loss function implementations (Normal/Beta/Kernelized ROLL)
|
||||
- =src/experiment.py= — Training loop, evaluation infra, =run_configurations()= entry point
|
||||
- =src/datasets.py= — 10+ dataset loaders (KEEL, UCI, Kaggle, synthetic)
|
||||
- =src/networks.py= — =KeelNet= MLP architecture
|
||||
- =src/summary.py= — Plotly HTML visualization (ROC, score distributions, ECDF)
|
||||
- =src/utils.py= — Logging, output dir creation (=init_experiment=)
|
||||
- =experiments/keel/=, =experiments/other/=, =experiments/large/= — experiment scripts
|
||||
- =src/datasets.py= — Dataset loaders incl. =Cifar10NDataset=, =ImbalancedCifar10Dataset=
|
||||
- =src/networks.py= — =KeelNet= MLP architecture + =ConvNet= (moved from experiment scripts for subprocess pickle compatibility)
|
||||
- =src/utils.py= — Logging, =init_experiment=, =get_device()= (CUDA/MPS/CPU)
|
||||
- =src/_episode_worker.py= — Subprocess entry point for parallel MPS episode execution
|
||||
- =experiments/keel/= — KEEL experiment scripts; =_base.py= shared runner
|
||||
- =experiments/other/= — CIFAR-10, CIFAR-10N, imbalanced CIFAR-10, adult, gaussian, etc.
|
||||
- =experiments/large/= — Higgs, credit card, home credit
|
||||
- =scripts/= — Remote runner: =setup_remote.sh=, =run_remote.sh=, =fetch_results.sh=, =tail_remote.sh=
|
||||
|
||||
** Conventions
|
||||
|
||||
@@ -19,23 +28,40 @@ ROLL (Rate Optimized Likelyhood-based Loss) — PyTorch research project impleme
|
||||
- KEEL experiments share =experiments/keel/_base.py= runner; individual files just call it
|
||||
- All datasets expose: =__getitem__=, =__len__=, =.x=, =.y= attributes
|
||||
- Episode-based eval: N independent train runs per config, results aggregated
|
||||
- GPU enabled via =cudaSupport = true= in flake.nix; =get_device()= in utils.py auto-selects GPU/CPU
|
||||
- Beta distribution variant (=roll_beta_loss_from_fpr=) is kept for thesis writing but is not actively developed or used
|
||||
- =get_device()= auto-selects CUDA → Apple MPS → CPU
|
||||
- *Two-tier results strategy*: =results/= holds ephemeral date-stamped run dirs (deletable); =results-final/= holds keeper results for the thesis — one flat copy per config per dataset, no date subfolders. Both are in =.gitignore=.
|
||||
- Resume scripts (=experiment-*-resume.py=) write directly into =results-final/<dataset>/= to land in the right place immediately. Delete after run completes.
|
||||
|
||||
** Gotchas
|
||||
|
||||
- Dataset paths injected as env vars by Nix shell hook (=$keel_wisconsin_dir=, etc.) — must use =nix develop=
|
||||
- Dataset paths injected as env vars by Nix shell hook (=$keel_wisconsin_dir=, etc.) — must use =nix develop= on Linux
|
||||
- =_calc_moments()= in roll.py is unused and has a variable typo (=array= vs =arr=)
|
||||
- CIFAR-10 binary: class 1 vs rest (not class 0)
|
||||
- Multiprocessing uses =spawn= method via =torch.multiprocessing=
|
||||
- CIFAR-10 binary: class 1 (automobile) vs rest; natural IR ~9
|
||||
- MPS tensors cannot cross multiprocessing process boundaries — =run_configurations()= auto-disables =is_mp= when =device.type == 'mps'=
|
||||
- MPS concurrency sweet spot: N=3 independent subprocesses for cifar10n on 16GB M1 Pro; N=5 caused OOM after ~15h.
|
||||
- cifar10n on remote now runs with =sequential_episodes=True= (one episode at a time in-process) to avoid MPS unified memory exhaustion
|
||||
- Subprocess workers (=_episode_worker.py=) write stdout+stderr to per-episode =worker.log= — 0 bytes is normal for successful runs (only WARNING+ logged)
|
||||
- Kaggle CLI: available via =nix run nixpkgs#kaggle= or inside =nix develop= (added to =flake.nix= buildInputs). Credentials at =~/.config/kaggle/kaggle.json= (username: anerzakobar). creditcard.csv downloaded to =~/.data/creditcard/creditcard.csv=.
|
||||
- Large datasets (creditcard, homecredit, higgs) are manual downloads; =credit_card_fraud_dir= / =home_credit_dir= env vars set by shell hook to =~/.data/...=
|
||||
|
||||
** Remote Machine
|
||||
|
||||
- Host: =chenzakobar@192.168.1.190= (Mac OS, Python 3.13 via Homebrew)
|
||||
- SSH key: =~/.ssh/roll_remote= — dedicated ed25519 key, NOT the YubiKey/GPG key
|
||||
- Remote project dir: =~/roll-impl=; venv: =~/roll-venv=; env vars: =~/roll-env.sh=
|
||||
- Use =--detach= flag with =run_remote.sh= for long experiments (survives sleep/disconnect)
|
||||
- Kill a detached run: =ssh -i ~/.ssh/roll_remote -o IdentitiesOnly=yes chenzakobar@192.168.1.190 'pkill -f <script-name>'=
|
||||
|
||||
** Key Files
|
||||
|
||||
- [[file:src/roll.py][src/roll.py]] — Core loss: =KernelizedROLLoss= custom autograd Function with KDE backward pass
|
||||
- [[file:src/experiment.py][src/experiment.py]] — =ExperimentConfiguration= dataclass, =Criteriorator= ABC, =run_configurations()=
|
||||
- [[file:experiments/keel/_base.py][experiments/keel/_base.py]] — shared KEEL runner =run_keel_experiment()=
|
||||
- [[file:flake.nix][flake.nix]] — Nix env with dataset downloads, hash-pinned, exports path env vars
|
||||
- [[file:AGENTS.md][AGENTS.md]] — Project guidelines (naming conventions, env, dataset list)
|
||||
- [[file:experiments/keel/_base.py][experiments/keel/_base.py]] — shared KEEL runner; config suite: roll-aoc, roll-tpr90, bce-weighted, libauc-auroc, gce-0.7, mae, focal-loss, asymmetric-loss
|
||||
- [[file:experiments/other/experiment-cifar10n.py][experiments/other/experiment-cifar10n.py]] — CIFAR-10N (noisy labels: clean/aggre/worse), sequential_episodes=True
|
||||
- [[file:experiments/large/experiment-creditcard.py][experiments/large/experiment-creditcard.py]] — Credit card fraud dataset
|
||||
- [[file:scripts/run_remote.sh][scripts/run_remote.sh]] — rsync + run on remote Mac; =--detach= for sleep-safe long runs; excludes =results-final/= from push, includes it in fetch
|
||||
- [[file:flake.nix][flake.nix]] — Nix env; kaggle + sshpass in buildInputs
|
||||
- =results-final/cifar10n/= — Keeper results: clean/aggre/worse × 7 configs × 3 episodes. As of 2026-07-25: clean+aggre complete; worse-roll-aoc/bce-weighted/mae complete; worse-gce-0.7/focal-loss/asymmetric-loss/libauc-auroc running on remote.
|
||||
|
||||
** Subnodes
|
||||
|
||||
@@ -43,3 +69,5 @@ ROLL (Rate Optimized Likelyhood-based Loss) — PyTorch research project impleme
|
||||
- [[id:a53cbe84-cd8d-45c2-a8cf-34ab520a3ea5][impl/experiments]] — Experiment structure, training flow, metrics, output layout
|
||||
- [[id:b8a9886a-d349-43e5-a745-817a148c1fd8][impl/datasets]] — Dataset catalog, KEEL list, eval metrics
|
||||
- [[id:151d5686-6f40-4158-a59a-b0be94cdc969][impl/research]] — Literature survey: competing methods, dataset gaps, key papers
|
||||
- [[id:fdc18323-e9c2-4fc6-ace5-065035d30c51][impl/todos]] — Project TODOs
|
||||
- [[id:863ca60e-6a3c-45d2-86fd-fb4ed386e31b][impl/work-history/recent]] — Daily work log
|
||||
|
||||
Reference in New Issue
Block a user