roam/impl.org

impl

• impl
  • Architecture
  • Conventions
  • Gotchas
  • Remote Machine
  • Key Files
  • Subnodes

impl

ROLL (Rate Optimized Likelyhood-based Loss) — PyTorch research project implementing custom loss functions for binary classification using kernel density estimation (KDE) to optimize TPR at target FPR thresholds. Targets imbalanced classification problems.

Architecture

• src/roll.py — Loss function implementations (Normal/Beta/Kernelized ROLL)
• src/experiment.py — Training loop, evaluation infra, run_configurations() entry point
• src/datasets.py — Dataset loaders incl. Cifar10NDataset, ImbalancedCifar10Dataset
• src/networks.py — KeelNet MLP architecture + ConvNet (moved from experiment scripts for subprocess pickle compatibility)
• src/utils.py — Logging, init_experiment, get_device() (CUDA/MPS/CPU)
• src/_episode_worker.py — Subprocess entry point for parallel MPS episode execution
• experiments/keel/ — KEEL experiment scripts; _base.py shared runner
• experiments/other/ — CIFAR-10, CIFAR-10N, imbalanced CIFAR-10, adult, gaussian, etc.
• experiments/large/ — Higgs, credit card, home credit
• scripts/ — Remote runner: setup_remote.sh, run_remote.sh, fetch_results.sh, tail_remote.sh

Conventions

• All hyperparameters in Python dataclasses (ExperimentConfiguration), no CLI parsing
• Experiments follow experiment-*.py naming; all call run_configurations()
• KEEL experiments share experiments/keel/_base.py runner; individual files just call it
• All datasets expose: __getitem__, __len__, .x, .y attributes
• Episode-based eval: N independent train runs per config, results aggregated
• get_device() auto-selects CUDA → Apple MPS → CPU
• Two-tier results strategy: results/ holds ephemeral date-stamped run dirs (deletable); results-final/ holds keeper results for the thesis — one flat copy per config per dataset, no date subfolders. Both are in .gitignore.
• Resume scripts (experiment-*-resume.py) write directly into results-final/<dataset>/ to land in the right place immediately. Delete after run completes.

Gotchas

• Dataset paths injected as env vars by Nix shell hook ($keel_wisconsin_dir, etc.) — must use nix develop on Linux
• _calc_moments() in roll.py is unused and has a variable typo (array vs arr)
• CIFAR-10 binary: class 1 (automobile) vs rest; natural IR ~9
• MPS tensors cannot cross multiprocessing process boundaries — run_configurations() auto-disables is_mp when device.type = 'mps'=
• MPS concurrency sweet spot: N=3 independent subprocesses for cifar10n on 16GB M1 Pro; N=5 caused OOM after ~15h.
• cifar10n on remote now runs with sequential_episodes=True (one episode at a time in-process) to avoid MPS unified memory exhaustion
• Subprocess workers (_episode_worker.py) write stdout+stderr to per-episode worker.log — 0 bytes is normal for successful runs (only WARNING+ logged)
• Kaggle CLI: available via nix run nixpkgs#kaggle or inside nix develop (added to flake.nix buildInputs). Credentials at ~/.config/kaggle/kaggle.json (username: anerzakobar). creditcard.csv downloaded to ~/.data/creditcard/creditcard.csv.
• Large datasets (creditcard, homecredit, higgs) are manual downloads; credit_card_fraud_dir / home_credit_dir env vars set by shell hook to ~/.data/...

Remote Machine

• Host: chenzakobar@192.168.1.190 (Mac OS, Python 3.13 via Homebrew)
• SSH key: ~/.ssh/roll_remote — dedicated ed25519 key, NOT the YubiKey/GPG key
• Remote project dir: ~/roll-impl; venv: ~/roll-venv; env vars: ~/roll-env.sh
• Use --detach flag with run_remote.sh for long experiments (survives sleep/disconnect)
• Kill a detached run: ssh -i ~/.ssh/roll_remote -o IdentitiesOnly=yes chenzakobar@192.168.1.190 'pkill -f <script-name>'

Key Files

• src/roll.py — Core loss: KernelizedROLLoss custom autograd Function with KDE backward pass
• src/experiment.py — ExperimentConfiguration dataclass, Criteriorator ABC, run_configurations()
• experiments/keel/_base.py — shared KEEL runner; config suite: roll-aoc, roll-tpr90, bce-weighted, libauc-auroc, gce-0.7, mae, focal-loss, asymmetric-loss
• experiments/other/experiment-cifar10n.py — CIFAR-10N (noisy labels: clean/aggre/worse), sequential_episodes=True
• experiments/large/experiment-creditcard.py — Credit card fraud dataset
• scripts/run_remote.sh — rsync + run on remote Mac; --detach for sleep-safe long runs; excludes results-final/ from push, includes it in fetch
• flake.nix — Nix env; kaggle + sshpass in buildInputs
• results-final/cifar10n/ — Keeper results: clean/aggre/worse × 7 configs × 3 episodes. As of 2026-07-25: clean+aggre complete; worse-roll-aoc/bce-weighted/mae complete; worse-gce-0.7/focal-loss/asymmetric-loss/libauc-auroc running on remote.

Subnodes

• impl/loss-functions — Loss variants, KDE internals, gradient computation
• impl/experiments — Experiment structure, training flow, metrics, output layout
• impl/datasets — Dataset catalog, KEEL list, eval metrics
• impl/research — Literature survey: competing methods, dataset gaps, key papers
• impl/todos — Project TODOs
• impl/work-history/recent — Daily work log