roam/impl.org

impl

• impl
  • Architecture
  • Conventions
  • Gotchas
  • Remote Machines
  • Key Files
  • Subnodes

impl

ROLL (Rate Optimized Likelyhood-based Loss) — PyTorch research project implementing custom loss functions for binary classification using kernel density estimation (KDE) to optimize TPR at target FPR thresholds. Targets imbalanced classification problems.

Architecture

• src/roll.py — Loss function implementations (Normal/Beta/Kernelized ROLL)
• src/experiment.py — Training loop, evaluation infra, run_configurations() entry point
• src/datasets.py — Dataset loaders incl. Cifar10NDataset, Animal10NDataset, Food101NDataset
• src/networks.py — KeelNet MLP + ConvNet(image_size=32|64) (parameterized for 32×32 or 64×64)
• src/utils.py — Logging, init_experiment, get_device() (CUDA/MPS/CPU)
• src/_episode_worker.py — Subprocess entry point for parallel MPS episode execution
• experiments/keel/ — KEEL experiment scripts; _base.py shared runner
• experiments/other/ — CIFAR-10, CIFAR-10N, ANIMAL-10N, adult, gaussian, etc.
• experiments/large/ — Higgs, credit card, home credit, Food-101N
• scripts/ — Remote runners: run_remote.sh (Mac), run_remote_beacon.sh (beacon-wsl)

Conventions

• All hyperparameters in Python dataclasses (ExperimentConfiguration), no CLI parsing
• Experiments follow experiment-*.py naming; all call run_configurations()
• KEEL experiments share experiments/keel/_base.py runner; individual files just call it
• All datasets expose: __getitem__, __len__, .x, .y attributes
• Episode-based eval: N independent train runs per config, results aggregated
• get_device() auto-selects CUDA → Apple MPS → CPU
• Two-tier results strategy: results/ holds ephemeral date-stamped run dirs (deletable); results-final/ holds keeper results for the thesis — one flat copy per config per dataset, no date subfolders. Both are in .gitignore.
• Resume scripts (experiment-*-resume.py) write directly into results-final/<dataset>/ to land in the right place immediately. Delete after run completes.
• ConvNet for 64×64 images: use partial(ConvNet, image_size=64) as net_creator (gives 4096-feature linear layer)
• KEEL runs: always use --cpu --no-mp + unset MLFLOW_TRACKING_URI — CPU is 6.6x faster than MPS for tiny batches; mp.Pool spawn overhead exceeds episode runtime; MLflow adds per-epoch I/O

Gotchas

• Dataset paths injected as env vars by Nix shell hook ($keel_wisconsin_dir, etc.) — must use nix develop on Linux
• CIFAR-10 binary: class 1 (automobile) vs rest; natural IR ~9
• MPS tensors cannot cross multiprocessing process boundaries — run_configurations() auto-disables is_mp when device.type = 'mps'=
• cifar10n on remote runs with sequential_episodes=True to avoid MPS unified memory exhaustion
• LibAUC does not support MPS — force device=torch.device('cpu') for libauc-auroc on Mac
• On beacon-wsl, first nix develop takes ~10 min (downloads all store paths incl. CUDA PyTorch)
• ANIMAL-10N download requires filling Google Form: https://forms.gle/8mbmbNgDFQ2rA1fLA
• sequential_episodes=True is dead code for CPU + is_mp=True: the flag only gates the MPS subprocess branch; CPU always spawns mp.Pool(n_episodes) when is_mp=True
• Per-epoch Python overhead in KEEL: ~54ms (ExperimentDataLoader shuffle+argwhere, pd.DataFrame ×2 in gen_criteria, extra loss forward in _get_loss) vs 0.5ms raw compute — not worth optimising unless runtime becomes critical
• gen_criteria logs per-epoch loss at DEBUG level (changed from INFO 2026-08-01) — visible in debug.log per run dir, not console
• MLflow: if MLFLOW_TRACKING_URI is set and points to a live server, per-epoch log_metric calls add significant overhead — always unset MLFLOW_TRACKING_URI for KEEL

Remote Machines

• Mac — chenzakobar@192.168.1.190 (macOS, Python 3.13 venv at ~/roll-venv, Apple MPS)

  • SSH key: ~/.ssh/roll_remote (dedicated ed25519, NOT YubiKey)
  • Runner: scripts/run_remote.sh + scripts/fetch_results.sh
  • Kill detached run: ssh ... 'pkill -f <script-name>'

• beacon-wsl — aner@192.168.1.200 (NixOS-WSL, RTX 3050 6GB VRAM, CUDA)

  • SSH key: YubiKey GPG key via agent (default SSH config)
  • Runner: scripts/run_remote_beacon.sh + scripts/fetch_results_beacon.sh
  • Uses nix develop --command python (no venv; flake has cudaSupport=true)
  • Kaggle creds copied to ~/.config/kaggle/kaggle.json

Key Files

• src/roll.py — Core loss: KernelizedROLLoss custom autograd Function with KDE backward pass
• src/experiment.py — ExperimentConfiguration dataclass, Criteriorator ABC, run_configurations()
• experiments/keel/_base.py — shared KEEL runner; config suite: roll-aoc, roll-tpr90, bce-weighted, libauc-auroc, gce-0.7, mae, focal-loss, asymmetric-loss
• experiments/keel/run_all_noise_levels.sh — runs 5%/10%/20% noise × all 12 datasets × 10 episodes; hardcodes --cpu --no-mp, unsets MLflow
• experiments/other/experiment-cifar10n.py — CIFAR-10N (noisy labels: clean/aggre/worse), sequential_episodes=True
• experiments/other/experiment-animal10n.py — ANIMAL-10N (human noise ~8%), runs on Mac MPS
• experiments/large/experiment-food101n.py — Food-101N (web noise ~18%, 7GB), runs on beacon-wsl CUDA
• scripts/run_remote.sh — rsync + run on remote Mac; --detach for sleep-safe long runs
• scripts/run_remote_beacon.sh — rsync + run on beacon-wsl via nix develop
• flake.nix — Nix env; kaggle + sshpass in buildInputs; cudaSupport=true
• results-final/cifar10n/ — Complete: clean/aggre/worse × 7 configs × 3 episodes (as of 2026-07-26)

Subnodes

• impl/loss-functions — Loss variants, KDE internals, gradient computation
• impl/experiments — Experiment structure, training flow, metrics, output layout
• impl/datasets — Dataset catalog, KEEL list, eval metrics
• impl/research — Literature survey: competing methods, dataset gaps, key papers
• impl/todos — Project TODOs
• impl/work-history/recent — Daily work log