backup: 2026-08-29 06:59

2026-08-29 06:59:35 +03:00
parent 07112f1a4c
commit 2a872c725b
19 changed files with 823 additions and 136 deletions
@@ -56,3 +56,4 @@ Skill descriptions are routing keys — they determine whether a skill activates
 - [[id:36fe6a01-ea1f-4785-8516-f6dcfecf05bb][azos/architecture]] — module registration, option namespace, file deployment, Claude Code wiring
 - [[id:c945da4f-de5c-4eb8-bd99-810576a2545a][azos/org-roam]] — setup, MCP server, SQLite internals, fork details
 - [[id:09ae970c-2995-43d9-a597-47bb904c98de][azos/pass]] — password store format, path mappings, mbsync conventions
 - [[id:ebf221fd-e5be-49d4-a078-6113e2e5e3f2][azos/beacon-wsl]] — NixOS-WSL machine, GPU/CUDA setup, remote deploy
@@ -0,0 +1,55 @@
 :PROPERTIES:
 :ID: ebf221fd-e5be-49d4-a078-6113e2e5e3f2
 :END:
 #+title: azos/beacon-wsl
 #+filetags: :project: :knowledge: :beacon-wsl:
 :PROPERTIES:
 :ID: azos-beacon-wsl
 :END:
 #+title: azos/beacon-wsl
 #+filetags: :project: :knowledge: :beacon-wsl:
 [[id:bf4dde56-0967-4374-94ed-771301e47c66][azos]]
 * Overview
 NixOS-WSL instance running on a Windows machine on the local network.
 - Static IP: =192.168.1.200=
 - GPU: NVIDIA RTX 3050, 6GB VRAM, CUDA 13.2
 - Config: =_machines/beacon-wsl.nix= → =nixos/configuration-beacon-wsl.nix=
 * SSH Access
 Uses YubiKey GPG key (=cardno:29_109_265=) — same key as lauretta. Authorized key
 is set in =users.users.aner.openssh.authorizedKeys.keys= in the NixOS config.
 Password auth disabled; root login disabled.
 * Remote Deployment
 Build locally and copy FAILS due to unsigned path rejection by remote Nix daemon.
 Must build on the target host itself:
 #+begin_src bash
 nixos-rebuild switch \
  --flake '/home/aner/azos?submodules=1#beacon-wsl' \
  --target-host aner@192.168.1.200 \
  --build-host aner@192.168.1.200 \
  --elevate=sudo
 #+end_src
 * CUDA / GPU
 - =programs.nix-ld.enable = true= required to run WSL-provided binaries (nvidia-smi, etc.)
 - =nvidia-smi= lives at =/usr/lib/wsl/lib/nvidia-smi=; needs =LD_LIBRARY_PATH=/usr/lib/wsl/lib=
 - =hardware.graphics.enable = true= exposes GPU via WSL2 paravirtualization
 - CUDA-enabled torch works via nix shell with =config.cudaSupport = true= (see =roll= project flake)
 * Windows-Side Setup
 - Mirrored networking: =C:\Users\<user>\.wslconfig= with =[wsl2] networkingMode=mirrored=
 - Windows Firewall inbound rule for TCP port 22 required (=-Profile Any=)
 - Static IP set via Windows network adapter settings
@@ -23,12 +23,18 @@ Attic (self-hosted Nix binary cache): cache name =main=, public key =main:9SZt/6
 Eurovision Vote: Django app sourced from external flake =github:anerisgreat/eurovote=, wrapped by =modules/services/eurovote.nix=. Uses DynamicUser + StateDirectory so systemd owns =/var/lib/eurovote/=; no tmpfiles entry needed.
 EmulatorJS: web game emulator at =games.zakobar.com= with an offline PWA + per-user save sync. See [[id:7148e7ef-7741-42ad-9d3d-4025001e1018][homey/emulatorjs]].
 Backup: Restic daily at 03:00 to S3 (Backblaze B2, bucket =zakobar-home-backup=). Pre-hook: Nextcloud maintenance mode on + pg_dump. Post-hook: maintenance mode off. Manual offload: =restic copy= to local disk. NAR content and media excluded.
 Reliability hardening in =hosts/pi-main/default.nix=: hardware watchdog (bcm2835_wdt), WiFi power save disabled, network watchdog timer, zramSwap zstd 25%, Nix build-dir on external HD.
 Bootstrap: =pi-main-bootstrap= config builds an SD image (=sd-image-aarch64.nix=) for first flash.
 ** Subnodes
 - [[id:7148e7ef-7741-42ad-9d3d-4025001e1018][homey/emulatorjs]] — EmulatorJS game emulator: offline PWA design, ROM management, save sync, PS1/large-ROM notes.
 ** Deployment
 *Always deploy from the dev machine using the dev shell command:*
@@ -76,6 +82,8 @@ Caddy Cloudflare plugin secrets: uses =LoadCredential= + =ExecStart= override (c
 Stirling-PDF login disable: =DOCKER_ENABLE_SECURITY=false= is build-time only. To disable the runtime login page, also set =SECURITY_ENABLELOGIN=false=.
 Fetching external static assets via =fetchFromGitHub= (used by EmulatorJS, same pattern works for any GitHub-hosted release): get hash with =nix-prefetch-url --unpack <tarball-url>=, then convert to SRI with =nix hash convert --hash-algo sha256 --to sri <base32>=. To upgrade: bump =rev=, re-prefetch, update =hash=. Full upgrade instructions are in a comment inside =modules/services/emulatorjs.nix=.
 ** Gotchas
 hdparm APM udev rule was removed — USB-SATA bridges often don't support APM commands and hdparm hangs indefinitely, causing boot-time crashes.
@@ -111,6 +119,7 @@ Admin SSH key is GPG-backed via YubiKey. Current key: =~/.ssh/gpg-as-ssh.pub= (=
 - =modules/storage.nix= — external HD mount, =extraDirs= option
 - =modules/backup.nix= — Restic, pre/post hooks, =extraPaths= option
 - =scripts/ssh-with-gpg-key.sh= — emergency SSH via GPG key file; sets up ephemeral GNUPGHOME and uses =[A]= subkey
 - =modules/services/emulatorjs.nix= + =emulatorjs-server.py= — EmulatorJS emulator; details in [[id:7148e7ef-7741-42ad-9d3d-4025001e1018][homey/emulatorjs]]
 ** TODOs
@@ -129,3 +138,7 @@ enableChunkedUpload = true
 maxChunkSize = 5_000_000   # 5 MiB per chunk
 #+end_src
 [[id:7148e7ef-7741-42ad-9d3d-4025001e1018][homey/emulatorjs]]
 [[id:2d455310-42a5-4ff5-be38-9af2b009aef7][homey/work-history/recent]]
@@ -0,0 +1,66 @@
 :PROPERTIES:
 :ID: 41466dc4-91a5-4578-bce3-7b4a0ba813d2
 :END:
 #+title: homey/emulatorjs
 #+filetags: :project: :knowledge: :emulatorjs:
 * homey/emulatorjs
 Self-hosted game emulator running on games.zakobar.com. Python sidecar at
 [[file:~/projects/selfhosted/homey/modules/services/emulatorjs-server.py][modules/services/emulatorjs-server.py]], NixOS module at
 [[file:~/projects/selfhosted/homey/modules/services/emulatorjs.nix][modules/services/emulatorjs.nix]].
 ** What works (stable baseline)
 - Game library launcher with search, favourites, thumbnails
 - Per-user server-side saves (GET/PUT /saves/<name>, keyed by Authelia Remote-User header)
 - Manual Save button on player page (Save → Saving... → Saved! feedback)
 - Auto-save every 30 seconds via setInterval
 - sendBeacon save on pagehide (iOS-reliable exit save)
 - pullSave on EJS_onGameStart loads server save into EmulatorJS FS
 ** Offline/PWA mode — designed but not yet working
 Intended design (revert this when debugging is done):
 *** Service Worker (SW_JS constant, served at /sw.js)
 - Cache name versioned: emulatorjs-vN. Bump N to wipe stale cache on deploy.
 - /static/ and /roms/<name>: cache-first, arrayBuffer() for caching (NOT clone() — iOS Safari bug: cloned response body is empty)
 - /play/<name>: network-first (so deployments take effect). Cache as offline fallback.
 - /roms/, /favorites, /: network-first, cache fallback
 - /saves/: pass-through to network only (page handles offline)
 - /sw.js: never cached
 - Install: cache.addAll(['/static/loader.js', '/']); skipWaiting() outside waitUntil
 - Activate: delete old cache names, clients.claim()
 *** IndexedDB offline save queue (_IDB_JS shared constant)
 - DB name: 'emulatorjs', version 1, store: 'pending-saves' (keyPath: 'name')
 - openIDB() MUST handle onblocked or Promise hangs indefinitely
 - idbPut/idbGet/idbGetAll/idbDelete helpers
 - syncPendingSaves(): flush IDB queue to server when online
 *** Player page flow
 - preflightCheck(): check cache for ROM, block if offline + ROM not cached, else return true
 - syncPendingSaves() runs inside EJS_onGameStart (NOT in startup path — if in startup, IDB blocks loader.js)
 - pushSave(): PUT to server if online, else idbPut to IDB queue
 - pullSave(): fetch from server if online, else idbGet from IDB
 *** Launcher additions
 - updateCacheStatus(): green dots on cached game cards (checks cache for /roms/<name>)
 - syncPendingSaves() on 'online' event
 ** Known bugs / gotchas
 - Python string.Template: \n inside triple-quoted template string is a LITERAL NEWLINE in generated JS → SyntaxError (unterminated string literal). Never use \n inside JS string literals embedded in the Python template. Use \\n or restructure.
 - iOS Safari response.clone() bug: cloned response body is empty (0 bytes). Always use response.arrayBuffer() + new Response(body, ...) for SW caching.
 - SW cache-first for /play/ pages means deployments don't take effect until cache is cleared. Use network-first for HTML pages.
 - Cache version bump alone is not enough if the browser serves the old page (from SW cache) before the new SW activates. User must clear site data OR the new SW must activate and claim before any navigation.
 - Python BaseHTTPRequestHandler is single-threaded. Heavy ROM downloads can block other requests.
 - preflightCheck() must call its first setDbg equivalent BEFORE any await, or the UI appears frozen.
 ** Ports / URLs
 - Port 8085, Caddy vhost games.zakobar.com, Authelia one_factor priority 57
 - Saves: /mnt/data/emulatorjs/saves/<user>/<name>.sav
 - ROMs: /mnt/data/emulatorjs/roms/
 - Static assets: Nix store (emulatorjsAssets = EmulatorJS v4.2.3 from GitHub)
@@ -0,0 +1,37 @@
 Part of [[id:c87d8901-bbee-4397-bff1-af8432b4f66e][homey]]. EmulatorJS self-hosted web game emulator at =games.zakobar.com= (port 8085), Authelia one_factor. A Python stdlib sidecar (=modules/services/emulatorjs-server.py=) serves the launcher, ROMs, per-user saves, and a PWA; static assets/cores come from GitHub via =fetchFromGitHub= (=modules/services/emulatorjs.nix=, currently v4.2.3). ROMs/saves/thumbnails live on the external HD under =/mnt/data/emulatorjs/=, outside Nix.
 ** Architecture — offline PWA (added 2026-07-31)
 Installable PWA with offline play, deliberately *network-first*: the Service Worker only reads cache when the network is unreachable, so offline support can never alter or break online play. (An earlier cache-first version rebuilt/buffered responses, broke EmulatorJS game loading, and was ripped out — the working tree that "fixed" it was simply the SW removed. Rebuilding responses via =arrayBuffer()=+=new Response()= also stripped range-request support and OOM'd large loads on iOS.)
 Server serves =/sw.js= (Service Worker, =Service-Worker-Allowed: /=), =/manifest.json=, and a stdlib-generated =/icon-256.png= (PNG encoded by hand with =zlib=/=struct=, built once at startup — no image deps). Both HTML pages carry a shared PWA head fragment.
 Two SW caches:
 - =emujs-shell-<BUILD_ID>= — app shell + =/static/= cores, cache-first. =BUILD_ID= = short sha256 of the server .py file + =STATIC_DIR=, so a redeploy auto-invalidates stale assets (no manual cache-version bump, unlike the old =CACHE_NAME='v4'=). NOTE: BUILD_ID changes on *any* edit to the .py file, so every deploy drops the shell cache and it must be repopulated by one online play.
 - =emujs-roms= — persistent across builds. ROMs are NOT auto-cached; download is opt-in per game via the launcher's ⬇ button (also caches the thumbnail + the =/play/<name>= document). Non-downloaded games dim out when offline.
 Core caching (the non-obvious bit — fixed 2026-07-31): the emulator core (e.g. =mgba-legacy-wasm.data=, shared by GB/GBC/GBA) is NOT reliably cached by the Service Worker, because on the first visit the SW is not yet controlling the page, so the core fetched during that first online play is never intercepted. Instead the *player page* caches it directly: after game start, =cacheLoadedStatic()= reads =performance.getEntriesByType('resource')=, filters to same-origin =/static/= URLs (the real files the browser loaded), and writes them into the shell cache from the page — bypassing SW-interception timing entirely and version-proof. Runs on game start (+5s delayed pass) and on the =online= event.
 Full design rationale is in the module docstring at the top of =emulatorjs-server.py=.
 ** Conventions
 *Adding/removing GB/GBC/GBA ROMs*: just drop/delete files in =/mnt/data/emulatorjs/roms/=. The =/roms/= listing does a live =os.listdir= on every request, so a page refresh picks up changes — no restart, no =nixos-rebuild=. Filename (minus extension) is the display title. Optional cover art: =<title>.png= in =/mnt/data/emulatorjs/thumbnails/= (else 🎮 placeholder). Files must be readable by the =emulatorjs= service user.
 *Upgrading EmulatorJS assets*: bump =rev= in =emulatorjs.nix=, re-prefetch (=nix-prefetch-url --unpack=), convert to SRI, update =hash=. Instructions are in a comment in the file.
 *To make a game fully offline-ready*: (1) tap ⬇ to download the ROM, and (2) play it once while online so the core caches. After a redeploy, step 2 must be repeated (shell cache is build-scoped).
 ** Gotchas
 *Save sync*: saves are independent of ROM download state. Two-tier: online → =PUT /saves/<game>= (keyed by =Remote-User=); offline/failed → queued in IndexedDB (=pending-saves= store). Queue flushes via =syncPendingSaves()= on the =online= event, on =EJS_onGameStart=, and on launcher load. NO Background Sync API — sync only runs while a page is open / on next app open. Last-write-wins, no timestamp merge (both deemed acceptable by owner 2026-07-31).
 *Core not cached → "Error downloading core (…-wasm.data)" offline*: symptom of the SW-timing gap above. Fixed by page-side =cacheLoadedStatic()=; requires one successful online play (after each deploy) to populate.
 *Filename charset*: a ROM loads only if its name matches =SAFE_RE= (=[A-Za-z0-9 _\-\.\(\)\[\]',!?:&+#]=). A name with other characters (accents, ~, @, %, ;) still appears in the listing but 404s/400s on open. Keep names in that set.
 *Offline library staleness*: the =/roms/= listing is network-first cached, so an offline device shows its last-cached list until it's online once.
 ** Future — hosting disc-based systems (e.g. PS1)
 Not done (owner declined 2026-07-31), but scoped. Would require: (1) map disc extensions to the =psx= core in =ROM_SYSTEMS= + zip detection (prefer single-file =.chd=; =.bin=/=.iso= are ambiguous across systems); (2) a user-supplied BIOS (copyrighted) served via a new =/bios/= route + =EJS_biosUrl=; (3) **stream ROM serving** — =send_body= currently does =f.read()=, loading the whole file into RAM, so a 600 MB disc image spikes 600 MB per request on the Pi. Fine for tiny cartridge ROMs; must switch to chunked/Range serving for large images. Offline download also impractical for 100s-of-MB ROMs (iOS Cache API quota).
@@ -0,0 +1,18 @@
 :PROPERTIES:
 :ID: 2d455310-42a5-4ff5-be38-9af2b009aef7
 :END:
 #+title: homey/work-history/recent
 #+filetags: :project: :work-history:
 * 2026-07-31 Fri
 - Restored EmulatorJS offline mode (previously removed because a cache-first Service Worker broke game loading). Rewrote =modules/services/emulatorjs-server.py= as a network-first PWA: SW only reads cache when offline, so it cannot break online play.
 - Added installable PWA (=/manifest.json=, stdlib-generated =/icon-256.png=), build-scoped =emujs-shell-<BUILD_ID>= cache with auto-invalidation on redeploy, and an opt-in per-game ⬇ download button caching ROMs into a persistent =emujs-roms= cache.
 - Kept IndexedDB offline save queue with sync-on-reconnect; player loads =loader.js= directly (removed the preflight gate).
 - Verified via module import + live HTTP smoke test (all routes 200; =/sw.js= has =Service-Worker-Allowed: /=).
 - Decided against Background Sync and timestamp-merge for saves (owner said not relevant). Decided against PS1/disc hosting for now (scoped it: needs psx core + BIOS + streamed ROM serving).
 - Recorded design in new org-roam subnode [[id:7148e7ef-7741-42ad-9d3d-4025001e1018][homey/emulatorjs]].
 - Fixed offline "Error downloading core (mgba-legacy-wasm.data)": the SW isn't controlling the page on first visit, so the core fetched during first online play was never cached. Added page-side =cacheLoadedStatic()= that reads =performance.getEntriesByType('resource')= and writes the actually-loaded =/static/= files into the shell cache directly (runs on game start + delayed + on reconnect). Download button now also caches the =/play/<name>= document.
 ** Archives
@@ -12,14 +12,14 @@ ROLL (Rate Optimized Likelyhood-based Loss) — PyTorch research project impleme
 - =src/roll.py= — Loss function implementations (Normal/Beta/Kernelized ROLL)
 - =src/experiment.py= — Training loop, evaluation infra, =run_configurations()= entry point
- =src/datasets.py= — Dataset loaders incl. =Cifar10NDataset=, =ImbalancedCifar10Dataset=
+- =src/datasets.py= — Dataset loaders incl. =Cifar10NDataset=, =Animal10NDataset=, =Food101NDataset=
- =src/networks.py= — =KeelNet= MLP architecture + =ConvNet= (moved from experiment scripts for subprocess pickle compatibility)
+- =src/networks.py= — =KeelNet= MLP + =ConvNet(image_size=32|64)= (parameterized for 32×32 or 64×64)
 - =src/utils.py= — Logging, =init_experiment=, =get_device()= (CUDA/MPS/CPU)
 - =src/_episode_worker.py= — Subprocess entry point for parallel MPS episode execution
 - =experiments/keel/= — KEEL experiment scripts; =_base.py= shared runner
- =experiments/other/= — CIFAR-10, CIFAR-10N, imbalanced CIFAR-10, adult, gaussian, etc.
+- =experiments/other/= — CIFAR-10, CIFAR-10N, ANIMAL-10N, adult, gaussian, etc.
- =experiments/large/= — Higgs, credit card, home credit
+- =experiments/large/= — Higgs, credit card, home credit, Food-101N
- =scripts/= — Remote runner: =setup_remote.sh=, =run_remote.sh=, =fetch_results.sh=, =tail_remote.sh=
+- =scripts/= — Remote runners: =run_remote.sh= (Mac), =run_remote_beacon.sh= (beacon-wsl)
 ** Conventions
@@ -31,37 +31,48 @@ ROLL (Rate Optimized Likelyhood-based Loss) — PyTorch research project impleme
 - =get_device()= auto-selects CUDA → Apple MPS → CPU
 - *Two-tier results strategy*: =results/= holds ephemeral date-stamped run dirs (deletable); =results-final/= holds keeper results for the thesis — one flat copy per config per dataset, no date subfolders. Both are in =.gitignore=.
 - Resume scripts (=experiment-*-resume.py=) write directly into =results-final/<dataset>/= to land in the right place immediately. Delete after run completes.
 - ConvNet for 64×64 images: use =partial(ConvNet, image_size=64)= as net_creator (gives 4096-feature linear layer)
 - *KEEL runs*: always use =--cpu --no-mp= + =unset MLFLOW_TRACKING_URI= — CPU is 6.6x faster than MPS for tiny batches; mp.Pool spawn overhead exceeds episode runtime; MLflow adds per-epoch I/O
 ** Gotchas
 - Dataset paths injected as env vars by Nix shell hook (=$keel_wisconsin_dir=, etc.) — must use =nix develop= on Linux
 - =_calc_moments()= in roll.py is unused and has a variable typo (=array= vs =arr=)
 - CIFAR-10 binary: class 1 (automobile) vs rest; natural IR ~9
 - MPS tensors cannot cross multiprocessing process boundaries — =run_configurations()= auto-disables =is_mp= when =device.type == 'mps'=
- MPS concurrency sweet spot: N=3 independent subprocesses for cifar10n on 16GB M1 Pro; N=5 caused OOM after ~15h.
+- cifar10n on remote runs with =sequential_episodes=True= to avoid MPS unified memory exhaustion
- cifar10n on remote now runs with =sequential_episodes=True= (one episode at a time in-process) to avoid MPS unified memory exhaustion
+- LibAUC does not support MPS — force =device=torch.device('cpu')= for libauc-auroc on Mac
- Subprocess workers (=_episode_worker.py=) write stdout+stderr to per-episode =worker.log= — 0 bytes is normal for successful runs (only WARNING+ logged)
+- On beacon-wsl, first =nix develop= takes ~10 min (downloads all store paths incl. CUDA PyTorch)
- Kaggle CLI: available via =nix run nixpkgs#kaggle= or inside =nix develop= (added to =flake.nix= buildInputs). Credentials at =~/.config/kaggle/kaggle.json= (username: anerzakobar). creditcard.csv downloaded to =~/.data/creditcard/creditcard.csv=.
+- ANIMAL-10N download requires filling Google Form: https://forms.gle/8mbmbNgDFQ2rA1fLA
- Large datasets (creditcard, homecredit, higgs) are manual downloads; =credit_card_fraud_dir= / =home_credit_dir= env vars set by shell hook to =~/.data/...=
+- =sequential_episodes=True= is dead code for CPU + =is_mp=True=: the flag only gates the MPS subprocess branch; CPU always spawns =mp.Pool(n_episodes)= when =is_mp=True=
 - Per-epoch Python overhead in KEEL: ~54ms (ExperimentDataLoader shuffle+argwhere, =pd.DataFrame= ×2 in =gen_criteria=, extra loss forward in =_get_loss=) vs 0.5ms raw compute — not worth optimising unless runtime becomes critical
 - =gen_criteria= logs per-epoch loss at DEBUG level (changed from INFO 2026-08-01) — visible in =debug.log= per run dir, not console
 - MLflow: if =MLFLOW_TRACKING_URI= is set and points to a live server, per-epoch =log_metric= calls add significant overhead — always =unset MLFLOW_TRACKING_URI= for KEEL
-** Remote Machine
+** Remote Machines
- Host: =chenzakobar@192.168.1.190= (Mac OS, Python 3.13 via Homebrew)
+- *Mac* — =chenzakobar@192.168.1.190= (macOS, Python 3.13 venv at =~/roll-venv=, Apple MPS)
- SSH key: =~/.ssh/roll_remote= — dedicated ed25519 key, NOT the YubiKey/GPG key
+  - SSH key: =~/.ssh/roll_remote= (dedicated ed25519, NOT YubiKey)
- Remote project dir: =~/roll-impl=; venv: =~/roll-venv=; env vars: =~/roll-env.sh=
+  - Runner: =scripts/run_remote.sh= + =scripts/fetch_results.sh=
- Use =--detach= flag with =run_remote.sh= for long experiments (survives sleep/disconnect)
+  - Kill detached run: =ssh ... 'pkill -f <script-name>'=
- Kill a detached run: =ssh -i ~/.ssh/roll_remote -o IdentitiesOnly=yes chenzakobar@192.168.1.190 'pkill -f <script-name>'=
+- *beacon-wsl* — =aner@192.168.1.200= (NixOS-WSL, RTX 3050 6GB VRAM, CUDA)
  - SSH key: YubiKey GPG key via agent (default SSH config)
  - Runner: =scripts/run_remote_beacon.sh= + =scripts/fetch_results_beacon.sh=
  - Uses =nix develop --command python= (no venv; flake has =cudaSupport=true=)
  - Kaggle creds copied to =~/.config/kaggle/kaggle.json=
 ** Key Files
 - [[file:src/roll.py][src/roll.py]] — Core loss: =KernelizedROLLoss= custom autograd Function with KDE backward pass
 - [[file:src/experiment.py][src/experiment.py]] — =ExperimentConfiguration= dataclass, =Criteriorator= ABC, =run_configurations()=
 - [[file:experiments/keel/_base.py][experiments/keel/_base.py]] — shared KEEL runner; config suite: roll-aoc, roll-tpr90, bce-weighted, libauc-auroc, gce-0.7, mae, focal-loss, asymmetric-loss
 - [[file:experiments/keel/run_all_noise_levels.sh][experiments/keel/run_all_noise_levels.sh]] — runs 5%/10%/20% noise × all 12 datasets × 10 episodes; hardcodes =--cpu --no-mp=, unsets MLflow
 - [[file:experiments/other/experiment-cifar10n.py][experiments/other/experiment-cifar10n.py]] — CIFAR-10N (noisy labels: clean/aggre/worse), sequential_episodes=True
- [[file:experiments/large/experiment-creditcard.py][experiments/large/experiment-creditcard.py]] — Credit card fraud dataset
+- [[file:experiments/other/experiment-animal10n.py][experiments/other/experiment-animal10n.py]] — ANIMAL-10N (human noise ~8%), runs on Mac MPS
- [[file:scripts/run_remote.sh][scripts/run_remote.sh]] — rsync + run on remote Mac; =--detach= for sleep-safe long runs; excludes =results-final/= from push, includes it in fetch
+- [[file:experiments/large/experiment-food101n.py][experiments/large/experiment-food101n.py]] — Food-101N (web noise ~18%, 7GB), runs on beacon-wsl CUDA
- [[file:flake.nix][flake.nix]] — Nix env; kaggle + sshpass in buildInputs
+- [[file:scripts/run_remote.sh][scripts/run_remote.sh]] — rsync + run on remote Mac; =--detach= for sleep-safe long runs
- =results-final/cifar10n/= — Keeper results: clean/aggre/worse × 7 configs × 3 episodes. As of 2026-07-25: clean+aggre complete; worse-roll-aoc/bce-weighted/mae complete; worse-gce-0.7/focal-loss/asymmetric-loss/libauc-auroc running on remote.
+- [[file:scripts/run_remote_beacon.sh][scripts/run_remote_beacon.sh]] — rsync + run on beacon-wsl via =nix develop=
 - [[file:flake.nix][flake.nix]] — Nix env; kaggle + sshpass in buildInputs; =cudaSupport=true=
 - =results-final/cifar10n/= — Complete: clean/aggre/worse × 7 configs × 3 episodes (as of 2026-07-26)
 ** Subnodes
@@ -7,7 +7,53 @@
 * impl/datasets
-** KEEL Dataset Class Balance
+** Datasets with Final Results
 *** CIFAR-10N (image)
 - *Size*: 50,000 training images (32×32 RGB, CIFAR-10 base)
 - *Noise*: clean 0% / aggre ~9% / worse ~40% (human annotator noise)
 - *Source*: UCSC-REAL/cifar-10-100n — standard noisy-label benchmark (Learning with Noisy Labels, NeurIPS 2021)
 - *Notes*: Positive class = automobile (class 1) vs rest; natural IR ~9:1. Three noise conditions all have final results. Labels replaced with real human annotation noise from crowdworkers.
 *** Credit Card Fraud (tabular)
 - *Size*: 284,807 transactions
 - *Noise*: 0% (clean; real confirmed fraud labels)
 - *Source*: Kaggle / ULB (Worldline + MLG research collaboration, 2013)
 - *Notes*: Extreme imbalance — only 492 frauds (IR ~577:1). 30 features: V1–V28 are PCA-transformed (original features confidential), plus Time and Amount. Time/Amount rescaled; V1–V28 left as-is.
 *** Bank Marketing (tabular)
 - *Size*: 41,188 samples; ~48 features after one-hot encoding
 - *Noise*: 0% (clean)
 - *Source*: UCI Machine Learning Repository (Moro et al., 2014 — Portuguese bank direct marketing campaigns)
 - *Notes*: IR ~8:1 (~11.3% positive). Target: whether client subscribed to term deposit. Mix of demographic, contact, and economic features.
 *** Home Credit (tabular)
 - *Size*: 307,511 loan applications; ~120 numerical features
 - *Noise*: 0% (clean; proxy labels — loan repayment used as credit default indicator)
 - *Source*: Kaggle Home Credit Default Risk competition (2018)
 - *Notes*: IR ~11.4:1 (~8% default rate). Features from application_train.csv only (no auxiliary tables). Non-numeric and ID columns dropped; NaNs filled with 0.
 *** KEEL Datasets (tabular, 12 with final results)
 - *Noise*: 0% (clean — standard imbalanced learning benchmarks)
 - *Source*: KEEL repository (Knowledge Extraction based on Evolutionary Learning)
 - *Notes*: All small/medium tabular datasets used as standard IR benchmarks in the literature. Loaded via keel_ds Python package.
 | Dataset      | Total | Pos | Neg | IR   | Notes                              |
 |--------------+-------+-----+-----+------+------------------------------------|
 | glass0       |   214 |  70 | 144 |  2.1 | Glass identification (UCI)         |
 | glass1       |   214 |  76 | 138 |  1.8 | Glass identification (UCI)         |
 | glass2       |   214 |  17 | 197 | 11.6 | Glass identification (UCI)         |
 | glass6       |   214 |  29 | 185 |  6.4 | Glass identification (UCI)         |
 | haberman     |   306 |  81 | 225 |  2.8 | Breast cancer survival (1958–1970) |
 | iris0        |   150 |  50 | 100 |  2.0 | Iris setosa vs others              |
 | new-thyroid1 |   215 |  35 | 180 |  5.1 | Thyroid function classification    |
 | pima         |   768 | 268 | 500 |  1.9 | Pima Indian diabetes (UCI)         |
 | vehicle2     |   846 | 218 | 628 |  2.9 | Vehicle silhouette (UCI)           |
 | vowel0       |   988 |  90 | 898 | 10.0 | Vowel recognition (class 0 vs rest)|
 | wisconsin    |   683 | 239 | 444 |  1.9 | Wisconsin breast cancer (UCI)      |
 | yeast3       |  1484 | 163 |1321 |  8.1 | Yeast protein localization (UCI)   |
 ** KEEL Dataset Class Balance (full list)
 Counts from =KeelDataset= loader. Ratio = false/true (imbalance factor).
 Two datasets fail to load (likely missing env vars or files): =led7digit=, =page-blocks=.
@@ -38,3 +84,6 @@ Two datasets fail to load (likely missing env vars or files): =led7digit=, =page
 - ISJ "Need more data" failure: cleveland (13 true), ecoli (20 true), glass4 (13), glass5 (9) — after 33% test split, too few samples in minority class for ISJ root-finding. Excluded from =run_all.sh=.
 - glass2 (17 true) is borderline — watch for failures.
 - led7digit and page-blocks fail to load entirely (missing dataset files or env vars).
 - Credit card V1–V28 are pre-PCA (do NOT re-scale them); only Time (col 0) and Amount (col 29) are rescaled.
 - Home Credit uses application_train.csv only — no auxiliary tables (bureau, previous applications, etc.) merged in.
 - CIFAR-10N aggre noise rate is ~9%, worse is ~40% — large gap; both have completed final results.
@@ -7,15 +7,72 @@
 * impl/experiments
-** Current Config Set (as of 2026-07-08)
+** Baseline Config Suite (current)
- =roll= — =kernelized_roll_tpr(0.95)=
+8 configs run on every dataset. All share StepLR(step_size=50, γ=0.5).
 - =roll+bce= — =kernelized_roll_tpr(0.95, bce_weight=0.5, bce_pos_weight=imbalance_ratio)=
 - =bce= — plain =BCEWithLogitsLoss=
 - =bce-weighted= — BCE with =pos_weight = num_false/num_true=
-FPR variants (=roll-fpr-0.40=, =roll-fpr-0.40+bce=) removed — less consistent, added complexity for marginal gain.
+| Config            | Loss                        | Optimizer | lr    | Notes                          |
-Target metric: FPR@TPR=0.95 (lower is better), written to =tpr_summary.csv=.
+|-------------------+-----------------------------+-----------+-------+--------------------------------|
 | roll-aoc          | kernelized_roll_aoc()       | RMSprop   | 1e-3  | wd=1e-5; KernelScheduler(32,20)|
 | roll-tpr90        | kernelized_roll_tpr(tpr=0.9)| RMSprop   | 1e-3  | wd=1e-5; KernelScheduler(32,20)|
 | bce-weighted      | BCEWithLogitsLoss           | Adam      | 1e-3  | pos_weight=imbalance_ratio     |
 | gce-0.7           | gce_loss(q=0.7)             | Adam      | 1e-3  | wd=1e-5                        |
 | mae               | mae_loss                    | Adam      | 1e-3  | wd=1e-5                        |
 | focal-loss        | focal_loss(γ=2.0)           | Adam      | 1e-3  | wd=1e-5                        |
 | asymmetric-loss   | asymmetric_loss             | Adam      | 1e-3  | wd=1e-5; γ_pos=0,γ_neg=4,clip=0.05 |
 | libauc-auroc      | AUCMLoss (Yuan ICCV 2021)   | PESG      | 0.01  | must be paired with PESG; CPU-only on MPS |
 RMSprop chosen for ROLL (not Adam) because KDE gradients are non-stationary — KernelScheduler
 anneals bandwidth during training, shifting the loss landscape. Adam's first-moment momentum
 accumulates stale directional bias from earlier kernel widths; RMSprop adapts step magnitude
 only, making it more reactive to the current gradient without directional memory.
 LibAUC/PESG: AUC is a pairwise (non-decomposable) metric; Yuan et al. reformulate as minimax
 with auxiliary scalars a≈E[f(x⁺)], b≈E[f(x⁻)], α (dual), enabling per-mini-batch updates.
 PESG updates all four variable sets jointly; loss module owns a,b,α so optimizer and loss
 are tightly coupled (hence opt_factory pattern in our code).
 ** KEEL Noise Experiment — Setup & Config Fix
 *Setup:* 12 datasets × 3 noise levels (5%, 10%, 20%) × 10 episodes × 8 configs.
 Label noise injected via =make_noisy_splitter= (random label flip on training set only).
 Results in =results-final/keel-noise-{05,10,20}/<dataset>/=.
 Runner: =experiments/keel/run_all_noise_levels.sh= → =run_all_noisy.sh= → =run_noisy.py=.
 CPU forced (6.6× faster than MPS for KEEL batch sizes); MLflow unset.
 *Config fairness bug (2026-08-28):* roll-aoc was using =max_iters=1500= / =patience=100=
 vs baselines =max_iters=5000= / =patience=500=. This is a structural training-budget
 disadvantage baked into =_base.py= (roll_config vs robust_config/bce_config).
 Fix: =experiments/keel/run_noisy_fair_roll.py= reruns roll-aoc only with fair settings;
 results swapped into =results-final= via =results-fair-roll/=.
 *Same bug applies to noiseless keel (results-final/keel/, 5 episodes) — still needs rerun.*
 ** KEEL Noise Experiment — Results (2026-08-28, fair roll-aoc config)
 Mean test AUC averaged across all 12 datasets, by noise level:
 | Config          |   5%   |  10%   |  20%   |  mean  |
 |-----------------+--------+--------+--------+--------|
 | *roll-aoc*      | 0.8713 | 0.8623 | *0.8471* | *0.8602* |
 | asymmetric-loss | 0.8719 | 0.8695 | 0.8330 | 0.8581 |
 | bce-weighted    | 0.8686 | 0.8622 | 0.8434 | 0.8581 |
 | focal-loss      | 0.8679 | 0.8573 | 0.8419 | 0.8557 |
 | gce-0.7         | 0.8704 | 0.8650 | 0.8305 | 0.8553 |
 | mae             | 0.8776 | 0.8663 | 0.8035 | 0.8491 |
 | libauc-auroc    | 0.8593 | 0.8544 | 0.8093 | 0.8410 |
 | roll-tpr90      | 0.7859 | 0.7705 | 0.7728 | 0.7764 |
 Per-dataset at 20% noise: roll-aoc wins 7/12 datasets.
 roll-aoc is the most noise-robust: degrades only −0.024 from 5%→20%;
 mae drops −0.074, libauc-auroc −0.050, asymmetric-loss −0.039.
 *Key findings for thesis:*
 - roll-aoc ranks #1 overall (mean 0.8602) once given a fair training budget.
 - Noise robustness is roll-aoc's clearest advantage — the gap widens at 20% noise.
 - mae is competitive at low noise but not noise-robust; not a strong baseline at 20%.
 - libauc-auroc consistently underperforms the simpler baselines (bce-weighted, focal).
 - asymmetric-loss is roll-aoc's closest competitor across all conditions.
 - roll-tpr90 is a poor AUC-maximiser (expected — it optimises TPR@FPR=0.1, not AUC).
 ** CIFAR-10N Experiment (2026-07-19 → 2026-07-20)
@@ -60,6 +117,14 @@ Episodes: 15.
 - =mae= and =gce-0.7= perform poorly under poisoning across all datasets.
 - =roll-aoc= and =libauc-auroc= are competitive on easy datasets (glass6, new-thyroid1, vowel0, wisconsin) where both reach ~0.99 AUC.
 ** Large Datasets
 | Experiment            | Dataset           | Status  | Notes                                        |
 |-----------------------+-------------------+---------+----------------------------------------------|
 | experiment-higgs      | HiggsDataset      | OK      | 500k samples, ~50/50 balance                 |
 | experiment-creditcard | CreditCardFraud   | MISSING | =/home/aner/.data/creditcard/creditcard.csv= |
 | experiment-homecredit | HomeCreditDataset | MISSING | =/home/aner/.data/homecredit/=               |
 ** KEEL Run 2026-07-08 — Setup
 Results folders: =/home/aner/projects/research-projects/roll/impl/results/*/2026-07-08-*/=
@@ -78,55 +143,6 @@ Changes from 2026-07-07 run:
 Scoring: =mean(fpr) + std(fpr)= across 5 episodes (penalises variance and instability; lower is better).
 Note: mean−std was tried first but incorrectly rewarded instability — corrected to mean+std.
 ** KEEL Run 2026-07-08 — Results (mean + std, FPR@TPR=0.95)
 | Dataset      | roll  | roll+bce | bce   | bce-weighted | Winner       |
 |--------------+-------+----------+-------+--------------+--------------|
 | glass0       | 0.568 | —        | 0.677 | 0.706        | roll         |
 | glass1       | 0.809 | 0.776    | 0.744 | 0.676        | bce-weighted |
 | glass2       | 0.702 | 0.642    | 0.793 | 0.683        | roll+bce     |
 | glass6       | 0.081 | 0.060    | 0.042 | 0.054        | bce          |
 | haberman     | 0.910 | 0.945    | 1.015 | 0.940        | roll         |
 | iris0        | 0.000 | 0.000    | 0.000 | 0.600        | roll/bce/roll+bce (tied) |
 | new-thyroid1 | 0.077 | 0.048    | 0.057 | 0.055        | roll+bce     |
 | pima         | 0.710 | 0.744    | 0.627 | 0.681        | bce          |
 | vehicle2     | 0.148 | 0.097    | 0.163 | 0.108        | roll+bce     |
 | vowel0       | 0.160 | 0.095    | 0.059 | 0.066        | bce          |
 | wisconsin    | 0.011 | 0.007    | 0.008 | 0.011        | roll+bce     |
 | yeast3       | 0.398 | 0.539    | 0.451 | 0.552        | roll         |
 Win count: roll+bce: 4, bce: 3, roll: 3, bce-weighted: 1
 ** KEEL Run 2026-07-08 — Findings
 - =roll+bce= is the most consistent ROLL variant: wins glass2, new-thyroid1, vehicle2, wisconsin.
 - =roll= alone wins where it has a large mean advantage that outweighs variance penalty (haberman, glass0, yeast3).
 - =bce= wins on glass6, pima, vowel0; =bce-weighted= wins glass1 but its iris0 score is inflated by one catastrophic episode.
 - No single method dominates. ROLL adds clear value on haberman, glass0, yeast3 (hard/imbalanced datasets).
 - High episode variance on haberman, yeast3, glass1 — small test sets make results noisy.
 ** Previous Run — KEEL Results (FPR@TPR=0.95, 2026-07-07, single episode, raw mean)
 | Dataset   | roll  | roll+bce | roll-fpr+bce | bce   | bce-weighted | Best          |
 |-----------+-------+----------+--------------+-------+--------------+---------------|
 | glass0    | 0.878 | 0.490    | 0.673        | 0.857 | 0.449        | bce-weighted  |
 | glass1    | 0.872 | 0.830    | 0.702        | 0.872 | 0.681        | bce-weighted  |
 | haberman  | 0.870 | 1.0      | 1.0          | 0.922 | 1.0          | roll (barely) |
 | pima      | 0.649 | 0.714    | 0.673        | 0.649 | 0.595        | bce-weighted  |
 | vehicle2  | 0.095 | 0.032    | 0.032        | 0.058 | 0.058        | roll+bce      |
 | vowel0    | 0.141 | 0.0      | 0.0          | 0.0   | 0.004        | tied          |
 | wisconsin | 0.964 | 0.0      | 0.0          | 0.0   | 0.0          | tied          |
 Config: 1 episode, ISJ bandwidth, no weight decay/dropout, lr=1e-3 for all, gamma=100/decay=500.
 ** Large Datasets
 | Experiment            | Dataset           | Status  | Notes                                        |
 |-----------------------+-------------------+---------+----------------------------------------------|
 | experiment-higgs      | HiggsDataset      | OK      | 500k samples, ~50/50 balance                 |
 | experiment-creditcard | CreditCardFraud   | MISSING | =/home/aner/.data/creditcard/creditcard.csv= |
 | experiment-homecredit | HomeCreditDataset | MISSING | =/home/aner/.data/homecredit/=               |
 ** run_all.sh
 Runs KEEL datasets via shell script in =experiments/keel/=.
@@ -276,6 +276,54 @@ All are image datasets. No standard noisy-label tabular benchmark exists.
 }
 #+end_src
 ** NIH ChestX-ray14 Experiment — Citations
 Approach: frozen pretrained DenseNet-121 (TorchXRayVision) as feature extractor → 1024-dim
 embeddings stored to disk → train MLP classifier using existing infrastructure. Justified by:
 1. *TorchXRayVision* (Cohen et al., MIDL 2022, arXiv:2111.00595) — open library with
   pretrained DenseNet-121 weights trained on NIH ChestX-ray14 + CheXpert + MIMIC-CXR.
   Explicitly designed for feature extraction / transfer learning. pip-installable.
   GitHub: https://github.com/mlmed/torchxrayvision
   Key quote: "pre-trained models can also be used as features extractors for semi-supervised
   training or transfer learning."
 2. *Transfer learning with chest X-rays for ER patient classification* (Scientific Reports 2020,
   https://www.nature.com/articles/s41598-020-78060-4) — uses frozen CheXNet output as features
   for downstream classification. Directly precedents our approach.
 BibTeX:
 #+begin_src bibtex
@inproceedings{cohen2022torchxrayvision,
  title={{TorchXRayVision}: A library of chest {X}-ray datasets and models},
  author={Cohen, Joseph Paul and Viviano, Joseph D. and Bertin, Paul and Morrison,
          Paul and Torabian, Parisa and Guarrera, Matteo and Lungren, Matthew P. and
          Chaudhari, Akshay and Brooks, Rupert and Hashir, Mohammad and Bertrand, Hadrien},
  booktitle={Medical Imaging with Deep Learning},
  year={2022},
  url={https://arxiv.org/abs/2111.00595}
 }
@article{bolourani2021transfer,
  title={Transfer learning with chest {X}-rays for {ER} patient classification},
  author={Bolourani, Sina and Brennan, Michael and Wang, Ping and Panjeton, Gregg and
          Naaraayan, Arun and Mukherjee, Vikramjit and Bhargava, Mohit and Schreiber,
          Theodore L. and Bhartia, Anubhav},
  journal={Scientific Reports},
  volume={11},
  pages={21906},
  year={2021},
  doi={10.1038/s41598-020-78060-4}
 }
 #+end_src
 Practical notes:
 - Dataset: 42GB, download to beacon-wsl (916GB free). Mac at 95% capacity, insufficient.
 - Feature extraction: one GPU pass after food101n finishes (~3.5 days from 2026-07-28)
 - Embeddings: ~450MB (112K × 1024 × float32), preload into RAM like tabular data
 - Binary task: one pathology vs rest (e.g. Effusion — highest prevalence ~12%, IR ~7)
 - Label noise: NLP-extracted from radiology reports, estimated ~20% noise rate
 ** Paper Subnodes
 - [[id:bf0fc08a-e806-48df-b188-7a2c4c41c693][impl/paper-tabpfn]] — TabPFN: in-context learning for small tabular classification (ICLR 2023)
@@ -5,23 +5,65 @@
 #+title: impl/todos
 #+filetags: :project: :todo:
-* TODO Look into previous results success/failures — understand failure modes of datasets where ROLL performs poorly (glass1, pima, glass6 from 2026-07-08 KEEL run; see [[id:a53cbe84-cd8d-45c2-a8cf-34ab520a3ea5][impl/experiments]] for full results)
+* DONE Look into previous results success/failures — understand failure modes of datasets where ROLL performs poorly (glass1, pima, glass6 from 2026-07-08 KEEL run; see [[id:a53cbe84-cd8d-45c2-a8cf-34ab520a3ea5][impl/experiments]] for full results)
-* TODO Run credit card fraud experiment
+* TODO Fill out ANIMAL-10N download form and download to Mac
-Experiment file already exists: =experiments/large/experiment-creditcard.py=.
+Form: https://forms.gle/8mbmbNgDFQ2rA1fLA
-Data at =~/.data/creditcard/creditcard.csv= (downloaded via Kaggle CLI).
+After receiving download link, run on Mac:
- Run all 7 configs (roll-aoc, bce-weighted, mae, gce-0.7, libauc-auroc, focal-loss, asymmetric-loss)
+=./scripts/run_remote.sh experiments/other/experiment-animal10n.py --detach=
- Tabular, ~285K rows, naturally imbalanced (~0.17% positive) — fast to run
+(data goes to =~/.data/animal10n/= on the Mac)
 - Supports the AUC-ROLL noisy dataset story (real-world messiness, severe imbalance)
-* TODO Track 1: controlled noise injection on KEEL datasets
+* DONE Run libauc-auroc on credit card fraud (CPU)
-Take 3–5 KEEL datasets with varying IR and inject label noise at 10%, 20%, 30% on minority class.
+Completed. Results in =results-final/creditcard/=.
-Goal: reproduce the CIFAR-10N noise-level ablation on imbalanced tabular data.
+
- Choose datasets spanning a range of IR (e.g. low/medium/high imbalance)
+* DONE Re-run credit card fraud weighted baselines
- Write noise injection utility (flip minority labels randomly at given rate)
+Completed. All 8 configs present in =results-final/creditcard/=.
- Run all 7 configs per dataset × noise level
+
- Expected result: ROLL-AUC advantage grows with noise level (mirrors CIFAR finding)
+* DONE Re-run all KEEL experiments with weighted baselines
- Key for thesis: validates the noise-sensitivity story on tabular domain
+Completed as of 2026-07-27. All 12 datasets × 8 configs in =results-final/keel/=.
 =_base.py= confirmed to pass =pos_weight=imbalance_ratio= to gce-0.7, mae, focal-loss, asymmetric-loss.
 * ACTIVE Food-101N experiment
 Running on beacon-wsl (as of 2026-07-28). =experiment-food101n-rerun-weighted.py= — 6 configs × 3 episodes.
 6/18 episodes done as of 2026-07-28 (~3 days remaining).
 Writes to =results-final/food101n/=.
 Tail: =ssh aner@192.168.1.200 'tail -f ~/roll-impl/run.log'=
 Fetch: =./scripts/fetch_results_beacon.sh=
 * TODO Resume NIH ChestX-ray14 experiment
 Crashed 2026-07-31 ~20:51 on Mac during =libauc-auroc= config.
 Error: =ValueError: All episodes have NaN/inf predictions — cannot build ROC curve=
 - Crash in =src/summary.py= ROC generation after training, not during training itself
 - The 6 non-libauc configs (roll-aoc, roll-tpr90, bce-weighted, gce-0.7, mae, focal-loss, asymmetric-loss) may have completed — check =results-final/chestxray14/= before resuming
 - Fix: diagnose why libauc produces NaN/inf outputs; likely learning rate or optimizer issue with PESG on MPS/CPU
 - Run script: =experiments/other/experiment-chestxray14.py= — writes directly to =results-final/chestxray14/=
 - Resume via: =./scripts/run_remote.sh experiments/other/experiment-chestxray14.py --detach=
  (after fixing the libauc NaN issue or skipping libauc-auroc)
 * TODO CIFAR-100N experiment
 Same paper and infrastructure as CIFAR-10N (Wei et al. 2022, arXiv:2110.12088).
 Download: ~160MB — same CIFAR-100 images + noise label file, no new image download needed.
 Key difference from CIFAR-10N: noise rate is 40.2% (vs 9–40% for CIFAR-10N) — harder noise regime.
 Setup: binarize same way as CIFAR-10N (one class vs rest), reuse ConvNet architecture.
 Run on Mac (trivial compute, same as CIFAR-10N).
 Priority: HIGH — near-zero effort, extends noise ablation to a harder regime.
 * TODO ISIC 2020 Melanoma experiment
 Binary dermoscopy classification: malignant (584 images, 1.8%) vs benign (32,542 images, 98.2%). IR ≈ 56:1.
 Label noise: real inter-dermatologist disagreement and clinical ambiguity (not annotator error).
 Download: ~2–4GB via Kaggle pre-resized 224×224 JPEG dataset (nischaydnk/isic-2020-jpg-224x224-resized).
 Architecture: pretrained CNN backbone (same pattern as ChestX-ray14 — frozen backbone + linear head or fine-tune).
 Run on beacon-wsl after Food-101N + ChestX-ray14 complete.
 Priority: MEDIUM-HIGH — strong clinical story, extreme imbalance + real noise, feasible size.
 Note: only 584 positive examples — conclusions will be limited; mention in thesis.
 * ACTIVE KEEL label noise ablation
 Running 5%, 10%, 20% label noise on all 12 KEEL datasets (10 episodes each) on Mac.
 Script: =experiments/keel/run_all_noise_levels.sh= (launched 2026-08-01)
 Results: =results-final/keel-noise-05/=, =results-final/keel-noise-10/=, =results-final/keel-noise-20/=
 (date-stamped subdirs per dataset, e.g. =keel-noise-05/glass0/YYYY-MM-DD-HH-MM/=)
 0% baseline: =results-final/keel/= (existing, 5 episodes)
 Fetch when done: =./scripts/fetch_results.sh=
 * TODO CheXpert experiment
 ~224K chest X-ray images, label noise from automated NLP extraction of radiology reports.
@@ -32,28 +74,21 @@ Dataset: https://stanfordmlgroup.github.io/competitions/chexpert/ (requires regi
 - Medium-large compute cost; run on remote Mac or cluster
 - Priority: lower than credit card fraud and Track 1; run after those confirm the story
 * TODO NIH ChestX-ray14 experiment
 ~112K chest X-ray images, label noise from NLP extraction of radiology reports.
 Similar story to CheXpert but smaller; could substitute if CheXpert access is delayed.
 Dataset: https://nihcc.app.box.com/v/ChestXray-NIHCC
 - Same setup considerations as CheXpert
 - Priority: same as CheXpert — run one or the other, not necessarily both
 * TODO ANIMAL-10N experiment
 Dataset loader written: =Animal10NDataset= in =src/datasets.py=.
-Data must be manually downloaded from https://nihalsid.github.io/animal-10n/ and extracted to =~/.data/animal10n/= (train/ and test/ subdirs).
+Experiment script written: =experiments/other/experiment-animal10n.py=.
-Next steps:
+Data must be downloaded after filling form (see todo above).
- Download the dataset on the remote Mac
+- ConvNet with image_size=64 (4096 linear), positive_class=0 (cat), IR ~9
- Write =experiments/other/experiment-animal10n.py= (all 7 configs: roll-aoc, bce-weighted, mae, gce-0.7, libauc-auroc, focal-loss, asymmetric-loss)
+- MPS-aware: sequential_episodes=True, libauc forced to CPU
 - ConvNet needs adjusted Linear layer: 64×64 images → after 3 MaxPool → 8×8 spatial → 64*8*8=4096 flatten (vs 1024 for CIFAR)
 - positive_class=0 (cat vs rest), IR ~9, same setup as CIFAR-10N
-* TODO Food-101N experiment
+* TODO Noisy Ostracods experiment
-~310K training images, ~18.4% web noise. Binary: one class vs rest.
+NeurIPS 2024 D&B track (arXiv:2412.02313). ~71k micro-crustacean specimens, specialist-annotated.
-Dataset: https://kuanghuei.github.io/Food-101N-Dataset/ (requires Google account for download)
+IR = 22,429 (extremely imbalanced), ~5.6% real expert-level label noise.
- Write =Food101NDataset= in =src/datasets.py= using torchvision.datasets.ImageFolder
+Download: estimated ~7–15GB JPEG (476×476px avg), hosted on SharePoint via GitHub repo.
- Images are 224×224; may need ResNet backbone or resize to 64×64 for ConvNet
+GitHub: https://github.com/H-Jamieu/Noisy_ostracods
- Lower priority than ANIMAL-10N (larger, more infra work)
+Binarization needed: multi-class (78 genera), would need to pick one vs rest.
 Caveat: paper reports existing noise-robust methods fail to beat CE — a strong ROLL result would be notable.
 Priority: LOW — more setup friction (SharePoint, not Kaggle), binarization decision needed, uncertain payoff.
 * TODO Clothing1M experiment
 ~1M images, ~38.5% web noise. Binary: one category vs rest.
@@ -5,6 +5,32 @@
 #+title: impl/work-history/recent
 #+filetags: :project: :work-history:
 * 2026-08-28 Thu
 - Returned from vacation; checked both remote machines for run status
 - beacon-wsl (food101n =experiment-food101n-rerun-weighted.py=): found crashed during gce-0.7 ep1 ~2 weeks ago — root cause: =MLFLOW_TRACKING_URI= was set pointing to =mlflow.zakobar.com=, server went down mid-training causing =MlflowException= (502s then DNS failure); bce-weighted (3/3) and mae (3/3) were complete, gce-0.7 ep0 saved, ep1 dir empty
 - Fixed MLflow crash vector: (1) unset =MLFLOW_TRACKING_URI= at top of resume script; (2) added try/except in =_MLFlowRun.track= to suppress per-metric exceptions without crashing training
 - Created =experiment-food101n-rerun-weighted-resume.py= with per-episode skip logic (checks =test-res.pkl= existence); launched detached on beacon-wsl; confirmed GPU active (RTX 3050), gce-0.7 ep1+2 and focal-loss ep0 completed, focal-loss ep1 in progress as of last check (~35h remaining)
 - mac (keel noise run from Aug 26): fully completed — all 12 datasets × 3 noise levels × 8 configs × 10 episodes done
 - Fetched keel noise results from mac; flattened =results-final/keel-noise-{05,10,20}/= structure (=init_experiment= had inserted timestamp subdirs; removed them, kept latest Aug-26 run per dataset, discarded stale Aug-01 partial dirs)
 - Discovered roll-aoc config disadvantage in KEEL runs: =max_iters=1500= and =patience=100= vs baselines =max_iters=5000= / =patience=500= (3.3× fewer epochs, 5× less patience); confirmed no override in =run_noisy.py= or =run_all_noisy.sh=
 - Created =experiments/keel/run_noisy_fair_roll.py=: reruns roll-aoc only across all 12 datasets × 3 noise levels × 10 episodes with fair config (=max_iters=5000=, =patience=500=); has skip logic; writes to =results-fair-roll/keel-noise-{05,10,20}/<dataset>/roll-aoc/=; launched detached on mac (CPU, no MLflow)
 - Plan: when =results-fair-roll= is done, swap =roll-aoc/= dirs into =results-final/keel-noise-*/= to replace unfair results
 * 2026-08-01 Fri
 - Checked experiment status: beacon-wsl running food101n (5/21 episodes done, bce-weighted ep3 in progress); Mac idle — chestxray14 had crashed Jul 31 20:51
 - Diagnosed chestxray14 crash: =ValueError: All episodes have NaN/inf predictions — cannot build ROC curve= during ROC summary generation after libauc-auroc training; non-libauc configs may have completed; logged as TODO with resume notes
 - Set up KEEL label noise ablation (5%, 10%, 20%) × 12 datasets × 10 episodes → =results-final/keel-noise-{05,10,20}/=; created =experiments/keel/run_all_noise_levels.sh=
 - Added =--results-dir=, =--cpu=, =--no-mp= flags to =run_noisy.py= and =run_all_noisy.sh=; added =force_cpu= param to =run_keel_experiment= in =_base.py=
 - Benchmarked MPS vs CPU for KEEL-sized batches (72 samples, 9 features, 3-layer MLP): CPU is 6.6x faster — MPS kernel dispatch overhead dominates at tiny batch sizes
 - Discovered =sequential_episodes=True= is dead code for CPU + =is_mp=True= path: flag only gates the MPS subprocess branch; CPU always uses =mp.Pool(n_episodes)= when =is_mp=True=, regardless of the flag
 - Discovered: MLflow was connected to remote server in the original MPS run, adding per-epoch network I/O that dominated wall time (~1.6s/epoch vs 54ms without)
 - Changed =gen_criteria= logging from =logging.info= to =logging.debug= (reduces StreamHandler flush overhead per epoch; per-epoch losses still captured in debug.log)
 - Settled on =--cpu --no-mp= + =unset MLFLOW_TRACKING_URI= for all KEEL noisy runs; hardcoded in =run_all_noise_levels.sh=
 - Identified residual per-epoch overhead: ~54ms (ExperimentDataLoader shuffle+argwhere per epoch, =pd.DataFrame= construction ×2 in =gen_criteria=, extra loss forward in =_get_loss=) vs 0.5ms raw compute
 - Launched final KEEL noise ablation on Mac: CPU, no-mp, no MLflow; estimated ~21h for all 3 noise levels
 * 2026-07-24 Thu
 - Investigated stalled cifar10n remote run: 5 workers from 6:45AM were still alive after 15.5h with 0 results; previous session's ps check had missed them, making the run appear dead
@@ -26,9 +26,32 @@ BGU MSc thesis on the ROLL method (Rate Optimized Likelihood Loss). See [[id:d23
 - Chapters added via =\input{content/<chapter>/<chapter>}= with a matching =\addtocontents{tocheb}= line
 - Build engine: pdflatex (uses =\usepackage[utf8x]{inputenc}=, not xelatex)
 - Math notation: data points are bold (=\mathbf{x}_i=); batches are =\mathcal{B}_0= / =\mathcal{B}_1=
- KDE background (general technique) in Ch. 2 (=sec:kde-background=); Ch. 3 back-references it; Ch. 2 ends with forward pointer to =sec:roll-kde=
+- KDE background (general technique) in Ch. 2 (=sec:kde-background=); Ch. 3 back-references it
 - =sec:imbalanced-tpr-fpr= and =sec:neyman-pearson= live in Ch. 2; Ch. 3 =sec:problem-formulation= \Cref's them — do not rename
 ** Mentor Direction (2026-08-17)
 Primary narrative focus agreed with mentor:
 - *Main story*: noise resistance and AUC optimization — this is the thesis core
 - *Hero method*: =roll-aoc= (AUC variant) — primary contribution
 - *Primary competitor*: =libauc-auroc= (Yuan et al. AUCMLoss + PESG) — already implemented
 - *Single-point optimization* (=roll-tpr90=): touch on briefly as secondary result
 - *Pacing*: get to the differentiating, strong results quickly — don't bury the lede
 ** Baselines (all in =impl/src/roll.py= + =impl/experiments/keel/_base.py=)
 | Experiment name | Class | What it is |
 |---|---|---|
 | =bce-weighted= | BCE + pos_weight | Standard imbalance fix |
 | =libauc-auroc= | =libauc_auc_loss= + PESG | Yuan et al. 2021 compositional AUC — primary AUC competitor |
 | =gce-0.7= | =gce_loss= | Generalized Cross-Entropy (noise-robust) |
 | =mae= | =mae_loss= | MAE loss (noise-robust) |
 | =focal-loss= | =focal_loss= | Focal Loss |
 | =roll-aoc= | =kernelized_roll_aoc= | *ROLL AUC variant — hero method* |
 | =roll-tpr90= | =kernelized_roll_tpr= | ROLL TPR@FPR variant — secondary result |
 Note: =libauc-auroc= requires PESG optimizer (not Adam) — already wired up in impl.
 ** Chapter Structure
 See subnodes for per-chapter guidelines and section details.
@@ -50,7 +73,7 @@ See subnodes for per-chapter guidelines and section details.
 - =nix develop= devShell includes python3+sympy — use it to verify math claims, never check mentally
 - KDE backward pass uses inverse-function-theorem approach deliberately (user must explain it verbally)
 - Correct gradient of threshold w.r.t. negative-class score: σ'(τ−x_i) / Σ_j σ'(τ−x_j) — sympy-verified
- **(NIX BUILD) New .tex files must be =git add='ed before =nix build=.** Nix flakes use git to enumerate source files; untracked files are invisible to the build. Symptom: silent LaTeX failure — TOC, LoF, and LoT all appear empty in the output PDF. Fix: =git add <new-file>=, then rebuild.
+- *(NIX BUILD) New .tex files must be =git add='ed before =nix build=.* Nix flakes use git to enumerate source files; untracked files are invisible to the build. Symptom: silent LaTeX failure — TOC, LoF, and LoT all appear empty in the output PDF. Fix: =git add <new-file>=, then rebuild.
 ** Key Files
@@ -63,9 +86,11 @@ See subnodes for per-chapter guidelines and section details.
 - [[id:c4a0d60f-49d3-4185-8c5b-f8a5a10a2ccc][thesis/introduction]] — Ch. 1 guidelines: structure, length, what to include
 - [[id:cd785954-008e-4b64-957d-25d1dbeac127][thesis/related-work]] — Ch. 2 section structure, placement markers, label conventions
 - [[id:0543ac39-af9b-40fd-9973-e60576a20695][thesis/method]] — Ch. 3 section status table, derivation gotchas
- [[id:59f84838-71a9-4aab-a494-8e6bea2080fb][thesis/experiments]] — Ch. 4 guidelines: datasets, baselines, metrics
+- [[id:c41756b5-fd5f-456e-ab2b-18f27db7d61d][thesis/experiments]] — Ch. 4 data sources, key results, section status
 - [[id:9b50caf2-d259-4848-b7f5-22a9288e8ba2][thesis/discussion]] — Ch. 5–6 guidelines: failure modes, future work
 - [[id:151159e8-1105-4a84-af9c-72d31c5e0660][thesis/todos]] — project TODOs
 - [[id:bd98ede2-d6ec-4e93-8d10-4800fe528ff7][thesis/citations]] — sources justifying implementation choices (architectures, baselines)
 - [[id:2c598296-d2f0-4333-9cdd-3625f8375e03][thesis/work-history/recent]] — session work log
 * todo
@@ -0,0 +1,35 @@
 :PROPERTIES:
 :ID: bd98ede2-d6ec-4e93-8d10-4800fe528ff7
 :END:
 #+title: thesis/citations
 #+filetags: :project: :knowledge: :citations:
 :PROPERTIES:
 :ID: thesis-citations
 :END:
 #+title: thesis/citations
 #+filetags: :project: :knowledge: :citations:
 Sources used to justify implementation choices in the thesis. Linked from [[id:6294e2be-6189-4473-b363-a1dd9a75ff9b][thesis]].
 * Bank Marketing Dataset — Model Architecture
 ** Justification
 The standard neural network architecture for the UCI Bank Marketing dataset in the literature is a 2-hidden-layer MLP with 128 units per layer and ReLU activations. This is used in our =experiment-bank-marketing.py= as:
 =Linear(input, 128) → ReLU → Dropout(0.3) → Linear(128, 128) → ReLU → Dropout(0.3) → Linear(128, 1)=
 ** Sources
 - [[https://github.com/ktamburi/Bank-Marketing-UCI-Dataset-Neural-Network][ktamburi/Bank-Marketing-UCI-Dataset-Neural-Network (GitHub)]]
  Most-cited student/practitioner reference for NN on this dataset. Uses exactly 128→128 architecture with ReLU, final softmax (we use sigmoid for binary). Justifies our hidden layer size choice.
 - [[https://arxiv.org/pdf/2407.00956][A Closer Look at Deep Learning Methods on Tabular Datasets (arXiv 2407.00956)]]
  Benchmarks MLP and TabNet on tabular datasets including bank marketing. Confirms that simple MLPs with 2-3 layers are the standard DL baseline for this domain.
 - [[https://doi.org/10.3390/e28030354][Risk-Sensitive ML for Financial Decision Modeling Under Imbalanced Data (Entropy 2025)]]
  Directly addresses class imbalance on the bank marketing dataset. Supports our use of pos_weight and AUC as the primary metric given IR ~8:1.
 - [[https://medium.com/@data-overload/bank-marketing-data-imbalanced-dataset-42140d6d4b95][Bank Marketing Data — Imbalanced Dataset (Medium)]]
  Confirms the dataset has ~11.3% positive rate (IR ~8:1), motivating use of imbalance-aware losses (GCE, MAE, focal, asymmetric, ROLL).
@@ -5,20 +5,91 @@
 #+title: thesis/experiments
 #+filetags: :project: :knowledge: :experiments:
-:PROPERTIES:
+Chapter 4 of the BGU MSc thesis on ROLL. Parent: [[id:6294e2be-6189-4473-b363-a1dd9a75ff9b][thesis]]
 :ID: thesis-experiments
 :END:
-#+title: thesis/experiments
+** Data Sources
 #+filetags: :project: :knowledge: :experiments:
-Chapter 4 of the BGU MSc thesis on ROLL. Not yet written.
+- =results-final/keel/= — 12 KEEL datasets × 8 methods × 5 episodes, clean labels
-Parent: [[id:6294e2be-6189-4473-b363-a1dd9a75fb9b][thesis]]
+- =results-final/keel-noise-05/10/20/= — same 12 datasets × 8 methods × 10 episodes, asymmetric noise
 - =results-final/cifar10n/= — 3 noise levels (clean/aggre/worse) × 7 methods × 3 episodes; no roll-tpr90
 - =results-final/bank-marketing/=, =creditcard/=, =homecredit/= — 5 episodes each, all methods
 - =results-final/chestxray14/= — 5 episodes × 7 methods (libauc missing due to bug); image dataset
-** Guidelines
+** ChestXray14 Setup (IMPORTANT)
- Target length: 15–20 pages
+- Dataset: NIH ChestX-ray14 (~112K frontal X-rays, 30,805 patients)
- Datasets: KEEL imbalanced benchmark suite + larger datasets TBD
+- Task: Effusion (pleural effusion) vs. rest; IR ~7.5:1, ~12% prevalence
- Baselines should include: cross-entropy + class weighting, AUC-surrogate methods, NP-inspired methods
+- Label noise: ~20% organic — labels from NLP extraction of radiology reports (NOT injected)
- Primary metric: TPR at fixed FPR; secondary: partial AUC, comparison across operating points
+- Feature extraction: DenseNet-121 pretrained on NIH via TorchXRayVision (=densenet121-res224-nih=)
- Report results for all three ROLL instantiations (Gaussian, Beta, KDE) to justify the progression
+  - Frozen; global avg pool of penultimate feature map → 1024-dim embeddings
  - Script: =scripts/extract_chestxray14_features.py=
 - Classifier: 1024→512→128→1 MLP with Dropout(0.3)
 - n=5 episodes; LibAUC excluded (PESG optimizer instability on this task)
 - Experiment: =experiments/other/experiment-chestxray14.py=
 ** Key Results
 *** ChestXray14 (primary new result — real-world organic noise)
 | Method   | AUC   | Std   |
 |----------|-------|-------|
 | roll-aoc | 0.839 | 0.001 | ← BEST, clear win
 | gce-0.7  | 0.829 | 0.004 |
 | roll-tpr | 0.828 | 0.002 |
 | mae      | 0.810 | 0.001 |
 | focal    | 0.798 | 0.009 |
 | bce-w    | 0.788 | 0.008 |
 | asym.    | 0.729 | 0.007 |
 | libauc   | ---   |       |
 *** KEEL Noise Experiment (primary thesis result)
 Mean AUC across all 12 KEEL datasets at increasing asymmetric label-noise rates:
 | Method    | 0%    | 5%    | 10%   | 20%   | Drop 0→20 |
 |-----------|-------|-------|-------|-------|-----------|
 | roll-aoc  | 0.880 | 0.871 | 0.862 | 0.847 | −0.033    |
 | bce-w     | 0.872 | 0.869 | 0.862 | 0.843 | −0.029    |
 | focal     | 0.879 | 0.868 | 0.857 | 0.842 | −0.037    |
 | asym.     | 0.892 | 0.872 | 0.870 | 0.833 | −0.059    |
 | gce-0.7   | 0.886 | 0.870 | 0.865 | 0.831 | −0.056    |
 | libauc    | 0.882 | 0.859 | 0.854 | 0.809 | −0.073    |
 | mae       | 0.875 | 0.878 | 0.866 | 0.804 | −0.071    |
 | roll-tpr  | 0.843 | 0.786 | 0.771 | 0.773 | −0.070    |
 *ROLL-AOC is best at 20% noise* despite being mid-pack at 0%. GCE/MAE degrade most.
 *** CIFAR-10N (image noise)
 | Noise  | roll-aoc | gce   | mae   | libauc | bce   | focal | asym  |
 |--------|----------|-------|-------|--------|-------|-------|-------|
 | clean  | 0.980    | 0.976 | 0.976 | 0.979  | 0.977 | 0.977 | 0.972 |
 | aggre  | 0.934    | 0.932 | 0.930 | 0.921  | 0.924 | 0.912 | 0.910 |
 | worse  | 0.821    | 0.774 | 0.809 | 0.752  | 0.738 | 0.734 | 0.720 |
 *** Large Datasets
 - Credit Card Fraud: roll-tpr90 best (0.974), roll-aoc close (0.968)
 - Home Credit: roll-aoc best (0.712), libauc very close (0.711)
 - Bank Marketing: ROLL FAILS — roll-aoc 0.511, roll-tpr90 0.438; GCE best (0.746). Open question.
 ** Gotchas
 - "Label-noise experiment" is NOT a poisoning attack — it's asymmetric noise injection (flip pos→neg)
 - keel-noise runs use 10 episodes; clean keel uses 5
 - libauc-auroc missing 2 episodes for glass2 in clean KEEL (n=58 not 60)
 - cifar10n has no roll-tpr90 variant; table excludes it
 - chestxray14 libauc missing entirely — PESG optimizer instability; needs rerun
 - Bank Marketing failure cause unknown
 ** Section Structure
 | Section | Label | Status |
 |---------|-------|--------|
 | Datasets (incl. ChestXray14) | sec:experiments:data_sets | Done |
 | Performance Measures | sec:experiments:performance_measures | Done |
 | Configurations | sec:experiments:configurations | Done |
 | KEEL Results | sec:results:keel | Done |
 | CIFAR-10N Results | sec:results:cifar10n | Done |
 | Large Datasets Results | sec:results:large | Done |
 | ChestXray14 Results | sec:results:chestxray14 | Done |
 | Label-Noise Robustness on KEEL | sec:experiments:noise | Done |
 | Ablation Study | sec:experiments:ablation | Stub |
 | Sensitivity Analysis | sec:experiments:sensitivity | Stub |
 | Summary | sec:experiments:summary | TODO |
@@ -0,0 +1,78 @@
 :PROPERTIES:
 :ID: c41756b5-fd5f-456e-ab2b-18f27db7d61d
 :END:
 #+title: thesis/experiments
 #+filetags: :project: :knowledge: :experiments:
 :PROPERTIES:
 :ID: 59f84838-71a9-4aab-a494-8e6bea2080fb
 :END:
 #+title: thesis/experiments
 #+filetags: :project: :knowledge: :experiments:
 Chapter 4 of the BGU MSc thesis on ROLL. Parent: [[id:6294e2be-6189-4473-b363-a1dd9a75fb9b][thesis]]
 ** Data Sources
 - =results-final/keel/= — 12 KEEL datasets × 8 methods × 5 episodes, clean labels
 - =results-final/keel-noise-05/10/20/= — same 12 datasets × 8 methods × 10 episodes, asymmetric noise
 - =results-final/cifar10n/= — 3 noise levels (clean/aggre/worse) × 7 methods × 3 episodes; no roll-tpr90
 - =results-final/bank-marketing/=, =creditcard/=, =homecredit/= — 5 episodes each, all methods
 - Each directory contains per-dataset =auc.csv= files (already aggregated from pkl)
 ** Key Results
 *** KEEL Noise Experiment (primary thesis result)
 Mean AUC across all 12 KEEL datasets at increasing asymmetric label-noise rates:
 | Method    | 0%    | 5%    | 10%   | 20%   | Drop 0→20 |
 |-----------|-------|-------|-------|-------|-----------|
 | roll-aoc  | 0.880 | 0.871 | 0.862 | 0.847 | −0.033    |
 | bce-w     | 0.872 | 0.869 | 0.862 | 0.843 | −0.029    |
 | focal     | 0.879 | 0.868 | 0.857 | 0.842 | −0.037    |
 | asym.     | 0.892 | 0.872 | 0.870 | 0.833 | −0.059    |
 | gce-0.7   | 0.886 | 0.870 | 0.865 | 0.831 | −0.056    |
 | libauc    | 0.882 | 0.859 | 0.854 | 0.809 | −0.073    |
 | mae       | 0.875 | 0.878 | 0.866 | 0.804 | −0.071    |
 | roll-tpr  | 0.843 | 0.786 | 0.771 | 0.773 | −0.070    |
 *ROLL-AOC is best at 20% noise* despite being mid-pack at 0%. GCE/MAE (designed for noise) degrade most.
 *** CIFAR-10N (image noise)
 | Noise  | roll-aoc | gce   | mae   | libauc | bce   | focal | asym  |
 |--------|----------|-------|-------|--------|-------|-------|-------|
 | clean  | 0.980    | 0.976 | 0.976 | 0.979  | 0.977 | 0.977 | 0.972 |
 | aggre  | 0.934    | 0.932 | 0.930 | 0.921  | 0.924 | 0.912 | 0.910 |
 | worse  | 0.821    | 0.774 | 0.809 | 0.752  | 0.738 | 0.734 | 0.720 |
 ROLL-AOC best at all three noise levels on CIFAR-10N.
 *** Large Datasets
 - Credit Card Fraud: roll-tpr90 best (0.974), roll-aoc close (0.968); both beat all baselines
 - Home Credit: roll-aoc best (0.712), libauc very close (0.711)
 - Bank Marketing: ROLL FAILS — roll-aoc 0.511, roll-tpr90 0.438; GCE best (0.746). Open question.
 ** Gotchas
 - "Label-noise experiment" is NOT a poisoning attack — it's asymmetric noise injection (flip pos→neg)
 - keel-noise runs use 10 episodes; clean keel uses 5. Mixing is fine for averages.
 - libauc-auroc missing 2 episodes for glass2 in clean KEEL (n=58 not 60) — known gap
 - cifar10n has no roll-tpr90 variant; table excludes it
 - Bank Marketing failure cause unknown; do not claim ROLL is universally superior on large tabular data
 ** Section Structure
 | Section | Label | Status |
 |---------|-------|--------|
 | Datasets | sec:experiments:data_sets | Done |
 | Performance Measures | sec:experiments:performance_measures | Done |
 | Configurations | sec:experiments:configurations | Done |
 | KEEL Results | sec:results:keel | Done — table filled |
 | CIFAR-10N Results | sec:results:cifar10n | Done — table filled |
 | Large Datasets Results | sec:results:large | Done — table filled |
 | Label-Noise Robustness on KEEL | sec:experiments:noise | Done — table added |
 | Ablation Study | sec:experiments:ablation | Stub — future work |
 | Sensitivity Analysis | sec:experiments:sensitivity | Stub — future work |
 | Summary | sec:experiments:summary | TODO — key points noted in comments |
@@ -5,16 +5,22 @@
 #+title: thesis/method
 #+filetags: :project: :knowledge: :method:
 :PROPERTIES:
 :ID: thesis-method
 :END:
 #+title: thesis/method
 #+filetags: :project: :knowledge: :method:
 Chapter 3 of the BGU MSc thesis on ROLL. Core contribution chapter. Currently ~30 pages.
 Parent: [[id:6294e2be-6189-4473-b363-a1dd9a75fb9b][thesis]]
 ** How roll-aoc Works (CRITICAL — post-pivot hero method)
 =roll-aoc= optimizes full AUC by:
 1. Sampling 100 evenly-spaced FPR points across [0, 1]
 2. Computing ROLL's TPR estimate at each point
 3. Averaging — this is a Monte Carlo approximation of the AUC integral
 This is *not* partial AUC (which integrates over a sub-range).
 This is *not* the pairwise ranking formulation (LibAUC/compositional AUC).
 It is a direct numerical integration of the ROC curve via 100 ROLL evaluations.
 This distinction must be stated clearly in Ch. 3 and Ch. 2 related work.
 ** Guidelines
 - sec:problem-formulation is now notation-only — motivation deferred to Ch. 2
@@ -48,4 +54,4 @@ Parent: [[id:6294e2be-6189-4473-b363-a1dd9a75fb9b][thesis]]
  σ'(τ − x_i) / Σ_j σ'(τ − x_j) where sum is over ALL of B_0 — sympy-verified
 - Beta ROLL: translation-invariance does NOT hold when scores pass through sigmoid before fitting;
  gradient-balance property breaks for Beta — documented in sec:roll-properties
- Gradient-balance proof uses a uniform-shift argument; requires translation-invariant CDF estimator
+- Gradient-balance proof uses a uniform-shift argument; requires translation-invariant CDF estimator
@@ -5,5 +5,70 @@
 #+title: thesis/todos
 #+filetags: :project: :todo:
 * Writing — chapters
 * TODO Write Ch. 1 (Introduction)
  - Hook: imbalance + noise are everywhere, standard losses fail silently
  - Gap: no existing method does AUC optimization + noise robustness together
  - ROLL's claim: distributional objective, robustness as structural consequence
  - Close with chapter roadmap
 * TODO Write Ch. 5 (Discussion)
  - Explain WHY ROLL-AOC is noise robust (KDE distributional fit vs isolated gradients)
  - Explain WHY Bank Marketing fails (open question, speculate honestly)
  - Explain WHY GCE/MAE fail under asymmetric imbalanced noise
  - Compare mechanism with LibAUC (same objective, different path)
 * TODO Write Ch. 6 (Conclusion)
  - Summary: differentiable AUC surrogate via score-distribution fitting
  - Key result: noise robustness validated on KEEL (5/10/20%) and CIFAR-10N (≤40%)
  - Limitations: Bank Marketing, 99-pass cost, bandwidth scheduler hyperparameter
  - Future work: learned bandwidth, partial-AUC variant, multi-class extension
 * TODO Finish Ch. 4 (Results) — remaining stubs
  - Write Summary section (sec:experiments:summary) — key points noted in comments
  - Decide: write or drop Ablation Study and Sensitivity Analysis sections
  - Fill all \cite{TODO} placeholders in experiments.tex
 * TODO Review Ch. 4 (Results) end-to-end
  - Check narrative flow after noise section was added
  - Confirm KEEL/CIFAR-10N/large-dataset table interpretations are accurate
  - Verify roll-tpr90 is correctly described as secondary/special-case throughout
 * Writing — front/back matter
 * TODO Write English abstract
 * TODO Write Hebrew abstract
 * TODO Fill Acknowledgements (Main_pages/acknowledgements.tex)
 * Writing — Ch. 2 & 3 loose ends
 * TODO Verify alpha notation w.r.t. FPR — eq:fpr-cdf writes FPR(τ) = F̂₀(τ) and threshold as τ = F̂₀⁻¹(α), but standard FPR = P(score > τ | y=0) = 1−F₀(τ) requires τ = F̂₀⁻¹(1−α). Check for sign/notation inconsistency throughout method.tex and appendix_gauss_grad.tex.
 * TODO Add graphs of CDF tracking for gaussian, KDE, roll.
 * TODO Cite in sec:related-work (Ch. 2):
  - Benchmarking Deep AUROC Optimization (arXiv 2203.14177) — cite in 2.2 as survey of AUROC losses; finding: composite losses best overall, barrier hinge best under noise
  - LibAUC pAUCLoss / SOPAs (arXiv 2203.01505) — cite in 2.3 as partial AUC baseline
  - "Closing the Approximation Gap" (arXiv 2512.01213, Dec 2025) — cite in 2.3 as recent concurrent work on partial AUC
  - "When AUC meets DRO" — cite in 2.3 as DRO-based partial AUC approach
 * Experiments — open questions
 * TODO Debug banking dataset — results unexpectedly low (roll-aoc 0.511 vs GCE 0.746), investigate root cause
 * TODO Check if LibAUC pAUCLoss (one-way partial AUC / SOPAs) should be added as baseline
  - Already have LibAUC as dependency; low-cost to add
  - Most direct competitor to roll-tpr90 (both target a fixed FPR region)
  - See: https://docs.libauc.org/examples/pauc.html
 * Final pass
 * TODO Build full PDF and proofread end-to-end
 * TODO Check all cross-references (\Cref, \label) are consistent after restructuring
 * TODO Verify Hebrew ToC entries match English section titles
 * DONE [#A] Rerun keel noised experiments — data exists in results-final/keel-noise-05/10/20
 * DONE Restructure thesis per mentor guidelines (2026-08-17)
  - [x] Focus framing on noise resistance + AUC solution (primary story)
  - [x] Minimize single-point optimization coverage (brief mention only)
  - [x] Ch. 2 restructured: ROC/AUC → Noisy Labels → KDE → Imbalanced → NP
  - [x] Ch. 3 restructured: AUC as primary objective, ROLL-AOC promoted
  - [x] Noise results table added to Ch. 4
@@ -0,0 +1,32 @@
 :PROPERTIES:
 :ID: 2c598296-d2f0-4333-9cdd-3625f8375e03
 :END:
 #+title: thesis/work-history/recent
 #+filetags: :project: :work-history:
 * 2026-08-28 Thu
 - Pivoted Ch. 3 (method.tex) from TPR@FPR to AUC as primary objective: rewrote Problem Formulation with AUC integral as eq:objective-auc, fixed TPR/FPR notation to functions of τ, demoted single-threshold objectives to special case
 - Reframed ROLL Framework intro to lead with AUC-via-numerical-integration story; NP framing moved to subordinate paragraph
 - Renamed "General ROLL Formulation and Derivation" → "Per-Threshold ROLL Loss"; renamed "ROLL variant - direct AUC optimisation" → "ROLL-AOC: AUC Optimization" with rewritten intro positioning it as the primary objective
 - Added ROLL-AOC framing sentence to each distribution instantiation (Gaussian, Beta, KDE)
 - Added Corollary (gradient balance under ROLL-AOC) and Remark (gradient coverage broadening under ROLL-AOC) to Properties section; requalified Observation as single-threshold
 - Explored all results-final CSVs; verified existing tables (KEEL clean, CIFAR-10N, large datasets) are correct — no changes needed
 - Computed KEEL noise experiment aggregates (asymmetric noise at 5/10/20%, 10 episodes each, 12 datasets)
 - Key finding: ROLL-AOC drops 3.7% from 0→20% noise; GCE/MAE drop 6–8% despite being designed for noise robustness; ROLL-AOC is #1 at 20% noise
 - Renamed Ch. 4 "Label-Noise Poisoning Experiment" → "Label-Noise Robustness on KEEL"; wrote protocol and results prose; added tab:results:keel-noise with 4 noise levels × 8 methods
 - Updated Summary section comments in experiments.tex with actual findings to synthesize
 - Created thesis/experiments org-roam subnode with full data source map, key numbers table, and section status
 - Restructured Ch. 2 (related_work.tex): new section order — ROC/AUC (2.1) → Noisy Labels (2.2) → KDE (2.3) → Imbalanced Classification (2.4) → NP (2.5) — to foreground AUC + noise as thesis core
 - Dropped section 2.3 (Partial AUC and TPR@FPR) entirely; TPR@FPR objective absorbed into Imbalanced section, NP moved up
 - Moved NP section from end of chapter to position 2.2, then later to 2.5 after deciding AUC+noise should open the chapter
 - Wrote Section 2.2 (Connection to Neyman-Pearson / The Neyman-Pearson Lemma) with subsections: lemma statement, practical gap
 - Fixed NP section typos: likelihood, cite, that
 - Rewrote NP practical gap subsection for clarity and precision
 - Fixed NP opening paragraph: corrected imprecise claim that AUC maximizes TPR at every FPR
 - Added yuan2021auc bib entry (Yuan et al., ICCV 2021, LibAUC/PESG) and wired citations in AUC optimization in literature subsection and imbalanced classification section
 - Added neyman1933ix bib entry (original 1933 NP paper)
 ** Archives