From 32009710da177bdb15bd153da1ca56dfc4e4f7a0 Mon Sep 17 00:00:00 2001 From: Aner Zakobar Date: Mon, 5 Oct 2026 09:48:25 +0300 Subject: [PATCH] backup: 2026-10-05 09:48 --- azos.org | 55 +--- azosarchitecture.org | 39 +++ azosemacs.org | 12 + azosexwm.org | 79 ++++++ azospass.org | 13 + big-jar-of-pickles.org | 44 +++ big-jar-of-picklesinfra.org | 30 ++ homey.org | 41 ++- homeybig-jar-of-pickles.org | 62 ++++ homeyntfy.org | 59 ++++ implloss-functions.org | 133 +++++++++ implwork-historyrecent.org | 10 + just-talking.org | 22 ++ just-talkingconversations.org | 195 +++++++++++++ just-talkingcyber-glossary.org | 103 +++++++ just-talkingcyber-reading-list.org | 17 ++ just-talkingcyber-terminology-ramp.org | 29 ++ just-talkingrelocation-budget.org | 47 +++ just-talkingrelocation.org | 31 ++ just-talkingsimlish-project.org | 79 ++++++ just-talkingtodos.org | 12 + just-talkinguser-profile.org | 65 +++++ just-talkinguser-profile_1.org | 20 ++ tax-returns.org | 25 ++ tax-returnsphoenix.org | 43 +++ tax-returnsstatus.org | 46 +++ tax-returnstodos.org | 11 + thesisexperiments_1.org | 33 ++- thesisintroduction.org | 49 ++-- thesismethod.org | 379 ++++++++++++++++++++++++- thesisrelated-work.org | 20 +- thesistodos.org | 105 ++++++- thesiswork-historyrecent.org | 31 +- tiny-trmn-tts.org | 85 ++++++ 34 files changed, 1914 insertions(+), 110 deletions(-) create mode 100644 azosemacs.org create mode 100644 azosexwm.org create mode 100644 big-jar-of-pickles.org create mode 100644 big-jar-of-picklesinfra.org create mode 100644 homeybig-jar-of-pickles.org create mode 100644 homeyntfy.org create mode 100644 just-talking.org create mode 100644 just-talkingconversations.org create mode 100644 just-talkingcyber-glossary.org create mode 100644 just-talkingcyber-reading-list.org create mode 100644 just-talkingcyber-terminology-ramp.org create mode 100644 just-talkingrelocation-budget.org create mode 100644 just-talkingrelocation.org create mode 100644 just-talkingsimlish-project.org create mode 100644 just-talkingtodos.org create mode 100644 just-talkinguser-profile.org create mode 100644 just-talkinguser-profile_1.org create mode 100644 tax-returns.org create mode 100644 tax-returnsphoenix.org create mode 100644 tax-returnsstatus.org create mode 100644 tax-returnstodos.org create mode 100644 tiny-trmn-tts.org diff --git a/azos.org b/azos.org index f53a842..4bf866c 100644 --- a/azos.org +++ b/azos.org @@ -1,59 +1,8 @@ -:PROPERTIES: -:ID: bf4dde56-0967-4374-94ed-771301e47c66 -:END: - -#+title: azos -#+filetags: :project: :knowledge: - -* Architecture - -Two-tier NixOS/home-manager feature system: =azos-core/= (shared submodule) + =features/= (machine-specific). Features auto-discovered via =import-tree=, collected into =config.flake.modules=. - -See [[id:36fe6a01-ea1f-4785-8516-f6dcfecf05bb][azos/architecture]] for module registration, option namespace, file deployment, and Claude Code wiring. - -* Conventions - -- azos-core features belong to the submodule; machine-specific features live in the outer repo -- Extensible options follow the pattern established by =azos.claude.globalSkills= and =azos.claude.globalMdSections=: attrsets any module can contribute to, deployed by the owning feature -- Emacs config is literate org: =azos-core/features/editor/emacs/config.org= -- Emacs base config: =azos-core/features/base/emacs/config.org= -- agent-shell defaults to Claude Code on lauretta via =(setq agent-shell-preferred-agent-config 'claude-code)= in =features/lauretta/emacs/config.org=; this symbol skips agent selection entirely (no prompt) -- Rebuild command: =azos-lauretta-update= -- Format: =nix fmt= -- Packages in overlay: =pkgs.= via =config.flake.overlayPkgs.= -- Unstable packages: =pkgs.unstable.= - -** Writing Claude Skills - -Skill descriptions are routing keys — they determine whether a skill activates. Guidelines (ref: https://stevekinney.com/writing/agent-skills): -- Open with =TRIGGER on:= and numbered conditions so activation is unambiguous -- One specific job per skill; overlapping descriptions cause routing competition -- Scope exclusions: state what the skill does NOT replace to prevent false positives -- Domain-specific language; no generic phrases like "helps with X" - -* Gotchas - - =azos-core= is a git submodule — =git add= must be run inside it separately from the outer repo - Nix flakes only evaluate git-tracked files; new files must be staged (=git add=) before =nix build= will see them - =import-tree= auto-discovers features but only sees tracked files — same constraint - The machine name is =lauretta=; machine config is at =_machines/lauretta.nix= - =postPatch= in Nix derivations: alejandra reformats indentation of multiline strings, which can change effective shell script content -* Key Files - -| File | Purpose | -|------+---------| -| =azos-core/features/claude-memory/default.nix= | org-roam MCP server integration | -| =azos-core/features/claude-skills/default.nix= | global skills + CLAUDE.md deployment | -| =azos-core/features/claude-skills/skills/todo.md= | per-project TODO management skill | -| =azos-core/features/claude-skills/skills/project-brain.md= | org-roam second brain skill | -| =azos/features/claude/default.nix= | installs claude-code, enables claude-memory + claude-skills | -| =features/lauretta/emacs/config.org= | lauretta-specific Emacs overrides (agent-shell, LLM, beacon, caldav) | -| =~/.claude/settings.json= | Claude Code global permissions; always-allowed org-roam MCP tools listed here | - -* Subnodes - -- [[id:36fe6a01-ea1f-4785-8516-f6dcfecf05bb][azos/architecture]] — module registration, option namespace, file deployment, Claude Code wiring -- [[id:c945da4f-de5c-4eb8-bd99-810576a2545a][azos/org-roam]] — setup, MCP server, SQLite internals, fork details -- [[id:09ae970c-2995-43d9-a597-47bb904c98de][azos/pass]] — password store format, path mappings, mbsync conventions -- [[id:ebf221fd-e5be-49d4-a078-6113e2e5e3f2][azos/beacon-wsl]] — NixOS-WSL machine, GPU/CUDA setup, remote deploy +** Subnodes +- [[id:b8094a02-528a-4eeb-a021-4ffc7396f0f3][azos/emacs]] — Emacs-specific gotchas (async-shell-command/pty, org-babel lexical binding) \ No newline at end of file diff --git a/azosarchitecture.org b/azosarchitecture.org index f46889e..538c17f 100644 --- a/azosarchitecture.org +++ b/azosarchitecture.org @@ -57,3 +57,42 @@ Three azos-core features wire up Claude Code: - =claude-memory= — registers org-roam-mcp as a global MCP server in =~/.claude.json= - =claude-skills= — deploys skills to =~/.claude/commands/= and content to =~/.claude/CLAUDE.md= - =claude= (=azos/features=) — installs claude-code, auto-enables the above two + +** Packaging Emacs packages not in nixpkgs + +=epkgs= (passed into =azos-core/features/editor/emacs/_pkg.nix=) is =pkgs.emacs.pkgs=, i.e. +nixpkgs' generated melpa package set — it lags real MELPA/GitHub by a while, so +brand-new packages (e.g. from an active author) are often missing. + +Pattern used for =gptel=, =gptel-autocomplete=, =latex-to-svg-backend=, +=agent-shell-math-renderer=: define a =let= binding in =_pkg.nix= using +=epkgs.callPackage epkgs.trivialBuild { pname; version; src = pkgs.fetchFromGitHub {...}; buildInputs = [...]; }=, +then reference that binding in the =packageRequires= list. Get the =rev= via +=git ls-remote HEAD=, and the =sha256= via +=nix-prefetch-url --unpack --type sha256 https://github.com///archive/.tar.gz= +then =nix hash convert --hash-algo sha256 --to sri = to get the SRI form used elsewhere in the repo. +Inter-package deps (e.g. agent-shell-math-renderer needs agent-shell + latex-to-svg-backend) +go in that package's =buildInputs=. + +Sandbox note: during the trivialBuild native-compile step you'll see +"Failed to install : Creating directory: Permission denied, /homeless-shelter" — +this is benign sandbox noise (no HOME during build), not a build failure; the derivation +still completes and installs correctly. Same warning already appears for gptel-autocomplete. + +** LaTeX-to-SVG rendering (latex-to-svg-backend / agent-shell-math-renderer) + +=latex-to-svg-backend= compiles equations with =\documentclass[varwidth,border=2pt]{standalone}= +plus =\usepackage{amsmath,amssymb,xcolor}=. The system texlive (=azos-core/features/editor/_pkg.nix=, +overlay =pkgs.azos-tex=, a =texliveSmall.withPackages= closure used for org-mode LaTeX export too) +did NOT originally include the =standalone= document class or its =varwidth= dependency — this +caused silent "LaTeX-to-SVG compile failed" warnings in agent-shell with no equation rendered. +Fixed by adding =standalone=, =varwidth=, =amsfonts= (provides =amssymb.sty=), =xcolor=, and +=mylatexformat= (optional but enables the package's precompiled-preamble speedup; +=latex-to-svg-backend-precompile= defaults to t and falls back gracefully if missing) to that +package list. Verified by hand: =latex= (from =pkgs.azos-tex=) then =dvisvgm= on a test equation +with the exact preamble above. + +General gotcha: if a new agent-shell/org LaTeX-preview package starts throwing +"compile failed" / missing =.sty= errors, check =azos-core/features/editor/_pkg.nix= (=azos-tex= +overlay) for the missing texlive package name — the error log path is printed in the Emacs +warning (e.g. =~/.cache/emacs/latex-to-svg/svg/.log=) and gives the exact missing file. diff --git a/azosemacs.org b/azosemacs.org new file mode 100644 index 0000000..03ed43d --- /dev/null +++ b/azosemacs.org @@ -0,0 +1,12 @@ +:PROPERTIES: +:ID: b8094a02-528a-4eeb-a021-4ffc7396f0f3 +:END: + +#+title: azos/emacs +#+filetags: :project: :knowledge: :emacs: + +** Gotchas +- =async-shell-command= spawns subprocesses with =TERM=dumb= and no real controlling tty (confirmed: no =INSIDE_EMACS=, pty exists but curses can't use it). Any gpg/pinentry prompt triggered from such a process (e.g. signed git commits) fails with "Inappropriate ioctl for device" once the pinentry PIN cache expires — pinentry-emacs mode silently isn't engaged either, since gpg only requests it when =INSIDE_EMACS= is set, which =async-shell-command= doesn't set (unlike =M-x shell=/comint, which does). Fix: run the command in a real pty via =vterm= instead (confirmed working — =vterm-term-environment-variable= is set to =xterm-256color=). Fixed in =azos/roam/backup= (=features/lauretta/emacs/config.org=) 2026-10-04. +- =org-babel= =emacs-lisp= source blocks evaluate with *dynamic* binding by default (=ob-emacs-lisp.el= defaults =:lexical= to ="no"=), and =config.org= sets no =:lexical= header anywhere (file-wide or per-block). This silently breaks closures in async code: a pattern like =(let ((output "")) (make-process :filter (lambda (_p chunk) (setq output ...)) :sentinel (lambda (_p _e) ... output ...)))= fails because the let-bound =output= unwinds before the async filter/sentinel callbacks run — they don't lexically close over it. The failure is silent: errors like "Symbol's value as variable is void: output" only appear in =*Messages*=, not as visible UI errors, and whatever the sentinel was supposed to set (e.g. =elfeed-feeds=) just stays =nil=. This hit the elfeed-protocol Nextcloud News auth block (fetches a =pass(1)= password asynchronously via =make-process= to avoid blocking Emacs's command loop and deadlocking with pinentry-emacs, then sets =elfeed-feeds= from the sentinel) — =elfeed-update= appeared to "do nothing" because of this exact bug. Fix/convention: for any async Emacs Lisp in this repo's org-babel blocks (process filters/sentinels, timers, callbacks) that needs to carry state across the async boundary, don't rely on closures over =let=-bound locals unless the block explicitly sets =:lexical yes=. Instead store state on the process object itself via =process-put=/=process-get= (works regardless of lexical/dynamic binding) — this is the pattern now used in the elfeed-protocol block. Fixed in =features/lauretta/emacs/config.org= (Elfeed Nextcloud News Sync section) 2026-10-05. + +[[id:bf4dde56-0967-4374-94ed-771301e47c66][azos]] \ No newline at end of file diff --git a/azosexwm.org b/azosexwm.org new file mode 100644 index 0000000..14db72d --- /dev/null +++ b/azosexwm.org @@ -0,0 +1,79 @@ +:PROPERTIES: +:ID: c52036a9-25ec-4b8b-8b72-90fcd9ea65dd +:END: + +#+title: azos/exwm +#+filetags: :project: :knowledge: :exwm: + +#+title: azos/exwm +#+filetags: :project: :knowledge: :exwm: + +Backlink: [[id:bf4dde56-0967-4374-94ed-771301e47c66][azos]] + +* Architecture + +EXWM feature: =azos-core/features/exwm/emacs/config.org= (literate), packaged via +=_pkg.nix= (=azos-emacs-exwm=), enabled by =azos.suites.exwm.enable=. + +Key custom keys: +- =s-r= -> =exwm-reset= -> calls =exwm-input-grab-keyboard= (switches window to + full "line-mode": Emacs captures all keys) +- =C-c C-k= (EXWM's own default binding in =exwm-mode-map=) -> =exwm-input-release-keyboard= + (switches to "char-mode": app captures keys) +- =exwm-input-prefix-keys= (config.org ~230): keys that always reach Emacs even in + char-mode: =C-x C-u C-h M-x M-& M-: M-\ s-d s-m s-r s-s s-q H-l C-w C-c C-@= +- =exwm-input-global-keys=: s-f2, s-tab, s-w, s-0..9, s-&, always grabbed regardless + of mode (system-wide, via =exwm-input--global-prefix-keys=) + +* Gotchas + +** Line-mode does NOT mean "all keys reach Emacs unconditionally" + +Even when an EXWM window is in full line-mode (grabbed via =exwm-input-grab-keyboard=, +an actual =XGrabKey Modifiers:Any Key:Any= on that window), EXWM's dispatcher +(=exwm-input--on-KeyPress-line-mode= -> =exwm-input--event-passthrough-p=) still +filters every keypress. A key is only sent to Emacs's command loop if: +- it's in =exwm-input--global-prefix-keys= (derived from =exwm-input-global-keys=), or +- it's in =exwm-input-prefix-keys=, or +- =(lookup-key (current-local-map) (vector event))= is non-nil — but + =current-local-map= only sees the buffer's own local map (effectively + =exwm-mode-map=), NOT minor-mode keymaps or =global-map=. + +Consequence: any key bound ONLY via a global minor mode (e.g. this repo's +=azos/global-minor-mode/keymap=, which is how =M-\= -> =split-window-horizontally= +is bound in =azos-core/features/base/emacs/config.org=) will silently fail inside +EXWM windows — the keypress gets bounced back to the underlying app instead of +running the Emacs command — even though =(key-binding (kbd "M-\\"))= correctly +resolves it when checked directly (that function walks the full keymap stack, +unlike the local-map-only check EXWM uses). + +Fix: add the key to =exwm-input-prefix-keys= in +=azos-core/features/exwm/emacs/config.org= (~230). This is unrelated to EXWM +package version — confirmed the relevant dispatch functions +(=exwm-input--event-passthrough-p=, =-on-KeyPress-line-mode=, +=-grab-keyboard=/=-release-keyboard=, =exwm-reset=) are byte-identical across +a =nix flake update= that bumped =emacs-exwm= 0.34.0.20250919 -> 0.34.0.20260204. + +** C-h k "not working" inside EXWM is often just mid-sequence + +=C-h k= is correctly wired (=C-h= is in =exwm-input-prefix-keys=). If it looks +inert, it's because =describe-key= is waiting for a third keypress (the key to +describe) — the minibuffer prompt is there, just easy to miss if focus/echo +isn't visible. + +** Debugging EXWM keybinding issues live + +=exwm-debug-mode= does NOT enable exwm's internal logging (=exwm--log= checks the +special var =exwm-debug= at macro-EXPANSION time, so it's compiled out entirely +if the package was byte-compiled with it off — toggling the mode at runtime is a +no-op for logging). Instead, use =emacsclient= to: +- check real state: =(with-current-buffer BUF (list exwm--input-mode exwm--id))= +- check real X input focus: =(slot-value (xcb:+request-unchecked+reply + exwm--connection (make-instance 'xcb:GetInputFocus)) 'focus)= +- add temporary =advice-add ... :before= on =exwm-input--on-KeyPress= / + =-on-KeyPress-line-mode= to log which keys actually reach the dispatcher +- simulate real physical keypresses (not just synthetic events aimed at a + window) with =nix shell nixpkgs#xdotool --command xdotool key --clearmodifiers + = — this uses XTEST and goes through the same X input pipeline as real + hardware, unlike =xdotool key --window ID= which uses SendEvent and can behave + differently. diff --git a/azospass.org b/azospass.org index 2bb9c1b..4283bc5 100644 --- a/azospass.org +++ b/azospass.org @@ -26,6 +26,7 @@ login: user@example.com - All =passwordCommand= fields in Nix/mbsync configs must pipe through =| head -1= to extract only the password line - =pass-git-helper= handles multi-line entries natively — no =head -1= needed there - mbsync app passwords live under the =mbsync/= namespace to keep them separate from account passwords +- Emacs' generic =auth-source-search= (used by any package supporting auth-source, e.g. lichess.el, gptel, mu4e, tramp) is wired to =pass= via =auth-source-pass-enable= in =azos-core/features/base/emacs/config.org= (added 2026-09-15). Store the secret under a pass entry matching the bare hostname the package searches on (e.g. =lichess.org=) — no per-package glue code needed, unlike the caldav/mbsync =shell-command-to-string "pass ... | head -1"= pattern. ** Path Mappings @@ -36,7 +37,19 @@ login: user@example.com | Zakobar mail (privateemail.com) | =privateemail.com= | =features/mail/default.nix= | | Nextcloud / caldav | =zakobar.com= | =features/lauretta/emacs/config.org= | | Git credentials (zakobar.com) | =zakobar.com= | =features/git-config/pass-git-mapping.ini= | +| Lichess API token | =lichess.org= (via auth-source-pass) | =azos-core/features/base/emacs/config.org= | ** Pending - =mbsync/anerisgreat@gmail.com= and =mbsync/anerz@post.bgu.ac.il= entries not yet created in the store — mail sync will fail until these are added via =pass insert= +- =lichess.org= entry not yet created — needed for lichess.el AI challenges/moves to authenticate (scopes: =board:play=, =challenge:write=, optionally =follow:read=) + +** Known secret leaked into repo (needs migration) + +=features/lauretta/emacs/config.org= hardcodes a live-looking Groq API key in plaintext (=gptel-make-openai "Groq" :key "gsk_..."=), committed to git history. Should migrate to =pass= + the new auth-source-pass wiring (gptel's =:key= slot accepts a function; gptel ships =gptel-api-key-from-auth-source= for exactly this). Flagged 2026-09-15, not yet fixed — user hasn't confirmed whether to rotate/migrate. + +** mu4e / mu database gotchas + +- =home.activation.muInit= in =features/mail/default.nix= only runs =mu init= if =~/.cache/mu= doesn't already exist — changing the =--maildir= path in that script (e.g. the 2026-09-07 =~/Mail= → =~/Maildir= fix) has no effect on machines where the cache dir already exists. The activation script is init-only, not a resync-on-config-change mechanism. +- Symptom of a corrupted Xapian index: mu4e fails with "Mu server process ended with exit code 1" / "error in process sentinel"; =mu info= or =mu server= directly report "error: failed to read database". Tell-tale sign in =~/.cache/mu/xapian/=: the small metadata files (=iamglass=, =flintlock=) have a much newer mtime than the bulk data files (=docdata.glass=, =position.glass=, =postlist.glass=, =termlist.glass=) — indicates a crashed/interrupted index run left the glass DB inconsistent. +- Fix: =mv ~/.cache/mu ~/.cache/mu.bak-$(date +%s)= (reversible backup, not delete), then =mu init --maildir="$HOME/Maildir" --my-address=... [repeat for all addresses]=, then =mu index=. Reindexing ~16k messages from local Maildir takes under a minute — safe to just do this rather than trying to repair the Xapian DB in place. diff --git a/big-jar-of-pickles.org b/big-jar-of-pickles.org new file mode 100644 index 0000000..7a3f62b --- /dev/null +++ b/big-jar-of-pickles.org @@ -0,0 +1,44 @@ +* big-jar-of-pickles +Swarm of weak/small local models ("Big Pickle," opencode's small model — a jar of them) wired with production-grade orchestration, attempting something none of them could do alone. Article-driven side project: does orchestration substitute for model capability, and where's the crossover point. + +** Architecture +Scope narrowed 2026-09-13: standalone local CLI, no self-hosted services required for now. LangGraph library + local SQLite checkpointer (BJP_CHECKPOINT_PATH, default ./bjp_checkpoints.sqlite) + Guardrails AI + LLM-judge, packaged via uv2nix. Langfuse tracing is opt-in (only wired if LANGFUSE_PUBLIC_KEY is set) rather than required. The homey-backed stack (Postgres+pgvector, self-hosted Langfuse, Aegra) is deferred to later, once the pipeline itself is proven out — reasoning and prior state preserved in [[big-jar-of-pickles/infra]]. Reason for the pivot: self-hosted Langfuse v4 hard-requires ClickHouse, which the target Homey Pi 4 (8GB) likely can't spare RAM for; rather than fight that, cut the self-hosted dependency entirely for the MVP. + +nixosModules.default (homey systemd wiring) was removed from flake.nix as part of this pivot — there's no service to deploy right now, just a CLI. If/when homey integration resumes, that module will need to be re-added (see [[big-jar-of-pickles/infra]] for the eurovote-pattern reference it followed). + +Phase 1 task: Research -> Write -> Critique pipeline. Specialist roles (researcher, writer, critic) played by many cheap/weak model calls on a fixed topic, producing a short technical explainer, with LLM-judge scoring tracked across iterations as the article's data. + +Phase 2 (maybe): same harness pointed at a quantified coding benchmark (HumanEval subset) for swarm vs. single-small-model vs. single-frontier-model comparison. + +LLM config is provider-agnostic by design (2026-09-13): Settings/llm.py use generic `llm_api_key`/`llm_base_url`/`llm_model` fields (env vars `LLM_API_KEY`/`LLM_BASE_URL`/`LLM_MODEL`), not provider-specific names — llm.py is just a bare `OpenAI(api_key=..., base_url=...)` client, so swapping providers is an env-var change, not a code change. Currently pointed at **OpenCode Zen** (opencode.ai/docs/zen/) — confirmed official name (not "Zen Cloud", an earlier placeholder guess). Zen is opencode's own model marketplace, OpenAI-compatible endpoint at `https://opencode.ai/zen/v1`, offers opencode's "Big Pickle" model among others. + +Secrets handling: no SOPS — this is a local-only CLI with no NixOS deployment yet, so the API key lives in a gitignored `.envrc` (direnv, `use flake` + exports), with `.envrc.example` committed as the fillable template. SOPS is the right tool for later, when/if homey/NixOS deployment resumes (matches the eurovote pattern already established in [[big-jar-of-pickles/infra]]) — introducing it now for a single-machine secret would be pure overhead. + +Debugging (2026-09-15): `bjp --debug` runs the graph via `graph.stream(..., stream_mode="updates")` instead of `.invoke()`, writing each node's output as a JSON line to `BJP_DEBUG_DIR/.jsonl` (debug.py). Replay with `bjp --show-debug `. This only captures node-level state deltas (plan/research/draft/critique/score), not raw prompts — actual system/user messages sent to the LLM are still unlogged since llm.py's `complete()` uses the raw `openai.OpenAI` client. The Langfuse `CallbackHandler` wired in cli.py also only sees LangGraph chain/node boundaries for the same reason (it hooks LangChain callback events, but nodes never go through a LangChain-wrapped LLM) — swapping to `from langfuse.openai import OpenAI` in llm.py would be the real fix for prompt/completion-level tracing, not yet done. + +** Deployment +None currently — runs as a local CLI (`nix run` / `uv run` / `bjp`). Inference is cloud (OpenCode Zen), so no local token-serving either way. + +** Conventions +(none yet) + +** Gotchas +- Guardrails integration (src/big_jar_of_pickles/guardrails_validation.py) uses the core Validator/register_validator API and is untested against a live run (no real credentials were available when it was scaffolded) — sanity check it actually validates before relying on it. +- Self-hosted Langfuse v4 requires ClickHouse (not just Postgres) — this is why observability was made opt-in rather than deploying self-hosted Langfuse on the resource-constrained Pi. See [[homey/big-jar-of-pickles]] for the ClickHouse memory-risk notes from the homey side. +- Nix flakes only see git-tracked files: `nix develop`/`nix build` silently fail to pick up newly created files until they're at least `git add`ed (staged is enough, doesn't need to be committed). Bit twice during dev — e.g. adding debug.py and running `nix develop` gave `ImportError: cannot import name 'debug'` until it was staged. `uv run` outside the nix devShell is a separate trap: it uses the ambient/ad-hoc venv, not the flake's env, so native-lib deps (grpc for guardrails' otel exporter needs libstdc++) can break there even when `nix develop` works fine — always test through `nix develop`. + +** Key Files +- /home/aner/projects/apps/big-jar-of-pickles/flake.nix — uv2nix flake: packages.default, devShells.default, apps.default (no nixosModule currently — removed in the 2026-09-13 local-only pivot) +- /home/aner/projects/apps/big-jar-of-pickles/src/big_jar_of_pickles/ — graph.py (StateGraph def), cli.py (entrypoint, SqliteSaver checkpointer, argparse with --debug/--show-debug), debug.py (JSONL per-node debug log writer/reader), nodes/{planner,researcher,writer,critic}.py, judge.py, guardrails_validation.py, config.py, llm.py +- /home/aner/projects/apps/big-jar-of-pickles/.envrc.example — committed template for the gitignored .envrc secrets file +- /home/aner/projects/apps/big-jar-of-pickles/README.md — project pitch + local dev instructions + +** Subnodes +- [[big-jar-of-pickles/infra]] — homey/Aegra/LangGraph-Server decision, homey integration pattern, packaging notes (historical/future — not current scope) + +** Related +Origin: brainstormed in [[just-talking/conversations]] session 2026-09-10, growing out of the AI-orchestration/career-track discussion (see the startup-offer entries in that log). + +[[id:d09e7464-e494-433b-beb8-e0742e904c93][big-jar-of-pickles/infra]] + +[[id:dc2ddfad-ada9-4f0d-ae35-806e8a1a2def][homey/big-jar-of-pickles]] \ No newline at end of file diff --git a/big-jar-of-picklesinfra.org b/big-jar-of-picklesinfra.org new file mode 100644 index 0000000..e90fa77 --- /dev/null +++ b/big-jar-of-picklesinfra.org @@ -0,0 +1,30 @@ +:PROPERTIES: +:ID: d09e7464-e494-433b-beb8-e0742e904c93 +:END: + +#+title: big-jar-of-pickles/infra +#+filetags: :project: :knowledge: :infra: + +* big-jar-of-pickles/infra +Detailed reasoning behind the two-repo infra split for [[big-jar-of-pickles]]. + +** Architecture +- Two-repo split: [[homey]] (/home/aner/projects/selfhosted/homey) hosts shared self-hosted infra — Postgres+pgvector, Redis, self-hosted Langfuse (all enabled/load-bearing) — plus a built-but-disabled Aegra module (services.aegra.enable = false). big-jar-of-pickles itself is purely the graph-running application. +- Rejected LangGraph's own self-hosted "Server" (langgraph-api): Elastic License 2.0, needs a paid enterprise key in production; only `langgraph dev` (in-memory, non-persistent) is free. Conflicts with the project's FOSS ethos (README already chose Langfuse over LangSmith for the same reason). +- Evaluated Aegra (github.com/ibbybuilds/aegra) as a FOSS (Apache 2.0, ~1.2k stars, active) reimplementation of that running layer — FastAPI+Postgres+Redis+OTLP, graphs registered via aegra.json (`{"graphs": {"id": "path/to/file.py:var"}}`). Confirmed this config is read only at process startup via filesystem paths, NOT dynamically hot-reloadable — the expected "persistent platform, swap graphs live" benefit doesn't actually hold. Decision: build it into homey but leave disabled; plain library use is simpler for an experimentation-focused project. +- This repo's actual approach: `langgraph` library directly (MIT) + a Postgres checkpointer (`langgraph.checkpoint.postgres`) against homey's shared Postgres for cross-run persistence, direct `langfuse` Python SDK v3 tracing (`from langfuse.langchain import CallbackHandler`, reads LANGFUSE_PUBLIC_KEY/SECRET_KEY/HOST from env automatically), Guardrails AI core `Validator`/`register_validator` API (not Guardrails Hub — Hub validators need a separate `guardrails configure`/`hub install` step that isn't nix-hermetic), and an LLM-as-judge node. +- Packaging: uv2nix (pyproject-nix + build-system-pkgs) built cleanly on the first `nix build` attempt for this dependency set (langgraph, langgraph-checkpoint-postgres, psycopg[binary], langfuse, guardrails-ai, openai) — no build-system overrides needed. flake.nix pins nixpkgs to nixos-25.05 to match homey. Confirmed working: `nix build .#default`, `nix develop`, `nix flake show`. +- homey integration pattern (confirmed by reading the eurovote flake + homey repo): an app flake exposes `nixosModules.default` -> homey adds it as a flake input -> wraps it in `modules/services/.nix` (adds a `homey..enable` option, sops secrets, Caddy vhost, Authelia rule, uptime-kuma monitor, backup path) -> the host (`hosts/pi-main/default.nix`) flips the enable flag. homey pins nixpkgs-25.05 (not unstable); pi-main is a real Raspberry Pi 4 (resource-constrained — zram swap sized for ~8GB RAM). + +** Gotchas +- The homey-side feature module (Postgres+pgvector, Redis, Langfuse oci-containers, dormant Aegra module) has NOT been implemented yet as of 2026-09-11 — only planned/researched. Separate work still to do in the homey repo. +- big-jar-of-pickles' nixosModules.default is not yet wired into homey / enabled on pi-main. + +** Key Files +- /home/aner/projects/selfhosted/homey/modules/services/eurovote.nix — reference pattern to replicate for wiring big-jar-of-pickles (and any future homey infra module) into homey +- /home/aner/projects/apps/eurovote/flake.nix — reference nixosModule shape (options, sops ExecStartPre, systemd service) + +** Related +[[big-jar-of-pickles]] + +[[id:dc2ddfad-ada9-4f0d-ae35-806e8a1a2def][homey/big-jar-of-pickles]] \ No newline at end of file diff --git a/homey.org b/homey.org index df0b634..79cf34b 100644 --- a/homey.org +++ b/homey.org @@ -25,7 +25,9 @@ Eurovision Vote: Django app sourced from external flake =github:anerisgreat/euro EmulatorJS: web game emulator at =games.zakobar.com= with an offline PWA + per-user save sync. See [[id:7148e7ef-7741-42ad-9d3d-4025001e1018][homey/emulatorjs]]. -Backup: Restic daily at 03:00 to S3 (Backblaze B2, bucket =zakobar-home-backup=). Pre-hook: Nextcloud maintenance mode on + pg_dump. Post-hook: maintenance mode off. Manual offload: =restic copy= to local disk. NAR content and media excluded. +Big Jar of Pickles: LangGraph swarm-pipeline app sourced from external flake =big-jar-of-pickles= (/home/aner/projects/apps/big-jar-of-pickles). Homey-side infra rollout across 4 phases (plan: /home/aner/.claude/plans/view-the-big-jar-dynamic-axolotl.md; spec: [[homey/big-jar-of-pickles]]). Phase 1 (native Postgres+pgvector + Redis) deployed and verified 2026-09-11. Phase 2 (Langfuse v4, containerized) written 2026-09-11 but not yet deployed — ships disabled pending sops secrets. Phases 3–4 (Aegra, app wrapper) not started. + +Backup: Restic daily at 03:00 to S3 (Backblaze B2, bucket =zakobar-home-backup=). Pre-hook: Nextcloud maintenance mode on + pg_dump (Nextcloud, native shared Postgres, and Langfuse's own Postgres all dumped separately). Post-hook: maintenance mode off. Manual offload: =restic copy= to local disk. NAR content and media excluded. Reliability hardening in =hosts/pi-main/default.nix=: hardware watchdog (bcm2835_wdt), WiFi power save disabled, network watchdog timer, zramSwap zstd 25%, Nix build-dir on external HD. @@ -34,6 +36,8 @@ Bootstrap: =pi-main-bootstrap= config builds an SD image (=sd-image-aarch64.nix= ** Subnodes - [[id:7148e7ef-7741-42ad-9d3d-4025001e1018][homey/emulatorjs]] — EmulatorJS game emulator: offline PWA design, ROM management, save sync, PS1/large-ROM notes. +- [[id:fd781dd9-e52c-4e3f-82ad-b27783b33177][homey/ntfy]] — Ntfy push notifications: CLI send commands, token management, pass integration. +- [[id:dc2ddfad-ada9-4f0d-ae35-806e8a1a2def][homey/big-jar-of-pickles]] — Postgres+pgvector/Redis/Langfuse/Aegra services for the big-jar-of-pickles app; rollout status, wiring pattern, gotchas. ** Deployment @@ -62,13 +66,15 @@ Both =--target-host= and =--build-host= point to the Pi — the build happens ON Nix evaluates git-tracked files from the flake. New/modified files must be at least =git add=-ed (staged) before deploying, or they will be invisible to Nix. Untracked files are silently ignored. +To validate a module that ships with =enable = false= by default (its =config = lib.mkIf cfg.enable {...}= block stays lazy/unevaluated by plain =nix flake check=), temporarily flip the flag on in =hosts/pi-main/default.nix=, rerun =nix flake check= to force full evaluation, then revert before committing/deploying. + ** Conventions =homeyConfig= specialArgs (passed to every module): =domain=, =organization=, =timezone=. Never hardcode domain strings. =users.mutableUsers = false= — all user config must be declared in Nix. -Secret injection pattern: =LoadCredential= stages sops-decrypted file into =$CREDENTIALS_DIRECTORY= before Exec*; shell script reads and exports. Ephemeral env files written to =/run/= and cleaned up in =ExecStopPost= / =postStop=. +Secret injection pattern: =LoadCredential= stages sops-decrypted file into =$CREDENTIALS_DIRECTORY= before Exec*; shell script reads and exports. Ephemeral env files written to =/run/= and cleaned up in =ExecStopPost= / =postStop=. A combined connection-string secret (e.g. a DB =DATABASE_URL= with the password embedded) can't use the usual "plain Nix env var + separate secret" split — the whole string has to be assembled inside the ExecStartPre script at runtime. Authelia =accessControlRules=: sorted by =priority= at build time (lower = first). Authelia stops at first match, so more-specific rules must have lower priority. Ranges: 0=bypass, 10–19=blanket bypass, 20–49=admin two_factor+deny pairs, 50–64=one_factor open, 65–79=per-path (resources + subject combos). @@ -76,13 +82,17 @@ Authelia =accessControlRules=: sorted by =priority= at build time (lower = first Attic token generation: run =atticadm make-token= inside the container. Tokens are stateless; losing one means just regenerating it with the same command. -DynamicUser services (Eurovision Vote): secrets must be mode =0444= (not =0400=) because DynamicUser gets a random UID that cannot be pre-assigned as owner. +DynamicUser services (Eurovision Vote): secrets must be mode =0444= (not =0400=) because DynamicUser gets a random UID that cannot be pre-assigned as owner. For a bind-mounted config file read by a podman container's own internal (unknown) UID, mode =0644= works the same way and doesn't require knowing that UID. Caddy Cloudflare plugin secrets: uses =LoadCredential= + =ExecStart= override (clears list with empty string first, then sets the real start command) to export =CLOUDFLARE_API_TOKEN= before exec-ing caddy. -Stirling-PDF login disable: =DOCKER_ENABLE_SECURITY=false= is build-time only. To disable the runtime login page, also set =SECURITY_ENABLELOGIN=false=. +Container UIDs for bind-mounted volumes should be verified (=skopeo inspect --config docker://=, checking the =User= field), not guessed — getting it wrong causes an activation-time failure, not a build-time one. See [[homey/big-jar-of-pickles]] for a worked example across 4 different images. -Fetching external static assets via =fetchFromGitHub= (used by EmulatorJS, same pattern works for any GitHub-hosted release): get hash with =nix-prefetch-url --unpack =, then convert to SRI with =nix hash convert --hash-algo sha256 --to sri =. To upgrade: bump =rev=, re-prefetch, update =hash=. Full upgrade instructions are in a comment inside =modules/services/emulatorjs.nix=. +Direct-LAN exposure without Caddy (e.g. Home Assistant, Langfuse's Phase-2 LAN-only rollout): bind the container port to all interfaces (omit the =127.0.0.1:= prefix) and add the port to =networking.firewall.allowedTCPPorts=. + +Fetching external static assets via =fetchFromGitHub= (used by EmulatorJS, same pattern works for any GitHub-hosted release): get hash with =nix-prefetch-url --unpack =, then convert to SRI with =nix hash convert --hash-algo sha256 --to sri =. Full upgrade instructions are in a comment inside =modules/services/emulatorjs.nix=. + +Shared platform infra (Postgres, Redis — not single-purpose UI services) lives at the top level next to =storage.nix=/=backup.nix=/=monitoring.nix=, not under =modules/services/=. See =modules/postgres.nix=/=modules/redis.nix=. ** Gotchas @@ -100,9 +110,13 @@ Attic: writing ephemeral TOML config via =ExecStartPre= shell script to =/run/at First deployment of a new container pulls the image during =nixos-rebuild switch=, which can block the activation for several minutes (e.g. stirling-pdf ~2 GB image took ~6 min on Pi). This is expected — not a hang. -SSH lockout recovery: =/etc/ssh/authorized_keys.d/admin= is OVERWRITTEN on every boot by NixOS activation (from the current system generation's Nix store). Manual edits to that file do not survive a reboot. To recover without a valid key: pull SD card, mount root partition (label =NIXOS_SD=), write the new public key to =/home/admin/.ssh/authorized_keys= (permissions: =.ssh/= 700, =authorized_keys= 600, owner UID 1000). This file is NOT managed by activation and survives reboots. Remove it after deploying the updated NixOS config. +SSH lockout recovery: =/etc/ssh/authorized_keys.d/admin= is OVERWRITTEN on every boot by NixOS activation. Manual edits don't survive a reboot. To recover: pull SD card, mount root partition (label =NIXOS_SD=), write the new public key to =/home/admin/.ssh/authorized_keys= (700/600, owner UID 1000) — not managed by activation, survives reboots. -Admin SSH key is GPG-backed via YubiKey. Current key: =~/.ssh/gpg-as-ssh.pub= (=openpgp:0x13E0C25D=, =cardno:29_109_265=). SSH client config uses =IdentityAgent ${XDG_RUNTIME_DIR}/gnupg/S.gpg-agent.ssh=. Old broken YubiKey was =cardno:24_758_470= — its private =[A]= subkey material was never exported (flash drive backup contains stubs only). +Admin SSH key is GPG-backed via YubiKey. Current key: =~/.ssh/gpg-as-ssh.pub= (=openpgp:0x13E0C25D=, =cardno:29_109_265=). SSH client config uses =IdentityAgent ${XDG_RUNTIME_DIR}/gnupg/S.gpg-agent.ssh=. + +Home Assistant first-boot: the =ExecStartPre= script writes a minimal =configuration.yaml= only if none exists, includes =http.trusted_proxies= and the =command_line= auth provider. After onboarding, create one local break-glass admin account; daily logins use LDAP. + +Native =services.postgresql=/=services.redis=, and containerized multi-service stacks (Langfuse): full gotcha list in [[homey/big-jar-of-pickles]] — custom Postgres =dataDir= needs the exact leaf pre-created before the unit's mount namespace sets up; default pg_hba already grants DynamicUser peer auth; named redis servers default to socket-only; a container's password can't always go through an env var (redis:7 has none — needs a generated config file instead). ** Key Files @@ -113,13 +127,16 @@ Admin SSH key is GPG-backed via YubiKey. Current key: =~/.ssh/gpg-as-ssh.pub= (= - =modules/services/authelia.nix= — access control rule rendering, =accessControlRules= option (unconditional) - =modules/services/uptime-kuma.nix= — =homey.monitoring.monitors= option (unconditional), sync script - =modules/services/attic.nix= — Nix binary cache, JWT token config, netrc injection for Nix daemon -- =modules/services/stirling-pdf.nix= — PDF tools (merge/split/OCR/etc), port 8084, auth disabled via =SECURITY_ENABLELOGIN=false= - =modules/monitoring.nix= — Prometheus + Grafana, proxy auth wiring, Node Exporter Full dashboard - =modules/common.nix= — Nix settings, podman network creation, sops global config - =modules/storage.nix= — external HD mount, =extraDirs= option - =modules/backup.nix= — Restic, pre/post hooks, =extraPaths= option -- =scripts/ssh-with-gpg-key.sh= — emergency SSH via GPG key file; sets up ephemeral GNUPGHOME and uses =[A]= subkey -- =modules/services/emulatorjs.nix= + =emulatorjs-server.py= — EmulatorJS emulator; details in [[id:7148e7ef-7741-42ad-9d3d-4025001e1018][homey/emulatorjs]] +- =modules/postgres.nix= / =modules/redis.nix= — native shared Postgres+pgvector and Redis, [[homey/big-jar-of-pickles]] Phase 1 (deployed) +- =modules/services/langfuse.nix= — containerized Langfuse v4 stack, [[homey/big-jar-of-pickles]] Phase 2 (written, not deployed) +- =scripts/ssh-with-gpg-key.sh= — emergency SSH via GPG key file +- =modules/services/emulatorjs.nix= + =emulatorjs-server.py= — details in [[id:7148e7ef-7741-42ad-9d3d-4025001e1018][homey/emulatorjs]] +- =modules/services/home-assistant.nix= + =modules/services/ha-ldap-auth.py= — HA module; LDAP auth script bundled as Nix derivation +- =modules/services/eurovote.nix= — reference pattern for wiring an external app flake's =nixosModules.default= in; template for [[id:dc2ddfad-ada9-4f0d-ae35-806e8a1a2def][homey/big-jar-of-pickles]] Phase 4 ** TODOs @@ -141,4 +158,6 @@ maxChunkSize = 5_000_000 # 5 MiB per chunk [[id:7148e7ef-7741-42ad-9d3d-4025001e1018][homey/emulatorjs]] -[[id:2d455310-42a5-4ff5-be38-9af2b009aef7][homey/work-history/recent]] \ No newline at end of file +[[id:2d455310-42a5-4ff5-be38-9af2b009aef7][homey/work-history/recent]] + +[[id:dc2ddfad-ada9-4f0d-ae35-806e8a1a2def][homey/big-jar-of-pickles]] diff --git a/homeybig-jar-of-pickles.org b/homeybig-jar-of-pickles.org new file mode 100644 index 0000000..16fa905 --- /dev/null +++ b/homeybig-jar-of-pickles.org @@ -0,0 +1,62 @@ +:PROPERTIES: +:ID: dc2ddfad-ada9-4f0d-ae35-806e8a1a2def +:END: + +#+title: homey/big-jar-of-pickles +#+filetags: :project: :knowledge: :big-jar-of-pickles: + +* homey/big-jar-of-pickles +Spec for the homey-side services that [[big-jar-of-pickles]] (a LangGraph swarm-pipeline app, /home/aner/projects/apps/big-jar-of-pickles) needs. Full architecture reasoning lives in [[big-jar-of-pickles/infra]]. Rollout plan: /home/aner/.claude/plans/view-the-big-jar-dynamic-axolotl.md (4 phases). + +** REVERTED 2026-09-15 — abandoned, not enough disk space on the Pi +The whole homey-side stack below (Postgres+pgvector, Redis, Langfuse module) was reverted out of the =homey= repo on 2026-09-15: =modules/postgres.nix=, =modules/redis.nix=, =modules/services/langfuse.nix= deleted, their imports/enables in =flake.nix= and =hosts/pi-main/default.nix= removed, the two new pg_dump blocks in =modules/backup.nix= reverted, and the =langfuse= secrets block removed from =secrets/secrets.yaml= via =sops unset '["langfuse"]'=. Reason given: the Pi 4's storage doesn't have room for this stack (Langfuse alone is Postgres+ClickHouse+Redis+MinIO). The =big-jar-of-pickles= app project itself (Phase 4, the LangGraph app) and its own infra notes are unaffected — only the homey-hosted services this page tracked are gone. If revisited, it'll need a different storage plan (e.g. external drive headroom check, or a smaller/hosted alternative to self-hosting ClickHouse) before re-attempting Phase 1. + +The Phase 1/2 implementation notes below are kept for reference in case this is retried, but do not reflect the current state of the repo. + +** Services to add (historical — reverted, see above) + +*** DONE Postgres + pgvector (Phase 1, deployed 2026-09-11) +=modules/postgres.nix=. Native =services.postgresql=. Two DBs: =big_jar_of_pickles=, =aegra= (reserved). Verified live: pgvector 0.8.0 installed. + +*** DONE Redis (Phase 1, deployed 2026-09-11) +=modules/redis.nix=. Native =services.redis.servers.big-jar-of-pickles=, socket-only. No consumer yet. + +*** Langfuse module written + secrets populated, NOT yet deployed (Phase 2, 2026-09-11) +=modules/services/langfuse.nix=. Ships with =homey.langfuse.enable= default false. All 10 sops secrets under =langfuse/= now populated in =secrets/secrets.yaml= (random-generated via =openssl rand=/uuid, written with a single =sops set '["langfuse"]' ''= call — one decrypt/re-encrypt cycle instead of ten, YubiKey PIN was already cached in gpg-agent so no interactive prompt was needed). **Remaining before deploy:** flip =homey.langfuse.enable = true= in =hosts/pi-main/default.nix=, deploy, then watch Grafana for ClickHouse's memory footprint before trusting it (Pi 4 8GB ceiling is the flagged risk) — deliberately left for explicit user go-ahead rather than done automatically, since it's the first ClickHouse deploy and genuinely resource-risky. + +Re-verified the real topology against upstream's actual =docker-compose.yml= (github.com/langfuse/langfuse) instead of trusting the org-roam spec's Sept-2026 assumption — **it had already changed**: this is Langfuse **v4** now (image tags =:4=, not v3), still web+worker+Postgres17+ClickHouse25.12+Redis7+MinIO, all on the =homey= podman network. Own private containerized Postgres, per the "don't reach into the native shared Postgres from a container" decision in Phase 1. + +Container UID handling (verified via =skopeo inspect --config=, not guessed, after Phase 1's dataDir mismatch cost a redeploy): +- langfuse-postgres: no baked-in image USER, entrypoint execs to "postgres" without self-chowning → pre-chowned host dir at UID 999 (mirrors nextcloud-postgres). +- langfuse-clickhouse: upstream's own compose forces =user: "101:101"= explicitly → mirrored with both the oci-container =user= override and matching chowned dirs. +- langfuse-minio (=cgr.dev/chainguard/minio=): image bakes in =User: "65532"= (chainguard's standard nonroot UID) → just needs matching dir ownership, no override. +- langfuse-redis: **no persistent volume at all** — sidesteps the UID question entirely; pure cache/queue state, same "disposable" classification as homey's own native Redis. + +Redis auth gotcha: the official =redis:7= image has no env-var password option — only =--requirepass= (would bake the secret into the Nix store) or a config file. Solved via a generated =redis.conf= from ExecStartPre (mode 644, deliberately world-readable — ephemeral /run file, sidesteps needing the container's internal UID), passed as redis-server's config-file argument via =cmd=. + +=DATABASE_URL= (and other secrets embedded in combined connection-string env vars) can't go through the usual "plain Nix-level env var + separate secret in the env-file" split used elsewhere (nextcloud etc.) — the whole URL string, password included, has to be assembled inside the ExecStartPre script at runtime. + +Exposure: LAN-only for now. Only web (3000) and MinIO's S3 API (9090) bound to all interfaces + opened in =networking.firewall.allowedTCPPorts= (mirrors the Home Assistant direct-LAN pattern). Everything else stays =127.0.0.1=-only. + +Deviated from the plan's digest-pinning decision: used version tags (=:4=, =:25.12=, =:7=, =:17=) instead of =pkgs.dockerTools.pullImage=-by-digest — upstream's own reference compose only pins by tag too. Documented as a deliberate deviation in the file header. + +Backup scope deliberately narrow: only langfuse-postgres gets a pg_dump (project/user/API-key config, hard to regenerate) via a new block in =modules/backup.nix=. ClickHouse/MinIO data excluded — treated as disposable analytics output. + +=LANGFUSE_INIT_*= env vars seed an initial org/project/user + API key pair on first boot so big-jar-of-pickles (Phase 4) has real =LANGFUSE_PUBLIC_KEY=/=SECRET_KEY= without a manual UI step. Assumed idempotent (not yet confirmed live). + +Validated via a **temporary** =homey.langfuse.enable = true= flip in =hosts/pi-main/default.nix= + =nix flake check= (forces full evaluation of the =mkIf cfg.enable= block, which stays lazy/unchecked while disabled) — passed clean, then reverted before committing. + +*** TODO Aegra (Phase 3 — built, disabled by default) +uv2nix build (pyproject-nix/uv2nix/pyproject-build-systems trio, same as big-jar-of-pickles' own flake). =homey.aegra.enable= default false. + +*** TODO big-jar-of-pickles app wrapper (Phase 4 — last) +New flake input + =modules/services/big-jar-of-pickles.nix=, eurovote pattern. Gated on upstream =nixosModules.default= being stable. Will consume Langfuse's seeded =LANGFUSE_PUBLIC_KEY=/=SECRET_KEY= from the Phase 2 init. + +** Gotchas learned during Phase 1 implementation (native services.postgresql/redis) +- =services.postgresql= with a custom =dataDir= sets =ReadWritePaths = [ cfg.dataDir ]= on the systemd unit, and systemd binds that **exact** path into the mount namespace *before* =ExecStartPre= (where initdb would normally create it) ever runs. Pre-create the exact schema-versioned leaf (e.g. =postgres/16=, not just =postgres=) via =homey.storage.extraDirs=; tmpfiles implicitly creates the missing parent. +- No custom =pg_hba=/=authentication= lines needed for DynamicUser peer auth — the module's built-in default already includes =local all all peer=. A DynamicUser service running as =User = "rolename"= authenticates for free over the Unix socket. +- A named =services.redis.servers.= (name != "") already defaults to =port = 0= (no TCP) and a per-name Unix socket path — no extra options needed for socket-only access. +- =modules/backup.nix='s =homey-backup-pre.service= has side effects (Nextcloud maintenance-mode ON, writes =/run/restic-homey-secrets.env=) only cleaned up by =restic-backups-homey='s =ExecStopPost=. **Never start =homey-backup-pre.service= standalone to test it** — trigger =restic-backups-homey.service= instead, or manually undo (=podman exec nextcloud php occ maintenance:mode --off= + =rm /run/restic-homey-secrets.env=) if you do. + +** Related +[[big-jar-of-pickles]], [[big-jar-of-pickles/infra]] diff --git a/homeyntfy.org b/homeyntfy.org new file mode 100644 index 0000000..2d7c4f8 --- /dev/null +++ b/homeyntfy.org @@ -0,0 +1,59 @@ +:PROPERTIES: +:ID: fd781dd9-e52c-4e3f-82ad-b27783b33177 +:END: + +#+title: homey/ntfy +#+filetags: :project: :knowledge: :ntfy: + +:PROPERTIES: +:ID: homey-ntfy +:END: + +#+title: homey/ntfy +#+filetags: :project: :knowledge: :ntfy: + +Self-hosted ntfy push notification server at =https://ntfy.zakobar.com=. Native NixOS service (not a container), port 2586. =auth-default-access = deny-all= — all clients need a token or password. Bypasses Authelia (ntfy enforces its own auth so the mobile app works without SSO). Topic for alerts: =alerts=. + +** Sending Notifications from CLI + +Token stored in =pass= at =zakobar-homecenter/ntfy-token=. + +#+begin_src bash +# Basic +curl -H "Authorization: Bearer $(pass zakobar-homecenter/ntfy-token)" \ + -d "your message" \ + https://ntfy.zakobar.com/alerts + +# With metadata +curl -H "Authorization: Bearer $(pass zakobar-homecenter/ntfy-token)" \ + -H "Title: Backup complete" \ + -H "Priority: default" \ + -H "Tags: white_check_mark" \ + -d "restic finished successfully" \ + https://ntfy.zakobar.com/alerts +#+end_src + +Priority values: =min=, =low=, =default=, =high=, =urgent=. +Tags are emoji shortcodes (e.g. =warning=, =rotating_light=, =white_check_mark=). + +** Token Management + +*Generating / rotating a token:* + +Two gotchas with =ntfy token add=: +1. Output goes to stderr — must redirect with =2>&1= or the pipe gets nothing. +2. No global =--config= flag — use =NTFY_AUTH_FILE= env var instead. +3. =pass insert -e= prompts from =/dev/tty=, not stdin — use =-m= (multiline) for piping. + +#+begin_src bash +ssh admin@192.168.1.100 'sudo NTFY_AUTH_FILE=/mnt/data/ntfy/auth.db ntfy token add admin 2>&1' \ + | awk '{print $2}' \ + | pass insert -m -f zakobar-homecenter/ntfy-token +#+end_src + +*Listing / removing tokens* (e.g. to clean up leaked tokens): + +#+begin_src bash +ssh admin@192.168.1.100 'sudo NTFY_AUTH_FILE=/mnt/data/ntfy/auth.db ntfy token list admin' +ssh admin@192.168.1.100 'sudo NTFY_AUTH_FILE=/mnt/data/ntfy/auth.db ntfy token remove admin tk_xxxx' +#+end_src \ No newline at end of file diff --git a/implloss-functions.org b/implloss-functions.org index d59c60e..91a1770 100644 --- a/implloss-functions.org +++ b/implloss-functions.org @@ -58,3 +58,136 @@ ISJ fails when a class has too few samples in a split (~<20). Affects: cleveland - Patience counter does not increment (early stopping cannot fire) All keel configs use =grace_period=50=. Prevents noisy early-epoch models from being selected as best. + +** Naming fix: =_kernelized_roll_fpr_internal='s =fpr= param renamed =cdf_target= (2026-09-10) + +Mentor-audit finding (screenshots, no direct cite): =kernelized_roll_fpr(fpr, ...)= calls +=partial(_kernelized_roll_fpr_internal, fpr=1 - fpr, ...)= — the INTERNAL function's =fpr= +parameter actually holds =1-fpr= (the CDF target, thesis's =1-α=), not the FPR itself, the +opposite of what its name implies. Logic was always correct (verified independently, see +next entry); only the name was misleading. Fixed by renaming the internal parameter (and +the keyword in the =partial(...)= call) to =cdf_target= — user's preferred name over +=alpha= (avoids clashing with the thesis's own use of α to mean the RAW target rate, not +its complement). The PUBLIC =kernelized_roll_fpr(fpr, ...)= parameter is untouched — it's +correctly named; the complement happens only when forwarding internally. No callers use +=fpr== as a keyword arg anywhere in the codebase, so the rename is fully self-contained. +Mirrored the identical rename in the thesis's own code listing +(=content/appendixs/appendix_roll_code.tex=, which reproduces this function verbatim) so +the two don't drift apart. + +** Verified correct (not a bug): =_kernelized_roll_tpr_internal='s =1-yh= and un-complemented =tpr= (2026-09-10) + +While investigating the =fpr=/=cdf_target= naming issue above, also audited two +look-alike oddities in the neighboring =_kernelized_roll_tpr_internal= — both checked out +as correct, for non-obvious reasons worth remembering before anyone "fixes" them later: + +1. *=1 - yh= instead of =-yh=* for the score transform (thesis's stated general rule says + "negate scores", i.e. =-f_theta(x)=). Directly compared =1-yh= vs =-yh= as inputs: + identical loss and gradients to ~3e-8 (float noise). Reason: =1-yh = -(yh-1)=, i.e. + negation plus a constant shift, and =KernelizedROLLoss= is translation-invariant in its + scores (same property underlying the thesis's gradient-balance proof) — so the shift is + silently absorbed and does nothing. Added a one-line code comment at the call site + (roll.py, right above the =KernelizedROLLoss.apply= call in + =_kernelized_roll_tpr_internal=) explaining this, so it doesn't look like a stray bug to + a future reader — this comment is the only thing that changed; the surrounding logic + was intentionally left as-is. +2. *=kernelized_roll_tpr(tpr)= passes =tpr= straight through with no complement*, while + =kernelized_roll_fpr(fpr)= complements (=1-fpr=). Looks asymmetric, but is required by + the math: reducing FPR@TPR(β) to the TPR@FPR machinery requires complementing the + target rate to =(1-β)= first (per the thesis's own reduction rule), and separately, + evaluating TPR@FPR at any rate =γ= uses CDF-target =1-γ= internally — chain the two + complements: =1-(1-β)=β=. They cancel exactly, so the raw =tpr= value reaching the + internal CDF-target IS =β= correctly, no complement needed. The apparent asymmetry + between the two wrappers is expected, not a bug. + +Closed the loop with an independent numerical check (not just internal +self-consistency): hand-computed =1 - F0_hat(F1_hat^{-1}(1-beta))= (thesis's +eq:roll-fpr-at-tpr) completely from scratch in a throwaway script — same normalization, +same KDEpy bandwidth selector, same sigmoid kernel, same bisection, but applied directly +to the ORIGINAL untransformed (yh, y) with no negation/swap trick at all. Result matched +=_kernelized_roll_tpr_internal='s actual output to ~3e-5 (bisection tolerance noise). +Confirms the function exactly reproduces the paper's formula. No code fix needed for +either point beyond the one comment above. + +** MAJOR: appendix_roll_code.tex was stale relative to impl/src/roll.py (2026-09-10) + +User asked directly: "did we ever update the code [listing] after we changed [the real +code]... is the code in the thesis matching actual code?" Answer at the time: NO. Did a +full AST-based function-by-function diff (Python =ast= module, not just eyeballing) of +every function in appendix_roll_code.tex against impl/src/roll.py. Found: + +Serious (pre-fix or actively-buggy code shown as current): +- =_icdf= / =_icdf_batch=: listing showed the OLD signatures entirely (single =v=/=scores= + instead of =v0=/=scores_neg=/=v1=/=scores_pos=, old fixed =+-10/v= bracket margin instead + of the exact =h0*log(alpha/(1-alpha))= bracket, old "stop on CDF residual" criterion + instead of the certified-loss-interval one) -- none of the bisection-bracket / + stopping-criterion audit fixes from earlier this session had been mirrored here. + =forward()='s calls to these were correspondingly stale (3-arg, not 5-arg). +- =backward()=: showed the OLD, numerically-vulnerable =kernel = s_neg*(1-s_neg)= gradient + formula -- the EXACT formula independently found to have a live bug (collapses to an + all-zero gradient when scores saturate) and replaced with the log-space + =_log_sigma_prime=/softmax version earlier this session. The listing never got that fix, + so it directly contradicted appendix_kde_grad.tex (which documents the fix as done) -- + a self-contradiction within the thesis between two appendices. +- =_log_sigma_prime= itself was MISSING from the listing entirely (not just stale) -- + =backward()= as shown couldn't even have called it. + +Minor (comments from this session's smaller fixes not mirrored): +- =_torch_mean_std=: listing still showed =torch.std(yh)= (Bessel-corrected), not the + =correction=0= MLE fix from earlier the same day (see next entry). +- =_kernelized_roll_tpr_internal=: missing the =1-yh= translation-invariance comment. + +Also softened the appendix's opening claim from "reproduced verbatim" to "reproduced ... +reformatted for print width ... logic is unchanged" -- it was never literally verbatim +(unicode sigma/tau stripped to ASCII, citation tags like =[eq:sigmoid-kernel]= dropped from +docstrings, lines rewrapped for column width), so "verbatim" was itself a small overclaim. + +Fixed all of the above directly in appendix_roll_code.tex: ported the current =_icdf=/ +=_icdf_batch=/=forward=/=backward= logic across (reformatted to the listing's existing +ASCII/wrapped style, not raw-pasted), added the missing =_log_sigma_prime= method right +after =_sigma_prime= (matching live file ordering), and added the two missing comments. + +VERIFICATION METHOD (worth remembering for next time this comes up): manual reading isn't +reliable for this kind of check -- used Python's own =ast= module to extract every +function's source from both files and diff them programmatically. First pass (raw source +diff) caught line-wrapping noise mixed in with real differences; had to write a stricter +second pass that parses both into ASTs, strips docstrings/comments via =ast.unparse=, and +diffs the RESULT -- that gave a clean "0 functions differ in actual executable logic" +confirmation across all 21 functions in the listing. Recommend this exact +=ast.parse=+=ast.unparse=-diff technique any time "does the thesis's code listing match +the real implementation" needs checking again -- eyeballing a big code appendix against a +live file is exactly how this drift went unnoticed for multiple fix-rounds in the first +place. + +Rebuilt thesis clean (nix build exit 0) after the fix, confirmed no new undefined +references introduced. + +** Fixed: Gaussian ROLL =_torch_mean_std= used Bessel-corrected std, not MLE (2026-09-10) + +Mentor-audit finding: Appendix A (appendix_gauss_grad.tex) states explicitly "The MLE +parameters are: ... σ_k² = (1/|B_k|)Σ(f_θ(x_j)-μ_k)²" — the biased, n-denominator MLE +variance. But =_torch_mean_std= (roll.py, Gaussian ROLL helpers) used =torch.std(yh)=, +whose default (=correction=1=) is Bessel-corrected (n-1 denominator) — NOT the MLE. Real +documentation-vs-implementation mismatch, same family as the AOC-divisor issue, but lower +stakes: verified =_torch_mean_std=/=_torch_normal_fit= are ONLY used by +=_roll_loss_from_fpr_internal= (Gaussian ROLL), which is NOT among the loss configs used +in any reported experiment (only =roll-aoc= and =roll-tpr90=, both KDE-based, are) — so +fixing this required no re-running of any experiment. + +Separately checked whether the ANALOGOUS =torch.var(yh, correction=1)= in KDE score +normalization (=score_scale=, roll.py ~line 271) has the same problem: it does NOT. +Thesis's own definition of the normalization constant (method.tex, sec:roll-numerical- +stability) is deliberately generic — =\hat\sigma_\mathcal{B} = sqrt(Var(...))= with no +n vs n-1 commitment anywhere, unlike Appendix A's explicit "MLE" language. Nothing textual +is being contradicted there. Also, per the thesis's own prop:normalization-invariance, +for Silverman's rule (scale-equivariant) the exact value of σ̂_B doesn't affect the loss +or gradient AT ALL; for ISJ (already flagged elsewhere as "a heuristic"), the n vs n-1 +choice is only a sqrt(n/(n-1)) deviation (~2.6% at batch 20, <1% by batch 64) -- verified +numerically. Left this one alone entirely, per user's explicit instruction to only fix +the Gaussian ROLL side. + +Fix applied: =torch.std(yh)= -> =torch.std(yh, correction=0)= in =_torch_mean_std=, with a +one-line comment noting this matches Appendix A's MLE formula rather than torch.std's +Bessel-corrected default. Verified: output now matches a hand-computed +=sqrt(mean((x-mean(x))^2))= MLE formula exactly (=torch.allclose= true), and the full +=roll_loss_from_fpr= forward+backward still runs cleanly end-to-end after the change. diff --git a/implwork-historyrecent.org b/implwork-historyrecent.org index ef71a4f..2df2bb2 100644 --- a/implwork-historyrecent.org +++ b/implwork-historyrecent.org @@ -5,6 +5,16 @@ #+title: impl/work-history/recent #+filetags: :project: :work-history: +* 2026-09-01 Tue + +- Fixed libauc-auroc sigmoid bug: =torch.sigmoid(logits)= before =AUCMLoss= collapsed outputs to ~0.047 causing inverted predictions (AUC ~0.34); removing sigmoid gave AUC 0.8368 on chestxray14 probe +- Fixed AUCMLoss =a=/=b=/=alpha= state carry-over between episodes (nn.Parameters not reset); added =reset_state()= call before each episode in all libauc resume scripts +- Applied grace_period=0 policy across all experiments: was causing early-stopping to fire before training had meaningfully started (best epoch 3-13 out of 5000) +- Completed chestxray14 libauc-auroc rerun (no sigmoid, lr=0.01, grace_period=0): mean AUC=0.8351 ± 0.0019 across 5 episodes +- Completed bank-marketing full rerun: roll-aoc mean=0.9312 ± 0.0019, libauc-auroc mean=0.9364 ± 0.0015, all 7 configs in 0.930–0.936 range (grace_period=0 fixed previously broken results of ~0.511) +- Completed food101n libauc-auroc (no sigmoid): mean AUC=0.8572 ± 0.0015 across 3 episodes; roll-aoc leads at 0.8667 +- Launched cifar10n full rerun on beacon-wsl (grace_period=0 everywhere, no-sigmoid libauc, writes to results/cifar10n); ~10h remaining as of 18:00 + * 2026-08-28 Thu - Returned from vacation; checked both remote machines for run status diff --git a/just-talking.org b/just-talking.org new file mode 100644 index 0000000..b2c8198 --- /dev/null +++ b/just-talking.org @@ -0,0 +1,22 @@ +** Architecture +Personal conversation and advice project — no code. Aner comes here to think through life, career, and other decisions. + +** Conventions +- Read [[just-talking/user-profile]] at session start for context on who Aner is +- Read [[just-talking/conversations]] for recent conversation history before engaging +- Add to conversations log at end of session or when user requests +- Add questions/notes to open documents directly without asking +- Give honest, direct takes — not just validation + +** Gotchas +n/a + +** Key Files +- CLAUDE.md — project instructions +- startup-talks-prep.md — action items for next conversation with startup founder + +** Subnodes +- [[just-talking/user-profile]] — who Aner is, current job, career goals, full background +- [[just-talking/conversations]] — running log of all conversations +- [[just-talking/simlish-project]] — scoping notes for Simlish speech synthesizer (for trmn) +- [[just-talking/todos]] — standing project TODOs diff --git a/just-talkingconversations.org b/just-talkingconversations.org new file mode 100644 index 0000000..27f4c46 --- /dev/null +++ b/just-talkingconversations.org @@ -0,0 +1,195 @@ +:PROPERTIES: +:ID: f9d8a4b3-8842-45f8-ad62-5f4cf237aacf +:END: + +#+title: just-talking/conversations +#+filetags: :project: :knowledge: :conversations: + +* just-talking/conversations +Running log of conversations with Aner. Most recent first. + +** 2026-09-29 — Cybersecurity Terminology: Gap Assessment + Study Methodology + +*** Topic +Follow-up to 2026-09-28: defined how to actually close the terminology gap before the 2026-10-28 start date. Ran a self-assessment, learned a critical detail about the startup's actual product domain, and agreed on a study methodology. + +*** Key Facts +- **Startup secures networks of utility companies** — this is OT/ICS/critical-infrastructure security, a distinct specialization from general enterprise security, with its own vocabulary (Purdue Model, Modbus/DNP3, NERC CIP, SIS), case studies (Stuxnet, Ukraine grid attacks, TRITON), and vendor landscape (Claroty, Dragos, Nozomi Networks — not CrowdStrike/Palo Alto). This reorders study priority significantly. +- Ran a category-by-category self-assessment (attack lifecycle, defensive stack, identity, cloud, vuln/offense, IR/forensics, frameworks/compliance, vendor landscape). Full results logged in [[just-talking/cyber-terminology-ramp]]. +- Two corrected misconceptions: IOC/IOA were backwards (IOC = reactive forensic evidence, IOA = proactive behavioral signal); Kerberoasting was misunderstood as attacking the KDC rather than offline-cracking a service account hash from a legitimately-issued ticket. +- Biggest blank spots: IR/forensics, frameworks/compliance, most of the defensive stack (EDR/XDR/SIEM/SOAR/NDR etc.), cloud security beyond self-taught k8s. + +*** Decision: Study Methodology +Hybrid, not one method for everything — full detail in [[just-talking/cyber-terminology-ramp]]: +- Conceptually rich + blank domains (OT/ICS fundamentals, IR/forensics, frameworks/compliance, defensive stack) → conversational deep-dive, one section at a time. +- Pure vocabulary/naming gaps (vendor landscape, acronym expansion) → quick reference pass, no deep discussion needed. +- Reordered plan to start with OT/ICS fundamentals first, ahead of generic enterprise defensive-stack coverage, since that's the actual job domain. + +*** Status +Assessment and methodology done. Next session: OT/ICS fundamentals deep-dive (Purdue Model, ICS protocols, NERC CIP, OT vendor landscape, canonical attack case studies). + +** 2026-09-28 — Job Accepted, Cybersecurity Terminology Ramp-Up, RSS Feed Aggregation + +*** Topic +Follow-up to the 2026-09-07 relocation counter-offer entry: Aner accepted the Index-backed cyber startup offer after a further negotiation/reoffer round. Starts in ~1 month (target ~2026-10-28). Also started building a daily-reading RSS aggregation habit. + +*** Key Facts +- Offer accepted; company reoffered and renegotiated to terms Aner is satisfied with (specifics not detailed in this session). +- Interview process surfaced a terminology gap: Aner didn't know "RAT" (remote access trojan) or "C2" (command & control server) when the team used them, despite having real cybersecurity background (8200 SDR/protocol work, Gita Technologies WiFi/Bluetooth security research per 2026-08-31 entry) and understanding the underlying concepts. +- Self-assessment: concept knowledge is solid, but current industry *terminology/lingo* has drifted or was never fully absorbed — needs a deliberate vocabulary-building push before start date. +- Has ~1 month before starting; wants to build a study strategy (not yet defined — flagged as next step). + +*** Action Taken This Session +- Built a curated list of 8 cybersecurity RSS sources for a daily aggregate reading feed, verified live via HTTP check: + - Krebs on Security — https://krebsonsecurity.com/feed/ + - The Hacker News — https://feeds.feedburner.com/TheHackersNews + - BleepingComputer — https://www.bleepingcomputer.com/feed/ + - Dark Reading — https://www.darkreading.com/rss.xml + - SANS Internet Storm Center — https://isc.sans.edu/rssfeed.xml + - Schneier on Security — https://www.schneier.com/feed/atom/ + - The Record (Recorded Future) — https://therecord.media/feed + - CISA Advisories — https://www.cisa.gov/cybersecurity-advisories/all.xml +- Recommended weighting Krebs + SANS ISC + CISA advisories highest for terminology acquisition specifically, since they use jargon densely in real sentences rather than as a glossary. + +*** Status +Job accepted, not yet started. Next steps (not yet done): (1) define the month-long terminology/lingo study strategy, (2) possibly turn the RSS list into an OPML file for import into a reader. + +** 2026-09-07 — Startup Offer Reopened: Relocation Counter-Offer + +*** Topic +Aner had said no to the Index-backed cyber startup offer (see 2026-08-31 entry), citing that he wants a relocation job. Founder countered with a "possible" US relocation tied to the company's client strategy. + +*** Key New Facts +- Founder's framing: "there's no startup if we don't have clients in the US, and if we have clients in the US, then you're going to go there" — relocation is logically tied to company survival, not a separate perk. +- Company has already relocated someone to Europe (not yet the US) — some relocation precedent exists. +- The role is shifting from pure "claude-first developer" IC work toward a client-facing "solutions" role — broader scope than originally discussed. +- Aner has customer-facing experience as an engineer but doesn't consider himself an expert at solutions/sales-engineering work. + +*** Analysis Given +- Equity: 0.5% for employee #4-5 is low-to-middling for that stage/seniority. Estimated 70-85% chance the grant ends up worth $0 — driven not just by startup failure rates (50-75% of seed-stage companies never return anything) but by liquidation preference stacks that can zero out common equity even in a nominal "exit" if the sale price doesn't clear the preference stack. Framed as: treat equity as a free lottery ticket, not part of the compensation plan. +- Risk mitigation: tie personal go/no-go checkpoints to concrete external milestones (Series A target Nov 2026 / March 2027 + buffer), ask directly for real US pipeline data rather than intent, watch for early-hire turnover as a leading indicator, get the relocation trigger condition specified concretely (e.g. tied to signed client / ARR threshold) rather than left as goodwill. +- Given broader "solutions" scope than originally discussed, flagged that 0.5% equity may be worth renegotiating upward if the role mandate is genuinely expanding. +- Interview prep for solutions/client-facing role: reframe existing customer-facing engineering stories, get a product demo ahead of time, practice mock discovery calls, be upfront about the sales-process gap while leaning on engineering credibility as differentiator. + +*** Status +Reconsidering the offer given the relocation counter — not yet decided. Next step is presumably an interview/conversation for the solutions-facing role. + +** 2026-08-31 — DevOpsDays TLV Talk Prep: NixOS Homelab CFP + +*** Topic +Preparing a CFP submission for DevOpsDays TLV December 2026. + +*** Key Facts +- Target event: DevOpsDays TLV, December 22, 2026, David Intercontinental +- CFP closes: September 25, 2026 (~25 days from conversation date) +- Talk subject: NixOS homelabbing on a Raspberry Pi, based on ~/projects/selfhosted/homey + +*** Research Findings on Past TLV Talks +- No Nix or homelab talks have ever appeared at DevOpsDays TLV (confirmed via 2025 full lineup and 2024 searches) +- 2025 topics: Docker keynote (Solomon Hykes), AI agents in DevOps, Kubernetes AI/ML inference, serverless patterns, Crossplane, cloud costs +- 2024 theme: "engineering horror stories" / postmortems +- NixOS+homelab talks exist globally (FOSDEM 2025, Grazer Linuxtage 2024) but not at DevOps-specific cons — the topic is fresh ground for TLV + +*** Talk Framing Decisions +- Framing: "enterprise benefit → homelab translation" at each stage — connects to what the audience cares about daily +- Rejected "GitOps" label: Aner's setup is declarative IaC; only qualifies as GitOps if his pipeline does continuous reconciliation (not just build/test) +- Rejected "for free!" and other clickbaity language +- Five core NixOS benefits to cover: + 1. Reproducibility + no drift (two sides of same coin — rebuild identical system; nothing silently diverges at runtime) + 2. Reproducible environments (dependencies declared, pinned, baked in) + 3. Atomic rollbacks / generations (broken upgrade reverts in seconds) + 4. Single-command deployment (any configured target, one place) + 5. Deployment flexibility (package anything, patch external deps at build time) + +*** Final Abstract (agreed) +A Raspberry Pi on your desk is a terrible production server. It's also one of the best learning environments you can have! + +NixOS is gaining ground in developer and Linux communities, but its core properties map directly onto problems DevOps engineers deal with every day: reproducibility, declared dependencies, atomic rollbacks, and zero configuration drift. This talk introduces NixOS through the lens of homelabbing. After running real services on a Pi for several years, I've found that even on cheap hardware I can host with enterprise standards, without the overkill. + +At each stage, I'll connect a NixOS feature to its enterprise equivalent and show how it plays out on a real home server: + +- Reproducibility and no drift — rebuild any system from config; trust that what's running matches it +- Reproducible environments — dependencies declared, pinned, and baked in +- Atomic rollbacks — every change creates a generation; a broken upgrade reverts in seconds +- Single-command deployment — configure once, deploy any target from one place +- Deployment flexibility — package anything, patch external dependencies at build time + +Whether you're curious about NixOS for home use or evaluating it for production, this talk gives you a concrete mental model and a working example to start from. + +*** Title (pending) +Not finalized. Leading candidate direction: "Declarative Homelab: Self-Hosting Everything on a Raspberry Pi with NixOS" or similar. Needs to be under 100 chars. + +** 2026-08-31 — Career Visibility: Using Unused Talent, Audio+ML Project Scoping + +*** Topic +How to advance career using underutilized skills; scoping a side project for visibility. + +*** Context +Aner decided (99%) against the startup offer. Wants to get ahead at current job or via a strategic move, without high-risk startup path. Has kids, can't trade stability for uncertain upside. + +*** Career Advancement Strategies Discussed +- Set a 6-month deadline at Applied Materials for a concrete outcome (Tech Lead title, scope expansion) — if it doesn't happen, start looking. +- Build external reputation: LinkedIn posts, Israeli ML/AI meetup talk, one published piece. +- Target "Head of AI" or "AI Lead" roles at 50–200 person Israeli companies — filled via network, not job boards. +- Get a formal "led X people" line on resume: lead interns, own a cross-team initiative. +- ML + LLM systems fluency is differentiating in Israel; Applied Materials industrial ML background is exotic to most conference audiences. + +*** Conference Topic Options Identified +1. ROLL thesis → talk or paper (AUC maximization under label noise) +2. ML in industrial/semiconductor manufacturing (small datasets, real-time, class imbalance) +3. SDR + ML intersection (unique combination from 8200 + Gita background) +4. NixOS for reproducible ML environments + +*** Side Project Scoping +Explored several audio+ML project angles (SDR ruled out — hardware import issues in Israel): +1. Ultrasonic protocol detection/classification (18–22kHz, SilverPush/SoniTalk) — interesting but likely too academic, not enough wild deployment to study +2. Acoustic anomaly detection with label noise robustness — extends ROLL thesis to industrial audio +3. Audio-as-SDR platform using sound card (0–96kHz at 192kHz sample rate) +4. Keyboard acoustic side-channel (done to death — Harrison et al. 2023 got 95% accuracy; 2025 papers add LLM typo correction) +5. **Simlish-style procedural speech synthesizer** ← selected + +*** Simlish Project (Selected) +- Goal: phonetically plausible speech that sounds like a language but isn't intelligible +- Application: trmn (/home/aner/projects/apps/trmn) — "The Truman Show" as a procedural low-poly life sim +- trmn already has: TinyStories-1M ONNX for character dialogue, SVG mouth/face sprites, deterministic PRNG, Nix packaging +- Simlish fills the missing piece: characters that audibly "speak" with emotion-appropriate prosody +- Technical approach: prosody extraction from LLM text → fake phoneme inventory → formant synthesis WASM vocoder → emotion conditioning from character traits +- Size target: <1MB WASM, browser-native, no server +- Research angle: "Procedural character voice synthesis conditioned on LLM output and character traits" — trmn as the demo +- Status: not started, scoping only + +*** Key Facts About Aner's Background (newly documented) +- MSc thesis: ROLL — novel AUC-maximizing loss under label noise (KDE-based, published-quality) +- 8200 IDF: built SDR infra and wrote the GNURadio curriculum for the intelligence corps +- Gita Technologies: WiFi/Bluetooth protocol security research, ML for cross-network target correlation +- Homey: production NixOS self-hosted home server (serious infrastructure project) +- Strengths pattern: rare breadth (SDR + cyber + ML + systems), full end-to-end ownership, teaching instinct + +** 2026-08-31 — Career Decision: Applied Materials vs. Cyber Startup + +*** Topic +Whether to leave Applied Materials for a first-5-employee offer at an Index Ventures-backed cyber startup. + +*** Key Facts Discussed +- Current: Applied Materials, 39k NIS/month + ~30k USD/year RSUs (4 years). ML/DL researcher role. Unhappy with day-to-day and org visibility, but stable. +- Offer: Cyber startup, employee #4-5. 46k NIS base + 0.5% equity. Role titled "claude-first developer." +- Startup: $10M seed, majority from Index Ventures (turned down other top VCs). Series A targeting Nov 2026 or March 2027. Founder is trusted friend, former VP. Co-founder established in cyber field. +- Index Ventures raised $2B across coordinated seed+venture+growth funds in 2026 — structured to follow on into Series A. + +*** Key Points Reached +- Financial delta is smaller than it looks: 39k + ~9k RSU equivalent ≈ 48k effective vs. 46k offered. Salary "raises coming" are promises, not contracts. +- 0.5% after dilution through A/B/C rounds realistically lands at 0.2–0.3% — still meaningful at $1B exit. +- Seed→Series A base rate is 15-20% in 2026 market, but Index's coordinated fund structure meaningfully improves those odds if they lead the A themselves. +- "Claude-first developer" concern is legitimate — but the title on resume is "founding engineer at Index-backed startup," not the internal job title. +- The real red flag: role feels like a step down AND Aner wants management, not more IC dev work. +- Fastest path to management is early startup that grows — not Applied Materials where management is blocked. +- Even if startup fails: founding engineer at Index-backed company is a more interesting resume than 3 more years at Applied Materials with no visibility. + +*** Career Path Discussion +- Aner wants engineering management. Has informal experience (project leadership, task delegation) but no formal title. +- Cold-applying for manager roles is hard. Better paths: Tech Lead role as bridge, mid-size Israeli company with real promotion track, Head of AI at small company. +- AI orchestration / LLM systems architect salary now comparable to ML researcher at senior level in Israel (~38-55k NIS/month). The gap has closed since 2023-2024 LLM demand surge. +- Aner's ML/DL background is additive for AI systems roles, not a liability. + +*** Action Item +- startup-talks-prep.md created in project directory with questions to ask in next founder conversation. diff --git a/just-talkingcyber-glossary.org b/just-talkingcyber-glossary.org new file mode 100644 index 0000000..b863b78 --- /dev/null +++ b/just-talkingcyber-glossary.org @@ -0,0 +1,103 @@ +:PROPERTIES: +:ID: 258e3562-4311-4897-9a5c-1202311b14ae +:END: + +#+title: just-talking/cyber-glossary +#+filetags: :project: :knowledge: :cybersecurity: :glossary: + +Running, quiz-ready glossary built up during Aner's cybersecurity terminology ramp-up (see [[just-talking/cyber-terminology-ramp]] for the study plan/methodology this supports). Term → one-line definition. Updated automatically as new terms are covered in session, no need to ask first. + +** General Attack/Defense Vocabulary +- RAT (Remote Access Trojan) — malware giving an attacker remote hands-on-keyboard control of a victim machine. +- C2 (Command & Control) — the infrastructure/server an attacker uses to control compromised machines and receive stolen data. Distinguish from **C2 payload/C2 traffic**: the actual encoded command or data moving across that channel at a given moment — "C2" names the infrastructure, "C2 payload" names a specific piece of data on it. +- Web shell — malicious code (often a small script like PHP) dropped directly onto a compromised web server, giving the attacker remote command execution through the web server's own HTTP interface. Distinct from a RAT: no separate persistent connection needed, the "channel" is just HTTP requests to the already-compromised server. Header-based C2 smuggling (e.g. WHIPSHOT, a 2026-observed PHP web shell) hides Base64-encoded command/response data inside HTTP header values instead of the body/URL, since headers get less scrutiny from most traffic-inspection tools. +- IOC (Indicator of Compromise) — forensic evidence something already happened (bad hash, known-bad IP, dropped file). Reactive. +- IOA (Indicator of Attack) — behavioral signal an attack is happening right now, regardless of the specific malware (e.g. PowerShell spawned with an encoded command). Proactive/technique-based. +- Kill chain — the staged sequence of an attack (recon → exploit → install/persist → escalate privileges → move laterally → achieve objective). +- MITRE ATT&CK — a public catalogue/matrix of real-world adversary tactics and techniques, organized by attack-lifecycle stage; the industry's shared reference vocabulary for describing attacker behavior. +- Lateral movement — moving from an initially compromised host to other hosts within the network. +- Living-off-the-land (LOTL) — using legitimate built-in OS tools (PowerShell, WMI, psexec) to carry out an attack instead of custom malware, to blend in and evade detection. +- Exfiltration — the act of stealing data out of a compromised network. +- Ransomware-as-a-Service (RaaS) — a ransomware operator provides the toolkit/infrastructure; affiliates carry out the actual intrusion and split the ransom revenue. +- Supply chain attack — compromising a trusted vendor/software update mechanism to reach that vendor's downstream customers. +- Zero-day — a vulnerability with no available patch (attacker often first to know). N-day — a known, patched vulnerability that's still exploitable wherever the patch hasn't been applied yet. +- APT (Advanced Persistent Threat) — a well-resourced, often nation-state-linked attacker group focused on long-term, stealthy access rather than smash-and-grab. +- TTP (Tactics, Techniques, and Procedures) — the standard way of describing *how* a specific threat actor operates, at three levels of specificity (tactic = goal, technique = general method, procedure = exact implementation). This is the language MITRE ATT&CK is built around. +- WAF (Web Application Firewall) — inspects/filters HTTP(S) requests against rules to block web-layer attacks. +- Sandboxing — running untrusted code in an isolated environment so it can't affect the surrounding system. +- Zero trust — never implicitly trust a device/user, even inside the "internal" network; always authenticate and authorize. +- SSO (Single Sign-On) — one login session grants access across multiple systems. +- MFA (Multi-Factor Authentication) — requiring more than one proof of identity to authenticate. +- SAML/OAuth — protocols for delegated authentication: a trusted identity source issues a token after authenticating the user, and that token is what other systems check instead of re-authenticating. +- Kerberoasting — requesting a Kerberos service ticket for a service account (any authenticated user can do this, no alert triggered), then cracking it offline to recover that account's password hash. +- CVE (Common Vulnerabilities and Exposures) — the standard ID assigned to a specific known vulnerability. +- CVSS (Common Vulnerability Scoring System) — the standard severity score (0-10) for a CVE. +- Red team / Blue team / Purple team — red = offense, simulates real attackers; blue = defense, detection & response; purple = collaboration between the two to improve detection. +- Pentest vs. vulnerability scan — a vuln scan is automated, passive, matches known signatures/CVEs; a pentest is human-driven, actively attempts real exploitation/chaining to validate actual risk. +- CNA (CVE Numbering Authority) — an organization authorized to mint CVE IDs for a defined scope (its own products, a coordinator role, etc.). Most CVEs today are assigned by vendor CNAs or bug-bounty-platform CNAs (HackerOne, Bugcrowd), not by MITRE directly. +- CNA-LR (CNA of Last Resort) — MITRE's role: assigns CVE IDs via a request form when no vendor/coordinator CNA covers the affected product. Slower than going through a dedicated CNA. +- CERT/CC (CERT Coordination Center, Carnegie Mellon) — a third-party coordinator CNA, useful when a vendor is unresponsive, has no security contact, or an issue spans multiple vendors. +- Coordinated disclosure — reporting a vulnerability privately to the affected vendor first (rather than publishing immediately), giving them time to patch before any public CVE/details are released; the norm the whole CVE submission process assumes. +- CWE (Common Weakness Enumeration) — the taxonomy of vulnerability *types* (e.g. "CWE-79: Cross-Site Scripting"). A CVE identifies one specific instance of a flaw in a specific product; CWE classifies what *kind* of flaw it is. +- NVD (National Vulnerability Database) — NIST's database that enriches published CVEs with a CVSS score and structured metadata after a CNA assigns the ID. + +** OT/ICS Fundamentals +- IT vs OT priority — IT security follows Confidentiality→Integrity→Availability (CIA); OT security flips it to Availability→Integrity→Confidentiality (AIC), because OT systems control physical processes where downtime/instability can cause physical harm. +- ICS (Industrial Control Systems) — umbrella term for equipment/software controlling physical industrial processes (PLCs, RTUs, HMIs, SCADA, DCS). +- SCADA (Supervisory Control and Data Acquisition) — a type of ICS focused on supervising geographically distributed systems (e.g. a utility grid or pipeline). +- DCS (Distributed Control System) — a type of ICS typically used within a single plant/site rather than spread over distance. +- PLC (Programmable Logic Controller) / RTU (Remote Terminal Unit) — the field-level controllers that read sensors and drive actuators in tight, fast loops (Purdue Level 1). +- HMI (Human-Machine Interface) — the operator-facing dashboard/screen showing live process data (Purdue Level 2). +- Historian — a database logging sensor/process data over time, typically at Purdue Level 3. +- Purdue Model — reference architecture mapping OT/IT network layers, Level 0 (physical sensors/actuators) up to Level 5 (corporate/enterprise); the IT/OT security boundary is conventionally drawn around Level 3/3.5. +- Level 3.5 / DMZ — the firewalled buffer zone mediating any traffic between OT (Levels 0-3) and IT (Levels 4-5); the main real-world security boundary in practice. +- Air-gapped — historically meant zero physical connectivity between OT and IT; in practice today almost nothing is truly air-gapped, so security work focuses on controlling the Level 3.5 boundary instead. +- Passive monitoring — watching a copy of network traffic (via a tap/span port) rather than actively scanning/injecting packets, because active scans can crash fragile field devices never built to handle unexpected input. +- Modbus — the oldest/most universal ICS protocol (1979); plaintext, no authentication; the canonical example of "ICS protocols have no security by design." +- DNP3 (Distributed Network Protocol) — dominant protocol for North American electric/water utility master-station-to-RTU communication; has an optional auth extension (DNP3-SA) rarely enabled in practice. +- IEC 61850 — modern substation automation standard for electric utilities; includes GOOSE messages. +- GOOSE (Generic Object Oriented Substation Event) — peer-to-peer protection-relay messages requiring single-digit-millisecond delivery, with essentially no authentication due to the speed requirement; a known attack surface (used in the 2016 Ukraine grid attack, Industroyer malware). +- IEC 60870-5-104 ("IEC 104") — DNP3's rough European/Middle-East equivalent protocol. +- OPC-UA — a newer IT/OT bridging protocol (e.g. historian-to-cloud) that, unlike the field-level protocols above, was built with authentication and encryption from the start. + +** OT Attack Case Studies +- Stuxnet (2010) — malware delivered via USB to Iran's Natanz facility, targeted Siemens PLCs to physically destroy uranium centrifuges by subtly altering their speed, while feeding operators falsified "everything normal" readings. Canonical example of an OT *integrity* attack — not data theft, not an obvious outage, just making the physical process do the wrong thing while lying to the humans monitoring it. Widely attributed to a US/Israel joint operation, never officially confirmed. +- Ukraine grid attacks (2015, 2016) — first confirmed cyberattacks to cause a power outage. 2015: Sandworm (Russian GRU-linked) phished into utility IT, pivoted to OT, manually opened substation breakers via SCADA HMI (~230,000 people lost power). 2016: follow-up used Industroyer malware, the first malware built to natively speak ICS protocols (including IEC 61850 GOOSE) to automate breaker control at scale instead of manual HMI clicks. +- TRITON / TRISIS (2017) — targeted a Saudi petrochemical plant's Safety Instrumented System (SIS) — the independent emergency-shutdown layer that exists to prevent physical catastrophe (explosion, toxic release). Reprogrammed Schneider Electric SIS controllers; a bug in the attacker's own code accidentally triggered a shutdown, likely preventing serious harm. The case study proving OT attacks carry real physical/safety risk, not just theoretical danger. +- SIS (Safety Instrumented System) — an independent hardware/software layer that triggers automatic emergency shutdown if a process becomes physically dangerous, separate from the normal control system. +- Metcalf sniper attack (2013) — attackers shot up a PG&E transmission substation in California, knocking out 17 transformers. Physical, not cyber, but the case study behind NERC CIP-014 (mandating physical security planning for critical substations) — cited to make the point that critical-infrastructure risk isn't purely digital. + +** OT Vendor Landscape +- Dragos — pure-play OT threat detection & intel vendor; known for naming/tracking specific ICS-targeting threat groups (e.g. XENOTIME, linked to TRITON), similar to how CrowdStrike names IT-side APT groups. +- Claroty — pure-play OT asset visibility + exposure/vulnerability management; also strong in healthcare device security under the term "xIoT" (extended IoT — medical devices, building systems, etc. alongside classic ICS). +- Nozomi Networks — pure-play OT asset visibility + passive network threat detection; notable distribution partnership with Palo Alto Networks. +- Dragos/Claroty/Nozomi are referred to as the OT security "big three" — all compete on roughly the same product shape: passive network monitoring → asset inventory → anomaly/threat detection → vulnerability/exposure management. +- Tenable.ot — Tenable's OT product (from acquiring Indegy), bundled with their IT vulnerability-management business. +- Microsoft Defender for IoT — Microsoft's OT offering (from acquiring CyberX). +- Forescout / Armis — broader IT/OT/IoT device-visibility vendors with Network Access Control (NAC) roots, less ICS-protocol-specialized than the big three. +- Vendor-native OT security — control-system manufacturers (Siemens, Schneider Electric, Honeywell) selling their own security add-ons for their own equipment. + +** 2026 OT Threat Landscape & Supply-Chain Targets +(from the Dragos 2026 OT Cybersecurity Year in Review — see [[just-talking/cyber-reading-list]]) +- KAMACITE, SYLVANITE, BAUXITE — named OT-targeting threat groups tracked by Dragos as active in 2026; BAUXITE notably deployed wiper malware alongside hacktivist messaging campaigns. +- MSP (Managed Service Provider) — a third party remotely managing IT/OT infrastructure (patching, monitoring, help desk) for multiple clients. A high-value supply-chain target because compromising one MSP can grant privileged access into all of its downstream clients at once (cf. the Kaseya ransomware incident). +- GIS (Geographic Information System) — software mapping/managing geospatial data. For utilities, GIS holds the physical topology (which pole feeds which transformer, pipeline routing, substation locations) — a reconnaissance goldmine for an attacker trying to map control loops and physical dependencies, which the 2026 Dragos report flagged as a rising adversary behavior. +- Engineering firms, system integrators, vendors, MSPs, and GIS providers are collectively the "trusted access periphery" around a utility — 2026 attacker focus has shifted toward this periphery rather than the utility's own network directly, since these firms hold privileged access or critical mapping data across multiple client environments. + +** Compliance / Frameworks +- NERC (North American Electric Reliability Corporation) — the organization overseeing bulk power system reliability across North America. +- NERC CIP (Critical Infrastructure Protection) — NERC's mandatory, legally-enforced cybersecurity standards for the electric grid; not a voluntary best-practice framework like NIST CSF. Overseen/enforced by FERC, with fines up to $1M+/day per violation. +- FERC (Federal Energy Regulatory Commission) — the US federal regulator that oversees and enforces NERC standards. +- BES (Bulk Electric System) — the scope of NERC CIP: roughly the high-voltage transmission-level grid (generally 100kV+), not every piece of electric infrastructure. Small distribution-only utilities may fall outside CIP scope. +- NERC CIP is electric-sector only — water utilities fall under EPA/AWIA requirements instead, and gas pipelines under TSA security directives, not NERC CIP. +- CIP-002 — categorize BES Cyber Systems by impact (High/Medium/Low); the mandatory starting point for all other CIP standards. +- CIP-005 — Electronic Security Perimeter (ESP): the legal/compliance formalization of the Purdue Level-3.5 segmentation boundary. +- CIP-007 — system security management (patching, ports/services hardening), with the OT-specific tension that you can't patch live field devices the way you'd patch a laptop. +- CIP-008 — incident reporting and response planning requirements. +- CIP-010 — configuration change management and vulnerability assessments. +- CIP-013 — supply chain risk management (added ~2020): the standard legally requiring utilities to vet third-party/vendor/MSP access — directly related to the MSP/GIS supply-chain targeting trend above. +- CIP-014 — physical security requirements for critical substations, added in response to the 2013 Metcalf sniper attack. + +** Networking / Crypto Primitives +- TLS (Transport Layer Security) — the standard encryption/authentication protocol behind HTTPS; assumes a reliable, ordered, connection-based transport (TCP). +- DTLS (Datagram Transport Layer Security) — TLS adapted to run over UDP instead of TCP, adding handshake retransmission and tolerance for packet loss/reordering/duplication since UDP gives none of that natively. Used wherever "HTTPS-grade" security is needed over a latency-sensitive, connectionless channel — VPN tunnel modes (e.g. Cisco AnyConnect), WebRTC, CoAP for constrained/IoT devices. Most classic field-level ICS protocols (Modbus, DNP3, GOOSE) predate DTLS and don't use it natively. diff --git a/just-talkingcyber-reading-list.org b/just-talkingcyber-reading-list.org new file mode 100644 index 0000000..36c2119 --- /dev/null +++ b/just-talkingcyber-reading-list.org @@ -0,0 +1,17 @@ +:PROPERTIES: +:ID: 0c5494ed-2727-4a48-a5e1-30cda084eeeb +:END: + +#+title: just-talking/cyber-reading-list + +:PROPERTIES: +:END: + +#+title: just-talking/cyber-reading-list +#+filetags: :project: :knowledge: :cybersecurity: :reading: + +Curated, high-quality cybersecurity resources Aner has found worth reading (and referring others to), surfaced during his [[just-talking/cyber-terminology-ramp]]. Distinct from the daily RSS feed habit (see the 2026-09-28 entry in [[just-talking/conversations]] for that list) — this node is for standalone reports/articles worth keeping, not a recurring feed. + +** OT/ICS Security +- [[https://www.sans.org/blog/top-takeaways-from-the-dragos-ot-cybersecurity-report-2026][SANS: Top Takeaways from the Dragos OT Cybersecurity Report 2026]] — free, ungated, substantial summary of Dragos's annual OT threat report. Covers 8 key 2026 OT threat shifts (adversaries staging environments months ahead, targeting HMIs/VFDs/meters/gateways to map process dependencies, initial-access-as-a-service mirroring IT ransomware ecosystems, disruption without ICS-specific payloads, engineering firms/integrators/MSPs as supply-chain targets, hacktivist groups going destructive) and names three threat groups: KAMACITE, SYLVANITE, BAUXITE. Recommended by Aner (2026-09-29) as genuinely good reading — worth recommending to others, not just personal reference. +- [[https://www.dragos.com/ot-cybersecurity-year-in-review][Dragos 2026 OT Cybersecurity Year in Review]] (full 91-page report + exec summary) — the primary source the SANS writeup above is based on. Gated behind a lead-gen form requiring a phone number; the SANS summary covers the substance without it. Worth going to the full report only if going deeper than the summary. diff --git a/just-talkingcyber-terminology-ramp.org b/just-talkingcyber-terminology-ramp.org new file mode 100644 index 0000000..21abfa1 --- /dev/null +++ b/just-talkingcyber-terminology-ramp.org @@ -0,0 +1,29 @@ +:PROPERTIES: +:END: + +#+title: just-talking/cyber-terminology-ramp +#+filetags: :project: :knowledge: :cybersecurity: + +Tracking Aner's month-long cybersecurity terminology/domain ramp-up before starting the Index-backed OT/ICS security startup (~2026-10-28). See [[just-talking/user-profile]] for the job context, [[just-talking/conversations]] for full session logs, [[just-talking/cyber-glossary]] for the running quiz-ready term glossary, and [[just-talking/cyber-reading-list]] for standout resources worth keeping/sharing. + +** Key Context +- Startup secures networks of utility companies — this is OT/ICS/critical-infrastructure security, not general enterprise security. +- Trigger: interview process surfaced that Aner didn't know "RAT" or "C2" despite solid underlying concept knowledge (8200 SDR, Gita WiFi/Bluetooth security research) — a terminology/lingo gap, not a concept gap. +- Daily RSS reading habit already established (Krebs, SANS ISC, CISA Advisories, etc. — see 2026-09-28 conversation entry for full list). + +** Self-Assessment Results (2026-09-29) +Full results and corrected misconceptions (IOC/IOA swap, Kerberoasting) logged in [[just-talking/cyber-glossary]]. Biggest blanks going in: APT, TTP, living-off-the-land, exfiltration, most of the defensive stack (EDR/XDR/SIEM/SOAR/NDR/etc.), cloud security beyond self-taught k8s, IR/forensics entirely, frameworks/compliance entirely, vendor landscape partial only. + +** Methodology (decided 2026-09-29) +Hybrid approach, not one-size-fits-all: +- Conceptually rich + currently blank domains (OT/ICS fundamentals, IR/forensics, frameworks/compliance, defensive stack) → conversational deep-dive, one section at a time, discuss before moving on. +- Pure vocabulary/naming gaps (vendor landscape, acronym expansion, APT/TTP-style terms) → quick reference pass, no deep discussion needed. +- Priority reorder given the utility-company detail: OT/ICS fundamentals first, ahead of generic defensive-stack coverage. +- **Standing workflow preference: auto-update [[just-talking/cyber-glossary]] and [[just-talking/cyber-reading-list]] with every new term/resource covered, without asking first or narrating "I'll add this" — just do it.** + +** OT/ICS Deep-Dive Progress (started 2026-09-29) +Covered so far (full definitions in [[just-talking/cyber-glossary]]): IT-vs-OT / AIC triad; Purdue Model levels 0-5 + Level 3.5 DMZ; ICS protocols (Modbus, DNP3, IEC 61850/GOOSE, IEC 104, OPC-UA); canonical attack case studies (Stuxnet, Ukraine grid/Industroyer, TRITON/SIS, Metcalf sniper attack); OT vendor landscape (Dragos/Claroty/Nozomi "big three" plus Tenable.ot, MS Defender for IoT, Forescout, Armis, vendor-native); 2026 threat landscape from the Dragos Year in Review (KAMACITE/SYLVANITE/BAUXITE, MSP/GIS as supply-chain targets — see [[just-talking/cyber-reading-list]]); NERC CIP compliance (NERC/FERC/BES scope, electric-only vs water/gas, CIP-002/005/007/008/010/013/014). +Not yet covered: IR/forensics (queued next), general frameworks/compliance beyond NERC CIP (NIST CSF, ISO 27001, SOC 2, CIS Controls, STIX/TAXII), remaining defensive-stack acronyms (EDR/XDR/SIEM/SOAR/NDR/NGFW/IDS/IPS/DLP/ZTNA), cloud security beyond k8s (CSPM/CWPP/CIEM). + +** Status +OT/ICS deep-dive in progress. Next: IR/forensics. diff --git a/just-talkingrelocation-budget.org b/just-talkingrelocation-budget.org new file mode 100644 index 0000000..5ead4af --- /dev/null +++ b/just-talkingrelocation-budget.org @@ -0,0 +1,47 @@ +:PROPERTIES: +:ID: 5a1bac62-e5af-47d4-9da9-2c60afb9ab7f +:END: + +#+title: just-talking/relocation-budget +#+filetags: :project: :knowledge: :relocation: :budget: + +Detailed budget-simulation numbers for Aner's US relocation, $200K single-earner household (see [[just-talking/relocation]] for criteria/context, [[just-talking/user-profile]] for who he is). All figures as of 2026-09-28/29, 2026 tax law. Rerun/update this node rather than re-deriving from scratch each session. + +** Tax Parameters Used (2026) +- Federal: standard deduction $16,100 (single); brackets 10%/$12,400, 12%/$50,400, 22%/$105,700, 24%/$201,775, 32%/$256,225, 35%/$640,600, 37% above. OBBBA (2025) made TCJA rates permanent. +- FICA: SS 6.2% up to wage base $184,500 (max $11,439); Medicare 1.45% uncapped; +0.9% additional Medicare only above $200K single. +- 401k 2026 employee limit: $24,500. 401k reduces federal+state taxable income but NOT FICA or state-payroll-program wages (SDI/PFML/FAMLI all computed on gross). +- CA: brackets 1%-9.3% up to $349,137 (higher brackets above that, irrelevant at $200K), ~9.3% marginal from $68,350. Std deduction ~$5,600 (est). SDI 1.3%, uncapped (wage ceiling removed in 2024). +- AZ: flat 2.5%, conforms to federal standard deduction. No state payroll program. +- CO: flat 4.4%, taxable base ~= federal taxable income. FAMLI employee share 0.44% up to SS wage base. +- WA: no income tax. PFML total 1.13%, employee pays 71.43% share (~0.807% effective) up to SS wage base. WA Cares 0.58% uncapped. Combined ~$2,650/yr on $200K. + +** Take-Home Results, $200K Single Filer (2026) +| 401k rate | CA take-home/mo | AZ take-home/mo | CO take-home/mo | WA take-home/mo | +|---|---|---|---|---| +| Max ($24,500) | $9,604 | $10,527 | $10,207 | $10,638 | +| 10% ($20,000) | $9,855 | (not rerun) | (not rerun) | (not rerun) | + +Israeli pension comparison (for deciding 401k %): mandatory pension = employee 6% + employer 6.5% (tagmulim) + employer 8.33% (severance/pitzuim) = 20.83% total. Keren hishtalmut (near-universal in Israeli tech, NOT confirmed whether Aner had one) = employee 2.5% + employer 7.5%, capped at ~₪15,712/mo (~$50K/yr) so doesn't scale as a flat % at $200K. To match Aner's own out-of-pocket share: 6% (pension only) or 8.5% (pension + study fund) — both well under the 401k cap. US employer 401k match policy at his new startup is UNKNOWN — worth him asking directly, since that's the real gap vs. Israel's automatic 14.83% employer contribution. + +** No-Credit / New-to-US Adjustments (methodology, applied to all auto insurance figures below) +- Auto insurance surcharge: ~25-35% above established-resident baseline in states using credit-based insurance scoring (AZ, CO, WA). CA/HI/MA/MI ban this practice, so CA avoids the credit-specific penalty (but CA's baseline is already high, roughly cancelling the advantage). +- Separately, ALL states apply some premium for lacking verifiable US driving/claims history, regardless of credit-scoring laws. +- 2-car combined "auto" line (payments + insurance w/ surcharge + gas) used: Phoenix $1,330/mo, Denver $1,320/mo, Seattle Eastside $1,350/mo, CA (SD/WC/SR) $1,410/mo. +- Rent: no-credit impact is mainly a bigger UPFRONT cash need (2-3 months deposit, or ~1 month's rent as one-time guarantor-service fee), not higher recurring monthly rent. Budget extra ~$6,400-9,600 upfront in Phoenix scaling up to ~$10,400-15,600 in pricier CA suburbs. + +** City Comparison Results (monthly leftover after full budget, max 401k unless noted) +- Phoenix/Chandler/Gilbert, AZ: rent $3,200 (4BR) → leftover **$2,989/mo** +- Denver Centennial/Cherry Creek, CO: rent $3,500 (4BR) → leftover **$2,379/mo** +- Seattle Eastside (Redmond/Kirkland/Bellevue), WA: rent $4,600 (4BR) → leftover **$1,580/mo** +- San Diego/Poway, CA: rent $4,200 (4BR) → leftover **$896/mo** +- Walnut Creek/San Ramon, CA (big 4BR, ~$5,200 rent): leftover **-$194/mo (deficit)** +- San Ramon, 3BR ~1,500sqft, Dougherty Valley area, rent $3,400, **10% 401k**: leftover **+$1,857/mo** +- San Ramon, 4BR, rent $4,200 (realistic range $4,200-4,800 per actual listings), **10% 401k**: leftover **+$1,057/mo** + +Budget line items used consistently across cities (family of 4, public school, employer-covered health insurance so no premium line): groceries ~$1,100-1,350, utilities ~$260-400, kids' activities $250 flat, annualized Israel trip $708/mo flat across all cities (~$8,500/yr family of 4, peak-season assumption — NOT researched per-departure-city, flagged as the least rigorous line), misc/discretionary ~$550-600. + +** San Ramon / Dougherty Valley Specifics +- Best school pocket in San Ramon: Dougherty Valley, feeds Dougherty Valley High School (ranked #25 in CA, top in San Ramon Valley Unified). District-wide SRVUSD is 94th percentile statewide; California High, Monte Vista High, San Ramon Valley High also all top-6%-in-state. +- Actual 4BR listings found (2026-09-28): 3107 Browntail Way (Dougherty Valley) $4,500/mo 4bd/3ba; 4685 Malayan St $4,800/mo 4bd/4ba; 117 Dahlia Ct $6,995/mo (outlier/premium). San Ramon-wide 4BR average $4,771/mo. +- Actual 3BR ~1,500sqft comps: Crow Canyon $3,327/mo, Highlands Point $3,200/mo. diff --git a/just-talkingrelocation.org b/just-talkingrelocation.org new file mode 100644 index 0000000..52c1bc0 --- /dev/null +++ b/just-talkingrelocation.org @@ -0,0 +1,31 @@ +:PROPERTIES: +:ID: 783813c3-4e26-442e-a955-3056d48cc0f4 +:END: + +#+title: just-talking/relocation +#+filetags: :project: :knowledge: :relocation: + +Durable facts about Aner's US relocation situation, criteria, and financial modeling. Linked from [[just-talking/user-profile]]. See [[just-talking/relocation-budget]] for detailed tax parameters, take-home math, and city-by-city budget simulation results/history. + +** Household & Career Context +- Sole earner: single-income household, $200K/year base salary (cyber startup job, accepted per [[just-talking/conversations]] 2026-09-28 entry). Spouse not employed in current planning scenarios. +- Family: 2 adults + 2 kids in budget scenarios (public school assumed, no daycare). Pregnancy under consideration (per 2026-08-31/09-07 conversation entries) — this is why abortion-rights durability matters, not just current law. +- Relocation is tied to the startup's US client strategy (company logic: "no startup if we don't have clients in the US; if we have clients in the US, you go there"). +- No US credit history, no prior US financial footprint. Effects modeled in detail in [[just-talking/relocation-budget]]. Needs 2 cars. + +** Relocation Criteria (as stated 2026-09-28) +- Weather: mild preferred; comfortable with heat AND cold snaps, but wants to avoid long/harsh snow-and-freeze winters. Baseline is Israeli (southern Israel) climate — hot, not much snow experience. +- Reproductive rights: needs non-restrictive abortion access, ideally durable/constitutionally protected rather than just current-legislature policy, given active pregnancy consideration. +- Taxes: wants significantly lower than California, not necessarily Texas-zero. +- Secondary/nice-to-have: presence of an Israeli community; career fit for cybersecurity industry (his new employer's US expansion play). +- Texas/Austin used only as a tax benchmark — ruled out early due to abortion restrictions (6-week-adjacent ban), not revisited as a live candidate. + +** Location Analysis Summary (as of 2026-09-28/29) +Monthly "leftover after budget" ranking, most to least favorable (full numbers in [[just-talking/relocation-budget]]): +1. Phoenix/Chandler/Gilbert, AZ — flat 2.5% tax, abortion rights locked in via 2024 ballot measure (Prop 139), no real winter, growing cyber/tech industry cluster. Best overall balance of the three original criteria. +2. Denver (Centennial/Cherry Creek), CO — flat 4.4% tax + FAMLI payroll tax, strong constitutionally-protected abortion rights, winters real but not prolonged/harsh, genuine cyber/defense industry cluster (Colorado Springs/Denver, NORAD/Space Command/NSA Colorado). +3. Seattle Eastside (Redmond/Kirkland/Bellevue/Issaquah), WA — zero income tax but its advantage gets erased by very high 4BR rent ($4,300-4,800). Strong abortion-rights protection. +4. San Diego/Poway, CA — CA tax burden + high coastal rent; only modestly better than Bay Area. +5. Walnut Creek/San Ramon, CA (Bay Area) — weakest on a big 4BR (~$5,200 rent ran a slight deficit at max 401k). BUT: a smaller 3BR (~1,500sqft) or a realistically-priced 4BR ($4,200) in San Ramon's Dougherty Valley area (top schools in district) at a 10% 401k rate both come back solidly positive ($1,057-1,857/mo leftover) — house size/type was the real budget-breaker, not the metro area itself. + +Nevada (Las Vegas): best pure tax play (zero income tax, abortion rights constitutionally protected) but thin cyber-industry presence versus AZ/CO. Israeli-community presence: none of the above are major hubs (that's LA/Bay Area/NY-NJ/South FL); Denver has the most organized Jewish/Israeli-American community of this set, Seattle has a modest Israeli tech-worker presence (Amazon/Microsoft). diff --git a/just-talkingsimlish-project.org b/just-talkingsimlish-project.org new file mode 100644 index 0000000..e8c5ab5 --- /dev/null +++ b/just-talkingsimlish-project.org @@ -0,0 +1,79 @@ +:PROPERTIES: +:ID: 2b1e1f91-6d2a-41df-b031-608a8c0f68f6 +:END: + +#+title: just-talking/simlish-project +#+filetags: :project: :knowledge: :audio: :ml: + +* just-talking/simlish-project +Scoping notes for Aner's Simlish-style procedural speech synthesizer project. + +** What It Is +Phonetically plausible speech synthesis that sounds like a language but isn't intelligible. +Primary application: trmn (/home/aner/projects/apps/trmn). + +** Why trmn Is the Perfect Host +- Characters already generate dialogue via TinyStories-1M ONNX (~15MB, bundled in Nix build) +- SVG face/mouth sprites already exist → ready to animate with audio +- Deterministic PRNG (masterSeed) → voice synthesis can be seeded per character +- Static site, Nix-packaged → WASM fits naturally + +** Technical Approach (Scoped, Not Started) +Two versions: browser module for trmn, research paper version. + +*** Browser / trmn version (<1MB WASM) +1. Prosody extraction from LLM text — sentence structure, punctuation, word length → pitch curve, stress, rate +2. Fake phoneme inventory — 10–15 phonemes that cohere but form no real language +3. Formant synthesis WASM vocoder — tiny (few KB of C → WASM), sounds speech-like, no training needed +4. Emotion conditioning — character traits + TinyStories output valence → voice parameters (happy = higher pitch/faster, sad = lower/slower) + +*** Research / paper version +- Use HuBERT speech units: extract discrete learned representations from real speech, scramble/remap to fake phoneme inventory, resynthesize with vocoder +- Preserves naturalness (working in learned speech space, not engineered phonemes) +- Could use LAFMA-style latent flow matching as the generative backbone (see below) +- Paper angle: "Procedural character voice synthesis conditioned on LLM output and character traits" + +** Audio ML Architecture Notes (from session 2026-08-31) + +*** Mel spectrogram +- STFT in sliding windows (25ms window, 10ms hop) → magnitude → mel filterbank (80 triangular filters, non-linear freq compression) → log +- Result: 80×T matrix, ~86 frames/second. Compresses 1025 FFT bins → 80 mel bins. +- Mel scale: perceptually motivated, ~linear below 1kHz, logarithmic above. mel = 2595 × log10(1 + Hz/700) +- Fixed preprocessing, not learned. Fully differentiable. + +*** GAN vs Diffusion vs Flow Matching (current landscape) +- GANs (HiFi-GAN, BigVGAN): still used as vocoders (mel → waveform). Fast inference. +- Diffusion (WaveGrad, DiffWave, Grad-TTS): had moment 2021–2022, now superseded. +- Flow matching: current standard for acoustic model. Voicebox, F5-TTS, E2-TTS all use it. + +*** Flow Matching +- Trains a vector field v_θ(x_t, t) in data space (R^(80×T) for mel) +- Path between noise x₀ ~ N(0,I) and data x₁ is linear: x_t = (1-t)·x₀ + t·x₁ +- Training target: predict (x₁ - x₀) — constant velocity along each conditional path +- Inference: integrate ODE from t=0 to t=1, ~10–20 steps (vs 100–1000 for diffusion) +- Marginal flow is curved (network averages over all crossing paths), but conditional paths are straight +- Latent flow matching: add VAE first, do flow matching in compressed latent z-space + +*** LAFMA (reference architecture — arXiv 2406.08203, INTERSPEECH 2024) +- Latent flow matching for text-to-audio generation +- Components: FLAN-T5-Large text encoder + VAEGAN + latent flow matching model +- VAEGAN: VAE with adversarial discriminator during training → sharper reconstructions than plain MSE VAE. Separate checkpoint from flow model. +- VAE is pretrained then FROZEN during flow matching training (separate checkpoints confirm this) +- Training stages: + 1. Train VAEGAN (audio → latent → audio, with discriminator) + 2. Freeze VAE. Train flow matching model in latent space. + 3. Inference: noise → [flow ODE] → latent → [VAE decoder] → mel → [vocoder] → audio +- If building on LAFMA: reuse VAEGAN checkpoint, only train flow matching part on own data + +** Size Target +- trmn browser module: <1MB WASM, formant synthesis, no server +- Research version: ~50–200MB, flow matching on mel or latent space + +** Status +Scoped 2026-08-31. Architecture researched. Not started. + +** Ruled-Out Alternatives +- Ultrasonic protocol detection: not enough wild deployment to study; too academic +- Keyboard acoustic side-channel: crowded (Harrison et al. 2023 = 95% accuracy; 2025 papers add LLM correction) +- Hebrew TTS: good idea but less personally motivating +- RTL-SDR hardware project: import restrictions in Israel for transmitting SDRs diff --git a/just-talkingtodos.org b/just-talkingtodos.org new file mode 100644 index 0000000..1245a69 --- /dev/null +++ b/just-talkingtodos.org @@ -0,0 +1,12 @@ +:PROPERTIES: +:ID: b65e1b4e-40c9-4229-9b82-62412d9cc7ff +:END: + +#+title: just-talking/todos +#+filetags: :project: :todo: + +* TODO Read and deep-dive CERT Polska Energy Sector Incident Follow-up Report 2025 (https://cert.pl/uploads/docs/CERT_Polska_Energy_Sector_Incident_Follow_up_Report_2025.pdf) — research any unfamiliar topics it raises +* TODO Deepen SNMP, Modbus, and related ICS/OT protocols (continuing from [[just-talking/cyber-terminology-ramp]] — Modbus already covered, SNMP not yet) +* TODO LangGraph deep dive +* TODO Python — refresher/deepening pass (already strong, per will-be-manager's pointers) +* TODO Node.js refresher (weaker area, needs review) diff --git a/just-talkinguser-profile.org b/just-talkinguser-profile.org new file mode 100644 index 0000000..f527b1b --- /dev/null +++ b/just-talkinguser-profile.org @@ -0,0 +1,65 @@ + +:PROPERTIES: +:ID: 6c349dbb-eb4a-4202-830b-b1881395d433 +:END: + +#+title: just-talking/user-profile +#+filetags: :project: :knowledge: :user-profile: + +* just-talking/user-profile +Profile of Aner Zakobar for context in future conversations. + +** Identity +- Name: Aner Zakobar +- Email: aner@zakobar.com +- Based in Israel (salaries in NIS), lives in Meitar +- Father of two + +** Current Job (as of 2026-08-31) +- Company: Applied Materials +- Role: ML / deep learning, research-oriented (Algorithm Team R&D) +- Base salary: 39k NIS/month +- RSUs: ~30k USD/year (locked in for at least 4 more years) +- Feeling: stable, good job security, but not happy with day-to-day or org visibility +- Self-perception: feels he is past "just developing" + +** Education +- BSc + MSc, Ben-Gurion University, 2019–present +- MSc thesis: ROLL (Rate Optimized Likelihood Loss) — a novel loss function for differentiable ROC/AUC optimization under label noise. Uses KDE to approximate output score distributions, custom backward pass, evaluated on KEEL, CIFAR-10N, NIH ChestX-ray14. Competitive under clean labels, leads under high noise rates. + +** Work History +- 8200 IDF Intelligence Corps (2013–2018): Designed/built end-to-end SDR communications systems. Built Linux-based DSP infrastructure. Created extensive GNURadio training curriculum — "wrote the book" on how the military uses GNURadio and DSP software development. Recognized for technical + general excellence. +- Gita Technologies (2018–2022): Protocol Security R&D. WiFi/Bluetooth protocol weakness research. Real-time protocol analysis systems. ML for cross-correlating targets across networks. +- Applied Materials (2022–present): ML/DL for classification and filtration in semiconductor manufacturing. Full ownership from data acquisition to deployment. Both DL and classical CV algorithms, real-time constraints. + +** Side Projects (proud of) +- Homey: Production-quality self-hosted home server on NixOS/Raspberry Pi. SOPS secrets, Podman containers, LDAP+Authelia auth, Nextcloud, Gitea, Jellyfin, Backblaze B2 backups, ntfy push notifications, Uptime Kuma. Serious infrastructure project with full disaster recovery docs. +- GNURadio military course: Built infra and guidelines from scratch — shaped how the intelligence corps does SDR/DSP development. +- Autonomous spacecraft landing software in Kerbal Space Program +- Custom NixOS+Emacs environment + +** Technical Strengths Pattern +- **Rare domain breadth**: SDR/DSP → Cyber/Protocol Security → ML/DL research → LLM era. Almost nobody has all four. +- **Research + engineering**: Can write novel ML loss functions AND ship production systems. Not just one or the other. +- **Full ownership instinct**: Repeatedly takes end-to-end ownership (data → deployment, infra → curriculum). Not a component person. +- **Teaching/training instinct**: Wrote the GNURadio curriculum for the military. This is a hidden leadership signal. +- **Deep systems builder**: Doesn't just use tools — builds infrastructure (Homey, 8200 DSP infra) and documents it properly. +- **Low-level + high-level**: From kernel DSP infrastructure to abstract ML loss function math. Rare combination. + +** Underutilized Assets +- ROLL thesis: publishable novel ML research. If not yet at a venue, this is high-ROI. +- SDR + ML intersection: almost no one has both. Huge niche in wireless security/RF anomaly detection. +- Teaching background: not on the CV in a way that reflects its real scope. + +** Career Values / Tensions +- Values: research depth, ML/DL work, feeling intellectually challenged +- Wants engineering management — has informal experience but no formal title +- Concern: both current job AND considered startup role feel like "just developing" +- Decided (99%) not to take the startup offer — kids, pay cut, high effort for uncertain return + +** Startup Offer (as of 2026-08-31) +- Cyber startup, early stage (first 5 employees) +- Founder is a trusted friend (former VP). Seed: $10M, majority Index Ventures. +- Offered role: "claude-first developer" — felt like a step down +- Offered salary: 46k NIS/month base + 0.5% equity +- Decision: almost certainly declining diff --git a/just-talkinguser-profile_1.org b/just-talkinguser-profile_1.org new file mode 100644 index 0000000..e39c8ed --- /dev/null +++ b/just-talkinguser-profile_1.org @@ -0,0 +1,20 @@ +:PROPERTIES: +:ID: 4b5921e3-9ce4-4e51-baf5-62274907a8af +:END: + +#+title: just-talking/user-profile +#+filetags: :project: :knowledge: :profile: + +Who Aner is and what's currently in motion. See [[just-talking/conversations]] for the running session log and [[just-talking/relocation]] for detailed US-relocation criteria/financial modeling. + +** Background +- Based in Israel, used to southern-Israel climate (hot, minimal cold/snow exposure) — this is the baseline he measures US weather against. +- ML/DL researcher background; 8200 IDF (SDR infra, GNURadio curriculum); Gita Technologies (WiFi/Bluetooth protocol security research); MSc thesis on AUC-maximizing loss under label noise (ROLL). +- Currently at Applied Materials (as of the 2026-08-31 entries), but accepted an offer at an Index Ventures-backed cyber startup as of 2026-09-28 — founding/early engineer, role shifting toward client-facing "solutions" work, starting around 2026-10-28. +- **Startup's product focus (learned 2026-09-29): scans and secures networks of utility companies** — i.e. OT/ICS/critical-infrastructure security, not general enterprise security. This should drive the vocabulary/domain vendor landscape (Claroty, Dragos, Nozomi Networks, not CrowdStrike/Palo Alto), frameworks (NERC CIP, Purdue Model), and protocols (Modbus, DNP3) he needs to learn. See [[just-talking/cyber-terminology-ramp]]. + +** Current Situation (2026-09-28) +- Planning a US relocation tied to the new job's US client strategy, as the sole household earner on a $200K base salary. +- Family: spouse + kids; considering pregnancy — this is why reproductive-rights durability is a hard filter, not a soft preference. +- No US credit history or financial footprint yet — a real constraint on insurance/rental terms during the move. See [[just-talking/relocation]]. +- Leading location candidate as of this date: Phoenix, AZ. Full ranked analysis in [[just-talking/relocation]]. diff --git a/tax-returns.org b/tax-returns.org new file mode 100644 index 0000000..2a3496a --- /dev/null +++ b/tax-returns.org @@ -0,0 +1,25 @@ +* tax-returns + +User (aner) is filing US taxes for the first time at 33, via the IRS Streamlined Procedure (has accountants). Project dir: /home/aner/projects/non-tech/tax-returns. CLAUDE.md at project root documents how to work in this repo. See [[tax-returns/status]] for current progress and [[tax-returns/todos]] for the open task list — load both before resuming work. + +** Architecture +Folder structure: tax-returns///.pdf. Source docs land in ~/Downloads first; organizing = copying (not moving), originals left in Downloads. Cross-institution whole-picture documents go in tax-returns/_Summaries/. A few non-institution documents (employer Form 106, accountant's own reference docs) sit loose at project root pending a filing-convention decision — see [[tax-returns/status]]. + +** Conventions +- Institution folder names use the English/common name (e.g. "Phoenix" not "הפניקס"). +- Same-year docs with near-identical generic filenames are often DIFFERENT PRODUCTS (pension/life insurance/study fund), not duplicates — always open and check content, never assume dupes from filename alone. Rename copies to disambiguate, e.g. "- קרן פנסיה.pdf" / "- ביטוח חיים.pdf". +- Also verify the filename's year actually matches the document's content year, and that the institution matches expectations — both have been wrong at least once. +- Broker-summary "start date" fields (e.g. "תחילת ביטוח") have repeatedly turned out to be money-vintage dates carried over from a transfer, not the account-open date at that institution — always check the actual document's "מועד הצטרפות" field before concluding how far back an institution's own documents need to go. + +** Gotchas +- User's surname changed over time in records: "לאופר ענר" (≤2017) → "זקוביץ/זקובר ענר" (2018+). Same person, ID 203340302. +- "דף פרטי הביטוח ונוסח פוליסה.pdf" in Downloads is Clal CAR insurance, unrelated — excluded. Clal is separately a real institution for a different product (life insurance) — don't conflate the two. +- poppler-utils (pdftotext/pdftoppm) not installed by default in this NixOS env; use `nix shell nixpkgs#poppler-utils` ephemerally when the Read tool's PDF renderer needs pdftoppm. + +** Key Files +- _Summaries/ — cross-institution snapshot reports (e.g. broker aggregator reports covering all institutions at once) +- Root-level "Streamline ..." PDFs — accountant's Streamlined Procedure engagement letter, checklist, and items-to-bring list (not year/institution specific, not yet relocated) + +** Subnodes +- [[tax-returns/status]] — current organization progress, full institution/policy list, known document gaps, and open filing-convention decisions +- [[tax-returns/todos]] — actionable task list for resuming work (which institutions to chase documents from, which gaps to resolve, which decisions are pending) diff --git a/tax-returnsphoenix.org b/tax-returnsphoenix.org new file mode 100644 index 0000000..73c2a31 --- /dev/null +++ b/tax-returnsphoenix.org @@ -0,0 +1,43 @@ +:PROPERTIES: +:ID: 95f929fa-b444-4c4c-9672-dad2868745fb +:END: + +#+title: tax-returns/phoenix +#+filetags: :project: :knowledge: :phoenix: + +* tax-returns/phoenix + +Detailed Phoenix (הפניקס) balance history and product map for [[tax-returns]] / [[tax-returns/status]]. Built 2026-10-04 by opening every PDF in tax-returns//Phoenix/ and reading account numbers + closing balances directly (pdftotext -layout via nix shell nixpkgs#poppler-utils; Hebrew RTL text but numbers/labels greppable). + +** Six distinct Phoenix products found (not three) +1. Pension "מקיפה", account 3766380012, fund #209. Active throughout. +2. Pension "משלימה"/"כללית", account 7189612018, started 01/02/2024. Active. +3. Life insurance, account 3766373017. Turns out to be a real savings-component policy (תגמולים+פיצויים), NOT pure risk as first assumed from 2018's all-zero figures (that was just a deposit backlog, see below). Went "מסולקת" (paid-up) in 2024 when user's status changed to עצמאי, then "מבוטלת" (cancelled) in 2025. +4. **NEW — study fund "אקסלנס/הפניקס השתלמות" account 007-229-296086-000**, joined 12/12/2018, employer גיטה טכנולוגיות. Not in the original known-products list. +5. **NEW — study fund "הפניקס השתלמות" account 007-907-538010-000**, joined 03/02/2022 as "שכיר ללא מעסיק", money vintage 30/11/2018 (received via transfer-in, not fresh deposits). +6. Health insurance (ביטוח בריאות), main policy 6433001010 (פרט, מחלות קשות) plus two cancelled group policies 1072734088/6577366013. Pure risk, no cash value/balance — appears under the generic filename "דוח לשנת 20XX.pdf" for 2016-2018 (that filename is NOT the pension, a trap for the usual generic-filename gotcha). Not tracked further. + +** Year-end balances (₪), from annual reports unless noted +| Year | Pension מקיפה 3766380012 | Pension משלימה 7189612018 | Life ins. 3766373017 | Study fund A 296086 | Study fund B 538010 | +|------|------|------|------|------|------| +| 2016 | no doc | N/A | N/A | N/A | N/A | +| 2017 | no doc | N/A | N/A | N/A | N/A | +| 2018 | 0.00 (inferred, 2019 report's comparison row; no standalone 2018 pension doc) | N/A | 0.00 (active; deposits backlogged, landed only in Jan 2019) | no doc (acct opened 12/12/2018, no 2018-specific figure found) | N/A | +| 2019 | 99,578.00 | N/A | 41,652.00 | 30,236.64 | N/A | +| 2020 | 119,432.00 | N/A | not found in doc (no standalone report; have pension+study fund only) | 46,905.32 | N/A | +| 2021 | 156,222.00 | N/A | not found in doc | 72,001.95 | N/A | +| 2022 | 194,501.00 | N/A | not found in doc | 60,255.35 | 4,929.41 (opened via transfer-in this year) | +| 2023 | 271,169.00 | N/A | 81,161.00 (inferred, 2024 report's comparison row; no standalone 2023 doc) | 0.00 — transferred out -65,997.71 during 2023 | 0.00 — transferred out -5,399.22 during 2023 | +| 2024 | 379,937.00 | not found in doc (no annual report exists for this fund yet) | 93,694.00 (status → מסולקת, paid-up) | closed | closed | +| 2025 | 515,703.00 | 35,769 (inferred, from 2026-Q1 report's "יתרת הכספים בקרן בתחילת השנה"; no standalone 2025 doc) | 0.00 — cancelled (מבוטלת), transferred out -101,603.71 during 2025 | closed | closed | +| 2026 (interim, year not over) | 578,849 as of 30/06/2026 | 51,189 as of 30/06/2026 | N/A (cancelled 2025) | closed | closed | + +** Major cross-institution resolutions (2026-10-04) +- **Mor mystery RESOLVED.** Phoenix life insurance 3766373017's 2025 "כספים שהעברת מהפוליסה" (-101,603.71: components -32,186.97/-26,636.30/-42,780.44) is an EXACT match to Mor account 1291666's 2025 incoming "כספים שהועברו לחשבון" (101,603.71 = same three components). Phoenix policy's "תום תקופת הביטוח" = 07/08/2025, close to user's Mor join date 28/07/2025. Source of the previously-UNRESOLVED Mor pre-2025 money is Phoenix life insurance 3766373017, not an unknown institution. Update [[tax-returns/status]]'s Mor/unresolved entries. +- **Migdal 44474197 source CONFIRMED** as Phoenix study fund account 007-229-296086-000 (transferred out -65,997.71 in 2023; close to 44474197's reported value after ~2 years growth). Previously only "presumed ex-Phoenix, not verified." +- **Migdal 44474194 full chain found**: Altshuler Shaham study-fund account 000772309 (vintage 01/11/2018, data only in their 2020 report) → transferred into this NEW Phoenix study-fund account 007-907-538010-000 (opened 03/02/2022, vintage recorded as 30/11/2018, transfer-in ~5,716.53) → transferred out -5,399.22 in 2023 → Migdal 44474194. The prior status note treated this as a direct Altshuler→Migdal transfer; it actually passed through Phoenix first. + +** Gotchas confirmed/reinforced here +- The generic filename "דוח לשנת 20XX.pdf" (2016-2018) is Phoenix HEALTH insurance, not pension — classic same-filename-different-product trap. +- "דוח שנתי מלא 20XX.pdf" (2019-2021) and "דוח מפורט לשנת 20XX.pdf" (2019-2023, unsuffixed) are STUDY FUND and PENSION reports respectively — only clear from opening the content, filenames don't disambiguate which product file is which across years (2024-2025 switch to explicit "- ביטוח חיים"/"- קרן פנסיה" suffixes). +- A policy/fund can show ₪0.00 for reasons other than closing: 2018 life insurance showed 0.00 simply because employer deposits were backlogged and only posted in Jan 2019 — not a cancellation. diff --git a/tax-returnsstatus.org b/tax-returnsstatus.org new file mode 100644 index 0000000..70c1562 --- /dev/null +++ b/tax-returnsstatus.org @@ -0,0 +1,46 @@ +* tax-returns/status + +Living status tracker for [[tax-returns]]. Update this whenever documents are organized or gaps get resolved so a new session can resume immediately without re-deriving anything. See [[tax-returns/todos]] for the actionable task list. See [[tax-returns/phoenix]] for the detailed Phoenix product map and year-by-year balance table. + +** Organized so far +- Phoenix: tax-returns/2016/Phoenix through tax-returns/2026/Phoenix populated. **Full product/balance audit done 2026-10-04 — see [[tax-returns/phoenix]].** Six distinct products found (not three): pension מקיפה 3766380012 (fund #209, since 01/07/2016), pension משלימה 7189612018 (since 01/02/2024, missing 2024 & 2025 annuals), life insurance 3766373017 (went מסולקת/paid-up 2024, cancelled 2025), plus two previously-unknown study-fund accounts (007-229-296086-000 since 12/12/2018, and 007-907-538010-000 since 03/02/2022) both fully transferred out to Migdal during 2023, and a health-insurance product (policy 6433001010 etc., pure risk, not tracked for balances). The generic filename "דוח לשנת 20XX.pdf" (2016-2018) turned out to be the HEALTH insurance report, not pension — another instance of the filename trap. +- Migdal: tax-returns/2022/Migdal through 2025/Migdal populated. Life insurance policy 22005480 (2022-2025, cancelled by 2025, pure risk/no cash value — nothing to withdraw). Study fund, 3 accounts: 44474194 and 44474197 (the two carrying rollover money) were BOTH FULLY WITHDRAWN during 2025 — confirmed via the 2025 annual report's ב.4 movements table, both end 2025 at exactly 0.00 ₪. This is a real taxable withdrawal event, worth flagging to the accountant. **Source chain fully resolved 2026-10-04 (see [[tax-returns/phoenix]]): 44474194's money is Altshuler Shaham study-fund account 000772309 (vintage 01/11/2018) → transferred into a Phoenix study-fund account (007-907-538010-000, opened 03/02/2022) → transferred out of Phoenix (-5,399.22 ₪ in 2023) → landed in Migdal 44474194. Not a direct Altshuler→Migdal transfer as previously logged; Phoenix was an intermediate stop.** 44474197 (66,750.92 ₪, much larger) is CONFIRMED ex-Phoenix: Phoenix study-fund account 007-229-296086-000 transferred out -65,997.71 ₪ in 2023, matching after ~2 years of growth. Account 44474199 (vintage 01/03/2023, tied to current employer אפלייד מטריאלס) is the only one still active, balance 63,441.44 ₪ as of end 2025, tax-free withdrawal eligibility date explicitly stated in-document as **29/03/2029** (standard 6-year study-fund lockup from first deposit). +- Mor: tax-returns/2025/Mor and 2026/Mor populated. קופת גמל תגמולים ופיצויים, account 1291666. User joined Mor 28/07/2025; money vintage 01/10/2018, employer גיטה טכנולוגיות. **Pre-2025 source RESOLVED 2026-10-04 (see [[tax-returns/phoenix]]): Phoenix life insurance 3766373017's 2025 "כספים שהעברת מהפוליסה" outflow is -101,603.71 ₪ with components -32,186.97/-26,636.30/-42,780.44 — an EXACT match to Mor's incoming 101,603.71 ₪ (32,186.97 עמית תגמולים + 26,636.30 מעסיק תגמולים + 42,780.44 מעסיק פיצויים). The Phoenix policy's תום תקופת הביטוח (07/08/2025) lines up with the user's Mor join date (28/07/2025). Source institution is Phoenix, not an unknown institution** — this closes out the long-standing Mor-source investigation; no further chasing needed (Mor direct/שגיא יוגב inquiry no longer necessary unless accountant wants independent confirmation). Severance portion (45,252 ₪) likely already withdrawable; contributions portion (62,222 ₪) pension-locked to age 60. +- Harel: tax-returns/2017/Harel through 2025/Harel populated, from ~/downloads (lowercase, different/messier folder). Harel pension 2017-2019 (tied to IDF career service, explains old Phoenix-pension question). Harel career-military study fund (קרן חיסכון לצבא קבע), account 39097425, continuous 2017-2025. +- Analyst: tax-returns/2025/Analyst and 2026/Analyst populated, from ~/Downloads (capital D). קופת גמל, account 11354243 ("אנליסט חיסכון לילד" originally — a childhood account opened by the user's mother). RESOLVED: tax-returns/2025/Analyst/טופס העברה מאלטשולר שחם.pdf (Form 1032) confirms the pre-2025 source was **אלטשולר שחם גמל ופנסיה בע"מ (Altshuler Shaham)**, fund "אלטשולר שחם גמל לבני 50 עד 60", account 43064837, transferred 23-24/07/2025. +- Altshuler Shaham: DONE for 2017-2025 (2026-10-04). tax-returns//Altshuler Shaham/ populated for 2017-2025, sourced from ~/attach/ (9 password-protected PDFs, password 0302, decrypted via qpdf; originals kept encrypted in ~/attach, untouched). Two distinct accounts/products found across these files: + 1. **קופת גמל לתגמולים** (provident fund), self-employed status, vintage 05/05/1994 (the childhood account from the user's mother). Account number 006218525 under Psagot branding (2017-2019, ₪75,999→₪82,167), renamed to 43064837 under Altshuler Shaham branding (2021-2025) — the 2021 report explicitly shows ₪91,171.55 "כספים שהועברו מפסגות" that year, confirming continuity (Psagot was acquired by Altshuler Shaham — same relationship, not a new account). Balance hits exactly ₪0.00 by end of 2025 — **this is the confirmed Analyst-transfer account** (see above), already resolved via Form 1032. **Gap: no 2020 document for this account** (the 2020 file in the batch is the other account, below) — history otherwise runs 2017-2019, 2021-2025. + 2. **קרן השתלמות** (study fund), salaried status, employer גיטה טכנולוגיות בע"מ, account 000772309, vintage 01/11/2018, eligible for tax-free withdrawal 30/11/2024, balance ₪5,291.44 end of 2020. Appears ONLY in the 2020 file — not present in any other year 2017-2019 or 2021-2025 among these 9 documents. **This money's full destination chain (2026-10-04): transferred into a Phoenix study-fund account in 2022, then from Phoenix into Migdal 44474194 in 2023 — see [[tax-returns/phoenix]].** Filed as tax-returns/2020/Altshuler Shaham/דוח שנתי מפורט לשנת 2020 - קרן השתלמות.pdf (distinct filename from the other years' קופת גמל לתגמולים reports, since it's a different product). +- **Menora Mivtachim: DONE for 2017-2022 (2026-10-04, newly discovered institution)**. tax-returns//Menora Mivtachim/ populated for 2017-2022, sourced from ~/menora/ (6 unencrypted PDFs, copied as-is; originals left in ~/menora untouched). Single small account: **קופת גמל לתגמולים ולחיסכון**, salaried status, account 39121432, vintage 02/07/2017, last employer **צה"ל - מת"ש - שירותי מס** (IDF payroll) — tied to the same career-military period as Harel, a previously-unknown third institution from that era. Balance grew modestly 2017 (₪176.57) → 2021 (₪1,859.88), then the 2022 report's movements table shows ₪1,824.09 "כספים שהועברו מהחשבון" (transferred out) during 2022, zeroing the account. RULED OUT as the Mor source (resolved to Phoenix, see above) — far too small and closed three years before Mor's 2025 join date. Likely consolidated into Harel's ongoing military study fund/pension around 2022, but not confirmed — low priority to chase given the trivial amount. +- tax-returns/_Summaries/ has the 2026-10-02 broker aggregator report (Sagi Yogev). Did NOT list Harel, undercounted Migdal, and didn't list Altshuler Shaham or Menora — treat as an incomplete, point-in-time snapshot, not authoritative. + +** Full institution/policy picture (as of 2026-10-04) +1. Phoenix — six products, see [[tax-returns/phoenix]] for full detail. Pension "מקיפה" 3766380012 (since 01/07/2016, fund #209, active); pension "כללית"/"משלימה" 7189612018 (since 01/02/2024, MISSING 2024 & 2025 annual, active); life insurance 3766373017 (paid-up 2024, cancelled 2025 — confirmed transferred to Mor); two study-fund accounts (007-229-296086-000 since 2018, 007-907-538010-000 since 2022) both fully transferred to Migdal in 2023; health insurance policy 6433001010 etc. (pure risk, no balance to track). +2. Migdal — DONE. Two of three study-fund accounts withdrawn in 2025 (see above); only 44474199 remains, unlockable tax-free 29/03/2029. Sources of both 44474194 and 44474197 now fully traced through Phoenix (see above) — RESOLVED, no longer "not independently re-verified." +3. Mor — DONE, including source. Pre-2025 money confirmed as a transfer from Phoenix life insurance 3766373017 (see above) — no longer an open question. +4. Harel — DONE, two products, no gaps in years available. +5. Analyst — DONE. Pre-2025 source confirmed as Altshuler Shaham account 43064837 (see above). +6. Altshuler Shaham (אלטשולר שחם) — DONE for 2017-2025 (see above). Password for ~/attach source PDFs: 0302. +7. Menora Mivtachim (מנורה מבטחים) — DONE for 2017-2022 (see above). Account closed 2022, nothing further to chase. +8. Clal — NOT STARTED. Pure risk life insurance policy 4672059 since 01/08/2025. Low priority. + +** Known gaps remaining +- Phoenix life insurance policy 3766373017 detailed reports missing 2020-2023 (have 2018, 2019, 2024, 2025) — end-2023 closing balance (81,161.00 ₪) is nonetheless recoverable from the 2024 report's prior-year comparison row, see [[tax-returns/phoenix]]. +- Phoenix policy 7189612018 ("כללית"/"משלימה") missing annual reports for 2024 and 2025 (only Q1/Q2 2026 quarterly) — end-2025 balance (35,769 ₪) is nonetheless recoverable from the 2026-Q1 report's opening-balance field. +- Phoenix pension מקיפה (3766380012) has no standalone document for 2016, 2017, or 2018; the 2018 closing balance (0.00 ₪, deposits were backlogged) is recoverable from the 2019 report's comparison row only. +- Altshuler Shaham provident account (006218525/43064837): missing a 2020 annual report specifically (have 2017-2019, 2021-2025). +- Clal: entire history missing (nothing downloaded yet) — low priority. +- Menora Mivtachim account 39121432's 2022 transfer-out destination (₪1,824.09) is unconfirmed — probably Harel, not verified. Low priority given the trivial amount. + +** Non-institution documents found, not yet filed (open decisions — ask user before acting) +- tax-returns/7391_15_12_2024_13_08_23.pdf — Israeli Form 106 (annual employer tax/withholding certificate, ~= W-2) for tax year 2018, employer גיטה טכנולוגיות בע"מ, via payroll processor Hilan. Loose at project root, filing convention undecided. +- Root-level "Streamline ..." PDFs (engagement letter, client checklist, items-to-bring list, all "2020-2022") — accountant's own Streamlined Procedure intake/reference docs. Loose at project root, filing convention undecided. + +** Context for accountant conversations (not a filing/action item, background only) +- "פיצויים" (severance) component in pension/provident/gemel accounts is pre-funded monthly by employers (~8.33%/month). Under a "Section 14" arrangement (standard in Israeli tech employment), this money is unconditionally the employee's and follows them between institutions — doesn't revert to the employer. Money stays tax-deferred in Israel inside the pension/provident-fund wrapper; relevant to PFIC/FBAR characterization for the Streamlined filing. +- Israeli provident-fund (קופת גמל) liquidity rules depend on deposit date: pre-2008 deposits are generally lump-sum/"הוני" eligible; post-2008 "תגמולים" deposits are pension-locked (annuity-only from age 60, with a narrow small-pension lump-sum exception); "פיצויים" (severance) deposits follow separate release rules tied to employment termination. Study funds (קרן השתלמות) become tax-free withdrawable 6 years after first deposit, per-account — confirmed exact dates are stated directly in each account's annual report ("מועד זכאות למשיכת כספים"). +- Israeli retirement fund transfers are siloed by TYPE: a קופת גמל (provident) can generally only transfer to another קופת גמל, a קרן פנסיה (pension) only to another pension, a קרן השתלמות (study fund) only to another study fund — cross-type moves require a specific triggering event (retirement/death/disability). This is the key logic used to rule out Migdal/Harel/Menora as Mor's source, and to trace the Altshuler→Phoenix→Migdal study-fund chain and the Phoenix-life-insurance→Mor transfer. +- **2025 withdrawal/transfer events worth flagging to the accountant**: Migdal study-fund accounts 44474194 and 44474197 were both fully withdrawn/emptied during 2025 (real taxable events, not just transfers). Phoenix life insurance 3766373017 was cancelled in 2025 with its ~101,603.71 ₪ balance transferred (not cashed out) into Mor. The Mor severance portion (45,252 ₪) may also already be withdrawable given the user's employer changed from גיטה טכנולוגיות to אפלייד מטריאלס before 2023. +- User had a career/standing military service track (צבא קבע / צה"ל) starting ~01/07/2016, with its own dedicated pension (Harel, ended ~2019), study fund (Harel, ongoing through 2025), and a third small provident account (Menora Mivtachim, 2017-2022, now closed) — all separate from civilian-employer retirement products. +- User's childhood savings account (started 05/05/1994, parent-opened) moved from Psagot to Altshuler Shaham (acquisition, same account/relationship, renumbered 006218525→43064837) before being transferred into Analyst in 2025 under the user's own name. +- User's employer גיטה טכנולוגיות (pre-אפלייד מטריאלס) is linked to multiple money trails now fully traced: the Mor provident account (via Phoenix life insurance), the Altshuler Shaham study-fund account (via a Phoenix intermediate stop → Migdal 44474194), and a second Phoenix study-fund account directly → Migdal 44474197. diff --git a/tax-returnstodos.org b/tax-returnstodos.org new file mode 100644 index 0000000..519f61c --- /dev/null +++ b/tax-returnstodos.org @@ -0,0 +1,11 @@ +* DONE Get documents from Harel (הראל) — filed into tax-returns//Harel/ for 2017-2025. Harel pension 2017-2019 (IDF career service, explains old Phoenix-pension "gap"). Harel career-military study fund 2017-2025, continuous. Ruled out as the pre-2025 source of the Mor money. +* DONE Resolve Phoenix gap: pension fund annual reports "missing" for 2016-2018 — RESOLVED, was Harel (IDF career service), not a real gap. +* DONE Get documents from Analyst (policy 11354243) — filed into tax-returns/2025/Analyst and 2026/Analyst. +* DONE Figure out which institution held the Analyst money before its 2025 transfer — RESOLVED via tax-returns/2025/Analyst/טופס העברה מאלטשולר שחם.pdf (Form 1032 transfer request): source was **Altshuler Shaham** (אלטשולר שחם גמל ופנסיה בע"מ), fund "אלטשולר שחם גמל לבני 50 עד 60", account 43064837, transferred 23-24/07/2025. +* TODO Ask mother for Altshuler Shaham (אלטשולר שחם) documents/history for the child-savings account (account 43064837, transferred to Analyst 11354243 on 23-24/07/2025). User's mother opened the account originally and is the one who will actually be able to get the info/statements from Altshuler Shaham — not something to chase through the broker or Downloads. User believes the money likely sat there since the account's vintage date (05/05/1994) essentially the whole time, i.e. close to the full ~30 years, though not yet confirmed by an actual document. +* TODO Figure out which institution held the Mor provident-fund money (קופת גמל, employer גיטה טכנולוגיות, tax-vintage 01/10/2018) before the 28/07/2025 transfer into Mor (exact transfer-in amount: 101,603.71 ₪ — עמית תגמולים 32,186.97 + מעסיק תגמולים 26,636.30 + מעסיק פיצויים 42,780.44). RULED OUT so far: Harel (wrong employer tag) and Migdal (amounts don't match — 85,802.45 ₪ withdrawn from the two Migdal study-fund accounts vs 101,603.71 ₪ transferred into Mor; also structurally impossible, since study funds never have a "מעסיק פיצויים" component and the Mor transfer-in explicitly includes 42,780.44 ₪ of exactly that, so the source MUST have been another קופת גמל/provident-fund-type account with both תגמולים and פיצויים, not a study fund). No confirmed candidate yet — ask the broker (שגיא יוגב) for the specific Mor ניוד confirmation, or check ~/Downloads and ~/downloads for a similarly named "טופס העברה" file tied to Mor/1291666. +* TODO Get documents from Clal (כלל חברה לביטוח) for the pure-risk life insurance policy 4672059 (since 01/08/2025). Low priority — no investment/accumulation component. Not the same as the already-excluded Clal car-insurance file. +* TODO Resolve Phoenix gap: life insurance policy 3766373017 detailed annual reports missing for 2020-2023 (have 2018, 2019, 2024, 2025). +* TODO Resolve Phoenix gap: pension policy 7189612018 ("כללית" / marketed as "הפניקס פנסיה משלימה") missing annual reports for 2024 and 2025 — only have Q1/Q2 2026 quarterly reports so far. +* TODO Decide filing convention for tax-returns/7391_15_12_2024_13_08_23.pdf — an Israeli Form 106 (annual employer tax/withholding certificate, ~W-2) for tax year 2018, employer גיטה טכנולוגיות, via payroll processor Hilan. Currently loose at project root. +* TODO Decide filing convention for the root-level "Streamline ..." PDFs (accountant's Streamlined Procedure engagement letter, client checklist, items-to-bring list, all dated "2020-2022"). Currently loose at project root. diff --git a/thesisexperiments_1.org b/thesisexperiments_1.org index 33b1a65..0cf03d2 100644 --- a/thesisexperiments_1.org +++ b/thesisexperiments_1.org @@ -5,20 +5,13 @@ #+title: thesis/experiments #+filetags: :project: :knowledge: :experiments: -:PROPERTIES: -:ID: 59f84838-71a9-4aab-a494-8e6bea2080fb -:END: - -#+title: thesis/experiments -#+filetags: :project: :knowledge: :experiments: - Chapter 4 of the BGU MSc thesis on ROLL. Parent: [[id:6294e2be-6189-4473-b363-a1dd9a75fb9b][thesis]] ** Data Sources - =results-final/keel/= — 12 KEEL datasets × 8 methods × 5 episodes, clean labels - =results-final/keel-noise-05/10/20/= — same 12 datasets × 8 methods × 10 episodes, asymmetric noise -- =results-final/cifar10n/= — 3 noise levels (clean/aggre/worse) × 7 methods × 3 episodes; no roll-tpr90 +- =results-final/cifar10n/= — 3 noise levels × 7 methods × 3 episodes; no roll-tpr90 - =results-final/bank-marketing/=, =creditcard/=, =homecredit/= — 5 episodes each, all methods - Each directory contains per-dataset =auc.csv= files (already aggregated from pkl) @@ -38,16 +31,21 @@ Mean AUC across all 12 KEEL datasets at increasing asymmetric label-noise rates: | mae | 0.875 | 0.878 | 0.866 | 0.804 | −0.071 | | roll-tpr | 0.843 | 0.786 | 0.771 | 0.773 | −0.070 | -*ROLL-AOC is best at 20% noise* despite being mid-pack at 0%. GCE/MAE (designed for noise) degrade most. +*ROLL-AOC is best at 20% noise* despite being mid-pack at 0%. -*** CIFAR-10N (image noise) -| Noise | roll-aoc | gce | mae | libauc | bce | focal | asym | -|--------|----------|-------|-------|--------|-------|-------|-------| -| clean | 0.980 | 0.976 | 0.976 | 0.979 | 0.977 | 0.977 | 0.972 | -| aggre | 0.934 | 0.932 | 0.930 | 0.921 | 0.924 | 0.912 | 0.910 | -| worse | 0.821 | 0.774 | 0.809 | 0.752 | 0.738 | 0.734 | 0.720 | +*** CIFAR-10N (image noise) — updated 2026-09-02 +Results from =results-final/cifar10n/auc.csv= (n=3 episodes, all 7 methods). +Regenerated via =scripts/regen_cifar10n_summaries.py= after fixing cuda→cpu map_location. -ROLL-AOC best at all three noise levels on CIFAR-10N. +| Noise | roll-aoc | libauc | gce | mae | asym | bce | focal | +|-------|----------|--------|-------|-------|-------|-------|-------| +| clean | 0.980 | 0.979 | 0.976 | 0.974 | 0.973 | 0.975 | 0.973 | +| aggre | 0.932 | 0.931 | 0.930 | 0.929 | 0.927 | 0.927 | 0.922 | +| worse | 0.819 | 0.810 | 0.811 | 0.810 | 0.810 | 0.806 | 0.809 | + +ROLL-AOC best at all noise levels. At aggre, leads LibAUC by 0.001 and GCE by 0.002. +At worse, all baselines cluster 0.806–0.811; ROLL leads GCE by 0.008. +Degradation (clean→worse): ROLL-AOC −0.161 (least); others −0.163 to −0.169. *** Large Datasets - Credit Card Fraud: roll-tpr90 best (0.974), roll-aoc close (0.968); both beat all baselines @@ -60,6 +58,7 @@ ROLL-AOC best at all three noise levels on CIFAR-10N. - keel-noise runs use 10 episodes; clean keel uses 5. Mixing is fine for averages. - libauc-auroc missing 2 episodes for glass2 in clean KEEL (n=58 not 60) — known gap - cifar10n has no roll-tpr90 variant; table excludes it +- =scripts/regen_cifar10n_summaries.py= originally only remapped mps→cpu, not cuda→cpu. Fixed 2026-09-02. - Bank Marketing failure cause unknown; do not claim ROLL is universally superior on large tabular data ** Section Structure @@ -70,7 +69,7 @@ ROLL-AOC best at all three noise levels on CIFAR-10N. | Performance Measures | sec:experiments:performance_measures | Done | | Configurations | sec:experiments:configurations | Done | | KEEL Results | sec:results:keel | Done — table filled | -| CIFAR-10N Results | sec:results:cifar10n | Done — table filled | +| CIFAR-10N Results | sec:results:cifar10n | Done — updated 2026-09-02, all 7 methods | | Large Datasets Results | sec:results:large | Done — table filled | | Label-Noise Robustness on KEEL | sec:experiments:noise | Done — table added | | Ablation Study | sec:experiments:ablation | Stub — future work | diff --git a/thesisintroduction.org b/thesisintroduction.org index 88d9090..c230495 100644 --- a/thesisintroduction.org +++ b/thesisintroduction.org @@ -5,16 +5,16 @@ #+title: thesis/introduction #+filetags: :project: :knowledge: :introduction: -:PROPERTIES: -:ID: thesis-introduction -:END: - -#+title: thesis/introduction -#+filetags: :project: :knowledge: :introduction: - Chapter 1 of the BGU MSc thesis on ROLL. Written last, after Ch. 2–4 are stable. Parent: [[id:6294e2be-6189-4473-b363-a1dd9a75fb9b][thesis]] +** Thesis Narrative (corrected 2026-08-29) + +The thesis is about learning under *label noise and class imbalance*, solved via +direct AUC optimization. The hero method is ROLL-AoC. The surprise finding is +*implicit* noise robustness — not a design goal, but a structural consequence of +distributional score fitting. + ** Guidelines - Target length: 4–6 pages @@ -24,20 +24,33 @@ Parent: [[id:6294e2be-6189-4473-b363-a1dd9a75fb9b][thesis]] ** Structure -1. *Hook* (~1–2 paragraphs) — concrete application where a wrong operating point is costly. - Candidates from existing prose: medical screening, fraud detection, astronomy/microscopy pipelines. +1. *Hook* (~1–2 paragraphs) — concrete scenario where noisy labels + imbalance combine badly. + Good candidates from experiments: ChestX-ray14 (NLP-extracted labels, IR≈7.5:1), + credit card fraud (IR≈577:1), CIFAR-10N (crowdworker noise, 40% worst-annotator). Make the reader feel the problem before naming it. -2. *Problem statement* (~1 paragraph) — class imbalance + why accuracy is the wrong metric + - the TPR@FPR objective in one crisp sentence. -3. *Contributions* (bulleted list, ~5 items) — what ROLL does that prior work does not: - - Differentiable TPR@FPR loss (and FPR@TPR by label/score symmetry) - - Gaussian, Beta, KDE instantiations with closed-form gradients - - Gradient-balance property (updates independent of class-size ratio) - - Gradient-locality property (threshold-focused gradient mass) - - Empirical evaluation on KEEL benchmark + larger datasets + +2. *Problem statement* (~1–2 paragraphs) — three interlocking failures: + - Standard losses (BCE, focal) fail under imbalance (accuracy wrong metric) AND noise + (mislabeled samples produce large isolated gradient outliers) + - Noise-robust losses (GCE, MAE) stay per-sample — no AUC connection + - AUC-optimizing methods (LibAUC) use pairwise surrogates that lose direct connection to the curve + Gap: no single method simultaneously optimizes AUC, handles imbalance structurally, and is + robust to label noise. + +3. *Contributions* (bulleted list): + - ROLL: general framework for differentiable ROC-based optimization via score distribution fitting + - Three instantiations: Gaussian, Beta, KDE (KDE is primary) + - ROLL-AoC: direct numerical AUC integration via 99 per-FPR evaluations — distinct from pairwise surrogates + - Gradient balance property: updates invariant to class-size ratio + - Gradient locality property: gradient mass concentrates near operating threshold + - Empirical: *implicit* noise robustness — ROLL-AoC outperforms noise-designed losses (GCE, MAE) + at 20% noise, consistent across tabular (KEEL), image (CIFAR-10N), medical imaging (ChestX-ray14) + 4. *Roadmap* (~1 paragraph) — one sentence per chapter, matching final chapter titles exactly. ** Gotchas - Roadmap must exactly match final chapter titles — write it last -- Don't use the inseparability / cascade-classifier tangent from the old draft; it dilutes the argument \ No newline at end of file +- Don't use the inseparability / cascade-classifier tangent from the old draft; it dilutes the argument +- Don't frame as "operating point optimization" — the pivot is AUC + label noise +- The noise robustness is *implicit* (emergent), not a design objective — this is the surprising result to sell diff --git a/thesismethod.org b/thesismethod.org index 19630bf..ea76846 100644 --- a/thesismethod.org +++ b/thesismethod.org @@ -28,6 +28,9 @@ This distinction must be stated clearly in Ch. 3 and Ch. 2 related work. - sec:imbalanced-tpr-fpr and sec:neyman-pearson were removed from this chapter and live in Ch. 2 - Both objectives (TPR@FPR and FPR@TPR) are equivalent by negating scores and swapping labels; all derivations given for TPR@FPR only — FPR@TPR follows by transformation +- EXCEPTION to the above (2026-09-10): sec:np-class-weighting and sec:np-noise-invariance are + NP-theoretic content that DOES live in this chapter now, despite NP itself being background — + see the dedicated section below for why and where. ** Section Structure @@ -38,14 +41,358 @@ This distinction must be stated clearly in Ch. 3 and Ch. 2 related work. | Differentiability Problem | sec:differentiability-problem | Written | | Score Distribution Fitting | sec:score-distribution-fitting | Written | | General ROLL Formulation and Derivation | sec:roll-formulation | Written — core theoretical contribution | -| Gaussian ROLL (forward + gradient) | sec:roll-gaussian | Written | -| Beta ROLL (forward + gradient) | sec:roll-beta | Stub — TBD | +| Distribution Instantiations (intro only) | sec:roll-instantiations | Rewritten — KDE only, parametric → appendix | | KDE ROLL (forward + gradient) | sec:roll-kde | Written | -| Properties | sec:roll-properties | Written — gradient balance + locality | +| Properties | sec:roll-properties | Written — gradient balance + locality + sign correctness | | Custom Backward Pass | sec:roll-backward | Stub — TBD | -| Numerical Stability | sec:roll-numerical-stability | Stub — TBD | +| Numerical Stability | sec:roll-numerical-stability | Written — normalization invariance proposition + frozen-c convention (2026-09-09) | | Bandwidth Estimation (ISJ + scheduling) | sec:kde-bandwidth | Written | +sec:np-class-weighting and sec:np-noise-invariance are NOT in this chapter as of the final +2026-09-10 move -- they live in Appendix L now (appendix:np-theory), pointed to from +Discussion 5.1.1. See the dedicated section below for the full story; don't re-add them +here if seen in stale context. + +Gaussian and Beta subsections removed from Ch. 3 body; full math lives in appendix: +- Gaussian (forward + analytic gradient): appendix:gauss-grad (appendix_gauss_grad.tex, expanded) +- Beta (forward + autograd note): appendix:beta-roll (appendix_beta_roll.tex, new) + +** Appendix Structure (after 2026-08-29 restructure, +2 appendices added 2026-09-09) + +| Label | Chapter title | File | Hebrew letter | +|---|---|---|---| +| appendix:gauss-grad | Gaussian ROLL: Forward Pass and Gradient Derivation | appendix_gauss_grad.tex | נספח ב | +| appendix:beta-roll | Beta ROLL: Forward Pass and Gradient | appendix_beta_roll.tex | נספח ג | +| appendix:kde-grad | KDE ROLL: Gradient Derivation | appendix_kde_grad.tex | נספח ד (now also has sec:kde-grad-stability, 2026-09-09) | +| appendix:kde-sigmoid | KDE ROLL: Sigmoid Kernel — Explicit Equations | appendix_kde_sigmoid.tex | נספח ה | +| appendix:kde-bisection | KDE ROLL: Bisection for CDF Inversion | appendix_kde_bisection.tex | נספח ו | +| appendix:kde-nr | KDE ROLL: Newton–Raphson Inversion (Superseded) | appendix_kde_nr.tex | נספח ז | +| appendix:roll-code | ROLL Loss: Implementation Listing | appendix_roll_code.tex | נספח ח | +| appendix:keel-noise | KEEL Benchmark: Noise Tables at 5% and 10% | appendix_keel_noise.tex | נספח ח (verify) | +| appendix:grad-balance | Proof of the Gradient Balance Proposition | appendix_grad_balance.tex | נספח ט | +| appendix:bandwidth-freeze | KDE ROLL: Adaptive-Bandwidth Gradients and the Freezing Convention | appendix_bandwidth_freeze.tex | נספח י (appended at end, 2026-09-09) | +| appendix:score-normalization | Score Normalisation: Invariance, ISJ, and the Frozen Chain Rule | appendix_score_normalization.tex | נספח יא (appended at end, 2026-09-09) | +| appendix:np-theory | Theoretical Justification for AUC-Optimizing Losses via the Neyman-Pearson Lemma | appendix_np_theory.tex | נספח יב (appended at end, 2026-09-10; NOT ROLL-derived math, holds sec:np-class-weighting/sec:np-noise-invariance, pointed to from Discussion 5.1.1) | + +Note: not every audit fix gets its own appendix — small, self-contained formula-level +fixes (like the sigmoid-derivative stability fix) go directly into the existing relevant +appendix as a new paragraph/section; only fixes needing real counterexamples/propositions +(bandwidth freezing, score normalization) got whole new appendix chapters. Keep it that +way — the user pushed back once already on the thesis getting too long. + +** Mentor Audit Progress (2026-09-07 audit, thesis_mathematical_audit.pdf) + +Full narrative for each item (derivations, counterexamples, code changes, test results) +lives in [[id:151159e8-1105-4a84-af9c-72d31c5e0660][thesis/todos]] under "Mentor mathematical audit fixes" — not duplicated here. +This audit list is now FULLY CLOSED (all items DONE as of 2026-09-09). + +DONE (2026-09-09), in priority order: bandwidth backprop convention (frozen v_0,v_1 + +appendix:bandwidth-freeze) → bisection stopping criterion (certified-loss bound, +appendix:kde-bisection + roll.py fix) → bisection bracket range (exact target-dependent +bracket, same files) → score-normalization contradiction (normalization-invariance +proposition + frozen-c convention, appendix:score-normalization) → cached-sigmoid +derivative underflow (log-space/softmax fix, sec:kde-grad-stability in appendix_kde_grad.tex ++ roll.py `_log_sigma_prime`) → indicator-derivative-at-jump wording (undefined, not +infinite, + finite-sample-vs-population-rate distinction) → score/label reversal scope +(footnote: negation trick needs reflection-equivariance; Beta uses s→1−s instead, verified +exact via the â↔b̂ swap identity) → softened "structural property" overclaim in Ch. 5's +noise-robustness discussion. + +CORRECTED framing of the common thread (2026-09-09 — user caught my oversimplification, +do NOT put the old version in the thesis): it is NOT "freeze anything numerically +estimated" — tau is also numerically estimated (via bisection) and we deliberately DO +differentiate through it, since that's the entire mechanism by which negative-class +scores get any gradient at all. The real distinction: differentiate through quantities +that are part of the TRUE objective's own definition (tau — TPR@FPR is defined via "the +threshold achieving this FPR," so tau's score-dependence is structural, not an artifact); +freeze quantities that exist ONLY because of the smoothing/approximation scheme used to +make the true (indicator-based) objective differentiable in the first place, and have no +counterpart in that true objective at all (v_0, v_1 bandwidth; c normalization scale). +User's framing: bandwidth is "an estimation of our approximation of a theoretical +distribution... could be wrong and flawed, and produces artifacts" when differentiated +through — that's the right way to say it. This principle is NOT yet written into the +thesis (user wants it kept out of the discussion chapter for now, maybe later) — this +memory entry is for continuity across sessions only. + +** Third Mentor Review (2026-09-10, screenshots) — AOC divisor mismatch + +Separate from the "second mentor review" 5-point list below (different reviewer/round, +delivered as screenshots so no direct file/page cite -- mentor referred to it loosely as +"equation 3.13" but the actual affected equation turned out to be eq:roll-aoc, now (3.15)). + +Mentor's claim: thesis defines L_ROLL-AOC with normalization 1/99 (eq:roll-aoc, +method.tex), but the implementation's _AOC_DIVISOR is 100. Mentor's argument: grid +spacing is 0.01 (i.e. 1/100), so a Riemann-sum approximation should divide by 100, +assuming the omitted alpha=1 endpoint is zero. + +VERIFIED: real mismatch, confirmed directly in impl/src/roll.py (sibling repo, NOT under +thesis/ -- lives at /home/aner/projects/research-projects/roll/impl/src/roll.py): +`_AOC_DIVISOR = 100` (line 114), used in BOTH the forward pass (`total_loss = +losses.sum() / divisor`, line 296) and the backward-pass gradients (lines 320, 333) -- +code's own comment even states "Sum divided by 100 approximates integral [sec:roll-aoc]", +i.e. the code was ALWAYS intended to match the thesis and the thesis just had the wrong +constant written down. Since the code is what actually produced every reported result, +the FIX belongs in the thesis text, not the code. + +IMPORTANT NUANCE (checked numerically before agreeing with the mentor's full framing): +g(1) = 1-TPR(FPR^-1(1)) = 1-TPR(-infinity) = 0 EXACTLY (verified: thresholding at -infinity +classifies everything positive) -- so (1/100)*sum_{i=1}^{99} IS a complete, valid +right-Riemann sum over 100 equal subintervals of [0,1] (the "missing" 100th term is known +to be exactly zero, not actually omitted). This gives a clean, PROVABLE error bound of +exactly 1/100=0.01 (classical monotone-function Riemann-sum bound: error <= range*step = +(g(0)-g(1))*(1/100) <= 1*(1/100)), tighter and cleaner than the old "1/99≈0.0101" claim. +BUT: numerically stress-tested both /99 and /100 against the true integral on 200 random +Gaussian-type AOC curves, and /99 had SMALLER actual error than /100 in ALL 200 trials +(though both stayed well under the 0.01 worst-case bound, ~0.005 typical). So it is NOT +mathematically true that "100 is more accurate than 99" for realistic AOC integrands -- +it's simply that 100 is what the code computes and is a cleanly-justifiable, provably- +bounded choice, not a superior one. Told the user this nuance directly before fixing, +rather than just conceding "99 was a bug, 100 is better" at face value. + +FIX applied (method.tex, sec:roll-aoc), all three affected spots: +1. eq:roll-aoc (now eq 3.15): 1/99 -> 1/100. +2. Darboux-bound paragraph right after it: rewritten around the g(1)=0-exact fact and the + right-Riemann-sum framing, replacing the old approximate "1/99≈0.0101 by a standard + Darboux-sum argument" with the new EXACT derivation and clean 0.01 bound (shown as + |L_ROLL-AOC - L_AOC| <= (g(0)-g(1))*(1/100) <= 1/100 = 0.01). +3. eq:roll-aoc-gradient (now eq 3.16): 1/99 -> 1/100 (gradient of a linear combination + scales the same way as the combination itself). +Checked for other stale "1/99" or "99-point average" mentions elsewhere in the thesis +(appendix_gauss_grad.tex, discussion_and_conclusions.tex, method.tex intro) -- all other +occurrences just say "99-point" or "99 evenly-spaced" descriptively (correctly describing +the NUMBER of evaluated points, unaffected by the divisor value), none needed changes. +Rebuilt thesis clean (nix build exit 0), confirmed eq 3.15/3.16 both show 1/100 and the +new bound derivation renders correctly in the compiled PDF. + +** Second Mentor Review (2026-09-09) — 5 points, status + +New round of feedback, distinct from the first audit. ALL 5 POINTS NOW RESOLVED as of +2026-09-10 (episode-count justification, the last open one, closed same day as this note). +Full detail in [[id:151159e8-1105-4a84-af9c-72d31c5e0660][thesis/todos]]; summary here: +1. Justify varying episode counts — RESOLVED 2026-09-10. User's corrections/actions: + CIFAR-10N was actually already n=5 (a stale "n=3" caption, not a real 3-episode run) — + just a caption fix, no data change. ChestXray14 and Food-101N WERE genuinely at n=3 — + user ran 2 more real episodes each and updated the mean/std numbers in + tab:results:large accordingly (git diff showed real numeric changes to those two rows + only, e.g. ChestXray14 ROLL-AOC 0.839->0.838, Food-101N 0.867->0.867 with updated std + devs) before asking me to verify entry + rewrite the counts. I caught one formatting + nit while verifying: Food-101N/Focal cell had "$\pm$0.01" (2 decimals) vs 3 decimals + everywhere else in that table -- fixed to "$\pm$0.010". + Final state: n=5 EVERYWHERE except KEEL noise experiments (5%/10%/20%), which stay at + n=10, with justification added inline in sec:experiments:configurations (experiments.tex + ~line 76): "KEEL's datasets are small enough that each episode trains quickly despite + the number of datasets involved, making it practical to run more episodes there for a + more reliable noise-robustness comparison" -- user's own justification, given verbatim + in the request ("keel dataset benchmarks ran very fast, despite their number, so they + ran for more episodes"). Also fixed the two stale table captions (tab:results:cifar10n + "n=3"->"n=5"; tab:results:large dropped the "Food-101N and ChestXray14: n=3" exception + entirely, now just states "n=5 episodes" for the whole table). Rebuilt thesis clean (nix + build exit 0), confirmed every episode-count mention in the compiled PDF is now n=5 + except the three KEEL-noise tables (4.3, H.1, H.2), which correctly stay n=10. +2. AUC integral definition: same variable (alpha) on page 5 (Ch.2) vs Ch.3? — RESOLVED + 2026-09-10. Was a real inconsistency: TPR is defined everywhere else (eq 3.1, eq 3.6) as + a function of a THRESHOLD tau; FPR^-1(alpha) is the separate function mapping an FPR + level to a threshold; Ch.2's page-5 definition and eq:objective-auc (eq 3.5) correctly + write the composition TPR(FPR^-1(alpha)), but sec:roll-aoc (method.tex, starting ~line + 300) silently dropped the FPR^-1 wrapper and wrote bare "TPR(alpha)" six separate times + (lines 300, 303, 306, 324, 325, and the fig:roll-aoc-roc caption) -- overloading TPR to + take an FPR-level directly, contradicting its own prior definition. Not a computational + bug (the AUC value computed is identical either way) but a real notation-consistency + flaw -- exactly the kind of thing a careful reader/mentor would catch. Fix: global + replace_all of the literal string "\operatorname{TPR}(\alpha)" -> + "\operatorname{TPR}(\operatorname{FPR}^{-1}(\alpha))" across method.tex (safe as a + blanket replace since grep confirmed all 6 occurrences in the whole file needed the + same fix, and no correctly-already-composed instance used that exact bare substring). + Note: $\mathcal{L}_{\text{ROLL-TPR@FPR}}(\alpha)$ notation right next to these was NOT + part of the problem -- that loss was already, by its own original definition + (eq:roll-tpr-at-fpr), a function of alpha directly, so no inconsistency there; only the + TPR symbol specifically was being overloaded. Rebuilt thesis clean (nix build exit 0), + confirmed all instances now render as "TPR(FPR⁻¹(α))" consistently throughout the PDF + (eq 3.5, eq 3.13, sec:roll-aoc prose, and the figure caption all match). +3. Theoretical justification for noise robustness — RESOLVED 2026-09-09, see below. +4. Exact gradient or stop-gradient approximation for the backward pass? — ALREADY FIXED, + pre-existing text. method.tex:470-476 explicitly: "We adopt this as a deliberate + stop-gradient convention"; appendix:bandwidth-freeze derives the fully-adaptive (exact) + gradient alongside specifically to show what's traded away. No ambiguity. +5. "Chapters A and B" should say "Appendix" — WAS NOT FIXED, now IS. Root cause: this + wasn't literal hardcoded text (source already used \Cref{appendix:...}) — it was a + cleveref config bug. Appendices are typeset via \chapter, so cleveref classified every + appendix label under the generic "chapter" counter and printed "Chapter X" everywhere + (~10+ instances across the whole PDF: "Chapter A", "Chapters A and B", "Chapter I", + "Chapter H", etc.), even though main.tex:170-171 already correctly defined + \Crefname{appendix}{Appendix}{Appendices} — that declaration was just never triggered. + Fix: added \crefalias{chapter}{appendix} right after \appendix in main.tex. Rebuilt and + confirmed every appendix ref now says "Appendix", the 5 real chapters still say + "Chapter N" (crefalias only affects labels defined after it in source order, and no + appendix file refers back to a main chapter{:...} label, so this was safe). + +** Neyman-Pearson Theory: now Appendix L, pointed to from Discussion 5.1.1 + +Answers second-mentor point 3 ("is there a theoretical justification for noise +robustness?"). Has moved TWICE now (2026-09-10, same day, both moves user-initiated): +Ch.2 (related_work.tex, original) -> Ch.3 (method.tex, inside sec:roll-formulation) -> +final location: Appendix L (content/appendixs/appendix_np_theory.tex, new chapter added +2026-09-10, label appendix:np-theory, \crefalias makes it render as "Appendix L" not +"Chapter L" per the cleveref fix). Do not be surprised by the two intermediate locations if +they show up in old context/diffs -- Appendix L + Discussion 5.1.1 is the FINAL, current +state; Ch.2 and Ch.3 no longer contain this material at all. + +User's reasoning for the final move: read the Ch.3-embedded version and felt it was still +in the wrong place -- genuinely new theoretical content shouldn't be buried inline in the +Method chapter either. Final structure: (1) Appendix L holds the full two-section analysis +verbatim (sec:np-class-weighting = L.1, sec:np-noise-invariance = L.2, both \section now, +not \subsection, matching appendix-chapter convention where each appendix's top-level +divisions are \section); (2) Discussion sec:discussion:np-justification (new subsection, +FIRST subsection of Section 5.1 "Discussion", before "Noise Robustness of ROLL-AOC") is a +short pointer paragraph, wording lifted close to verbatim from what the user dictated, +ending "The full analysis is given in \Cref{appendix:np-theory}"; (3) Ch.3 (method.tex) has +ZERO remaining trace -- fully reverted to how it read before any of this was added; (4) Ch.2 +sec:neyman-pearson also has zero trace, ends at "The practical gap" as it always did. + +Generalization the user explicitly asked to be stated: this is NOT ROLL-specific. It +justifies any order-dependent/ranking-based loss -- primarily ROLL-AOC, but also LibAUC and +similar AUC-optimizing methods -- since none of them ever form a class-weight ratio or +compare to a fixed eta in the first place. Added this framing to: the appendix's own lead-in +paragraph, the end of L.1 (explicit LibAUC mention), and the Discussion pointer paragraph. + +Appendix L.1's "ROLL avoids this" sentence now says "The TPR@FPR objective of +\Cref{sec:roll-formulation}, \eqref{eq:roll-tpr-at-fpr}" (explicit chapter-crossing +reference, since the appendix is no longer inside the section that defines it) -- still +resolves correctly cross-chapter via \eqref/\Cref, no different from how other appendices +already reference Ch.3 equations (e.g. appendix:kde-bisection referencing method.tex +equations). + +Follow-up (2026-09-10, same day): user asked to add a per-sample "outcome" framing to L.1, +calling it "the main driver" showing the practical difference between sample-wise and +order-dependent losses. Not a new derivation -- it's the same q*/eta result from the +existing footnote, unpacked concretely: since q*(x)>1/2 iff L(x)>eta, for ANY given x the +classifier ends up calling it positive or negative purely by which side of eta its own +L(x) falls on, independent of how separable x actually is. Two samples with nearly +identical L(x) straddling eta can land on opposite sides purely because of where eta sits +-- and since eta is fixed entirely by the chosen weight ratio, the weight choice (not the +data) decides borderline samples' fate. Added as a new paragraph in appendix_np_theory.tex +sec:np-class-weighting (L.1), right after the eta-derivation paragraph, before "The +standard heuristic..." paragraph. Closing sentence makes the contrast explicit: "This is +the central difference from an order-dependent loss: a per-sample loss must commit to some +eta before training even begins, and every sample's outcome is filtered through that +fixed, weight-determined cutoff." Rebuilt thesis clean (nix build exit 0), confirmed the +new paragraph renders correctly in Appendix L.1. + +Follow-up 2 (2026-09-10, same day): user asked why eta = (w0/w1)*(pi0/pi1) specifically had +the pi0/pi1 factor -- explained it's NOT part of NP itself (NP just says "some eta exists, +thresholds the likelihood ratio"); the pi0/pi1 comes from Bayes' rule converting between +POSTERIOR odds (what a weighted loss naturally compares: w1*P(y=1|x) vs w0*P(y=0|x)) and +LIKELIHOOD ratio (NP's native currency) -- posterior odds = likelihood ratio * prior odds, +so translating the weighted-loss condition into likelihood-ratio form introduces pi1/pi0. + +Follow-up 3 (2026-09-10, same day): user then asked to SIMPLIFY the whole section to avoid +needing eta's exact formula at all -- "it exists, proportional to the class-weight ratio, +same for all samples; changing weights changes target eta; noise changes interaction with +eta, regardless of how eta is defined; don't need the eta=1 part, keep it simple." Realized +the cleanest fix wasn't to hedge/soften the existing likelihood-ratio version, but to +REWRITE L.1 entirely in POSTERIOR-ODDS terms (matching L.2, which already worked this way) +instead of likelihood-ratio terms -- this eliminates the Bayes'-rule/prior-odds conversion +entirely, since a weighted loss compares posteriors NATIVELY, with no translation needed: +q*>1/2 iff O(x)=P(y=1|x)/P(y=0|x) > w0/w1, exactly, no pi0/pi1 anywhere. Kept this as an +EXACT stated value (not hedged as merely "proportional") since the footnote genuinely +proves the exact identity for weighted CE -- hedging it as merely proportional right next +to a footnote proving exact equality would have been inconsistent, so instead added one +sentence generalizing the QUALITATIVE takeaway (one constant, same for every sample, moves +with the weight ratio) as what carries over to other per-sample losses, without claiming +the exact w0/w1 formula does. REMOVED entirely per explicit request: the "eta=1 exactly" +class-imbalance-cancellation paragraph (was the whole reason pi0/pi1 existed in the old +version). Also removed L.2's opening footnote bridging its posterior-odds eta to L.1's old +likelihood-ratio eta -- no longer needed since both sections now use the SAME O(x) object +throughout, L.2 just directly continues from L.1's result. L.2's OWN "shrinks to w=1" +result (the noise-validity-window boundary case) was NOT removed -- that's a different, +still-wanted finding (about noise, not about class-imbalance cancellation), user's "don't +need the eta=1 part" referred specifically to L.1's paragraph. Rebuilt thesis clean (nix +build exit 0), confirmed the removed content is gone and no new undefined refs. + +CAVEAT ADDED BY ME (not yet confirmed by user): translated the new appendix chapter title +and the Discussion subsection's Hebrew TOC lines myself (I am not a native speaker) -- +"הצדקה תיאורטית לפונקציות הפסד ממטבות AUC לפי למת ניימן-פירסון" (appendix) and "הצדקה +תיאורטית לפונקציות הפסד ממטבות AUC" (discussion subsection). Flagged to the user directly +that these should be reviewed/corrected before treating this as final, unlike the rest of +this thesis's Hebrew content which presumably had more careful authorship. + +*** sec:np-class-weighting — class weighting as an implicit choice of eta + +Core claim, sympy-verified before writing: for a weighted per-sample loss (weighted CE), +minimizing pointwise risk and thresholding at q>1/2 is exactly equivalent to thresholding +the likelihood ratio p(x|y=1)/p(x|y=0) > eta with eta = (w0/w1)*(pi0/pi1). Standard +inverse-class-frequency weighting (w1/w0 = pi0/pi1) makes eta collapse to EXACTLY 1, +regardless of noise/separability/distribution shape — i.e. it cancels the prior imbalance +rather than targeting any actual desired FPR; this is a pure algebraic cancellation baked +into the recipe (w0/w1 is DEFINED as the reciprocal of pi0/pi1), not a data-dependent +coincidence — shown explicitly via a displayed substitution, added after the user asked why +eta=1 held in general, not just in one example. + +Derivation placed as a footnote (short, ~4 lines), not a full appendix — user's call +("footnote if it's a few lines, appendicise if more"). No new bib entry added; self-contained +derivation, not attributed to a specific citation (Elkan-style cost-sensitive learning +literature is the likely place this already exists, but wasn't verified/added — flag for the +user's own citation pass if wanted later). + +Scope note (user corrected an earlier, weaker framing before this was even written into the +thesis): an early draft claimed ROLL-AOC "doesn't commit to a single eta" by sweeping 99 +alpha values — user rejected this: at any actual decision point a classifier commits to one +threshold, hence one implied eta; sweeping alphas during AOC training isn't the same as +avoiding commitment to one eta at inference. Dropped; only the TPR@FPR "discovers eta as it +goes" point was kept. + +*** sec:np-noise-invariance — class weighting under label noise: an implicit, distorted eta + +User's own idea (not mine), refined through two full passes: + +PASS 1 (superseded): user proposed noise gives each sample a different "target eta." +Checked with sympy — NOT what happens under class-conditional (x-independent) noise: the +noisy likelihood ratio L~(x) depends on x ONLY through the true ratio L(x), via a fixed +Mobius function g(L), strictly monotone for rho<1/2 — i.e. ONE global bijection, not +per-sample. Written up using this density-ratio g(L) form directly. + +PASS 2 (current thesis text): while explaining the g(L) derivation to the user, caught a +real bug — p(x|ytilde=k) needs dividing by the normalizing constant D_k=P(ytilde=k) to be a +proper conditional density, which the written g(L) omitted; true L~(x) = (D0/D1)*g(L(x)), +not g(L(x)) exactly (sympy-verified). Didn't affect the monotonicity conclusion (D0/D1 is a +positive x-independent constant) but was imprecise as written. Fixing it surfaced the real +goal more clearly: not just "ranking is preserved" abstractly, but "noise implies an eta +implicitly chosen by score/class weighting, one ROLL never has to choose." Reframed entirely +in POSTERIOR-ODDS space instead of density-ratio space, which sidesteps the bug rather than +patching it (no priors/normalizing constants needed at all): O(x)=P(y=1|x)/P(y=0|x), and +P(ytilde=1|x) = (1-rho)P(y=1|x)+rho*P(y=0|x) via elementary total probability (no Bayes'-rule +density manipulation, no joint-vs-conditional distinction to get wrong). Gives +P(ytilde=1|x)/P(ytilde=0|x) = h(O(x)), h(O)=[rho+O(1-rho)]/[(1-rho)+O*rho], h'(O)= +(1-2*rho)/[...]^2 > 0 for rho<1/2 (sympy-verified; h(O)->O at rho=0 sanity check passes). + +Core claim: training with weight ratio w=w0/w1 on NOISY labels doesn't threshold O(x) at w +-- it thresholds h(O(x)) at w, equivalent to thresholding the TRUE posterior odds at +eta_true = h^{-1}(w) = [w(1-rho)-rho]/[(1-rho)-w*rho]. Whatever w you pick, you actually get +eta_true, which depends on the (usually unknown) rho — choosing w no longer means choosing +eta, it means choosing eta up to an unknown noise-dependent distortion. Extra result found +and verified (2000-trial sympy/numeric check, 0 mismatches): eta_true is only a valid +(positive) posterior odds when rho/(1-rho) < w < (1-rho)/rho; outside that window no +classifier could produce the implied threshold at all. As rho->1/2 this window shrinks to +just {w=1} — heavy noise leaves ONLY the no-tilt threshold achievable by weighting, +regardless of intent. ROLL sidesteps the whole mechanism: no weight, no eta, no need to know +rho — it targets a rate directly, and ranking by O(x) is preserved under this noise model +(same h monotonicity fact) so the target still means what it's supposed to. + +Caveat (user's explicit request, carried unchanged across both passes and the Ch.2->Ch.3 +move — "show it's problematic under the specific condition, doesn't mean it's untrue in +general"): the whole argument needs rho to depend on y only (class-conditional), not x. +Instance-dependent noise rho(x) breaks THIS ARGUMENT (h would vary with x directly) — did +NOT claim ranking invariance actually fails there, only that this proof doesn't cover it. + +*** Build status + +Rebuilt thesis clean (nix build exit 0) at every step: first Ch.2 writeup, the pass-2 +rewrite, the eta=1 follow-up, and the final Ch.2->Ch.3 move. Confirmed after the move: +sections now number 3.2.4/3.2.5 (Method chapter), \Cref{eq:roll-tpr-at-fpr} resolves to +"(3.8)" instead of a forward-pointer, and the Ch.2 back-references (Section 2.4, Section +2.5 for sec:imbalanced-tpr-fpr/sec:neyman-pearson) and Ch.4 forward-reference +(\Cref{chapter:experiments} -> "Chapter 4") all resolve correctly. + ** Gotchas - KDE backward pass uses inverse-function-theorem approach deliberately; implicit differentiation @@ -55,3 +402,27 @@ This distinction must be stated clearly in Ch. 3 and Ch. 2 related work. - Beta ROLL: translation-invariance does NOT hold when scores pass through sigmoid before fitting; gradient-balance property breaks for Beta — documented in sec:roll-properties - Gradient-balance proof uses a uniform-shift argument; requires translation-invariant CDF estimator +- eq:roll-tpr-at-fpr-gaussian and eq:gauss-grad-combined now defined in appendix_gauss_grad.tex + (moved from method.tex) — appendix_gauss_grad.tex references both, so they must stay there +- New .tex files must be git-added before nix build (untracked files are invisible to the + flake's source enumeration) — bit both appendix:bandwidth-freeze and appendix:score-normalization + on this, now routine to check +- Never verify a derivation mentally — use sympy in the nix devShell (`nix develop --command + python3`), and cross-check with an independent finite-difference derivative when possible +- impl/src/roll.py's KernelizedROLLoss is a custom torch.autograd.Function with a hand-written + backward() — it never auto-differentiates through forward()'s intermediate computations, so + it was already implementing "frozen bandwidth" and "frozen normalization scale" by construction + (omission), not by explicit design, until the thesis text caught up to state this as a + deliberate convention (2026-09-09) +- When a KEEL regression test doesn't need bit-exact old-vs-new comparison (user will say so + explicitly, e.g. "don't need numerical exactness"), a single clean run (no NaN, sane AUC in + the range prior runs on the same dataset/config produced) is sufficient — no need for the + git-stash old-vs-new dance every time +- cleveref config gotcha: \Crefname{appendix}{Appendix}{Appendices} alone does NOT make + \Cref{appendix:...} say "Appendix" if appendices are typeset with \chapter (shares the + "chapter" counter) — need \crefalias{chapter}{appendix} after \appendix too, otherwise every + appendix cross-reference silently renders as "Chapter X" with no error/warning +- Chapter boundaries aren't fixed by topic (prior work vs. original contribution) alone — + NP the LEMMA is prior work (Ch.2), but original NP-flavored ANALYSIS the thesis contributes + can live in Ch.3 even though it's built on Ch.2 background; don't assume "cites prior work" + means "belongs in related work" for a whole section wholesale diff --git a/thesisrelated-work.org b/thesisrelated-work.org index 4f311b1..abe11b3 100644 --- a/thesisrelated-work.org +++ b/thesisrelated-work.org @@ -30,12 +30,30 @@ Parent: [[id:6294e2be-6189-4473-b363-a1dd9a75fb9b][thesis]] - sec:imbalanced-tpr-fpr and sec:neyman-pearson are \Cref-referenced in Ch. 3 (sec:problem-formulation) — do not rename these labels - KDE section (sec:kde-background) must end with a forward pointer to sec:roll-kde in Ch. 3 — established convention from project brain root - Ch. 3 KDE section opens with a back-reference to sec:kde-background — keep this pairing consistent -- sec:imbalanced-tpr-fpr arc (per TODO in file): accuracy gameable under imbalance → class weighting standard fix → no direct operating-point control → therefore need TPR@FPR directly +- sec:imbalanced-tpr-fpr arc (per TODO in file): accuracy gameable under imbalance → class weighting standard fix → no direct operating-point control → therefore need TPR@FPR directly. + This arc is now closed rigorously by sec:np-class-weighting + sec:np-noise-invariance, but + as of 2026-09-10 those two subsections live in Ch. 3 (method.tex), not here — see + [[id:0543ac39-af9b-40fd-9973-e60576a20695][thesis/method]] for full detail. sec:neyman-pearson in THIS chapter ends at "The practical + gap" again, as it originally did. - AUC section should distinguish whole-curve AUC from operating-point-specific TPR@FPR — this distinction motivates the partial AUC / TPR@FPR section - sec:noisy-labels added 2026-07-13; NP moved to 2.6 (was 2.5); sec:neyman-pearson label is UNCHANGED so Ch. 3 \Cref references still work without modification - sec:noisy-labels should be forward-referenced from Ch. 4 poisoning experiment; bridge: ROLL's robustness is a structural consequence of the distributional objective, not a design goal - Noisy labels bibtex entries staged in impl/research ** Noisy Labels Literature; move to thesis.bib when writing this section +** Class weighting / NP noise theory — MOVED to Ch. 3 (2026-09-10) + +sec:np-class-weighting and sec:np-noise-invariance (originally added here 2026-09-09, +directly after sec:neyman-pearson's "The practical gap") were relocated to method.tex on +2026-09-10, per the user's explicit call: NP itself is prior work and correctly stays in +Ch. 2, but these two subsections are original theoretical contributions of this thesis, not +background, and the user wants them visible to reviewers who skip the related-work chapter. +Full history of how these subsections were derived (the g(L) density-ratio bug catch, the +posterior-odds reframe, eta_true and its valid-window result, the eta=1 follow-up) now +lives entirely in [[id:0543ac39-af9b-40fd-9973-e60576a20695][thesis/method]] — not duplicated here, since the content itself moved. + +This chapter's sec:neyman-pearson is back to ending at "The practical gap," as it did +before 2026-09-09. + ** Noisy Labels Literature (sec:noisy-labels) *** wu2025preserving — AUC Fairness + Noisy Protected Groups (ICML 2025) diff --git a/thesistodos.org b/thesistodos.org index 0ac6ee4..24aa812 100644 --- a/thesistodos.org +++ b/thesistodos.org @@ -30,11 +30,7 @@ - Decide: write or drop Ablation Study and Sensitivity Analysis sections - Fill all \cite{TODO} placeholders in experiments.tex -* TODO Review Ch. 4 (Results) end-to-end - - Check narrative flow after noise section was added - - Confirm KEEL/CIFAR-10N/large-dataset table interpretations are accurate - - Verify roll-tpr90 is correctly described as secondary/special-case throughout - +* TODO Review Ch. 4 (Results) end-to-end - Check narrative flow after noise section was added - Confirm KEEL/CIFAR-10N/large-dataset table interpretations are accurate - Verify roll-tpr90 is correctly described as secondary/special-case throughout * Writing — front/back matter * TODO Write English abstract @@ -59,6 +55,102 @@ - Most direct competitor to roll-tpr90 (both target a fixed FPR region) - See: https://docs.libauc.org/examples/pauc.html +* Mentor mathematical audit fixes (2026-09-07, thesis_mathematical_audit.pdf) + +** DONE [#A] Decide bandwidth backprop convention (Eq. 3.16, Appendix C, Section 3.4.2, thesis pp. 25/19) — Appendix C's backward pass treats only the j=i kernel term as score-dependent, which is only valid for a FIXED bandwidth, but Section 3.4.2 estimates bandwidth from the class scores (ISJ/Silverman). Audit gave an exact counterexample where the true adaptive loss is constant but Eq. (3.16) gives a nonzero gradient. + RESOLVED (2026-09-09): chose (a) frozen-bandwidth (stop-gradient on v_0,v_1 in the backward pass). Full details in [[id:0543ac39-af9b-40fd-9973-e60576a20695][thesis/method]] under "Audit: Bandwidth Backprop Convention". Added Observation 3.4 "Gradient Sign Correctness" to sec:roll-properties in method.tex; new Appendix appendix:bandwidth-freeze with two sympy-verified counterexamples (exact cancellation + persistent sign reversal with closed-form proof). CONFIRMED with user: impl/src/roll.py's bandwidth computation already assumes frozen bandwidth — writeup-only fix, no code change needed. Fully closed. +** DONE [#A] Fix bisection stopping criterion to certify loss accuracy, not just CDF residual (Appendix E, p. 31) — current rule |F̂₀(τ̂) − (1−α)| < 1e-4 can pass while F̂₁(τ̂) is off by ~1 (counterexample given with h₀=1, h₁=0.01). + RESOLVED (2026-09-09): replaced the CDF-residual stopping rule with a certified loss-interval rule — keep bracket [lo,hi] valid, stop when F̂₁(hi) − F̂₁(lo) ≤ ε_L (exact via monotonicity of F̂₁ + the bisection bracket invariant containing the true root; no derivative/Lipschitz bound needed — that alternative was considered and rejected as strictly weaker, see [[id:0543ac39-af9b-40fd-9973-e60576a20695][thesis/method]]). + - Rewrote appendix_kde_bisection.tex (appendix:kde-bisection): new "Stopping criterion" + "Why the residual-only rule fails" paragraphs, with a constructed, code-verified counterexample (B_0={-30,-25,5,45,60}, v0=0.01, v1=1000: old rule stops 8 iterations in with tau off by only 0.014, but loss F̂_1 off by 0.40; new rule gets F̂_1 accurate to 3e-5). Rebuilt thesis clean. + - Code fix in impl/src/roll.py: both `_icdf` and `_icdf_batch` in KernelizedROLLoss now take v1/scores_pos as extra args and stop on the certified F̂_1(hi)-F̂_1(lo) bound instead of the F̂_0 residual; forward() call site updated to pass v1, scores_pos through. + - Regression-tested on KEEL glass1 with roll-aoc (seed 0, CPU, 1 episode) via git-stash old-vs-new: AUC 0.8052 (old) vs 0.8106 (new), essentially identical. Fully closed. +** DONE Restrict initial bisection bracket to valid α range — use exact target-dependent bracket (Appendix E, p. 30) — old bracket [s⁰_min−10h₀, s⁰_max+10h₀] only guaranteed containment for α∈(4.5e-5, 1-4.5e-5), not all of (0,1). + RESOLVED (2026-09-09): switched to the exact bracket lo/hi = s⁰_min/max + h₀·log(p/(1−p)), p=1−α. + - Proved in appendix_kde_bisection.tex (new Proposition "Exact bracket containment" + short proof): the kernel term at the min/max score equals p exactly at lo/hi (direct substitution), and monotonicity of σ forces every other term below/above it, so F̂₀(lo) < p < F̂₀(hi) holds for EVERY p∈(0,1), whenever the batch isn't degenerate (mininfinity, the closer sample's normalized weight converges to 1/(1+e^-v0) (~0.731 at v0=1), NOT to 0, confirmed stable from M=20 to M=1000 even though the raw (unnormalized) sigma_0' values have themselves underflowed to exactly 0.0 in float32 by M=1000. + - Nice bonus connection: verified the log-space fix from the cached-sigmoid-derivative item (sec:kde-grad-stability) ALREADY correctly preserves this ratio out to M=1000 (via softmax's max-subtraction), where the OLD cached-identity approach would have collapsed to a 0/0-type failure. No new code fix needed for this item — the earlier fix already covers it as a side effect. + - Writeup: rewrote the "Gradient locality" paragraph in sec:roll-properties (method.tex) to explicitly split the two cases (absolute for y=1, relative-share for y=0) with the verified counterexample inline; split Observation 3.3 into explicit (a)/(b) sub-claims matching the two mechanisms; softened the figure's lead-in sentence and caption (fig:threshold-locality) to note it illustrates the case where both classes are broadly spread around tau (where the two behaviors happen to agree), rather than claiming universality. Did not need to regenerate the figure itself — it's a fair illustration of the typical/generic case, only the caption's blanket wording was the problem. + - Rebuilt thesis clean. +** DONE Separate exact fitted-distribution AOC from empirical/population AOC of original scores (Sections 3.1–3.3, Figure 3.2, p. 11) — the continuous integral L^fit_AOC = P(T1≤T0) is exact only for the FITTED distributions; it is not generally the empirical/population AUC of the original scores. + RESOLVED (2026-09-09): verified against the real kernelized_roll_aoc code on a perfectly-separated finite sample (empirical AUC=1.0 exactly via sklearn) — ROLL-AOC's implied fitted AUC is 0.99994 (loss 6e-5) for widely-separated classes, and 0.971 (loss 0.029) when the separating margin is tiny relative to within-class spread. Confirms KDE smoothing means the fitted-distribution AOC structurally never reaches the ideal step-function value even at perfect empirical separation (same "smoothing gap" any smooth surrogate has relative to its sharp target). + - User's call: don't add the numeric example to the thesis, these are extreme/edge cases — just soften the overclaiming wording. Reworded the two "faithful surrogate...regardless of how scores evolve" / "most faithful surrogate" sentences in sec:roll-instantiations (method.tex, right before sec:roll-kde) to "adapts to the score distribution's shape... without committing to a parametric assumption" and "most adaptable surrogate" — drops the implied exactness/fidelity claim while keeping the real comparative point (non-parametric > parametric when the shape is unknown). Rebuilt thesis clean. + - Note for future reference (not in thesis): the verified gap numbers are in this org entry if a stronger, example-backed version of this fix is ever wanted later. +** DONE Fix Figure 3.2 caption / distinguish 99-point average from the integral identity (p. 9–10, Eq. 3.13) — L_99 is a quadrature approximation, not identical to the integral AOC. + RESOLVED (2026-09-09): derived and verified the Darboux bound — since TPR(alpha) is monotone along the ROC curve (structural, always true), g(alpha)=1-TPR(alpha) is monotone and bounded in [0,1], so a 99-point average differs from the true continuous integral by at most 1/99≈0.0101 by a standard Darboux-sum argument. Stress-tested numerically against 500 random monotonic bounded functions: worst observed gap 0.0033, comfortably under 1/99. + - Added one sentence right after eq:roll-aoc in method.tex stating this bound, with its precise scope: covers ONLY the quadrature step, does NOT extend to the gradient (eq:roll-aoc-gradient), does NOT cover the KDE-fitted-vs-empirical gap (previous fix), does NOT cover residual bisection numerical error in tau_k. + - Also lightly adjusted the fig:roll-aoc-roc caption ("gives the value of" -> "approximates the continuous integral, giving the value of") so it doesn't read as if the discrete average and the true integral are definitionally the same thing. + - User's call: no separate appendix/proposition needed, one sentence + caption tweak sufficient. Rebuilt thesis clean, confirmed the forward reference to eq:roll-aoc-gradient resolves. +** DONE Narrow "direct integration is not tractable" claim before Eq. (3.13) (p. 9) — false for the Gaussian case, which has closed form L_AOC^Gaussian = Φ((μ₀−μ₁)/√(σ₀²+σ₁²)). + RESOLVED (2026-09-09): verified the closed form numerically (matches direct quadrature to 1e-9) and re-derived it via the standard Gaussian identity ∫Φ(a+bu)φ(u)du=Φ(a/√(1+b²)). + - method.tex sec:roll-aoc: scoped the claim to "not tractable in general" + parenthetical noting the Gaussian exception and that the 99-point quadrature is used uniformly across instantiations for implementation consistency, not necessity. + - appendix_gauss_grad.tex: added new "Closed-form AOC" paragraph (right before "Gradient derivation") with the full derivation and eq:gauss-aoc-closed-form, since this fact wasn't documented anywhere previously. + - CAUGHT AND FIXED a real notational collision while writing this: sec:roll-formulation (method.tex ~line 156) already used \Phi for the ABSTRACT DISTRIBUTION FAMILY'S PARAMETER SPACE ({P_phi : phi in Phi}), clashing with \Phi as standard-normal-CDF (needed here, and already used in the translation-equivariance counterexample fix earlier this session). Renamed the parameter-space symbol phi/Phi -> psi/Psi throughout sec:roll-formulation (6 occurrences: distribution family def, hat-phi_1/hat-phi_0 fitted params, CDF/quantile-function definitions) — theta/Theta was considered and rejected since it'd collide with f_theta (model parameters), used everywhere. psi/Psi unused elsewhere in the thesis, clean choice. Confirmed related_work.tex's unrelated phi (hypothesis-testing notation, different chapter) doesn't collide. + - Rebuilt thesis clean, confirmed no undefined refs in final pass. +** DONE Fix index collision in Eq. (3.14) (p. 9) — index i was reused for both the differentiated sample and the threshold-summation index. + RESOLVED (2026-09-09): renamed the threshold-summation index from i to r in eq:roll-aoc-gradient (method.tex, right after eq:roll-aoc), keeping i for the differentiated sample per the rest of the chapter's convention. Added a one-line parenthetical clarifying the two roles. Left eq:roll-aoc (3.13) itself untouched — it has no outer per-sample index to collide with, so its use of i as the threshold-sum index there is fine in isolation. Trivial fix, rebuilt thesis clean. +** DONE Handle degenerate batches and regularity assumptions explicitly (Section 3.4.2 area, pp. 19–20). Bundled 5 sub-issues; verified each against real code before deciding scope with user. + RESOLVED (2026-09-09). Verified findings: + - Constant/singleton class scores: tested directly — does NOT crash. KDEpy's own silvermans_rule internally detects "too many identical values" and substitutes bw=1.0 (with a warning) before our max_v=10000 cap even applies. Doesn't happen with real continuous model outputs; user confirmed treat as non-issue, just document the fallback. + - Single-class batch (one class entirely empty): tested directly — DOES crash (ValueError from KDEpy, "Data must be of length > 0"). User: important point, doesn't happen in practice because every experiment uses a balanced batch sampler (is_balanced=True), and this requirement isn't ROLL-specific — ANY ROC-based loss needs both classes present since TPR/FPR are each conditional on one class alone. + - Bandwidth cap (v<=10000, i.e. h-floor 0.0001): breaks the exact scale-equivariance hypothesis of prop:normalization-invariance once triggered. User: last-resort-only, rare, and even rarer after this session's other numerical fixes (log-space grad, certified bisection) — footnote, not a big deal. + - Gaussian zero-variance (sigma_k=0): analogous issue, unsupported currently. User: edge case footnote, doesn't happen under normal circumstances. + - General regularity (nonzero density at the inverted quantile): user wanted this included as the umbrella statement tying 1+3 together. + User's final scope decision: state #2 (both classes present) prominently as an important standing assumption tied to the balanced sampler; discuss #3 and #5 together using #1 (constant scores) as the motivating example, as a footnote-weight aside not a big fix; add #4 as a separate short footnote. Implemented exactly that: + - New "Regularity assumptions" paragraph at the end of sec:roll-formulation (method.tex, right before sec:roll-aoc) stating the both-classes-present assumption plainly, explaining it's inherent to any ROC-based objective, and noting it's enforced via the balanced sampler rather than in the loss math. + - sec:kde-bandwidth (method.tex): corrected "Silverman's rule... always produces a valid bandwidth" (not quite true without the library's internal override) to explain the constant-scores case and KDEpy's bw=1.0 fallback, with a footnote on the v<=10000 cap (ties to prop:normalization-invariance, notes it's essentially never reached now) and a closing sentence generalizing to the nonzero-density-at-quantile assumption. + - appendix_gauss_grad.tex: footnote on sigma_k=0 attached to the first mention of sigma_k^2 in the lead sentence (not bare after the equation — footnotes need inline text to attach to). + - Rebuilt thesis clean, no undefined refs in final pass. +** DONE Justify or flag as heuristic: applying ISJ (Gaussian KDE) bandwidth to the logistic kernel scale parameter. + RESOLVED (2026-09-09): derived the actual AMISE-optimal Gaussian-to-logistic-kernel conversion factor for context (h*_logistic/h*_gaussian ≈ 0.559, using sigma_K^2=pi^2/3, R(K)=1/6 for logistic vs sigma_K^2=1, R(K)=1/(2*sqrt(pi)) for Gaussian, both verified numerically to full precision) — i.e. ISJ's raw output is ~79% too large relative to what would be logistic-kernel-optimal, if the theory is taken at face value. + - Chose option B (don't apply the correction): unlike every other item this session, actually applying this fix would CHANGE the bandwidth used in every training run — a real algorithmic change requiring rerunning all reported experiments, not a no-op/writeup fix. User's call: leave the code as-is, be honest in the writeup that it's a heuristic. + - method.tex sec:kde-bandwidth: added 2 sentences right after introducing ISJ — states plainly that ISJ's optimality is proven for Gaussian kernels, the logistic kernel is used instead, this is "a plug-in heuristic approximation chosen for its ready availability in the existing implementation," with a pointer to future work. + - discussion_and_conclusions.tex sec:conclusions:future: added new "Kernel-appropriate bandwidth selection" paragraph (distinct from the existing "Adaptive bandwidth estimation" paragraph, which covers per-sample/local adaptivity — a different concern from the Gaussian-vs-logistic kernel-convention mismatch), stating deriving/validating a logistic-kernel-appropriate bandwidth rule is left to future work. + - Did NOT include the derived 0.559 factor itself in the thesis text (kept out per not wanting to imply a correction that wasn't applied); it's preserved here in org-roam if wanted later. + - Rebuilt thesis clean, confirmed the cross-chapter reference (method.tex -> discussion_and_conclusions.tex) resolves. +** DONE Fix "small absolute score scale → undersmoothing" explanation (Section 3.4.2.1, pp. 19–20). + RESOLVED (2026-09-09): verified the audit's claim directly — under the homogeneous (uncapped Silverman) bandwidth rule, raw gradient magnitude scales as EXACTLY 1/a under s->as rescaling (confirmed numerically: grad*a constant to full precision across a=1e1 down to a=1e-6). So the thesis's claim ("small sigma_hat -> small h -> low gradients") is actually backwards: shrinking absolute scale GROWS the raw gradient, not shrinks it, under exact scale equivariance. + - Considered but did not pursue further: two more precise candidate mechanisms (poor class separation early in training; ISJ's non-scale-equivariance already documented in appendix:score-normalization). User chose the simpler of the two to cite: poor class separation, framed as "a plausible explanation," not proven. + - sec:kde-bandwidth-scaling (method.tex): replaced the disproven causal claim with an empirical statement (scheduler helps convergence, observed in practice) + the class-separation explanation, explicitly hedged as plausible rather than asserted as fact. Kept the existing "solution: bandwidth scheduler" paragraph as-is (already empirical, not overclaiming). + - Rebuilt thesis clean. +** DONE Qualify exact-FPR-target assumption (Section 3.1, p. 2) — assumes every α∈(0,1) is attained by a deterministic threshold, which fails for atomic/empirical score distributions. + RESOLVED (2026-09-09): user wanted the simple version — state the continuity/strict-monotonicity requirement directly in the FPR^-1 definition (method.tex:34), plus a footnote pointing forward to the instantiations used in this thesis (KDE sec:roll-kde, Gaussian appendix:gauss-grad, Beta appendix:beta-roll) confirming each has a continuous strictly-increasing CDF by construction, so the assumption holds automatically for what ROLL actually optimizes over. Did not pursue the heavier options (generalized ROC convention, randomized thresholding) — not needed given the fitted distributions already satisfy the requirement. Rebuilt clean, confirmed the three forward references resolve. +** DONE Fix indicator-derivative-at-jump description (p. 4) — the derivative of an indicator at its jump is undefined, not "an ordinary derivative equal to infinity." Also distinguish finite-sample indicator objectives from population rates (which can be differentiable under distributional assumptions). + RESOLVED (2026-09-09): method.tex sec:differentiability-problem (~line 114). Replaced "the derivative of TPR/FPR w.r.t. f_theta(x_i) becomes infinite" with a precise one-sided-limit account: approaching the jump from below leaves the indicator at 0 while approaching from above sends the difference quotient to +infinity, so the two one-sided derivatives disagree and the derivative at the jump is undefined — not a two-sided limit that converges to infinity (unlike, e.g., a vertical tangent on a continuous function). + - Added a new paragraph distinguishing the finite-sample indicator objective from the population rate: TPR/FPR as written in this section are indicator averages over the finite sample D, where each summand 1[f_theta(x_i) > tau] is a step function of that one sample's score — this per-sample step is what's non-differentiable. The population quantities TPR(tau)=1-F_1(tau), FPR(tau)=1-F_0(tau) (F_1, F_0 the underlying class-conditional score distributions, left unhatted to distinguish from ROLL's later fitted hat-F_1/hat-F_0) are differentiable in tau whenever F_1, F_0 have continuous densities, with d(TPR)/d(tau) = -f_1(tau). So non-differentiability lives in the discrete per-sample indicators used to estimate the rate, not in the rate itself under a smooth distributional model — this is exactly the gap ROLL's score-distribution-fitting approach (sec:score-distribution-fitting, immediately following) exploits. + - Writeup-only fix, no code change. Rebuilt thesis clean (nix build exit 0). +** DONE Qualify score/label reversal transformation (pp. 6–7, 14) — using −s, complemented labels, and target 1−α is correct only for reflection-equivariant fits (Gaussian, symmetric KDE); it is not automatic for every distribution family, and literal negation leaves the Beta score domain — needs a support-preserving transformation + compatible fitting rule for Beta. + RESOLVED (2026-09-09): user chose option (a), keep the negation-based derivation as the general-case default, and add a footnote scoping it plus giving the Beta-specific fix — no rederivation of the main text needed. + - method.tex: added a footnote right after the existing negation/label-swap sentence (end of the paragraph introducing eq:roll-tpr-at-fpr / eq:roll-fpr-at-tpr, ~line 220) stating the negation trick requires the fitted family to be closed under score negation (true for Gaussian, symmetric KDE) and is invalid for Beta ROLL since its scores live in (0,1) and negation leaves that support. + - Beta fix (user's proposal, verified before writing up): replace $f_\theta(\mathbf{x}) \to -f_\theta(\mathbf{x})$ with $f_\theta(\mathbf{x}) \to 1-f_\theta(\mathbf{x})$ — stays in (0,1), and is already computed in the Beta forward pass as the complementary score $t^{(k)}=1-s^{(k)}$ (appendix_beta_roll.tex eq:beta-denom/eq:beta-params). + - Verified with sympy/numpy before writing anything (never trusted mentally, per usual convention): (1) numerically confirmed $X\sim\text{Beta}(a,b) \Rightarrow 1-X\sim\text{Beta}(b,a)$ exactly (sympy's symbolic beta() didn't auto-cancel $B(a,b)$ vs $B(b,a)$, so checked numerically instead — matched to 1e-9 across 5 random parameter draws); (2) confirmed the log-moment estimator $D_k$ (eq:beta-denom) is exactly symmetric under swapping which vector is called $s^{(k)}$ vs $t^{(k)}$, so applying the estimator to $1-s^{(k)}$ exactly swaps $\hat a_k \leftrightarrow \hat b_k$ (numerically exact, not approximate); (3) end-to-end check that the same "target rate complemented to $1-\alpha$" recovery claim used for the Gaussian/negation case holds identically for the Beta/$1-s$ case — built both a Gaussian direct-vs-transform check (sanity check of the *existing* thesis claim, confirmed exact match) and a Beta direct-vs-transform check using scipy.stats.beta (also confirmed exact match to float precision) using the swapped $(\hat b,\hat a)$ params as the "new" class parameters. + - Rebuilt thesis clean (nix build exit 0), footnote renders correctly with the appendix_beta_roll.tex cross-references (\Cref{appendix:beta-roll}, eq:beta-params) resolving. +** TODO Remove/soften claim that gradient balance implies noise robustness or population FPR control (general, throughout Ch. 3 prose) — these require additional statistical assumptions/arguments or appropriately scoped empirical evidence; should not be presented as direct consequences of the audited identities. + * Final pass * TODO Build full PDF and proofread end-to-end @@ -72,3 +164,6 @@ - [x] Ch. 2 restructured: ROC/AUC → Noisy Labels → KDE → Imbalanced → NP - [x] Ch. 3 restructured: AUC as primary objective, ROLL-AOC promoted - [x] Noise results table added to Ch. 4 +* DONE Get rid of TPR results +* TODO re-integrate results for banking +* TODO integrate food101n results when done diff --git a/thesiswork-historyrecent.org b/thesiswork-historyrecent.org index d1dcd30..bea3cf7 100644 --- a/thesiswork-historyrecent.org +++ b/thesiswork-historyrecent.org @@ -5,6 +5,36 @@ #+title: thesis/work-history/recent #+filetags: :project: :work-history: +* 2026-08-31 Mon + +- Added roll_aoc_roc.py figure (ROC curve with green AUC / red AOC shading, 10 sampled threshold points) to illustrate ROLL-AOC; inserted \begin{figure} in method.tex after the ROLL-AOC gradient equation +- Added Food-101N results to thesis: new bib entry (lee2018cleannet), dataset description item, updated episode-count note, narrative analysis, and table row in tab:results:large (roll-aoc leads at 0.867, libauc second at 0.857) +- Full thesis re-read via subagent: identified and fixed ~20 issues across all chapters +- Fixed all \textsc{Roll} → \textsc{ROLL} across experiments.tex, discussion.tex, appendix_keel_noise.tex +- Fixed ChestX-ray14 → ChestXray14 everywhere for consistent naming +- Fixed symmetric vs asymmetric noise contradiction: table captions and discussion.tex now all say "symmetric label noise" +- Fixed BCE-W margin number in CIFAR-10N results (0.047 → 0.083) +- Removed broken \Cref{fig:density-fit-comparison} (figure never inserted) +- Fixed duplicate "interchangeable at the implementation level" clause in method.tex +- Fixed typos: "reffer", "w.r.t the", "At it's core", "round about", "based off of" (×3), "32x32" → $32\times32$, duplicate \cite, "throwing distance" → academic phrasing +- Removed related_work TODO comment (AUC/ROC formalization now present) +- Fixed incomplete sentence "by fitting." in method.tex problem formulation +- Ran parallel 4-agent bibliography verification across all 36 entries +- Fixes from bib verification: feng2020can author Fengmei → Fengmao Lv; oakdenrayner2020exploring author Lauren → Luke; removed jigsaw-puzzle template leftovers (demaine2007jigsaw, altman1989solving) +- Fixed broken \Cref{rem:aoc-coverage} rendering as "?? 3.3.2.0.1": added missing \newtheorem{remark}[theorem]{Remark} and \Crefname{remark}{Remark}{Remarks} to main.tex + +* 2026-08-29 Sat + +- Investigated label noise claims for large tabular datasets (Bank Marketing, Credit Card Fraud, Home Credit) via web search; confirmed Credit Card Fraud has structural "hidden fraud" (class-conditional false negatives), found no evidence of inherent label noise in Bank Marketing or Home Credit +- Updated Credit Card Fraud dataset description to note undetected fraud as structural source of class-conditional false negatives; removed blanket "All labels clean" claim from large tabular paragraph +- Researched NIH ChestX-ray14 ~20% noise figure — found it unsourced; Wang 2017 claims >90% accuracy, Oakden-Rayner 2019 audit found PPV 10–30pp lower; rewrote description to cite both with a defensible formulation instead of asserting a specific percentage +- Added bib entries: wang2017chestxray14 (CVPR 2017), oakdenrayner2020exploring (Academic Radiology 2020) +- Replaced all \cite{TODO} in experiments.tex with real citations; added 8 new bib entries: alcalafdez2011keel, wei2022cifar10n, moro2014bank, dalpozzolo2015calibrating, tittelfitz2023fpr, homecredit2018kaggle, cohen2022torchxrayvision, lin2017focal +- Merged standalone "Label-Noise Robustness on KEEL" section into KEEL Benchmark subsection; integrated protocol description inline; removed duplicate content +- Replaced aggregate KEEL noise table (methods × 4 noise levels) with three full per-dataset tables (5%, 10%, 20%), computed from raw results-final CSVs; updated discussion to reference all three and added 20%-noise win-count observation (Roll-AoC leads 6/12 datasets) +- Fixed stale back-reference in ChestX-ray14 results (sec:experiments:noise → sec:results:keel); updated stale ~20% noise mentions in results body and table caption +- Added TODOs: audit bold/emphasis in all result tables; review KEEL noise tables for main body vs appendix; remove Roll-TPR from large datasets table; reword Bank Marketing after rerun (suspected bug in current results) + * 2026-08-28 Thu - Pivoted Ch. 3 (method.tex) from TPR@FPR to AUC as primary objective: rewrote Problem Formulation with AUC integral as eq:objective-auc, fixed TPR/FPR notation to functions of τ, demoted single-threshold objectives to special case @@ -18,7 +48,6 @@ - Renamed Ch. 4 "Label-Noise Poisoning Experiment" → "Label-Noise Robustness on KEEL"; wrote protocol and results prose; added tab:results:keel-noise with 4 noise levels × 8 methods - Updated Summary section comments in experiments.tex with actual findings to synthesize - Created thesis/experiments org-roam subnode with full data source map, key numbers table, and section status - - Restructured Ch. 2 (related_work.tex): new section order — ROC/AUC (2.1) → Noisy Labels (2.2) → KDE (2.3) → Imbalanced Classification (2.4) → NP (2.5) — to foreground AUC + noise as thesis core - Dropped section 2.3 (Partial AUC and TPR@FPR) entirely; TPR@FPR objective absorbed into Imbalanced section, NP moved up - Moved NP section from end of chapter to position 2.2, then later to 2.5 after deciding AUC+noise should open the chapter diff --git a/tiny-trmn-tts.org b/tiny-trmn-tts.org new file mode 100644 index 0000000..c3fec9d --- /dev/null +++ b/tiny-trmn-tts.org @@ -0,0 +1,85 @@ +** Architecture +Simlish-style procedural speech synthesizer for trmn. Output: ONNX model loaded by JS in trmn. +Pipeline: text → prosody (JS) → phoneme sequence → ONNX flow matching → mel → Griffin-Lim → audio. +Deterministic: seeded per character via trmn's masterSeed PRNG. +See [[just-talking/simlish-project]] for full scoping notes and research version design. + +Two-stage inference: +1. Acoustic model (flow matching, ONNX): (phone_ids, durations, pitch, energy, emotion) → mel[80×T] +2. Vocoder (Griffin-Lim, pure JS): mel → waveform. No weights. Swappable later. + +** Decided: Acoustic Model +Adapted Matcha-TTS (ICASSP 2024, arXiv:2309.03199). ConvNeXt-based OT-CFM flow decoder. +- Phoneme encoder: embedding table (N_PHONEMES=16) + 2 Conv1d layers → hidden[128, P] +- Duration predictor: DROPPED — durations come from prosody.mjs at inference time +- Length regulator: kept — repeat_interleave phoneme hidden states by duration counts +- Flow decoder: 5 ConvNeXt blocks, 128 channels. FiLM emotion conditioning per block. +- Pitch + energy: linear-embedded per frame, concatenated with phoneme hidden before decoder +- ~2–5M params. Trained with OT-CFM (MSE on predicted velocity x1-x0). + +** Decided: Phoneme Inventory (16 total) +0= 1= +2=p 3=t 4=k 5=b 6=d 7=m 8=n 9=s 10=l +11=a 12=e 13=i 14=o 15=u +No / — not needed for non-autoregressive model. + +** Decided: Training Data — LUT Approach +English words → letter LUT → mapped text → Piper TTS → audio → mel extraction. +LUT: letters in inventory map to themselves; others map to nearest class (r→l, f→s, g→d, etc.) +Phone IDs come from re-applying LUT at extract time — no forced alignment. +Duration heuristic: vowels get 1.5× frames, silences 0.4×, consonants 1.0×, proportional to total T. +Length-scale varied 0.8–1.3× per sample for duration variety. + +Piper voice: en_US-lessac-medium (fetched via Nix, both .onnx and .onnx.json pinned by sha256). +~10–50h synthetic audio expected to suffice. + +** Decided: Nix Targets +Separate derivations with strict dep isolation (mirrors trmn pattern): + packages.onnx-model — ONNX artifact trmn pulls in; placeholder until checkpoint pinned + packages.piper-voice — fetchurl'd lessac-medium model + config, cached independently + apps.gen-data — Piper + librosa + numpy; no torch, no JS + apps.train — torchWithCuda + onnx; no Piper, no MFA, no JS + apps.test — nodejs only; no Python + devShells.data — same deps as gen-data, PIPER_MODEL/PIPER_CONFIG set via env + devShells.training — torchWithCuda + onnx; for LAN laptop (RTX 3050) + devShells.default — nixfmt + nodejs; lightweight + +** Decided: Training Environment +LAN laptop: aner@192.168.1.200, no password. WSL2 on Windows. RTX 3050 6GB VRAM, CUDA 13.2. +CUDA accessible via /dev/dxg (WSL2 WDDM). nvidia-smi at /usr/lib/wsl/lib/nvidia-smi. +Disk: 899GB available (plenty). RAM: 7.6GB WSL2 limit — may need to raise via C:\Users\\.wslconfig. +PyTorch: use torchWithCuda (not torch) in Nix — default torch is CPU-only. + +** Key Files +- flake.nix — all Nix targets; Piper voice fetched here +- data/gen_corpus.py — English words → LUT → .txt (Piper input) + .ids (phone IDs) +- data/synthesize.py — Piper TTS → .wav per sentence (length-scale varied) +- data/extract.py — audio + .ids → mel/pitch/energy/durs → .npz dataset +- model/acoustic.py — AcousticModel: PhonemeEncoder + LengthRegulator + FlowDecoder +- model/train.py — OT-CFM training loop +- model/export.py — PyTorch checkpoint → ONNX FP32 → INT8 +- inference/prosody.mjs — text + emotion → phone_ids/durations/pitch/energy (pure JS) +- inference/griffin_lim.mjs — mel → waveform (pure JS, replaceable vocoder) +- test/infer.test.mjs — ONNX load + forward pass + shape checks (node:test) +- test/prosody.test.mjs — prosody extraction unit tests: determinism, emotion, shapes + +** Conventions +- No runtime server calls at inference +- Same seed must always produce same voice (deterministic) +- Vocoder is a swappable box: mel[80×T] in, waveform out +- Nix: no npm/pip lock files — Nix is the build orchestrator +- Only git-tracked files included in builds (src = ./. in derivations) +- node:test for JS tests (no framework, follows trmn no-bundler pattern) +- Piper CLI: echo "text" | piper -m model.onnx -c config.json -f out.wav + +** Gotchas +- trmn is static Nix-packaged — ONNX artifact must be added to trmn's flake.nix (not this one) +- SVG mouth sprites in trmn are ready to animate but animation API not yet defined +- ODE integration loop (Euler, N steps) runs in JS — only single-step velocity exported to ONNX +- extract.py reads .ids files from corpus/ subdir, audio from audio/ subdir — path is hardcoded relative +- dataPython defined twice in flake.nix (packages and apps sections) — intentional, avoids cross-section ref +- torchWithCuda (not torch) required in Nix for GPU training — default torch is CPU-only +- WSL2: nvidia-smi not on PATH, use /usr/lib/wsl/lib/nvidia-smi; CUDA works via /dev/dxg + +** Subnodes +(none yet)