backup: 2026-10-05 09:48
This commit is contained in:
@@ -25,7 +25,9 @@ Eurovision Vote: Django app sourced from external flake =github:anerisgreat/euro
|
||||
|
||||
EmulatorJS: web game emulator at =games.zakobar.com= with an offline PWA + per-user save sync. See [[id:7148e7ef-7741-42ad-9d3d-4025001e1018][homey/emulatorjs]].
|
||||
|
||||
Backup: Restic daily at 03:00 to S3 (Backblaze B2, bucket =zakobar-home-backup=). Pre-hook: Nextcloud maintenance mode on + pg_dump. Post-hook: maintenance mode off. Manual offload: =restic copy= to local disk. NAR content and media excluded.
|
||||
Big Jar of Pickles: LangGraph swarm-pipeline app sourced from external flake =big-jar-of-pickles= (/home/aner/projects/apps/big-jar-of-pickles). Homey-side infra rollout across 4 phases (plan: /home/aner/.claude/plans/view-the-big-jar-dynamic-axolotl.md; spec: [[homey/big-jar-of-pickles]]). Phase 1 (native Postgres+pgvector + Redis) deployed and verified 2026-09-11. Phase 2 (Langfuse v4, containerized) written 2026-09-11 but not yet deployed — ships disabled pending sops secrets. Phases 3–4 (Aegra, app wrapper) not started.
|
||||
|
||||
Backup: Restic daily at 03:00 to S3 (Backblaze B2, bucket =zakobar-home-backup=). Pre-hook: Nextcloud maintenance mode on + pg_dump (Nextcloud, native shared Postgres, and Langfuse's own Postgres all dumped separately). Post-hook: maintenance mode off. Manual offload: =restic copy= to local disk. NAR content and media excluded.
|
||||
|
||||
Reliability hardening in =hosts/pi-main/default.nix=: hardware watchdog (bcm2835_wdt), WiFi power save disabled, network watchdog timer, zramSwap zstd 25%, Nix build-dir on external HD.
|
||||
|
||||
@@ -34,6 +36,8 @@ Bootstrap: =pi-main-bootstrap= config builds an SD image (=sd-image-aarch64.nix=
|
||||
** Subnodes
|
||||
|
||||
- [[id:7148e7ef-7741-42ad-9d3d-4025001e1018][homey/emulatorjs]] — EmulatorJS game emulator: offline PWA design, ROM management, save sync, PS1/large-ROM notes.
|
||||
- [[id:fd781dd9-e52c-4e3f-82ad-b27783b33177][homey/ntfy]] — Ntfy push notifications: CLI send commands, token management, pass integration.
|
||||
- [[id:dc2ddfad-ada9-4f0d-ae35-806e8a1a2def][homey/big-jar-of-pickles]] — Postgres+pgvector/Redis/Langfuse/Aegra services for the big-jar-of-pickles app; rollout status, wiring pattern, gotchas.
|
||||
|
||||
** Deployment
|
||||
|
||||
@@ -62,13 +66,15 @@ Both =--target-host= and =--build-host= point to the Pi — the build happens ON
|
||||
|
||||
Nix evaluates git-tracked files from the flake. New/modified files must be at least =git add=-ed (staged) before deploying, or they will be invisible to Nix. Untracked files are silently ignored.
|
||||
|
||||
To validate a module that ships with =enable = false= by default (its =config = lib.mkIf cfg.enable {...}= block stays lazy/unevaluated by plain =nix flake check=), temporarily flip the flag on in =hosts/pi-main/default.nix=, rerun =nix flake check= to force full evaluation, then revert before committing/deploying.
|
||||
|
||||
** Conventions
|
||||
|
||||
=homeyConfig= specialArgs (passed to every module): =domain=, =organization=, =timezone=. Never hardcode domain strings.
|
||||
|
||||
=users.mutableUsers = false= — all user config must be declared in Nix.
|
||||
|
||||
Secret injection pattern: =LoadCredential= stages sops-decrypted file into =$CREDENTIALS_DIRECTORY= before Exec*; shell script reads and exports. Ephemeral env files written to =/run/= and cleaned up in =ExecStopPost= / =postStop=.
|
||||
Secret injection pattern: =LoadCredential= stages sops-decrypted file into =$CREDENTIALS_DIRECTORY= before Exec*; shell script reads and exports. Ephemeral env files written to =/run/= and cleaned up in =ExecStopPost= / =postStop=. A combined connection-string secret (e.g. a DB =DATABASE_URL= with the password embedded) can't use the usual "plain Nix env var + separate secret" split — the whole string has to be assembled inside the ExecStartPre script at runtime.
|
||||
|
||||
Authelia =accessControlRules=: sorted by =priority= at build time (lower = first). Authelia stops at first match, so more-specific rules must have lower priority. Ranges: 0=bypass, 10–19=blanket bypass, 20–49=admin two_factor+deny pairs, 50–64=one_factor open, 65–79=per-path (resources + subject combos).
|
||||
|
||||
@@ -76,13 +82,17 @@ Authelia =accessControlRules=: sorted by =priority= at build time (lower = first
|
||||
|
||||
Attic token generation: run =atticadm make-token= inside the container. Tokens are stateless; losing one means just regenerating it with the same command.
|
||||
|
||||
DynamicUser services (Eurovision Vote): secrets must be mode =0444= (not =0400=) because DynamicUser gets a random UID that cannot be pre-assigned as owner.
|
||||
DynamicUser services (Eurovision Vote): secrets must be mode =0444= (not =0400=) because DynamicUser gets a random UID that cannot be pre-assigned as owner. For a bind-mounted config file read by a podman container's own internal (unknown) UID, mode =0644= works the same way and doesn't require knowing that UID.
|
||||
|
||||
Caddy Cloudflare plugin secrets: uses =LoadCredential= + =ExecStart= override (clears list with empty string first, then sets the real start command) to export =CLOUDFLARE_API_TOKEN= before exec-ing caddy.
|
||||
|
||||
Stirling-PDF login disable: =DOCKER_ENABLE_SECURITY=false= is build-time only. To disable the runtime login page, also set =SECURITY_ENABLELOGIN=false=.
|
||||
Container UIDs for bind-mounted volumes should be verified (=skopeo inspect --config docker://<image>=, checking the =User= field), not guessed — getting it wrong causes an activation-time failure, not a build-time one. See [[homey/big-jar-of-pickles]] for a worked example across 4 different images.
|
||||
|
||||
Fetching external static assets via =fetchFromGitHub= (used by EmulatorJS, same pattern works for any GitHub-hosted release): get hash with =nix-prefetch-url --unpack <tarball-url>=, then convert to SRI with =nix hash convert --hash-algo sha256 --to sri <base32>=. To upgrade: bump =rev=, re-prefetch, update =hash=. Full upgrade instructions are in a comment inside =modules/services/emulatorjs.nix=.
|
||||
Direct-LAN exposure without Caddy (e.g. Home Assistant, Langfuse's Phase-2 LAN-only rollout): bind the container port to all interfaces (omit the =127.0.0.1:= prefix) and add the port to =networking.firewall.allowedTCPPorts=.
|
||||
|
||||
Fetching external static assets via =fetchFromGitHub= (used by EmulatorJS, same pattern works for any GitHub-hosted release): get hash with =nix-prefetch-url --unpack <tarball-url>=, then convert to SRI with =nix hash convert --hash-algo sha256 --to sri <base32>=. Full upgrade instructions are in a comment inside =modules/services/emulatorjs.nix=.
|
||||
|
||||
Shared platform infra (Postgres, Redis — not single-purpose UI services) lives at the top level next to =storage.nix=/=backup.nix=/=monitoring.nix=, not under =modules/services/=. See =modules/postgres.nix=/=modules/redis.nix=.
|
||||
|
||||
** Gotchas
|
||||
|
||||
@@ -100,9 +110,13 @@ Attic: writing ephemeral TOML config via =ExecStartPre= shell script to =/run/at
|
||||
|
||||
First deployment of a new container pulls the image during =nixos-rebuild switch=, which can block the activation for several minutes (e.g. stirling-pdf ~2 GB image took ~6 min on Pi). This is expected — not a hang.
|
||||
|
||||
SSH lockout recovery: =/etc/ssh/authorized_keys.d/admin= is OVERWRITTEN on every boot by NixOS activation (from the current system generation's Nix store). Manual edits to that file do not survive a reboot. To recover without a valid key: pull SD card, mount root partition (label =NIXOS_SD=), write the new public key to =/home/admin/.ssh/authorized_keys= (permissions: =.ssh/= 700, =authorized_keys= 600, owner UID 1000). This file is NOT managed by activation and survives reboots. Remove it after deploying the updated NixOS config.
|
||||
SSH lockout recovery: =/etc/ssh/authorized_keys.d/admin= is OVERWRITTEN on every boot by NixOS activation. Manual edits don't survive a reboot. To recover: pull SD card, mount root partition (label =NIXOS_SD=), write the new public key to =/home/admin/.ssh/authorized_keys= (700/600, owner UID 1000) — not managed by activation, survives reboots.
|
||||
|
||||
Admin SSH key is GPG-backed via YubiKey. Current key: =~/.ssh/gpg-as-ssh.pub= (=openpgp:0x13E0C25D=, =cardno:29_109_265=). SSH client config uses =IdentityAgent ${XDG_RUNTIME_DIR}/gnupg/S.gpg-agent.ssh=. Old broken YubiKey was =cardno:24_758_470= — its private =[A]= subkey material was never exported (flash drive backup contains stubs only).
|
||||
Admin SSH key is GPG-backed via YubiKey. Current key: =~/.ssh/gpg-as-ssh.pub= (=openpgp:0x13E0C25D=, =cardno:29_109_265=). SSH client config uses =IdentityAgent ${XDG_RUNTIME_DIR}/gnupg/S.gpg-agent.ssh=.
|
||||
|
||||
Home Assistant first-boot: the =ExecStartPre= script writes a minimal =configuration.yaml= only if none exists, includes =http.trusted_proxies= and the =command_line= auth provider. After onboarding, create one local break-glass admin account; daily logins use LDAP.
|
||||
|
||||
Native =services.postgresql=/=services.redis=, and containerized multi-service stacks (Langfuse): full gotcha list in [[homey/big-jar-of-pickles]] — custom Postgres =dataDir= needs the exact leaf pre-created before the unit's mount namespace sets up; default pg_hba already grants DynamicUser peer auth; named redis servers default to socket-only; a container's password can't always go through an env var (redis:7 has none — needs a generated config file instead).
|
||||
|
||||
** Key Files
|
||||
|
||||
@@ -113,13 +127,16 @@ Admin SSH key is GPG-backed via YubiKey. Current key: =~/.ssh/gpg-as-ssh.pub= (=
|
||||
- =modules/services/authelia.nix= — access control rule rendering, =accessControlRules= option (unconditional)
|
||||
- =modules/services/uptime-kuma.nix= — =homey.monitoring.monitors= option (unconditional), sync script
|
||||
- =modules/services/attic.nix= — Nix binary cache, JWT token config, netrc injection for Nix daemon
|
||||
- =modules/services/stirling-pdf.nix= — PDF tools (merge/split/OCR/etc), port 8084, auth disabled via =SECURITY_ENABLELOGIN=false=
|
||||
- =modules/monitoring.nix= — Prometheus + Grafana, proxy auth wiring, Node Exporter Full dashboard
|
||||
- =modules/common.nix= — Nix settings, podman network creation, sops global config
|
||||
- =modules/storage.nix= — external HD mount, =extraDirs= option
|
||||
- =modules/backup.nix= — Restic, pre/post hooks, =extraPaths= option
|
||||
- =scripts/ssh-with-gpg-key.sh= — emergency SSH via GPG key file; sets up ephemeral GNUPGHOME and uses =[A]= subkey
|
||||
- =modules/services/emulatorjs.nix= + =emulatorjs-server.py= — EmulatorJS emulator; details in [[id:7148e7ef-7741-42ad-9d3d-4025001e1018][homey/emulatorjs]]
|
||||
- =modules/postgres.nix= / =modules/redis.nix= — native shared Postgres+pgvector and Redis, [[homey/big-jar-of-pickles]] Phase 1 (deployed)
|
||||
- =modules/services/langfuse.nix= — containerized Langfuse v4 stack, [[homey/big-jar-of-pickles]] Phase 2 (written, not deployed)
|
||||
- =scripts/ssh-with-gpg-key.sh= — emergency SSH via GPG key file
|
||||
- =modules/services/emulatorjs.nix= + =emulatorjs-server.py= — details in [[id:7148e7ef-7741-42ad-9d3d-4025001e1018][homey/emulatorjs]]
|
||||
- =modules/services/home-assistant.nix= + =modules/services/ha-ldap-auth.py= — HA module; LDAP auth script bundled as Nix derivation
|
||||
- =modules/services/eurovote.nix= — reference pattern for wiring an external app flake's =nixosModules.default= in; template for [[id:dc2ddfad-ada9-4f0d-ae35-806e8a1a2def][homey/big-jar-of-pickles]] Phase 4
|
||||
|
||||
** TODOs
|
||||
|
||||
@@ -141,4 +158,6 @@ maxChunkSize = 5_000_000 # 5 MiB per chunk
|
||||
|
||||
[[id:7148e7ef-7741-42ad-9d3d-4025001e1018][homey/emulatorjs]]
|
||||
|
||||
[[id:2d455310-42a5-4ff5-be38-9af2b009aef7][homey/work-history/recent]]
|
||||
[[id:2d455310-42a5-4ff5-be38-9af2b009aef7][homey/work-history/recent]]
|
||||
|
||||
[[id:dc2ddfad-ada9-4f0d-ae35-806e8a1a2def][homey/big-jar-of-pickles]]
|
||||
|
||||
Reference in New Issue
Block a user