homey/big-jar-of-pickles

homey/big-jar-of-pickles

homey/big-jar-of-pickles

Spec for the homey-side services that big-jar-of-pickles (a LangGraph swarm-pipeline app, home/aner/projects/apps/big-jar-of-pickles) needs. Full architecture reasoning lives in big-jar-of-pickles/infra. Rollout plan: /home/aner.claude/plans/view-the-big-jar-dynamic-axolotl.md (4 phases).

REVERTED 2026-09-15 — abandoned, not enough disk space on the Pi

The whole homey-side stack below (Postgres+pgvector, Redis, Langfuse module) was reverted out of the homey repo on 2026-09-15: modules/postgres.nix, modules/redis.nix, modules/services/langfuse.nix deleted, their imports/enables in flake.nix and hosts/pi-main/default.nix removed, the two new pg_dump blocks in modules/backup.nix reverted, and the langfuse secrets block removed from secrets/secrets.yaml via sops unset '["langfuse"]'. Reason given: the Pi 4's storage doesn't have room for this stack (Langfuse alone is Postgres+ClickHouse+Redis+MinIO). The big-jar-of-pickles app project itself (Phase 4, the LangGraph app) and its own infra notes are unaffected — only the homey-hosted services this page tracked are gone. If revisited, it'll need a different storage plan (e.g. external drive headroom check, or a smaller/hosted alternative to self-hosting ClickHouse) before re-attempting Phase 1.

The Phase 1/2 implementation notes below are kept for reference in case this is retried, but do not reflect the current state of the repo.

Services to add (historical — reverted, see above)

DONE Postgres + pgvector (Phase 1, deployed 2026-09-11)

modules/postgres.nix. Native services.postgresql. Two DBs: big_jar_of_pickles, aegra (reserved). Verified live: pgvector 0.8.0 installed.

DONE Redis (Phase 1, deployed 2026-09-11)

modules/redis.nix. Native services.redis.servers.big-jar-of-pickles, socket-only. No consumer yet.

Langfuse module written + secrets populated, NOT yet deployed (Phase 2, 2026-09-11)

modules/services/langfuse.nix. Ships with homey.langfuse.enable default false. All 10 sops secrets under langfuse/ now populated in secrets/secrets.yaml (random-generated via openssl rand=/uuid, written with a single =sops set '["langfuse"]' '<json>' call — one decrypt/re-encrypt cycle instead of ten, YubiKey PIN was already cached in gpg-agent so no interactive prompt was needed). Remaining before deploy: flip homey.langfuse.enable = true in hosts/pi-main/default.nix, deploy, then watch Grafana for ClickHouse's memory footprint before trusting it (Pi 4 8GB ceiling is the flagged risk) — deliberately left for explicit user go-ahead rather than done automatically, since it's the first ClickHouse deploy and genuinely resource-risky.

Re-verified the real topology against upstream's actual docker-compose.yml (github.com/langfuse/langfuse) instead of trusting the org-roam spec's Sept-2026 assumption — it had already changed: this is Langfuse v4 now (image tags :4, not v3), still web+worker+Postgres17+ClickHouse25.12+Redis7+MinIO, all on the homey podman network. Own private containerized Postgres, per the "don't reach into the native shared Postgres from a container" decision in Phase 1.

Container UID handling (verified via skopeo inspect --config, not guessed, after Phase 1's dataDir mismatch cost a redeploy):

langfuse-postgres: no baked-in image USER, entrypoint execs to "postgres" without self-chowning → pre-chowned host dir at UID 999 (mirrors nextcloud-postgres).
langfuse-clickhouse: upstream's own compose forces user: "101:101" explicitly → mirrored with both the oci-container user override and matching chowned dirs.
langfuse-minio (cgr.dev/chainguard/minio): image bakes in User: "65532" (chainguard's standard nonroot UID) → just needs matching dir ownership, no override.
langfuse-redis: no persistent volume at all — sidesteps the UID question entirely; pure cache/queue state, same "disposable" classification as homey's own native Redis.

Redis auth gotcha: the official redis:7 image has no env-var password option — only --requirepass (would bake the secret into the Nix store) or a config file. Solved via a generated redis.conf from ExecStartPre (mode 644, deliberately world-readable — ephemeral /run file, sidesteps needing the container's internal UID), passed as redis-server's config-file argument via cmd.

DATABASE_URL (and other secrets embedded in combined connection-string env vars) can't go through the usual "plain Nix-level env var + separate secret in the env-file" split used elsewhere (nextcloud etc.) — the whole URL string, password included, has to be assembled inside the ExecStartPre script at runtime.

Exposure: LAN-only for now. Only web (3000) and MinIO's S3 API (9090) bound to all interfaces + opened in networking.firewall.allowedTCPPorts (mirrors the Home Assistant direct-LAN pattern). Everything else stays 127.0.0.1-only.

Deviated from the plan's digest-pinning decision: used version tags (:4, :25.12, :7, :17) instead of pkgs.dockerTools.pullImage-by-digest — upstream's own reference compose only pins by tag too. Documented as a deliberate deviation in the file header.

Backup scope deliberately narrow: only langfuse-postgres gets a pg_dump (project/user/API-key config, hard to regenerate) via a new block in modules/backup.nix. ClickHouse/MinIO data excluded — treated as disposable analytics output.

LANGFUSE_INIT_* env vars seed an initial org/project/user + API key pair on first boot so big-jar-of-pickles (Phase 4) has real LANGFUSE_PUBLIC_KEY=/=SECRET_KEY without a manual UI step. Assumed idempotent (not yet confirmed live).

Validated via a temporary homey.langfuse.enable = true flip in hosts/pi-main/default.nix + nix flake check (forces full evaluation of the mkIf cfg.enable block, which stays lazy/unchecked while disabled) — passed clean, then reverted before committing.

TODO Aegra (Phase 3 — built, disabled by default)

uv2nix build (pyproject-nix/uv2nix/pyproject-build-systems trio, same as big-jar-of-pickles' own flake). homey.aegra.enable default false.

TODO big-jar-of-pickles app wrapper (Phase 4 — last)

New flake input + modules/services/big-jar-of-pickles.nix, eurovote pattern. Gated on upstream nixosModules.default being stable. Will consume Langfuse's seeded LANGFUSE_PUBLIC_KEY=/=SECRET_KEY from the Phase 2 init.

Gotchas learned during Phase 1 implementation (native services.postgresql/redis)

services.postgresql with a custom dataDir sets ReadWritePaths = [ cfg.dataDir ] on the systemd unit, and systemd binds that exact path into the mount namespace before ExecStartPre (where initdb would normally create it) ever runs. Pre-create the exact schema-versioned leaf (e.g. postgres/16, not just postgres) via homey.storage.extraDirs; tmpfiles implicitly creates the missing parent.
No custom pg_hba=/=authentication lines needed for DynamicUser peer auth — the module's built-in default already includes local all all peer. A DynamicUser service running as User = "rolename" authenticates for free over the Unix socket.
A named services.redis.servers.<name> (name != "") already defaults to port = 0 (no TCP) and a per-name Unix socket path — no extra options needed for socket-only access.
modules/backup.nix's homey-backup-pre.service has side effects (Nextcloud maintenance-mode ON, writes /run/restic-homey-secrets.env) only cleaned up by restic-backups-homey's ExecStopPost. Never start homey-backup-pre.service standalone to test it — trigger restic-backups-homey.service instead, or manually undo (podman exec nextcloud php occ maintenance:mode --off + rm /run/restic-homey-secrets.env) if you do.

big-jar-of-pickles, big-jar-of-pickles/infra

7.8 KiB Raw Blame History

homey/big-jar-of-pickles