roam/big-jar-of-pickles.org

* big-jar-of-pickles
Swarm of weak/small local models ("Big Pickle," opencode's small model — a jar of them) wired with production-grade orchestration, attempting something none of them could do alone. Article-driven side project: does orchestration substitute for model capability, and where's the crossover point.

** Architecture
Scope narrowed 2026-09-13: standalone local CLI, no self-hosted services required for now. LangGraph library + local SQLite checkpointer (BJP_CHECKPOINT_PATH, default ./bjp_checkpoints.sqlite) + Guardrails AI + LLM-judge, packaged via uv2nix. Langfuse tracing is opt-in (only wired if LANGFUSE_PUBLIC_KEY is set) rather than required. The homey-backed stack (Postgres+pgvector, self-hosted Langfuse, Aegra) is deferred to later, once the pipeline itself is proven out — reasoning and prior state preserved in [[big-jar-of-pickles/infra]]. Reason for the pivot: self-hosted Langfuse v4 hard-requires ClickHouse, which the target Homey Pi 4 (8GB) likely can't spare RAM for; rather than fight that, cut the self-hosted dependency entirely for the MVP.

nixosModules.default (homey systemd wiring) was removed from flake.nix as part of this pivot — there's no service to deploy right now, just a CLI. If/when homey integration resumes, that module will need to be re-added (see [[big-jar-of-pickles/infra]] for the eurovote-pattern reference it followed).

Phase 1 task: Research -> Write -> Critique pipeline. Specialist roles (researcher, writer, critic) played by many cheap/weak model calls on a fixed topic, producing a short technical explainer, with LLM-judge scoring tracked across iterations as the article's data.

Phase 2 (maybe): same harness pointed at a quantified coding benchmark (HumanEval subset) for swarm vs. single-small-model vs. single-frontier-model comparison.

LLM config is provider-agnostic by design (2026-09-13): Settings/llm.py use generic `llm_api_key`/`llm_base_url`/`llm_model` fields (env vars `LLM_API_KEY`/`LLM_BASE_URL`/`LLM_MODEL`), not provider-specific names — llm.py is just a bare `OpenAI(api_key=..., base_url=...)` client, so swapping providers is an env-var change, not a code change. Currently pointed at **OpenCode Zen** (opencode.ai/docs/zen/) — confirmed official name (not "Zen Cloud", an earlier placeholder guess). Zen is opencode's own model marketplace, OpenAI-compatible endpoint at `https://opencode.ai/zen/v1`, offers opencode's "Big Pickle" model among others.

Secrets handling: no SOPS — this is a local-only CLI with no NixOS deployment yet, so the API key lives in a gitignored `.envrc` (direnv, `use flake` + exports), with `.envrc.example` committed as the fillable template. SOPS is the right tool for later, when/if homey/NixOS deployment resumes (matches the eurovote pattern already established in [[big-jar-of-pickles/infra]]) — introducing it now for a single-machine secret would be pure overhead.

Debugging (2026-09-15): `bjp --debug` runs the graph via `graph.stream(..., stream_mode="updates")` instead of `.invoke()`, writing each node's output as a JSON line to `BJP_DEBUG_DIR/<thread_id>.jsonl` (debug.py). Replay with `bjp --show-debug <thread_id>`. This only captures node-level state deltas (plan/research/draft/critique/score), not raw prompts — actual system/user messages sent to the LLM are still unlogged since llm.py's `complete()` uses the raw `openai.OpenAI` client. The Langfuse `CallbackHandler` wired in cli.py also only sees LangGraph chain/node boundaries for the same reason (it hooks LangChain callback events, but nodes never go through a LangChain-wrapped LLM) — swapping to `from langfuse.openai import OpenAI` in llm.py would be the real fix for prompt/completion-level tracing, not yet done.

** Deployment
None currently — runs as a local CLI (`nix run` / `uv run` / `bjp`). Inference is cloud (OpenCode Zen), so no local token-serving either way.

** Conventions
(none yet)

** Gotchas
- Guardrails integration (src/big_jar_of_pickles/guardrails_validation.py) uses the core Validator/register_validator API and is untested against a live run (no real credentials were available when it was scaffolded) — sanity check it actually validates before relying on it.
- Self-hosted Langfuse v4 requires ClickHouse (not just Postgres) — this is why observability was made opt-in rather than deploying self-hosted Langfuse on the resource-constrained Pi. See [[homey/big-jar-of-pickles]] for the ClickHouse memory-risk notes from the homey side.
- Nix flakes only see git-tracked files: `nix develop`/`nix build` silently fail to pick up newly created files until they're at least `git add`ed (staged is enough, doesn't need to be committed). Bit twice during dev — e.g. adding debug.py and running `nix develop` gave `ImportError: cannot import name 'debug'` until it was staged. `uv run` outside the nix devShell is a separate trap: it uses the ambient/ad-hoc venv, not the flake's env, so native-lib deps (grpc for guardrails' otel exporter needs libstdc++) can break there even when `nix develop` works fine — always test through `nix develop`.

** Key Files
- /home/aner/projects/apps/big-jar-of-pickles/flake.nix — uv2nix flake: packages.default, devShells.default, apps.default (no nixosModule currently — removed in the 2026-09-13 local-only pivot)
- /home/aner/projects/apps/big-jar-of-pickles/src/big_jar_of_pickles/ — graph.py (StateGraph def), cli.py (entrypoint, SqliteSaver checkpointer, argparse with --debug/--show-debug), debug.py (JSONL per-node debug log writer/reader), nodes/{planner,researcher,writer,critic}.py, judge.py, guardrails_validation.py, config.py, llm.py
- /home/aner/projects/apps/big-jar-of-pickles/.envrc.example — committed template for the gitignored .envrc secrets file
- /home/aner/projects/apps/big-jar-of-pickles/README.md — project pitch + local dev instructions

** Subnodes
- [[big-jar-of-pickles/infra]] — homey/Aegra/LangGraph-Server decision, homey integration pattern, packaging notes (historical/future — not current scope)

** Related
Origin: brainstormed in [[just-talking/conversations]] session 2026-09-10, growing out of the AI-orchestration/career-track discussion (see the startup-offer entries in that log).

[[id:d09e7464-e494-433b-beb8-e0742e904c93][big-jar-of-pickles/infra]]

[[id:dc2ddfad-ada9-4f0d-ae35-806e8a1a2def][homey/big-jar-of-pickles]]