6.2 KiB
big-jar-of-pickles
Swarm of weak/small local models ("Big Pickle," opencode's small model — a jar of them) wired with production-grade orchestration, attempting something none of them could do alone. Article-driven side project: does orchestration substitute for model capability, and where's the crossover point.
Architecture
Scope narrowed 2026-09-13: standalone local CLI, no self-hosted services required for now. LangGraph library + local SQLite checkpointer (BJP_CHECKPOINT_PATH, default ./bjp_checkpoints.sqlite) + Guardrails AI + LLM-judge, packaged via uv2nix. Langfuse tracing is opt-in (only wired if LANGFUSE_PUBLIC_KEY is set) rather than required. The homey-backed stack (Postgres+pgvector, self-hosted Langfuse, Aegra) is deferred to later, once the pipeline itself is proven out — reasoning and prior state preserved in big-jar-of-pickles/infra. Reason for the pivot: self-hosted Langfuse v4 hard-requires ClickHouse, which the target Homey Pi 4 (8GB) likely can't spare RAM for; rather than fight that, cut the self-hosted dependency entirely for the MVP.
nixosModules.default (homey systemd wiring) was removed from flake.nix as part of this pivot — there's no service to deploy right now, just a CLI. If/when homey integration resumes, that module will need to be re-added (see big-jar-of-pickles/infra for the eurovote-pattern reference it followed).
Phase 1 task: Research -> Write -> Critique pipeline. Specialist roles (researcher, writer, critic) played by many cheap/weak model calls on a fixed topic, producing a short technical explainer, with LLM-judge scoring tracked across iterations as the article's data.
Phase 2 (maybe): same harness pointed at a quantified coding benchmark (HumanEval subset) for swarm vs. single-small-model vs. single-frontier-model comparison.
LLM config is provider-agnostic by design (2026-09-13): Settings/llm.py use generic `llm_api_key`/`llm_base_url`/`llm_model` fields (env vars `LLM_API_KEY`/`LLM_BASE_URL`/`LLM_MODEL`), not provider-specific names — llm.py is just a bare `OpenAI(api_key=…, base_url=…)` client, so swapping providers is an env-var change, not a code change. Currently pointed at OpenCode Zen (opencode.ai/docs/zen/) — confirmed official name (not "Zen Cloud", an earlier placeholder guess). Zen is opencode's own model marketplace, OpenAI-compatible endpoint at `https://opencode.ai/zen/v1`, offers opencode's "Big Pickle" model among others.
Secrets handling: no SOPS — this is a local-only CLI with no NixOS deployment yet, so the API key lives in a gitignored `.envrc` (direnv, `use flake` + exports), with `.envrc.example` committed as the fillable template. SOPS is the right tool for later, when/if homey/NixOS deployment resumes (matches the eurovote pattern already established in big-jar-of-pickles/infra) — introducing it now for a single-machine secret would be pure overhead.
Debugging (2026-09-15): `bjp –debug` runs the graph via `graph.stream(…, stream_mode="updates")` instead of `.invoke()`, writing each node's output as a JSON line to `BJP_DEBUG_DIR/<thread_id>.jsonl` (debug.py). Replay with `bjp –show-debug <thread_id>`. This only captures node-level state deltas (plan/research/draft/critique/score), not raw prompts — actual system/user messages sent to the LLM are still unlogged since llm.py's `complete()` uses the raw `openai.OpenAI` client. The Langfuse `CallbackHandler` wired in cli.py also only sees LangGraph chain/node boundaries for the same reason (it hooks LangChain callback events, but nodes never go through a LangChain-wrapped LLM) — swapping to `from langfuse.openai import OpenAI` in llm.py would be the real fix for prompt/completion-level tracing, not yet done.
Deployment
None currently — runs as a local CLI (`nix run` / `uv run` / `bjp`). Inference is cloud (OpenCode Zen), so no local token-serving either way.
Conventions
(none yet)
Gotchas
- Guardrails integration (src/big_jar_of_pickles/guardrails_validation.py) uses the core Validator/register_validator API and is untested against a live run (no real credentials were available when it was scaffolded) — sanity check it actually validates before relying on it.
- Self-hosted Langfuse v4 requires ClickHouse (not just Postgres) — this is why observability was made opt-in rather than deploying self-hosted Langfuse on the resource-constrained Pi. See homey/big-jar-of-pickles for the ClickHouse memory-risk notes from the homey side.
- Nix flakes only see git-tracked files: `nix develop`/`nix build` silently fail to pick up newly created files until they're at least `git add`ed (staged is enough, doesn't need to be committed). Bit twice during dev — e.g. adding debug.py and running `nix develop` gave `ImportError: cannot import name 'debug'` until it was staged. `uv run` outside the nix devShell is a separate trap: it uses the ambient/ad-hoc venv, not the flake's env, so native-lib deps (grpc for guardrails' otel exporter needs libstdc++) can break there even when `nix develop` works fine — always test through `nix develop`.
Key Files
- /home/aner/projects/apps/big-jar-of-pickles/flake.nix — uv2nix flake: packages.default, devShells.default, apps.default (no nixosModule currently — removed in the 2026-09-13 local-only pivot)
- home/aner/projects/apps/big-jar-of-pickles/src/big_jar_of_pickles — graph.py (StateGraph def), cli.py (entrypoint, SqliteSaver checkpointer, argparse with –debug/–show-debug), debug.py (JSONL per-node debug log writer/reader), nodes/{planner,researcher,writer,critic}.py, judge.py, guardrails_validation.py, config.py, llm.py
- home/aner/projects/apps/big-jar-of-pickles.envrc.example — committed template for the gitignored .envrc secrets file
- /home/aner/projects/apps/big-jar-of-pickles/README.md — project pitch + local dev instructions
Subnodes
- big-jar-of-pickles/infra — homey/Aegra/LangGraph-Server decision, homey integration pattern, packaging notes (historical/future — not current scope)
Related
Origin: brainstormed in just-talking/conversations session 2026-09-10, growing out of the AI-orchestration/career-track discussion (see the startup-offer entries in that log).