roam/tiny-trmn-tts.org

• Architecture
• Decided: Acoustic Model
• Decided: Phoneme Inventory (16 total)
• Decided: Training Data — LUT Approach
• Decided: Nix Targets
• Decided: Training Environment
• Key Files
• Conventions
• Gotchas
• Subnodes

Architecture

Simlish-style procedural speech synthesizer for trmn. Output: ONNX model loaded by JS in trmn. Pipeline: text → prosody (JS) → phoneme sequence → ONNX flow matching → mel → Griffin-Lim → audio. Deterministic: seeded per character via trmn's masterSeed PRNG. See just-talking/simlish-project for full scoping notes and research version design.

Two-stage inference:

1. Acoustic model (flow matching, ONNX): (phone_ids, durations, pitch, energy, emotion) → mel[80×T]
2. Vocoder (Griffin-Lim, pure JS): mel → waveform. No weights. Swappable later.

Decided: Acoustic Model

Adapted Matcha-TTS (ICASSP 2024, arXiv:2309.03199). ConvNeXt-based OT-CFM flow decoder.

• Phoneme encoder: embedding table (N_PHONEMES=16) + 2 Conv1d layers → hidden[128, P]
• Duration predictor: DROPPED — durations come from prosody.mjs at inference time
• Length regulator: kept — repeat_interleave phoneme hidden states by duration counts
• Flow decoder: 5 ConvNeXt blocks, 128 channels. FiLM emotion conditioning per block.
• Pitch + energy: linear-embedded per frame, concatenated with phoneme hidden before decoder
• ~2–5M params. Trained with OT-CFM (MSE on predicted velocity x1-x0).

Decided: Phoneme Inventory (16 total)

0=<pad> 1=<sil> 2=p 3=t 4=k 5=b 6=d 7=m 8=n 9=s 10=l 11=a 12=e 13=i 14=o 15=u No <bos>/<eos> — not needed for non-autoregressive model.

Decided: Training Data — LUT Approach

English words → letter LUT → mapped text → Piper TTS → audio → mel extraction. LUT: letters in inventory map to themselves; others map to nearest class (r→l, f→s, g→d, etc.) Phone IDs come from re-applying LUT at extract time — no forced alignment. Duration heuristic: vowels get 1.5× frames, silences 0.4×, consonants 1.0×, proportional to total T. Length-scale varied 0.8–1.3× per sample for duration variety.

Piper voice: en_US-lessac-medium (fetched via Nix, both .onnx and .onnx.json pinned by sha256). ~10–50h synthetic audio expected to suffice.

Decided: Nix Targets

Separate derivations with strict dep isolation (mirrors trmn pattern): packages.onnx-model — ONNX artifact trmn pulls in; placeholder until checkpoint pinned packages.piper-voice — fetchurl'd lessac-medium model + config, cached independently apps.gen-data — Piper + librosa + numpy; no torch, no JS apps.train — torchWithCuda + onnx; no Piper, no MFA, no JS apps.test — nodejs only; no Python devShells.data — same deps as gen-data, PIPER_MODEL/PIPER_CONFIG set via env devShells.training — torchWithCuda + onnx; for LAN laptop (RTX 3050) devShells.default — nixfmt + nodejs; lightweight

Decided: Training Environment

LAN laptop: aner@192.168.1.200, no password. WSL2 on Windows. RTX 3050 6GB VRAM, CUDA 13.2. CUDA accessible via /dev/dxg (WSL2 WDDM). nvidia-smi at /usr/lib/wsl/lib/nvidia-smi. Disk: 899GB available (plenty). RAM: 7.6GB WSL2 limit — may need to raise via C:\Users\<you>\.wslconfig. PyTorch: use torchWithCuda (not torch) in Nix — default torch is CPU-only.

Key Files

• flake.nix — all Nix targets; Piper voice fetched here
• data/gen_corpus.py — English words → LUT → .txt (Piper input) + .ids (phone IDs)
• data/synthesize.py — Piper TTS → .wav per sentence (length-scale varied)
• data/extract.py — audio + .ids → mel/pitch/energy/durs → .npz dataset
• model/acoustic.py — AcousticModel: PhonemeEncoder + LengthRegulator + FlowDecoder
• model/train.py — OT-CFM training loop
• model/export.py — PyTorch checkpoint → ONNX FP32 → INT8
• inference/prosody.mjs — text + emotion → phone_ids/durations/pitch/energy (pure JS)
• inference/griffin_lim.mjs — mel → waveform (pure JS, replaceable vocoder)
• test/infer.test.mjs — ONNX load + forward pass + shape checks (node:test)
• test/prosody.test.mjs — prosody extraction unit tests: determinism, emotion, shapes

Conventions

• No runtime server calls at inference
• Same seed must always produce same voice (deterministic)
• Vocoder is a swappable box: mel[80×T] in, waveform out
• Nix: no npm/pip lock files — Nix is the build orchestrator
• Only git-tracked files included in builds (src = ./. in derivations)
• node:test for JS tests (no framework, follows trmn no-bundler pattern)
• Piper CLI: echo "text" | piper -m model.onnx -c config.json -f out.wav

Gotchas

• trmn is static Nix-packaged — ONNX artifact must be added to trmn's flake.nix (not this one)
• SVG mouth sprites in trmn are ready to animate but animation API not yet defined
• ODE integration loop (Euler, N steps) runs in JS — only single-step velocity exported to ONNX
• extract.py reads .ids files from corpus/ subdir, audio from audio/ subdir — path is hardcoded relative
• dataPython defined twice in flake.nix (packages and apps sections) — intentional, avoids cross-section ref
• torchWithCuda (not torch) required in Nix for GPU training — default torch is CPU-only
• WSL2: nvidia-smi not on PATH, use /usr/lib/wsl/lib/nvidia-smi; CUDA works via /dev/dxg

Subnodes

(none yet)