86 lines
5.1 KiB
Org Mode
86 lines
5.1 KiB
Org Mode
** Architecture
|
||
Simlish-style procedural speech synthesizer for trmn. Output: ONNX model loaded by JS in trmn.
|
||
Pipeline: text → prosody (JS) → phoneme sequence → ONNX flow matching → mel → Griffin-Lim → audio.
|
||
Deterministic: seeded per character via trmn's masterSeed PRNG.
|
||
See [[just-talking/simlish-project]] for full scoping notes and research version design.
|
||
|
||
Two-stage inference:
|
||
1. Acoustic model (flow matching, ONNX): (phone_ids, durations, pitch, energy, emotion) → mel[80×T]
|
||
2. Vocoder (Griffin-Lim, pure JS): mel → waveform. No weights. Swappable later.
|
||
|
||
** Decided: Acoustic Model
|
||
Adapted Matcha-TTS (ICASSP 2024, arXiv:2309.03199). ConvNeXt-based OT-CFM flow decoder.
|
||
- Phoneme encoder: embedding table (N_PHONEMES=16) + 2 Conv1d layers → hidden[128, P]
|
||
- Duration predictor: DROPPED — durations come from prosody.mjs at inference time
|
||
- Length regulator: kept — repeat_interleave phoneme hidden states by duration counts
|
||
- Flow decoder: 5 ConvNeXt blocks, 128 channels. FiLM emotion conditioning per block.
|
||
- Pitch + energy: linear-embedded per frame, concatenated with phoneme hidden before decoder
|
||
- ~2–5M params. Trained with OT-CFM (MSE on predicted velocity x1-x0).
|
||
|
||
** Decided: Phoneme Inventory (16 total)
|
||
0=<pad> 1=<sil>
|
||
2=p 3=t 4=k 5=b 6=d 7=m 8=n 9=s 10=l
|
||
11=a 12=e 13=i 14=o 15=u
|
||
No <bos>/<eos> — not needed for non-autoregressive model.
|
||
|
||
** Decided: Training Data — LUT Approach
|
||
English words → letter LUT → mapped text → Piper TTS → audio → mel extraction.
|
||
LUT: letters in inventory map to themselves; others map to nearest class (r→l, f→s, g→d, etc.)
|
||
Phone IDs come from re-applying LUT at extract time — no forced alignment.
|
||
Duration heuristic: vowels get 1.5× frames, silences 0.4×, consonants 1.0×, proportional to total T.
|
||
Length-scale varied 0.8–1.3× per sample for duration variety.
|
||
|
||
Piper voice: en_US-lessac-medium (fetched via Nix, both .onnx and .onnx.json pinned by sha256).
|
||
~10–50h synthetic audio expected to suffice.
|
||
|
||
** Decided: Nix Targets
|
||
Separate derivations with strict dep isolation (mirrors trmn pattern):
|
||
packages.onnx-model — ONNX artifact trmn pulls in; placeholder until checkpoint pinned
|
||
packages.piper-voice — fetchurl'd lessac-medium model + config, cached independently
|
||
apps.gen-data — Piper + librosa + numpy; no torch, no JS
|
||
apps.train — torchWithCuda + onnx; no Piper, no MFA, no JS
|
||
apps.test — nodejs only; no Python
|
||
devShells.data — same deps as gen-data, PIPER_MODEL/PIPER_CONFIG set via env
|
||
devShells.training — torchWithCuda + onnx; for LAN laptop (RTX 3050)
|
||
devShells.default — nixfmt + nodejs; lightweight
|
||
|
||
** Decided: Training Environment
|
||
LAN laptop: aner@192.168.1.200, no password. WSL2 on Windows. RTX 3050 6GB VRAM, CUDA 13.2.
|
||
CUDA accessible via /dev/dxg (WSL2 WDDM). nvidia-smi at /usr/lib/wsl/lib/nvidia-smi.
|
||
Disk: 899GB available (plenty). RAM: 7.6GB WSL2 limit — may need to raise via C:\Users\<you>\.wslconfig.
|
||
PyTorch: use torchWithCuda (not torch) in Nix — default torch is CPU-only.
|
||
|
||
** Key Files
|
||
- flake.nix — all Nix targets; Piper voice fetched here
|
||
- data/gen_corpus.py — English words → LUT → .txt (Piper input) + .ids (phone IDs)
|
||
- data/synthesize.py — Piper TTS → .wav per sentence (length-scale varied)
|
||
- data/extract.py — audio + .ids → mel/pitch/energy/durs → .npz dataset
|
||
- model/acoustic.py — AcousticModel: PhonemeEncoder + LengthRegulator + FlowDecoder
|
||
- model/train.py — OT-CFM training loop
|
||
- model/export.py — PyTorch checkpoint → ONNX FP32 → INT8
|
||
- inference/prosody.mjs — text + emotion → phone_ids/durations/pitch/energy (pure JS)
|
||
- inference/griffin_lim.mjs — mel → waveform (pure JS, replaceable vocoder)
|
||
- test/infer.test.mjs — ONNX load + forward pass + shape checks (node:test)
|
||
- test/prosody.test.mjs — prosody extraction unit tests: determinism, emotion, shapes
|
||
|
||
** Conventions
|
||
- No runtime server calls at inference
|
||
- Same seed must always produce same voice (deterministic)
|
||
- Vocoder is a swappable box: mel[80×T] in, waveform out
|
||
- Nix: no npm/pip lock files — Nix is the build orchestrator
|
||
- Only git-tracked files included in builds (src = ./. in derivations)
|
||
- node:test for JS tests (no framework, follows trmn no-bundler pattern)
|
||
- Piper CLI: echo "text" | piper -m model.onnx -c config.json -f out.wav
|
||
|
||
** Gotchas
|
||
- trmn is static Nix-packaged — ONNX artifact must be added to trmn's flake.nix (not this one)
|
||
- SVG mouth sprites in trmn are ready to animate but animation API not yet defined
|
||
- ODE integration loop (Euler, N steps) runs in JS — only single-step velocity exported to ONNX
|
||
- extract.py reads .ids files from corpus/ subdir, audio from audio/ subdir — path is hardcoded relative
|
||
- dataPython defined twice in flake.nix (packages and apps sections) — intentional, avoids cross-section ref
|
||
- torchWithCuda (not torch) required in Nix for GPU training — default torch is CPU-only
|
||
- WSL2: nvidia-smi not on PATH, use /usr/lib/wsl/lib/nvidia-smi; CUDA works via /dev/dxg
|
||
|
||
** Subnodes
|
||
(none yet)
|