:PROPERTIES:
:ID: 2b1e1f91-6d2a-41df-b031-608a8c0f68f6
:END:

#+title: just-talking/simlish-project
#+filetags: :project: :knowledge: :audio: :ml:

* just-talking/simlish-project
Scoping notes for Aner's Simlish-style procedural speech synthesizer project.

** What It Is
Phonetically plausible speech synthesis that sounds like a language but isn't intelligible.
Primary application: trmn (/home/aner/projects/apps/trmn).

** Why trmn Is the Perfect Host
- Characters already generate dialogue via TinyStories-1M ONNX (~15MB, bundled in Nix build)
- SVG face/mouth sprites already exist → ready to animate with audio
- Deterministic PRNG (masterSeed) → voice synthesis can be seeded per character
- Static site, Nix-packaged → WASM fits naturally

** Technical Approach (Scoped, Not Started)
Two versions: browser module for trmn, research paper version.

*** Browser / trmn version (<1MB WASM)
1. Prosody extraction from LLM text — sentence structure, punctuation, word length → pitch curve, stress, rate
2. Fake phoneme inventory — 10–15 phonemes that cohere but form no real language
3. Formant synthesis WASM vocoder — tiny (few KB of C → WASM), sounds speech-like, no training needed
4. Emotion conditioning — character traits + TinyStories output valence → voice parameters (happy = higher pitch/faster, sad = lower/slower)

*** Research / paper version
- Use HuBERT speech units: extract discrete learned representations from real speech, scramble/remap to fake phoneme inventory, resynthesize with vocoder
- Preserves naturalness (working in learned speech space, not engineered phonemes)
- Could use LAFMA-style latent flow matching as the generative backbone (see below)
- Paper angle: "Procedural character voice synthesis conditioned on LLM output and character traits"

** Audio ML Architecture Notes (from session 2026-08-31)

*** Mel spectrogram
- STFT in sliding windows (25ms window, 10ms hop) → magnitude → mel filterbank (80 triangular filters, non-linear freq compression) → log
- Result: 80×T matrix, ~86 frames/second. Compresses 1025 FFT bins → 80 mel bins.
- Mel scale: perceptually motivated, ~linear below 1kHz, logarithmic above. mel = 2595 × log10(1 + Hz/700)
- Fixed preprocessing, not learned. Fully differentiable.

*** GAN vs Diffusion vs Flow Matching (current landscape)
- GANs (HiFi-GAN, BigVGAN): still used as vocoders (mel → waveform). Fast inference.
- Diffusion (WaveGrad, DiffWave, Grad-TTS): had moment 2021–2022, now superseded.
- Flow matching: current standard for acoustic model. Voicebox, F5-TTS, E2-TTS all use it.

*** Flow Matching
- Trains a vector field v_θ(x_t, t) in data space (R^(80×T) for mel)
- Path between noise x₀ ~ N(0,I) and data x₁ is linear: x_t = (1-t)·x₀ + t·x₁
- Training target: predict (x₁ - x₀) — constant velocity along each conditional path
- Inference: integrate ODE from t=0 to t=1, ~10–20 steps (vs 100–1000 for diffusion)
- Marginal flow is curved (network averages over all crossing paths), but conditional paths are straight
- Latent flow matching: add VAE first, do flow matching in compressed latent z-space

*** LAFMA (reference architecture — arXiv 2406.08203, INTERSPEECH 2024)
- Latent flow matching for text-to-audio generation
- Components: FLAN-T5-Large text encoder + VAEGAN + latent flow matching model
- VAEGAN: VAE with adversarial discriminator during training → sharper reconstructions than plain MSE VAE. Separate checkpoint from flow model.
- VAE is pretrained then FROZEN during flow matching training (separate checkpoints confirm this)
- Training stages:
  1. Train VAEGAN (audio → latent → audio, with discriminator)
  2. Freeze VAE. Train flow matching model in latent space.
  3. Inference: noise → [flow ODE] → latent → [VAE decoder] → mel → [vocoder] → audio
- If building on LAFMA: reuse VAEGAN checkpoint, only train flow matching part on own data

** Size Target
- trmn browser module: <1MB WASM, formant synthesis, no server
- Research version: ~50–200MB, flow matching on mel or latent space

** Status
Scoped 2026-08-31. Architecture researched. Not started.

** Ruled-Out Alternatives
- Ultrasonic protocol detection: not enough wild deployment to study; too academic
- Keyboard acoustic side-channel: crowded (Harrison et al. 2023 = 95% accuracy; 2025 papers add LLM correction)
- Hebrew TTS: good idea but less personally motivating
- RTL-SDR hardware project: import restrictions in Israel for transmitting SDRs