roam/just-talkingsimlish-project.org

just-talking/simlish-project

• just-talking/simlish-project
  • What It Is
  • Why trmn Is the Perfect Host
  • Technical Approach (Scoped, Not Started)
    • Browser / trmn version (<1MB WASM)
    • Research / paper version
  • Audio ML Architecture Notes (from session 2026-08-31)
    • Mel spectrogram
    • GAN vs Diffusion vs Flow Matching (current landscape)
    • Flow Matching
    • LAFMA (reference architecture — arXiv 2406.08203, INTERSPEECH 2024)
  • Size Target
  • Status
  • Ruled-Out Alternatives

just-talking/simlish-project

Scoping notes for Aner's Simlish-style procedural speech synthesizer project.

What It Is

Phonetically plausible speech synthesis that sounds like a language but isn't intelligible. Primary application: trmn (/home/aner/projects/apps/trmn).

Why trmn Is the Perfect Host

• Characters already generate dialogue via TinyStories-1M ONNX (~15MB, bundled in Nix build)
• SVG face/mouth sprites already exist → ready to animate with audio
• Deterministic PRNG (masterSeed) → voice synthesis can be seeded per character
• Static site, Nix-packaged → WASM fits naturally

Technical Approach (Scoped, Not Started)

Two versions: browser module for trmn, research paper version.

Browser / trmn version (<1MB WASM)

1. Prosody extraction from LLM text — sentence structure, punctuation, word length → pitch curve, stress, rate
2. Fake phoneme inventory — 10–15 phonemes that cohere but form no real language
3. Formant synthesis WASM vocoder — tiny (few KB of C → WASM), sounds speech-like, no training needed
4. Emotion conditioning — character traits + TinyStories output valence → voice parameters (happy = higher pitch/faster, sad = lower/slower)

Research / paper version

• Use HuBERT speech units: extract discrete learned representations from real speech, scramble/remap to fake phoneme inventory, resynthesize with vocoder
• Preserves naturalness (working in learned speech space, not engineered phonemes)
• Could use LAFMA-style latent flow matching as the generative backbone (see below)
• Paper angle: "Procedural character voice synthesis conditioned on LLM output and character traits"

Audio ML Architecture Notes (from session 2026-08-31)

Mel spectrogram

• STFT in sliding windows (25ms window, 10ms hop) → magnitude → mel filterbank (80 triangular filters, non-linear freq compression) → log
• Result: 80×T matrix, ~86 frames/second. Compresses 1025 FFT bins → 80 mel bins.
• Mel scale: perceptually motivated, ~linear below 1kHz, logarithmic above. mel = 2595 × log10(1 + Hz/700)
• Fixed preprocessing, not learned. Fully differentiable.

GAN vs Diffusion vs Flow Matching (current landscape)

• GANs (HiFi-GAN, BigVGAN): still used as vocoders (mel → waveform). Fast inference.
• Diffusion (WaveGrad, DiffWave, Grad-TTS): had moment 2021–2022, now superseded.
• Flow matching: current standard for acoustic model. Voicebox, F5-TTS, E2-TTS all use it.

Flow Matching

• Trains a vector field v_θ(x_t, t) in data space (R^(80×T) for mel)
• Path between noise x₀ ~ N(0,I) and data x₁ is linear: x_t = (1-t)·x₀ + t·x₁
• Training target: predict (x₁ - x₀) — constant velocity along each conditional path
• Inference: integrate ODE from t=0 to t=1, ~10–20 steps (vs 100–1000 for diffusion)
• Marginal flow is curved (network averages over all crossing paths), but conditional paths are straight
• Latent flow matching: add VAE first, do flow matching in compressed latent z-space

LAFMA (reference architecture — arXiv 2406.08203, INTERSPEECH 2024)

• Latent flow matching for text-to-audio generation
• Components: FLAN-T5-Large text encoder + VAEGAN + latent flow matching model
• VAEGAN: VAE with adversarial discriminator during training → sharper reconstructions than plain MSE VAE. Separate checkpoint from flow model.
• VAE is pretrained then FROZEN during flow matching training (separate checkpoints confirm this)

• Training stages:

  1. Train VAEGAN (audio → latent → audio, with discriminator)
  2. Freeze VAE. Train flow matching model in latent space.
  3. Inference: noise → [flow ODE] → latent → [VAE decoder] → mel → [vocoder] → audio
• If building on LAFMA: reuse VAEGAN checkpoint, only train flow matching part on own data

Size Target

• trmn browser module: <1MB WASM, formant synthesis, no server
• Research version: ~50–200MB, flow matching on mel or latent space

Status

Scoped 2026-08-31. Architecture researched. Not started.

Ruled-Out Alternatives

• Ultrasonic protocol detection: not enough wild deployment to study; too academic
• Keyboard acoustic side-channel: crowded (Harrison et al. 2023 = 95% accuracy; 2025 papers add LLM correction)
• Hebrew TTS: good idea but less personally motivating
• RTL-SDR hardware project: import restrictions in Israel for transmitting SDRs