80 lines
4.4 KiB
Org Mode
80 lines
4.4 KiB
Org Mode
:PROPERTIES:
|
||
:ID: 2b1e1f91-6d2a-41df-b031-608a8c0f68f6
|
||
:END:
|
||
|
||
#+title: just-talking/simlish-project
|
||
#+filetags: :project: :knowledge: :audio: :ml:
|
||
|
||
* just-talking/simlish-project
|
||
Scoping notes for Aner's Simlish-style procedural speech synthesizer project.
|
||
|
||
** What It Is
|
||
Phonetically plausible speech synthesis that sounds like a language but isn't intelligible.
|
||
Primary application: trmn (/home/aner/projects/apps/trmn).
|
||
|
||
** Why trmn Is the Perfect Host
|
||
- Characters already generate dialogue via TinyStories-1M ONNX (~15MB, bundled in Nix build)
|
||
- SVG face/mouth sprites already exist → ready to animate with audio
|
||
- Deterministic PRNG (masterSeed) → voice synthesis can be seeded per character
|
||
- Static site, Nix-packaged → WASM fits naturally
|
||
|
||
** Technical Approach (Scoped, Not Started)
|
||
Two versions: browser module for trmn, research paper version.
|
||
|
||
*** Browser / trmn version (<1MB WASM)
|
||
1. Prosody extraction from LLM text — sentence structure, punctuation, word length → pitch curve, stress, rate
|
||
2. Fake phoneme inventory — 10–15 phonemes that cohere but form no real language
|
||
3. Formant synthesis WASM vocoder — tiny (few KB of C → WASM), sounds speech-like, no training needed
|
||
4. Emotion conditioning — character traits + TinyStories output valence → voice parameters (happy = higher pitch/faster, sad = lower/slower)
|
||
|
||
*** Research / paper version
|
||
- Use HuBERT speech units: extract discrete learned representations from real speech, scramble/remap to fake phoneme inventory, resynthesize with vocoder
|
||
- Preserves naturalness (working in learned speech space, not engineered phonemes)
|
||
- Could use LAFMA-style latent flow matching as the generative backbone (see below)
|
||
- Paper angle: "Procedural character voice synthesis conditioned on LLM output and character traits"
|
||
|
||
** Audio ML Architecture Notes (from session 2026-08-31)
|
||
|
||
*** Mel spectrogram
|
||
- STFT in sliding windows (25ms window, 10ms hop) → magnitude → mel filterbank (80 triangular filters, non-linear freq compression) → log
|
||
- Result: 80×T matrix, ~86 frames/second. Compresses 1025 FFT bins → 80 mel bins.
|
||
- Mel scale: perceptually motivated, ~linear below 1kHz, logarithmic above. mel = 2595 × log10(1 + Hz/700)
|
||
- Fixed preprocessing, not learned. Fully differentiable.
|
||
|
||
*** GAN vs Diffusion vs Flow Matching (current landscape)
|
||
- GANs (HiFi-GAN, BigVGAN): still used as vocoders (mel → waveform). Fast inference.
|
||
- Diffusion (WaveGrad, DiffWave, Grad-TTS): had moment 2021–2022, now superseded.
|
||
- Flow matching: current standard for acoustic model. Voicebox, F5-TTS, E2-TTS all use it.
|
||
|
||
*** Flow Matching
|
||
- Trains a vector field v_θ(x_t, t) in data space (R^(80×T) for mel)
|
||
- Path between noise x₀ ~ N(0,I) and data x₁ is linear: x_t = (1-t)·x₀ + t·x₁
|
||
- Training target: predict (x₁ - x₀) — constant velocity along each conditional path
|
||
- Inference: integrate ODE from t=0 to t=1, ~10–20 steps (vs 100–1000 for diffusion)
|
||
- Marginal flow is curved (network averages over all crossing paths), but conditional paths are straight
|
||
- Latent flow matching: add VAE first, do flow matching in compressed latent z-space
|
||
|
||
*** LAFMA (reference architecture — arXiv 2406.08203, INTERSPEECH 2024)
|
||
- Latent flow matching for text-to-audio generation
|
||
- Components: FLAN-T5-Large text encoder + VAEGAN + latent flow matching model
|
||
- VAEGAN: VAE with adversarial discriminator during training → sharper reconstructions than plain MSE VAE. Separate checkpoint from flow model.
|
||
- VAE is pretrained then FROZEN during flow matching training (separate checkpoints confirm this)
|
||
- Training stages:
|
||
1. Train VAEGAN (audio → latent → audio, with discriminator)
|
||
2. Freeze VAE. Train flow matching model in latent space.
|
||
3. Inference: noise → [flow ODE] → latent → [VAE decoder] → mel → [vocoder] → audio
|
||
- If building on LAFMA: reuse VAEGAN checkpoint, only train flow matching part on own data
|
||
|
||
** Size Target
|
||
- trmn browser module: <1MB WASM, formant synthesis, no server
|
||
- Research version: ~50–200MB, flow matching on mel or latent space
|
||
|
||
** Status
|
||
Scoped 2026-08-31. Architecture researched. Not started.
|
||
|
||
** Ruled-Out Alternatives
|
||
- Ultrasonic protocol detection: not enough wild deployment to study; too academic
|
||
- Keyboard acoustic side-channel: crowded (Harrison et al. 2023 = 95% accuracy; 2025 papers add LLM correction)
|
||
- Hebrew TTS: good idea but less personally motivating
|
||
- RTL-SDR hardware project: import restrictions in Israel for transmitting SDRs
|