backup: 2026-10-05 09:48

just-talkingsimlish-project.org

@@ -0,0 +1,79 @@
+:PROPERTIES:
+:ID: 2b1e1f91-6d2a-41df-b031-608a8c0f68f6
+:END:
+
+#+title: just-talking/simlish-project
+#+filetags: :project: :knowledge: :audio: :ml:
+
+* just-talking/simlish-project
+Scoping notes for Aner's Simlish-style procedural speech synthesizer project.
+
+** What It Is
+Phonetically plausible speech synthesis that sounds like a language but isn't intelligible.
+Primary application: trmn (/home/aner/projects/apps/trmn).
+
+** Why trmn Is the Perfect Host
+- Characters already generate dialogue via TinyStories-1M ONNX (~15MB, bundled in Nix build)
+- SVG face/mouth sprites already exist → ready to animate with audio
+- Deterministic PRNG (masterSeed) → voice synthesis can be seeded per character
+- Static site, Nix-packaged → WASM fits naturally
+
+** Technical Approach (Scoped, Not Started)
+Two versions: browser module for trmn, research paper version.
+
+*** Browser / trmn version (<1MB WASM)
+1. Prosody extraction from LLM text — sentence structure, punctuation, word length → pitch curve, stress, rate
+2. Fake phoneme inventory — 10–15 phonemes that cohere but form no real language
+3. Formant synthesis WASM vocoder — tiny (few KB of C → WASM), sounds speech-like, no training needed
+4. Emotion conditioning — character traits + TinyStories output valence → voice parameters (happy = higher pitch/faster, sad = lower/slower)
+
+*** Research / paper version
+- Use HuBERT speech units: extract discrete learned representations from real speech, scramble/remap to fake phoneme inventory, resynthesize with vocoder
+- Preserves naturalness (working in learned speech space, not engineered phonemes)
+- Could use LAFMA-style latent flow matching as the generative backbone (see below)
+- Paper angle: "Procedural character voice synthesis conditioned on LLM output and character traits"
+
+** Audio ML Architecture Notes (from session 2026-08-31)
+
+*** Mel spectrogram
+- STFT in sliding windows (25ms window, 10ms hop) → magnitude → mel filterbank (80 triangular filters, non-linear freq compression) → log
+- Result: 80×T matrix, ~86 frames/second. Compresses 1025 FFT bins → 80 mel bins.
+- Mel scale: perceptually motivated, ~linear below 1kHz, logarithmic above. mel = 2595 × log10(1 + Hz/700)
+- Fixed preprocessing, not learned. Fully differentiable.
+
+*** GAN vs Diffusion vs Flow Matching (current landscape)
+- GANs (HiFi-GAN, BigVGAN): still used as vocoders (mel → waveform). Fast inference.
+- Diffusion (WaveGrad, DiffWave, Grad-TTS): had moment 2021–2022, now superseded.
+- Flow matching: current standard for acoustic model. Voicebox, F5-TTS, E2-TTS all use it.
+
+*** Flow Matching
+- Trains a vector field v_θ(x_t, t) in data space (R^(80×T) for mel)
+- Path between noise x₀ ~ N(0,I) and data x₁ is linear: x_t = (1-t)·x₀ + t·x₁
+- Training target: predict (x₁ - x₀) — constant velocity along each conditional path
+- Inference: integrate ODE from t=0 to t=1, ~10–20 steps (vs 100–1000 for diffusion)
+- Marginal flow is curved (network averages over all crossing paths), but conditional paths are straight
+- Latent flow matching: add VAE first, do flow matching in compressed latent z-space
+
+*** LAFMA (reference architecture — arXiv 2406.08203, INTERSPEECH 2024)
+- Latent flow matching for text-to-audio generation
+- Components: FLAN-T5-Large text encoder + VAEGAN + latent flow matching model
+- VAEGAN: VAE with adversarial discriminator during training → sharper reconstructions than plain MSE VAE. Separate checkpoint from flow model.
+- VAE is pretrained then FROZEN during flow matching training (separate checkpoints confirm this)
+- Training stages:
+  1. Train VAEGAN (audio → latent → audio, with discriminator)
+  2. Freeze VAE. Train flow matching model in latent space.
+  3. Inference: noise → [flow ODE] → latent → [VAE decoder] → mel → [vocoder] → audio
+- If building on LAFMA: reuse VAEGAN checkpoint, only train flow matching part on own data
+
+** Size Target
+- trmn browser module: <1MB WASM, formant synthesis, no server
+- Research version: ~50–200MB, flow matching on mel or latent space
+
+** Status
+Scoped 2026-08-31. Architecture researched. Not started.
+
+** Ruled-Out Alternatives
+- Ultrasonic protocol detection: not enough wild deployment to study; too academic
+- Keyboard acoustic side-channel: crowded (Harrison et al. 2023 = 95% accuracy; 2025 papers add LLM correction)
+- Hebrew TTS: good idea but less personally motivating
+- RTL-SDR hardware project: import restrictions in Israel for transmitting SDRs