Lipsync

lipsync — audio-driven mouths for XR Blocks

lipsync turns any avatar with a head pivot into a face that visibly mouths along to a MediaStream. Mental model: a per-frame FFT + formant analyser writes viseme weights into a target; the target (typically xb.StylizedFace) re-rasterises a small canvas decal anchored to the head sphere's local -Z. No model download, no ML runtime.

> Full reference (DSP pipeline, samples, caveats, public surface): README.md. Samples: samples/.

When to use

Pair with netblocks so every remote peer's voice stream drives their own avatar's mouth, turning shared rooms from silent spheres into faces that visibly speak. Standalone use is fine too: a TTS playback, an NPC, or a single-user puppet head. The face primitive (xb.StylizedFace) is in xrblocks core, so any consumer can drive it via setVisemes(VisemeWeights), not just lipsync.

For ML-grade phoneme accuracy, the lower-level pieces (FormantVisemeMapper, MfccExtractor, computeAudioFeatures) and types are exported so a model-based mapper can slot in without touching the addon's public surface.

Mental model

• MediaStream → WebAudio AnalyserNode → byte frequency + time-domain buffers each frame.
• computeAudioFeatures extracts RMS, voicing, F1, F2, and a few band energies.
• FormantVisemeMapper maps F1/F2 to six viseme weights (jawOpen, aa, oo, oh, ee, consonant) with frame-rate-independent smoothing (1 - exp(-dt / tau)).
• LipsyncMouth writes the weights to its target via setVisemes(...).
• xb.StylizedFace re-rasterises a 256×256 canvas (one dark mouth ellipse, optional eyes) and uploads it as a CanvasTexture on a small plane.

Public surface

| Symbol | Where | Purpose |

| ----------------------------------------------------------------------------------------------------------------------------------- | ------------------------------------------------------- | ----------------------------------------------------------------------------------------------------------------------------------- |

| LipsyncMouth | xrblocks/addons/lipsync/index.js | The driver xb.Script. Constructor: (stream: MediaStream, {target, audioContext?, fftSize?, silenceThreshold?, silenceHoldMs?}). |

| xb.StylizedFace | xrblocks core (import {StylizedFace} from 'xrblocks') | The face decal. Options: headRadius, textureSize, showEyes. |

| RemoteUserAvatar.face | xrblocks/addons/netblocks | Already a StylizedFace, ready for LipsyncMouth to drive. |

| FormantVisemeMapper, MfccExtractor, computeAudioFeatures | xrblocks/addons/lipsync/index.js | Pure modules a future ML mapper can plug into. |

| Types: VisemeWeights, VisemeTarget, FormantVisemeMapperOptions, MfccExtractorOptions, AudioFeatures, AudioFeatureInputs | xrblocks/addons/lipsync/index.js | Shared shapes for custom drivers. |

Lifecycle and ownership

• The caller owns the MediaStream. LipsyncMouth.dispose() disconnects audio nodes but never stops tracks. If you got the stream from getUserMedia, stop the tracks yourself when done.
• The caller owns the AudioContext when passed in. Always pass audioContext to reuse a shared context for any scene with more than one mouth (browsers cap contexts at around six per page). If omitted, LipsyncMouth creates its own and closes it on dispose.
• The caller owns the target face. dispose() resets it to rest pose so a speaker who stops mid-vowel never leaves their avatar's mouth frozen open, but never disposes the face itself.
• Instances are one-shot. After dispose, construct a new LipsyncMouth.

Browser quirks worth knowing

• Microphone access requires HTTPS in modern browsers. Use localhost or a real cert for cross-device testing.
• Browsers can drop a MediaStreamAudioSourceNode unless the same stream is also being pumped by an HTMLMediaElement. LipsyncMouth creates a muted off-DOM <audio> primer per stream to keep WebAudio alive. Same workaround SpatialVoice uses.
• High-pitched voices (children, sopranos) push formants up and reduce vowel separation. Speaker-relative normalisation would help and is a sensible follow-up.