Moss Voice

Live: moss-voice.sliverkeigo.fun

A browser-native text-to-speech workspace built on MOSS-TTS-Nano. The model runs entirely on-device via ONNX Runtime Web; nothing about the script ever leaves the tab.

This is a real product surface, not a marketing page. The code is shaped around honest browser-first inference boundaries so it stays compatible with Vercel's static-frontend deployment model.

Screencast

A screen recording of the clone flow goes here. Until then, hit the Live link above to try it directly.

What's here

app/                       Next.js 15 App Router shell
  layout.tsx               Fonts + global CSS + theme noflash
  page.tsx                 Mounts <Workspace/>
  globals.css              Paper / ink palette, theme tokens, animations

components/                React UI
  TopNav.tsx               Sticky nav + live model-status pill
  TopProgressBar.tsx       2px sticky progress bar (idle/downloading/initializing)
  Hero.tsx                 Inline value-prop strip
  Editor.tsx               Script editor with char/word/secs HUD
  Voices.tsx               Picker w/ built-in groups, My voices, Clone CTA
  CloneVoiceSheet.tsx      Modal: upload + decode + encoder dispatch
  Controls.tsx             Generate CTA + status copy
  Player.tsx               Dark playback card with waveform
  Onboarding.tsx           First-run modal w/ real byte progress
  Workspace.tsx            Glue: state, hotkeys, streaming playback, WAV download
  Footer.tsx               Attribution + license line
  ThemeSwitcher.tsx        4-palette switch (moss / slate / sand / ink)
  LocaleSwitcher.tsx       en / zh-CN / ja
  I18nProvider.tsx         next-intl client provider

lib/tts/                   Browser inference module
  types.ts                 Worker wire protocol + UI-facing types
  manifest.ts              Hugging Face repo / file inventory
  voiceCatalog.ts          8 demo-vetted voices + i18n-aware display names
  cloned_voices.ts         OPFS store for user-cloned voices (rename/delete)
  model-store.ts           OPFS-backed cache w/ parallel streaming download
  runtime.ts               ORT-Web sessions + prefill / decode_step / codec encode+decode
  audio_stream_player.ts   Web Audio prebuffer + underrun-safe scheduler
  worker.ts                Module worker entry point, transferable Float32 chunks
  client.ts                Main-thread client + event bus + reference-audio resampler
  useMossVoice.ts          React hook
  wav.ts                   Float32 PCM → 16-bit WAV + waveform summary

lib/i18n/                  next-intl message catalogs
  messages/{en,zh-CN,ja}.ts

lib/data/
  supported_languages.ts   19 endonym entries surfaced in the picker strip

scripts/
  copy-ort-assets.mjs      Stages onnxruntime-web/wasm bundles into /ort/
  copy-voice-previews.mjs  Mirrors the 8 reference clips into /voices/

Architecture

flowchart LR
  UI[Workspace UI] -- request --> Client[TtsClient]
  Upload[Reference clip] -- AudioBuffer --> Client
  Client -- resample 48k stereo --> Client
  Client -- postMessage --> Worker[Inference Worker]
  Worker -- fetch + stream --> Edge[/api/model · Vercel edge cache/]
  Edge -- miss --> HF[(Hugging Face)]
  HF -- bytes --> Edge
  Edge -- bytes --> OPFS[(OPFS cache)]
  Worker -- read --> OPFS
  Worker -- create sessions (parallel) --> ORT[onnxruntime-web WASM · threaded]
  ORT -- frame chunks --> Worker
  ORT -- prompt codes --> Worker
  Worker -- transferable Float32 --> Client
  Worker -- prompt_audio_codes --> Cloned[(OPFS cloned voices)]
  Client -- streaming PCM --> Player[AudioStreamPlayer]
  Player -- Web Audio --> Speakers
  Client -- final buffer --> UI -- encode --> WAV[/WAV download/]

• Main thread: React + Next.js App Router. A single TtsClient singleton owns the worker handle and an AudioStreamPlayer that feeds Web Audio with a 0.5 s prebuffer + underrun guard.
• Worker: spawned with new Worker(new URL("./worker.ts", import.meta.url)). Owns the OPFS cache, the ORT-Web sessions, and the autoregressive decode loop. Audio frames are transferred back as Float32Array chunks (zero-copy).
• OPFS cache: model files live at nano-reader-browser-model-store/<repo>/<file> so the same cache layout used by the official MOSS-TTS-Nano-Reader browser extension can read the same bytes.
• Multi-threaded WASM: we ship onnxruntime-web's threaded build (ort-wasm-simd-threaded.{wasm,mjs}) directly from /ort/ and request numThreads = min(4, cores - 1). Cross-origin isolation is enforced via the COOP/COEP headers in next.config.mjs.
• Edge-cached origin proxy: a single route handler at /api/model/[...path] reverse-proxies the Hugging Face resolve URLs and stamps the response with Cache-Control: public, max-age=1y, immutable. The first user in each Vercel POP warms the edge cache; everyone after gets warm bytes a few ms away instead of crossing the public internet to HF. An allowlist derived from ALL_REPOS keeps the route from being used as a free CDN for arbitrary HF files.

Model lifecycle (real states, not a mock)

idle  →  downloading  →  initializing  →  ready
                              │              │
                              ▼              ▼
                            error  ←─── synthesis success / failure

Synthesis pipeline

sequenceDiagram
  participant UI as Workspace
  participant W as Worker
  participant T as SentencePiece
  participant M as MOSS-TTS-Nano
  participant C as MOSS-Audio-Tokenizer-Nano
  participant P as AudioStreamPlayer

  UI->>W: synthesize(text, voice)
  W->>T: encode(text)
  T-->>W: token ids
  W->>M: prefill(tokens, prompt_audio_codes)
  loop until <eos>
    W->>M: local_fixed_sampled_frame + decode_step
    M-->>W: next audio frame (16 codebooks)
    W->>C: codec_decode_step(frame)
    C-->>W: PCM chunk
    W-->>P: Float32 transferable
    P-->>UI: scheduleBuffer (Web Audio)
  end
  W-->>UI: final Blob URL + duration

Streaming decode is live: the codec model emits PCM frame-by-frame while the prosody model is still generating, so playback starts within the prebuffer window instead of after the whole utterance.

Voice catalog

8 voices, taken from the OpenMOSS demo site and renamed to describe timbre, not the content of the reference clip. Names below are the English locale; the picker reads zh-CN and ja from the i18n catalog.

Each voice maps onto a row in MOSS-TTS-Nano's builtin_voices[] inside browser_poc_manifest.json. Those entries ship pre-encoded prompt_audio_codes, so the picker can render them without touching the encoder.

Cloned voices

The picker also exposes a Clone a voice CTA that runs the encoder side of the codec on-device:

1. User drops a 10–30 s wav / mp3 / m4a clip.
2. Main thread decodes it with AudioContext.decodeAudioData and resamples to 48 kHz stereo via OfflineAudioContext (the worker has no such API).
3. PCM is transferred to the worker, which runs moss_audio_tokenizer_encode and returns prompt_audio_codes in the exact shape builtin_voices[] produces.
4. The result is persisted to OPFS under moss-voice-cloned/cv-*.json (independent of the model-cache directory, so a re-download of the model can't wipe a user's clones).
5. Synthesis treats cloned and built-in voices symmetrically — the main thread attaches the prompt_audio_codes to the request when the selected voice is a cv-… id.

Cloned voices can be inline-renamed (✎) or deleted (×) from the picker; the underlying file is rewritten with the new name. The clone CTA stays disabled until the model status is ready.

Voice ≠ language. Any voice can synthesize text in any of the 19 languages MOSS-TTS-Nano supports — the voice defines timbre, the script defines content. The full list (zh / en / de / es / fr / ja / it / hu / ko / ru / fa / ar / pl / pt / cs / da / sv / el / tr) is rendered in the picker's language strip.

Upstream's prose says "20 languages" but its own table only lists 19; we follow the table, since that's the verifiable surface.

Theming & i18n

• 4 palettes: moss (default), slate, sand, ink. Selection persists to localStorage; a tiny <script> runs before paint to avoid a flash of unstyled theme.
• 3 locales: English, Simplified Chinese, Japanese. Powered by next-intl in client-only mode so locale can switch without a route refresh.

Required browser features

• OPFS (navigator.storage.getDirectory) for persistent caching of ~745 MB of model weights (TTS + audio tokenizer with encoder).
• Web Workers with type: "module".
• WebAssembly + SIMD + threads (SharedArrayBuffer). Cross-origin isolation must be active — the COOP/COEP headers are already wired in next.config.mjs.

Recent Chrome / Edge / Safari 17+ on desktop satisfies all three.

Running locally

npm install
npm run dev
# Visit http://localhost:3000

The first model load is large (~745 MB pulled from Hugging Face, including the ~45 MB codec encoder used for voice cloning). After that, OPFS reuses the cached bytes across visits.

End-to-end tests:

npm run test:e2e          # full Playwright suite (~2 min)
npm run test:e2e -- --headed  # watch it run

Deploying to Vercel

vercel link
vercel --prod

No additional configuration needed. The repo ships:

• next.config.mjs with COOP/COEP headers for cross-origin isolation and a webpack alias that pins onnxruntime-web to its /wasm subentry (the default bundle ships a self-reentrant new Worker(new URL(import.meta.url)) that Vercel's file:// resolution rejects).
• A prebuild hook that stages the threaded WASM assets into /ort/ and the 8 reference clips into /voices/.
• An /api/model/[...path] route handler that proxies Hugging Face with a one-year immutable Cache-Control header, so Vercel's edge network serves repeat visitors. Origin is still Hugging Face — no build-step uploads of multi-hundred-MB binaries.

Roadmap

The full text→speech pipeline (SentencePiece tokenization → prefill → local_fixed_sampled_frame autoregressive loop → decode_step KV-cache update → streaming codec_decode_step → PCM) plus the voice-clone encoder path (codec_encode → prompt_audio_codes → OPFS) are wired end-to-end and run entirely in the worker / main-thread pair. The work below is what's planned next.

Next up

• Microphone capture for voice cloning. Today the clone flow only accepts uploaded files. Add an in-app recorder with a 30 s timer and a level meter, then feed the resulting Blob through the same decode → encode path.
• Long-form chunking. We cap at 500 characters to match the model's prompt-template length comfortably. Splitting longer inputs into multiple synthesis passes and concatenating PCM is the obvious extension.
• WeTextProcessing-style normalization passes. Free text goes straight into SentencePiece with only minimal English nudges (capitalize first letter, ensure terminal punctuation). The upstream Reader runs a multi-stage text normalizer before tokenization.

Changelog

0.3.0

• Voice cloning is live. moss_audio_tokenizer_encode joins the worker's session set; main-thread OfflineAudioContext handles 48 kHz / stereo resampling; OPFS persists cloned voices under moss-voice-cloned/cv-*.json independent of the model cache.
• Voice picker grew a Clone a voice CTA (BETA) and a My voices group with rename + delete, header chip switches to N custom · M built-in once at least one clone exists.
• Picker also surfaces the 19 actual supported languages via an endonym chip strip (upstream's prose says "20" but the table lists 19; we follow the table). -