Skip to content

feat(smallestai): use Waves continuous WebSocket streaming protocol for TTS#6363

Draft
harshitajain165 wants to merge 1 commit into
livekit:mainfrom
harshitajain165:feat/smallest-tts-streaming-protocol
Draft

feat(smallestai): use Waves continuous WebSocket streaming protocol for TTS#6363
harshitajain165 wants to merge 1 commit into
livekit:mainfrom
harshitajain165:feat/smallest-tts-streaming-protocol

Conversation

@harshitajain165

@harshitajain165 harshitajain165 commented Jul 9, 2026

Copy link
Copy Markdown
Contributor

Summary

The SmallestAI SynthesizeStream currently buffers all incoming tokens locally and sends one isolated request per segment, without using the Waves continuous-streaming protocol (continue / flush / max_buffer_flush_ms) that comparable LiveKit TTS plugins (ElevenLabs, Cartesia, Rime, Neuphonic, Inworld, …) use.

This PR switches the plugin to the continuous protocol so it speaks the Waves streaming API the way the official SDK and other providers' plugins do:

  • Forwards tokens to /waves/v1/tts/live as they arrive with "continue": true, instead of buffering the whole segment locally.
  • Closes each segment with an explicit "flush": true frame.
  • Exposes the server-side max_buffer_flush_ms buffer bound as a TTS(...) constructor arg and update_options(...) option (default 0).
  • Runs send/receive as concurrent tasks over the pooled connection. The connection is only returned to the pool after a clean complete; the pool discards it on error/interruption, so reuse stays safe.
  • Collapses to one SynthesizeStream instance = one segment, matching the current base-class contract (multi-segment instances are deprecated upstream).

Response-frame handling (chunk / word_timestamp / complete / error) is unchanged, and the HTTP ChunkedStream path is untouched.

Scope / note

This change is about protocol parity — using the continuous-streaming fields the Waves WS API exposes, matching the official SDK and other plugins. It is not a latency change: with current server behavior the endpoint synthesizes on the explicit flush, so output timing is unchanged from the previous buffer-until-flush implementation. The client is now ready to benefit from any future server-side incremental emission with no further plugin changes.

Testing

Verified against the live Waves /tts/live endpoint:

  • Continuous streaming: tokens streamed with continue/flush/max_buffer_flush_ms are accepted and return audio (lightning_v3.1 / magnus).
  • Word timestamps over streaming: with the correct voice/model pairing (lightning_v3.1_pro / meher, word_timestamps=True), the endpoint returns word_timestamp frames — all words surfaced as TimedString transcript entries with sensible timings alongside audio. The word_timestamps flag on the streamed frames is accepted (no "Invalid input data").
  • Interruption / reuse: cancelling a stream mid-synthesis leaves the pool clean — a subsequent stream on the same TTS instance completes normally.
  • HTTP path: synthesize() (ChunkedStream) is unchanged and still returns audio.
  • Hermetic tests (local mock WS server) cover the wire protocol and the word_timestamps flag — kept local for now, not included in this PR.
  • ruff format, ruff check, mypy all clean.

…or TTS

The SynthesizeStream previously buffered all tokens locally and sent one
isolated request per segment, so synthesis could not begin until the full
segment text was assembled — adding TTFB latency and ignoring the Waves
continuous-streaming protocol that every comparable LiveKit TTS plugin uses.

Stream tokens to /waves/v1/tts/live as they arrive with `continue: true`,
close each segment with an explicit `flush: true`, and expose the server-side
`max_buffer_flush_ms` buffer bound as a constructor/update_options option.
Send and receive run as concurrent tasks over the pooled connection; the
connection is only returned to the pool after a clean `complete` (the pool
discards it on error/interruption), keeping reuse safe.

One SynthesizeStream instance now maps to one segment, matching the current
base-class contract (extra segments per instance are deprecated upstream).

Verified against the live Waves endpoint: it accepts the continue/flush
protocol and returns audio.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant