feat(smallestai): use Waves continuous WebSocket streaming protocol for TTS#6363
Draft
harshitajain165 wants to merge 1 commit into
Draft
feat(smallestai): use Waves continuous WebSocket streaming protocol for TTS#6363harshitajain165 wants to merge 1 commit into
harshitajain165 wants to merge 1 commit into
Conversation
…or TTS The SynthesizeStream previously buffered all tokens locally and sent one isolated request per segment, so synthesis could not begin until the full segment text was assembled — adding TTFB latency and ignoring the Waves continuous-streaming protocol that every comparable LiveKit TTS plugin uses. Stream tokens to /waves/v1/tts/live as they arrive with `continue: true`, close each segment with an explicit `flush: true`, and expose the server-side `max_buffer_flush_ms` buffer bound as a constructor/update_options option. Send and receive run as concurrent tasks over the pooled connection; the connection is only returned to the pool after a clean `complete` (the pool discards it on error/interruption), keeping reuse safe. One SynthesizeStream instance now maps to one segment, matching the current base-class contract (extra segments per instance are deprecated upstream). Verified against the live Waves endpoint: it accepts the continue/flush protocol and returns audio.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
The SmallestAI
SynthesizeStreamcurrently buffers all incoming tokens locally and sends one isolated request per segment, without using the Waves continuous-streaming protocol (continue/flush/max_buffer_flush_ms) that comparable LiveKit TTS plugins (ElevenLabs, Cartesia, Rime, Neuphonic, Inworld, …) use.This PR switches the plugin to the continuous protocol so it speaks the Waves streaming API the way the official SDK and other providers' plugins do:
/waves/v1/tts/liveas they arrive with"continue": true, instead of buffering the whole segment locally."flush": trueframe.max_buffer_flush_msbuffer bound as aTTS(...)constructor arg andupdate_options(...)option (default0).complete; the pool discards it on error/interruption, so reuse stays safe.SynthesizeStreaminstance = one segment, matching the current base-class contract (multi-segment instances are deprecated upstream).Response-frame handling (
chunk/word_timestamp/complete/error) is unchanged, and the HTTPChunkedStreampath is untouched.Scope / note
This change is about protocol parity — using the continuous-streaming fields the Waves WS API exposes, matching the official SDK and other plugins. It is not a latency change: with current server behavior the endpoint synthesizes on the explicit
flush, so output timing is unchanged from the previous buffer-until-flush implementation. The client is now ready to benefit from any future server-side incremental emission with no further plugin changes.Testing
Verified against the live Waves
/tts/liveendpoint:continue/flush/max_buffer_flush_msare accepted and return audio (lightning_v3.1/magnus).lightning_v3.1_pro/meher,word_timestamps=True), the endpoint returnsword_timestampframes — all words surfaced asTimedStringtranscript entries with sensible timings alongside audio. Theword_timestampsflag on the streamed frames is accepted (no "Invalid input data").synthesize()(ChunkedStream) is unchanged and still returns audio.word_timestampsflag — kept local for now, not included in this PR.ruff format,ruff check,mypyall clean.