Skip to content

feat(telemetry): trace RPC calls made and handled - #2498

Draft
davidzhao wants to merge 1 commit into
dz/telemetry-startup-shutdownfrom
dz/telemetry-rpc
Draft

davidzhao wants to merge 1 commit into
dz/telemetry-startup-shutdownfrom
dz/telemetry-rpc

Conversation

@davidzhao

@davidzhao davidzhao commented Sep 15, 2026

Copy link
Copy Markdown
Member

Port of livekit/agents#7134. Stacked on #2497.

Description

Neither user-registered RPC methods nor the avatar datastream RPCs appeared in a trace. This installs a tracing interceptor on the job's local participant and turns each RPC into a span.

Span Kind Parent
rpc_call CLIENT the current span, so an RPC issued from a tool nests under function_tool
rpc_handler SERVER the primary session's root span, so it lands on the session timeline; the job's root (job_entrypoint) before or after the session

Attributes: rpc.method, lk.rpc.request_id, lk.rpc.caller_identity, lk.rpc.destination_identity, lk.rpc.payload_size, lk.rpc.response_size, lk.rpc.response_timeout, lk.rpc.error_code, lk.rpc.handler_registered. Request and response bodies are under PII keys lk.pii.rpc.payload / lk.pii.rpc.response, truncated to 1 KiB. RpcError and handler exceptions set error status.

Installed from JobContext.connect() after the room connects, and from RoomIO.start() on every connect for a room connected elsewhere. One interceptor per process; the SDK dedups registrations by identity.

The SDK dispatches incoming invocations from its FFI event path, outside the job's AsyncLocalStorage, so install captures the job (passed explicitly by JobContext.connect() and by RoomIO, which runs on that event path too) and the interceptor runs the handler chain inside it. That is what parents rpc_handler to the session root, and it gives the user's handler getJobContext(), as a Python handler has.

Depends on the SDK

The interceptor hook is LocalParticipant.addRpcInterceptor from livekit/node-sdks#724, not yet in the installed @livekit/rtc-node. Like the Python layer, this degrades to a no-op with one debug log on an SDK without it: the interceptor shapes are declared structurally in telemetry/rpc.ts, install feature-detects the method, and no dependency bump is needed now. Once the SDK ships, bumping @livekit/rtc-node turns tracing on without further code changes.

Note that the FFI layer answers UNSUPPORTED_METHOD for unregistered methods before the SDK's handler runs, so lk.rpc.handler_registered=false is only set on the defensive path where the chain sees that error.

Changes Made

  • telemetry/rpc.ts (new): TracingRpcInterceptor, install; exported as telemetry.rpc.
  • telemetry/traces.ts: StartSpanOptions.kind passed through startSpan / startActiveSpan / startActiveSpanSync.
  • telemetry/trace_types.ts: eleven new attributes (the two lk.pii.rpc.* keys are PII).
  • job.ts, voice/room_io/room_io.ts: install points.

Adaptations from the Python source

  • Python's CancelledError / cancel_reason branch and its tests are not ported: Node promises are not cancelled and the SDK never cancels the handler chain.
  • install(undefined) is tolerated since room.localParticipant is optional in the JS SDK types.

Testing

  • New telemetry/rpc.test.ts (11 tests) with fake continuations: attributes and parenting both directions, truncation and byte sizes, error codes and status, the unregistered flag on the defensive path, handler exceptions, install once / degrade; room_io.test.ts install-on-connect case.
  • Full agents suite green; build, typecheck, lint, API report updated.

🤖 Generated with Claude Code

@changeset-bot

changeset-bot Bot commented Sep 15, 2026

Copy link
Copy Markdown

🦋 Changeset detected

Latest commit: 87f7817

The changes in this PR will be included in the next version bump.

This PR includes changesets to release 39 packages
Name Type
@livekit/agents Patch
@livekit/agents-plugin-anam Patch
@livekit/agents-plugin-anthropic Patch
@livekit/agents-plugin-assemblyai Patch
@livekit/agents-plugin-azure Patch
@livekit/agents-plugin-baseten Patch
@livekit/agents-plugin-bey Patch
@livekit/agents-plugin-cartesia Patch
@livekit/agents-plugin-cerebras Patch
@livekit/agents-plugin-deepgram Patch
@livekit/agents-plugin-did Patch
@livekit/agents-plugin-elevenlabs Patch
@livekit/agents-plugin-fishaudio Patch
@livekit/agents-plugin-google Patch
@livekit/agents-plugin-hume Patch
@livekit/agents-plugin-inworld Patch
@livekit/agents-plugin-krisp Patch
@livekit/agents-plugin-lemonslice Patch
@livekit/agents-plugin-liveavatar Patch
@livekit/agents-plugin-livekit Patch
@livekit/agents-plugin-meta Patch
@livekit/agents-plugin-minimax Patch
@livekit/agents-plugin-mistral Patch
@livekit/agents-plugin-mistralai Patch
@livekit/agents-plugin-neuphonic Patch
@livekit/agents-plugin-openai Patch
@livekit/agents-plugin-perplexity Patch
@livekit/agents-plugin-phonic Patch
@livekit/agents-plugin-protoface Patch
@livekit/agents-plugin-resemble Patch
@livekit/agents-plugin-rime Patch
@livekit/agents-plugin-runway Patch
@livekit/agents-plugin-sarvam Patch
@livekit/agents-plugin-silero Patch
@livekit/agents-plugin-soniox Patch
@livekit/agents-plugin-tavus Patch
@livekit/agents-plugins-test Patch
@livekit/agents-plugin-trugen Patch
@livekit/agents-plugin-xai Patch

Not sure what this means? Click here to learn what changesets are.

Click here if you're a maintainer who wants to add another changeset to this PR

Port of livekit/agents#7134. `telemetry.rpc` installs one tracing
interceptor on the job's local participant and turns each RPC into a
span following the OpenTelemetry RPC semantic conventions: `rpc_call`
(CLIENT, under the current span, so an RPC issued from a tool nests
under `function_tool`) and `rpc_handler` (SERVER, under the primary
session's root span). Attributes cover method, ids, identities, payload
and response sizes, the response timeout in seconds, the RpcError code
and whether a handler was registered; request and response bodies are
recorded truncated to 1 KiB under `lk.pii.rpc.*` keys.

Installed from JobContext.connect() after the room connects and from
RoomIO on every connected transition; both are idempotent, the SDK
dedups the singleton interceptor by identity. `StartSpanOptions` gains
`kind`.

Adaptation: the installed @livekit/rtc-node predates
LocalParticipant.addRpcInterceptor (it is in an open SDK PR), so the
interceptor and call shapes are declared structurally here and `install`
feature-detects the hook, degrading to one debug log and a no-op like
the Python layer on an older livekit-rtc. Not ported: the Python
handling of asyncio cancellation of the handler chain (promises are not
cancelled in Node; the SDK maps timeouts on the caller's side).

Incoming invocations are dispatched by the SDK from its FFI event path,
outside the job's AsyncLocalStorage, so the session and job roots did not
resolve there and rpc_handler spans came out as roots of their own (seen in
a live run). install() now captures the job (passed explicitly by
JobContext.connect() and RoomIO, which itself runs on the SDK's event path)
and the interceptor runs the handler chain inside it, which also gives the
user's handler getJobContext(), as a Python handler has.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant