feat(langchain-agents)!: emit invoke_agent, chat and execute_tool spans by apucacao · Pull Request #35 · launchdarkly/python-ai-sdk

apucacao · 2026-08-11T20:13:29Z

Replaces one flat span per call with the tree the TypeScript SDK emits, for langchain-agents. This is the last of the six handlers.

invoke_agent                     one per call, carries the LD identity and the run total
├── chat {model}                 one per model turn, with that turn's own tokens
└── execute_tool {name}          one per tool call, a sibling of chat

The per-turn data was already being summed from each message's usage_metadata; it now drives a span per turn.

The same two provider-attribute fixes as #34

gen_ai.system is now the literal string langchain rather than the configured provider name: the key names the instrumentation, and here that is the framework.

gen_ai.provider.name is new and is a binary choice rather than a passthrough. It names who served the model, so it follows the client actually instantiated: anthropic when the config says so, openai for everything else, including Bedrock, Azure and an unset value. That mirrors the handler's own model resolution.

Other changes

Cached tokens are read from usage_metadata.input_token_details and reported per turn, not added to the input figure, which LangChain already reports inclusive of them.
Finish reasons go through the shared LangChain helper rather than being dropped, so a turn that stopped to call a tool is distinguishable from one that finished.
The streaming path gets a finally, so a consumer that stops reading no longer leaves the root span unended and unexported.

The graph span is untouched. ld.ai.graph and its two attributes already matched the TypeScript SDK and are out of scope.

Breaking change

The span is renamed from langchain.agent to invoke_agent. Queries selecting on the old name will not match. gen_ai.system changes value, as above. Prompt and completion content is no longer on spans unless the caller passes capture_content=True.

Where this sits

Needs the usage layer (#28) and the content layer (#29). Independent of the other five handler PRs; the stack orders them only because gh stack is linear. The oracle above (#36) needs all six.

Tests: 824 to 844.

Note

Overview
Replaces one flat langchain.agent span per call with the same invoke_agent → chat / execute_tool tree the TypeScript SDK emits. Span logic moves into spans.py, driven by LangChain callbacks on ainvoke and astream.

The root invoke_agent span holds LaunchDarkly identity, feature_flag, and run-level token totals; each model turn gets chat {model}; each tool call gets execute_tool {name} as a sibling of chat under the root. gen_ai.system is langchain; gen_ai.provider.name is anthropic or openai (not a passthrough of the config name). Finish reasons and cache token details are reported per turn; usage reconciles message usage_metadata with callback llm_output.token_usage.

capture_content defaults off; prompts and completions appear on spans only when enabled. Streaming adds finally cleanup so early consumer exit ends and exports spans (launchdarkly.stream.abandoned, not ERROR). langchain_agents() now passes capture_content into the handler factory instead of config().

Breaking: queries on langchain.agent or old gen_ai.system values need updating; span content is opt-in.

^{Reviewed by Cursor Bugbot for commit 4862dc9. Bugbot is set up for automated code reviews on this repo. Configure here.}

apucacao · 2026-08-11T20:20:46Z

bugbot run

apucacao · 2026-08-11T20:44:59Z

bugbot run

apucacao · 2026-08-11T21:02:22Z

bugbot run

apucacao · 2026-08-11T21:19:34Z

bugbot run

apucacao · 2026-08-11T21:46:46Z

bugbot run

apucacao · 2026-08-12T17:23:34Z

bugbot run

cursor

✅ Bugbot reviewed your changes and found no new issues!

Comment @cursor review or bugbot run to trigger another review on this PR

^{Reviewed by Cursor Bugbot for commit c643fe3. Configure here.}

apucacao · 2026-08-12T17:46:40Z

bugbot run

cursor

✅ Bugbot reviewed your changes and found no new issues!

Comment @cursor review or bugbot run to trigger another review on this PR

^{Reviewed by Cursor Bugbot for commit 5d7d073. Configure here.}

apucacao · 2026-08-12T18:03:38Z

bugbot run

cursor

✅ Bugbot reviewed your changes and found no new issues!

Comment @cursor review or bugbot run to trigger another review on this PR

^{Reviewed by Cursor Bugbot for commit a2e14f3. Configure here.}

apucacao · 2026-08-12T19:22:26Z

bugbot run

cursor

✅ Bugbot reviewed your changes and found no new issues!

Comment @cursor review or bugbot run to trigger another review on this PR

^{Reviewed by Cursor Bugbot for commit a2e14f3. Configure here.}

apucacao · 2026-08-13T00:34:24Z

bugbot run

cursor

✅ Bugbot reviewed your changes and found no new issues!

Comment @cursor review or bugbot run to trigger another review on this PR

^{Reviewed by Cursor Bugbot for commit 93d24bc. Configure here.}

apucacao · 2026-08-13T00:56:48Z

bugbot run

cursor

✅ Bugbot reviewed your changes and found no new issues!

Comment @cursor review or bugbot run to trigger another review on this PR

^{Reviewed by Cursor Bugbot for commit abe5db7. Configure here.}

apucacao · 2026-08-13T01:10:03Z

bugbot run

cursor

✅ Bugbot reviewed your changes and found no new issues!

Comment @cursor review or bugbot run to trigger another review on this PR

^{Reviewed by Cursor Bugbot for commit 4c46e9f. Configure here.}

One flat span named langchain.agent becomes the tree the TypeScript SDK emits: an invoke_agent root, one `chat {model}` child per model turn, one `execute_tool {name}` child per tool call. The per-turn data was already being summed from each message's usage_metadata; it now drives a span per turn. BREAKING CHANGE: the span this handler emits is renamed from `langchain.agent` and `langchain.agent.stream` to `invoke_agent`. Queries selecting on the old names will not match. Prompt and completion content is no longer on spans unless the caller passes capture_content=True. gen_ai.system changes value; see below. gen_ai.system is now the literal string langchain rather than the configured provider name, matching the TypeScript SDK: the key names the instrumentation, and here that is the framework. gen_ai.provider.name is new and is a binary choice rather than a passthrough. It names who served the model, so it follows the client actually instantiated: anthropic when the config says so, openai for everything else, including Bedrock, Azure and an unset value. That mirrors the handler's own model resolution. Cached tokens are now read from usage_metadata.input_token_details and reported per turn. They are not added to the input figure, which LangChain already reports inclusive of them. Finish reasons go through the shared LangChain helper rather than being dropped, so a turn that stopped to call a tool is distinguishable from one that finished. The streaming path gets a finally, so a consumer that stops reading no longer leaves the root span unended and unexported. The graph span is untouched. ld.ai.graph and its two attributes already matched the TypeScript SDK and are out of scope for this change. Tests: 54 to 74.

…oes not fail them The abandonment path reused close_open_spans, which records a synthetic exception and sets ERROR on every span still open, so an early consumer stop was indistinguishable from a provider failure in a trace. The comment three lines above it already claimed the opposite. Adds abandon_open_spans, mirroring the method openai-agents already had: every open span ends through end_span_once, staying UNSET and carrying launchdarkly.stream.abandoned. The failure path keeps its behaviour. Tested on the callback handler directly rather than through the streaming path. Reaching the state that matters, a chat or tool span still open at the break, needs a fake model that yields mid-turn, and with the fixtures here LangGraph has already run every callback by the time the first chunk reaches the consumer. My first attempt went through stream() and passed whether or not the fix was present, which is worse than no test. The test file says so, so the next person does not repeat it. Found by Bugbot on #35.

…rapper The wrapper never passed capture_content to the factory, so it stayed in kwargs and reached config(), which takes no such argument. A caller asking for content on spans got a TypeError rather than content. Lifted out alongside variables, which was already handled the same way and for the same reason: one configures the handler, the other belongs to the invocation, and config() accepts neither. Two tests, one per branch, asserting the flag reaches the factory and does not reach config(). Found by Bugbot on #33 against openai-agents. Five of the six wrappers had it; each is fixed in its own layer.

…nreachable Two mirror-image leaks in the callback handler, both reachable through content serialisation, which raises on any tool argument or result that is not JSON-serialisable. The end callbacks popped the span before doing that work. After the pop nothing else can reach it, so close_open_spans could not recover it and the span was never ended: the exporter never saw the turn or the tool call at all. Both now end it on the way out. on_tool_start had the reverse problem: it created the span, wrote the arguments, and only then inserted it into the tracking dict. A raise in between left a span no cleanup path knew about. It is now tracked first, so every later path can still close it. Two tests, each failing on the exact leak when the fix is reverted. The tracer patch has to stay active while the callbacks run rather than only while they are built, which is what my first attempt got wrong. Found by Bugbot on #35.

…t can raise _start_model created the span, wrote the conversation onto it, and only then inserted it into the tracking dict. Serialising conversation content raises on anything that is not JSON-serialisable, and a span created but never inserted is unreachable by close_open_spans, abandon_open_spans and the end callbacks alike: it never ends, so the exporter never sees it. on_tool_start already had this fix. The model-start path is the mirror of it and did not. Found by Bugbot on #35.

…r reads Span construction moved to spans.py, which holds the real _HAS_OTEL. The handler kept its own copy, plus the two imports it needed, alive only by a noqa. Nothing read any of it. That mattered because the tests patched the dead one. 7 tests set handler._HAS_OTEL to False and believed they were exercising the install without the otel extra; the flag was unread, so they exercised nothing and passed either way. They now patch spans._HAS_OTEL, which is the flag start_root_span actually consults: with it patched, span creation returns None, and with it set it does not. Found by Bugbot on #32. Five of the six handlers carried the dead gate, and four had tests aimed at it.

extract_llm_usage read the llm_output fallback with `or`, so a genuine 0 was skipped in favour of the next key. With both counts at zero the bag came back all None, lang_chain_span_usage read that as the provider having said nothing, and the run went unreported: a turn that completed and cost nothing became indistinguishable from one that never reported, which is the distinction the reported flag exists to preserve. Now keyed on presence rather than truthiness. Three tests: a zero prompt count survives, both-zero still counts as reported, and a genuinely absent count is still absent. Found by Bugbot on #35.

…wn spans The success path sums usage_metadata off each message. The callbacks read the whole LLMResult and fall back to llm_output.token_usage, which some providers use instead. For those providers the chat spans carried real tokens while the successful run's root, and the bag handed back to the caller, both stayed at zero: a config-scoped cost query undercounted completed work, and the two figures in one trace contradicted each other. The message-level sum stays authoritative wherever it has anything to say, so a provider that reports in both places cannot be counted twice. Only when it saw nothing at all do the callbacks stand in, because then they are the only record of what the run cost. The docstring claimed both sides computed the same numbers. They did not, and it now says which fields each one reads. Two tests: llm_output-only usage reaches the root, and usage_metadata still wins when both are present. Found by Bugbot on #35.

…rite fails on_llm_end accumulated the turn's usage after the content write. A raise while serialising completion content dropped a turn the provider had already billed, and that accumulator is what a failed run's root reports and what a successful run falls back to when the messages carry no usage of their own. The accumulation now happens before the write, matching what the other five handlers do. Found by auditing every handler for the ordering Bugbot reported on #30 and #34.

The prompt write ran before the try that fails the root, so a raise while serialising it left the root open: never ended, never exported, so the run disappeared from AI Config Monitoring along with the feature_flag event it carries. Both paths had it. Two tests, one per path. Found by Bugbot on #34, which is this shape in langchain-messages. All five handlers that had it are fixed in their own layers.

apucacao · 2026-08-13T01:38:29Z

bugbot run

cursor

✅ Bugbot reviewed your changes and found no new issues!

Comment @cursor review or bugbot run to trigger another review on this PR

^{Reviewed by Cursor Bugbot for commit 4862dc9. Configure here.}

apucacao mentioned this pull request Aug 11, 2026

test: fail the build when the six handlers drift apart again #36

Draft

cursor Bot reviewed Aug 11, 2026

View reviewed changes

Comment thread packages/langchain-agents/src/launchdarkly_ai_langchain_agents/handler.py Outdated

apucacao force-pushed the ag/py-telemetry-langchain-agents branch from 082eaaf to 9ccef53 Compare August 11, 2026 20:43

cursor Bot reviewed Aug 11, 2026

View reviewed changes

Comment thread packages/langchain-agents/src/launchdarkly_ai_langchain_agents/spans.py Outdated

apucacao force-pushed the ag/py-telemetry-langchain-agents branch from 9ccef53 to 326b8a6 Compare August 11, 2026 21:01

cursor Bot reviewed Aug 11, 2026

View reviewed changes

Comment thread packages/langchain-agents/src/launchdarkly_ai_langchain_agents/spans.py Outdated

apucacao force-pushed the ag/py-telemetry-langchain-agents branch from 326b8a6 to 65fc5f8 Compare August 11, 2026 21:18

cursor Bot reviewed Aug 11, 2026

View reviewed changes

Comment thread packages/langchain-agents/src/launchdarkly_ai_langchain_agents/spans.py Outdated

cursor Bot reviewed Aug 11, 2026

View reviewed changes

Comment thread packages/langchain-agents/src/launchdarkly_ai_langchain_agents/spans.py

apucacao force-pushed the ag/py-telemetry-langchain-agents branch from 65fc5f8 to c643fe3 Compare August 12, 2026 17:23

cursor Bot reviewed Aug 12, 2026

View reviewed changes

apucacao force-pushed the ag/py-telemetry-langchain-agents branch from c643fe3 to 5d7d073 Compare August 12, 2026 17:45

cursor Bot reviewed Aug 12, 2026

View reviewed changes

apucacao force-pushed the ag/py-telemetry-langchain-agents branch from 5d7d073 to 0ec7a76 Compare August 12, 2026 18:02

cursor Bot reviewed Aug 12, 2026

View reviewed changes

Comment thread packages/langchain-agents/src/launchdarkly_ai_langchain_agents/spans.py

apucacao force-pushed the ag/py-telemetry-langchain-agents branch from 0ec7a76 to dfedf11 Compare August 12, 2026 18:07

cursor Bot reviewed Aug 12, 2026

View reviewed changes

apucacao force-pushed the ag/py-telemetry-langchain-agents branch from a2e14f3 to 93d24bc Compare August 13, 2026 00:33

cursor Bot reviewed Aug 13, 2026

View reviewed changes

apucacao force-pushed the ag/py-telemetry-langchain-agents branch from 93d24bc to abe5db7 Compare August 13, 2026 00:55

cursor Bot reviewed Aug 13, 2026

View reviewed changes

apucacao force-pushed the ag/py-telemetry-langchain-agents branch 2 times, most recently from abe5db7 to 4c46e9f Compare August 13, 2026 01:09

cursor Bot reviewed Aug 13, 2026

View reviewed changes

apucacao added 10 commits August 12, 2026 21:36

apucacao force-pushed the ag/py-telemetry-langchain-agents branch from 4c46e9f to 4862dc9 Compare August 13, 2026 01:37

cursor Bot reviewed Aug 13, 2026

View reviewed changes

Uh oh!

Conversation

apucacao commented Aug 11, 2026 • edited by cursor Bot Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

The same two provider-attribute fixes as #34

Other changes

Breaking change

Where this sits

Uh oh!

apucacao commented Aug 11, 2026

Uh oh!

Uh oh!

apucacao commented Aug 11, 2026

Uh oh!

Uh oh!

apucacao commented Aug 11, 2026

Uh oh!

Uh oh!

apucacao commented Aug 11, 2026

Uh oh!

Uh oh!

apucacao commented Aug 11, 2026

Uh oh!

Uh oh!

apucacao commented Aug 12, 2026

Uh oh!

cursor Bot left a comment

Choose a reason for hiding this comment

Uh oh!

apucacao commented Aug 12, 2026

Uh oh!

cursor Bot left a comment

Choose a reason for hiding this comment

Uh oh!

apucacao commented Aug 12, 2026

Uh oh!

Uh oh!

cursor Bot left a comment

Choose a reason for hiding this comment

Uh oh!

apucacao commented Aug 12, 2026

Uh oh!

cursor Bot left a comment

Choose a reason for hiding this comment

Uh oh!

apucacao commented Aug 13, 2026

Uh oh!

cursor Bot left a comment

Choose a reason for hiding this comment

Uh oh!

apucacao commented Aug 13, 2026

Uh oh!

cursor Bot left a comment

Choose a reason for hiding this comment

Uh oh!

apucacao commented Aug 13, 2026

Uh oh!

cursor Bot left a comment

Choose a reason for hiding this comment

Uh oh!

apucacao commented Aug 13, 2026

Uh oh!

cursor Bot left a comment

Choose a reason for hiding this comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

1 participant

apucacao commented Aug 11, 2026 •

edited by cursor Bot

Loading