docs(cookbook): instrument an existing app and verify it end to end by abhijaisrivastava15 · Pull Request #786 · future-agi/docs

abhijaisrivastava15 · 2026-08-05T06:42:14Z

What

A cookbook quickstart that takes an app with no tracing at all to a verified integration, and then proves it worked rather than asking the reader to eyeball the dashboard.

Today the closest thing we have is observe/troubleshooting/no-traces-appearing, which diagnoses a symptom. Nothing tells you your integration is correct: a trace that arrives looking fine can still be missing cost, sessions, users, or half its spans.

Page added

src/pages/docs/cookbook/quickstart/instrument-and-verify.mdx
src/lib/navigation.ts - one entry, first in Cookbooks > Quickstart > Observability, because the five siblings there all assume tracing already works

Asset added

public/fi_verify.py (246 lines) - serves at docs.futureagi.com/fi_verify.py, which the Install section curls

It is in this PR rather than its own so the page never ships a live link to a 404. The page publishes the file's line count and sha256 (9b331742...) so a reader can prove their copy in one command.

The worked example

Chintan's note was that a cookbook is only finished when it shows an agent end to end. The page now does, on a repository nobody wrote for us.

openai/openai-agents-python, its examples/customer_service airline support agent: a triage agent, a handoff, two specialist agents and a local tool. It is a fair target because it has all four of the page's problems at once. It emits no trace we can read, one message fans out across four agents, main.py is an interactive REPL so there is no entry point that runs once, and it already carries a conversation id and a passenger context, so Step 3 has a real session and a real user rather than invented ones.

The six steps were then followed literally against it. No business logic edited, main.py imported not modified, one new file for the entry point the repository does not have.

  PASS  G1   preflight ok, delivery ok
  PASS  G2   project_name='support-agent-quickstart' project_type='observe'
  PASS  G3   13 spans, 1 trace(s), 1 root(s), 0 orphan(s)
  PASS  G4   3 LLM span(s), 0 untyped
  PASS  G5   prompt and completion on every LLM span
  PASS  G6   session.id=['conv_94ac3e74fcc4']
  PASS  G7   user.id=['acct_10427']
  PASS  G8   model on every LLM span
  PASS  G9   tokens and cost on LLM spans, rolled up onto the root; root cost=0.00108604
  PASS  G10  no credential in any span attribute

  Future AGI integrated
  GREEN LIGHT achieved

That is a real run against api.futureagi.com with a real model call, real tokens and a real cost, not a transcript written by hand.

Screenshots

Three, from that run, on the platform. Every one is the genuine surface with nothing reconstructed.

The trace list reads the root and nothing else, which is why Step 4 puts the totals there. Latency, tokens and status arrive on their own; cost and the model do not.

The FutureAGI trace list showing two support.turn traces with input, output, latency, tokens, total cost and model

Inside a trace, the instrumentor typed the agents, the model calls and the handoff. faq_lookup_tool is a local function the instrumentor cannot see, so its TOOL span is the three lines from Step 4. The attributes panel is the same list Step 4 sends, read back off a real span.

The trace tree for one support turn, from the support.turn root through the triage agent, the handoff, the FAQ agent, the tool span and three LLM spans

Because session.id was set once at the edge, both turns group without either turn knowing about the other.

The FutureAGI sessions view showing one session with its first and last message, duration, total cost, three traces and the user id

This supersedes the earlier note on this PR that said there was nothing to screenshot. That was true of the tutorial alone and stopped being true once the worked example gave us a real trace to point at.

Notebook

quickstart/instrument-and-verify.ipynb is now on cookbook/quickstart-notebooks in future-agi/cookbooks, the 22nd in that folder, and the page carries the Colab and GitHub badge row like its 21 siblings. Both badge URLs return 200.

This also supersedes the earlier note saying there would be no badge row.

Shape

Written to the section's own skeleton, checked against all 34 existing quickstarts rather than one of them:

## Install (28/34), ## Tutorial with <Steps>/<Step> (33/34), ## What you built with <Check> (34/34), ## Next steps with <CardGroup cols={4}> (33/34)
<TLDR>, <Prerequisites>, the badge row (21/34), and the | Time | Difficulty | Package | table, all present
Frontmatter is title + description only, as every sibling has

One deliberate deviation: <Tabs> for the two integration routes (Future AGI SDK and plain OpenTelemetry). distributed-tracing sets the precedent for tabs on a multi-variant page, and the alternative is two pages for one procedure.

Tested by an agent that had never seen it

A fresh agent was given the page URL, real keys and a sandbox, and told to reach GREEN LIGHT or stop honestly. It reached all ten gates on the first real run, in about seven minutes against the page's own estimate of twenty. It also reached green on the printed directory layout with no __init__.py files at all, which confirms the tree as printed is correct.

It found four things, all fixed in this branch:

The tabs are a trap for anything that flattens the page to text. Steps 2 to 4 present the two tracks as tab-switched UI, so flattened they concatenate with no marker: setup.py twice, futureagi_rollup.py twice, and futureagi_spans.py twice as structurally different files under the same name. Mixing them is a guaranteed ImportError. Our own "Copy page as Markdown for LLMs" button and the docs MCP server both invite exactly that flattening. Every listing now names its track on the first line, and one sentence up front says to pick one track and stay on it.

The worked example described its one new file without printing it. It is now printed in full, including the line that points the agents SDK at an OpenAI-compatible endpoint that is not OpenAI.

The install list omitted -openai-agents, which is the instrumentor the worked example installs.

An undocumented stderr line on every run (ai-evaluation is not installed, please install it to trace protect) now has one sentence saying it is the optional Protect package and affects no gate.

Checks run

node scripts/audit-links.mjs - 0 broken nav links, 0 broken content links, in nav so not an orphan
Renders 200 on a local dev server off this branch; TLDR, Prerequisites, Steps, Step, Check, Tabs, Note, Warning and CardGroup all render as components, none leak as text
All three images serve 200 from the branch
Both notebook badge URLs return 200
The rendered page flattened to plain text keeps all six track markers, so the two tracks stay separable without the tab UI
Zero em-dashes, zero HTML entities
Every internal link target verified to exist on dev

Preview

npm run dev

http://localhost:4321/docs/cookbook/quickstart/instrument-and-verify

Follow-up, not in this PR

observe/troubleshooting/no-traces-appearing currently ends at "recheck the causes above in order". A one-line link out to this page would close that dead end. Kept separate to keep this diff to one page.

Adds a cookbook quickstart that takes an app with no tracing to a verified integration, covering both the Future AGI SDK and plain OpenTelemetry routes, and ships fi_verify.py as a downloadable asset so the ten gates can be run against the spans an app really produced.

Install, Tutorial with Steps, What you built with Check, Next steps, matching the 34 sibling quickstarts rather than the one that deviates from them, and restoring the done-condition and the blocked-report format the first draft dropped.

…boundary Time and difficulty were out of family: 45 min was the longest value on any of the 35 quickstarts and this page is half the length of the one that says 20. The Python prerequisite read as a constraint on the reader's app; only fi_verify.attach() is Python-bound, so the not-Python section now says what to do without it.

Runs the six steps against openai/openai-agents-python's customer service agent and shows the trace list, the trace tree and the session from that run. Name the track inside every listing so the two tracks stay separable when the page is flattened to text, print the entry point the worked example adds, and link the Colab notebook.

Resolve navigation.ts: dev restructured the cookbook nav into collapsible groups while this branch added a single entry. Took dev's structure and placed the entry first in Tracing and Debugging, ahead of Manual Tracing, because the siblings in that group assume tracing already works.

/docs/tracing/auto moved to /docs/integrations/traceai. The three references on this page were written before the move and now fail the link audit.

khushalsonawat

Check comments

khushalsonawat · 2026-08-26T07:13:04Z

@@ -0,0 +1,668 @@
+---
+title: "Instrument an Existing App and Verify It End to End"


Full frontmatter like every sibling ships now: slug, date, author, products, frameworks, difficulty, time-to-complete, tags, canonical, code-repo-url, page-type

khushalsonawat · 2026-08-26T07:13:04Z

+</TLDR>
+
+<div style={{display: "flex", gap: "8px", flexWrap: "wrap", margin: "0.5rem 0 1rem"}}>
+<a href="https://colab.research.google.com/github/future-agi/cookbooks/blob/cookbook/quickstart-notebooks/quickstart/instrument-and-verify.ipynb" target="_blank" style={{display: "inline-flex"}}><img src="https://colab.research.google.com/assets/colab-badge.svg" alt="Open in Colab" style={{height: "28px"}} /></a>


Point both badges at blob/main, the notebook is already there. The quickstart-notebooks branch is due for deletion

khushalsonawat · 2026-08-26T07:13:04Z

+
+| Time | Difficulty | Package |
+|------|-----------|---------|
+| 20 min | Beginner | `fi-instrumentation-otel` |


Beginner doesn't fit, SpanProcessors and contextvars make this intermediate

khushalsonawat · 2026-08-26T07:13:04Z

+| 20 min | Beginner | `fi-instrumentation-otel` |
+
+<Prerequisites>
+- FutureAGI account → [app.futureagi.com](https://app.futureagi.com)


Future AGI, two words. 14 on this page counting tab titles and alt text

khushalsonawat · 2026-08-26T07:13:04Z

+- FutureAGI account → [app.futureagi.com](https://app.futureagi.com)
+- API keys: `FI_API_KEY` and `FI_SECRET_KEY` (see [Get your API keys](/docs/admin-settings))
+- An app that makes at least one real LLM call, with an entry point you can run once
+- Python 3.10+ to run `fi_verify.py`. The listings below are Python, but the app you are tracing can be in any language: see [If your app is not Python](#if-your-app-is-not-python)


3.11 is the floor for cookbooks, here and in the install comments

khushalsonawat · 2026-08-26T07:13:04Z

+
+Future AGI is integrated, and you can show that rather than assert it.
+
+## Next steps


Closer is one forward link, no card grid or Next steps heading. And there's no Troubleshooting section, all 70 cookbooks ship one with 5+ recipe-specific rows

khushalsonawat · 2026-08-26T07:13:04Z

            title: 'Tracing & Debugging',
            collapsible: true,
            items: [
+              { title: 'Instrument and Verify', href: '/docs/cookbook/quickstart/instrument-and-verify' },


Sidebar item and page title should match. And add the card to platform/index.mdx, bump the hub's Tracing & Debugging count

khushalsonawat · 2026-08-26T07:13:04Z

+    python fi_verify.py preflight    closes G1, before any code is touched
+    python fi_verify.py check        all ten gates, after one real request
+
+Python 3.10+. No dependency beyond the OpenTelemetry SDK.


3.11 floor here too

khushalsonawat · 2026-08-26T07:13:04Z

+        out.append(WHY.get(code, "unexpected status; the body above is the collector's."))
+        put("preflight", False, {"http": code, "body": body})
+        return False, out
+    for label, k, kw in (("wrong key   ", key[:-4] + "0000", {}),


A real key ending 0000 makes the wrong-key control the real key, and preflight false-fails

khushalsonawat · 2026-08-26T07:13:04Z

+         "model on every LLM span" if not nomodel else "missing on " + str(nomodel[:3])),
+        # Cost is what the trace list is read for and the one most often missing. It is not
+        # computed for you, and a zero is not a cost, so a zero does not pass.
+        ("G9", bool(counted and rolled and isinstance(cost, (int, float)) and cost > 0),


This needs a call: requiring user-sent cost > 0 fails an integration that leans on platform pricing. If platform pricing is the mainline, G9 softens and the roll-up becomes the unknown-model path

The collector prices any span carrying a model and non-zero tokens, so requiring a user-sent root cost failed an integration that leans on platform pricing. G9 now asks that every LLM span is priceable or carries its own cost, and names the double count when a root total sits on top. Also: a real key ending 0000 made the wrong-key control the real key, so preflight false-failed. Python floor is 3.11.

Cost: the collector prices LLM spans from the model and tokens (LiteLLM table, then per-org rates) and a sent value wins. Names which surface reads the root and which sum every span, so the roll-up is now the unpriced-model path rather than the mainline. Shape: full frontmatter, Future AGI as two words, 3.11 floor, badges on blob/main, Troubleshooting table, one forward link instead of the card grid, image captions and borderRadius, hub card and count.

All three are now uniform 3024x1890 full-window dark, taken off the same run in the platform. The trace list is filtered to traces that carried tokens, so the failed first attempt and the preflight probe are out of frame, and the cost annotation now says what is actually true: our table keys this model as groq/llama-3.3-70b-versatile, so nothing prices it and the number on the root is the one the roll-up put there.

abhijaisrivastava15 requested a review from khushalsonawat August 5, 2026 06:42

abhijaisrivastava15 added 3 commits August 5, 2026 12:21

abhijaisrivastava15 marked this pull request as ready for review August 5, 2026 09:12

abhijaisrivastava15 added 2 commits August 24, 2026 13:08

docs(cookbook): point instrumentor links at the traceAI reference

0ea85b1

/docs/tracing/auto moved to /docs/integrations/traceai. The three references on this page were written before the move and now fail the link audit.

khushalsonawat requested changes Aug 26, 2026

View reviewed changes

abhijaisrivastava15 added 4 commits August 26, 2026 15:42

docs(cookbook): refresh the printed fi_verify.py checksum

94ef0cc

Navigation Menu

Uh oh!

docs(cookbook): instrument an existing app and verify it end to end - #786

docs(cookbook): instrument an existing app and verify it end to end#786
abhijaisrivastava15 wants to merge 10 commits into
devfrom
cookbook/instrument-and-verify

abhijaisrivastava15 commented Aug 5, 2026 •

edited

Loading

Uh oh!

khushalsonawat left a comment

Uh oh!

khushalsonawat Aug 26, 2026

Uh oh!

khushalsonawat Aug 26, 2026

Uh oh!

khushalsonawat Aug 26, 2026

Uh oh!

khushalsonawat Aug 26, 2026

Uh oh!

khushalsonawat Aug 26, 2026

Uh oh!

khushalsonawat Aug 26, 2026

Uh oh!

khushalsonawat Aug 26, 2026

Uh oh!

khushalsonawat Aug 26, 2026

Uh oh!

khushalsonawat Aug 26, 2026

Uh oh!

khushalsonawat Aug 26, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

		@@ -0,0 +1,668 @@
		---
		title: "Instrument an Existing App and Verify It End to End"


		Future AGI is integrated, and you can show that rather than assert it.

		## Next steps

Uh oh!

Conversation

abhijaisrivastava15 commented Aug 5, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

What

Page added

Asset added

The worked example

Screenshots

Notebook

Shape

Tested by an agent that had never seen it

Checks run

Preview

Follow-up, not in this PR

Uh oh!

khushalsonawat left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

abhijaisrivastava15 commented Aug 5, 2026 •

edited

Loading