[WIP] feat(usage): local usage analytics — ledger, RPC, settings page#4449
[WIP] feat(usage): local usage analytics — ledger, RPC, settings page#4449t3dotgg wants to merge 1 commit into
Conversation
… RPC, settings page Records token/cost usage facts from live provider events into a replayable projection, exposes a dashboard-shaped RPC, and adds Settings > Usage. Server: - New thread.usage-recorded orchestration event (internal thread.usage.record command) carrying immutable per-model interval facts with provider-native session identity; emitted from runtime ingestion for Codex cumulative token-usage updates and Claude turn.completed modelUsage settlements (exact cost in integer micro-USD; stale completions recorded, flagged). - Registered projection.usage projector -> projection_usage_facts (fact_id upsert, migration 033). Deliberately does NOT prune on thread.reverted (billing history is immutable); thread deletion redacts linkage. - Cumulative->delta baselines keyed by native session + model, reseeded from fact sums after restart to prevent double counting. - ClaudeAdapter: providerRefs now carry the SDK session id. CodexAdapter: snapshots now include cumulative total* fields; native thread id attached to usage events (separates concurrent subagent sessions on one thread). - usage.getSummary RPC (orchestration:read scope): totals, by-model, daily, hour-of-week, by-project buckets grouped in the caller's IANA timezone. - Pricing: versioned effective-dated list-price catalog in shared runtime code; estimates computed at query time only, never stored, always flagged. Unpriced models surface as token-only, never $0. Web: - Settings > Usage per the approved mock: harness-level twin donuts (tokens + cost, model shades within family, striped estimate slices), KPI tiles, tokens-per-day stacks, hour x weekday heatmap, models table, project bars. Tests: pure accounting (delta/reset/normalization), projector (idempotent replay, revert retention, deletion redaction, session-model sums), adapter fixtures updated for the new snapshot fields. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
|
Important Review skippedAuto reviews are disabled on this repository. Please check the settings in the CodeRabbit UI or the ⚙️ Run configurationConfiguration used: Repository UI Review profile: CHILL Plan: Pro Plus Run ID: You can disable this status message by setting the Use the checkbox below for a quick retry:
✨ Finishing Touches🧪 Generate unit tests (beta)
Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out. Comment |
github-actions
Bot
added
vouch:trusted
![macroscopeapp[bot]](https://avatars.githubusercontent.com/in/900172?s=60&v=4){kind=small}
| const reset = | ||
| cumulative.inputTokens < baseline.inputTokens || | ||
| cumulative.cachedInputTokens < baseline.cachedInputTokens || | ||
| cumulative.outputTokens < baseline.outputTokens; |
There was a problem hiding this comment.
🟡 Medium orchestration/usageAccounting.ts:44
usageDelta fails to detect provider session resets when only cacheCreationTokens, reasoningOutputTokens, or costMicroUsd decrease. The reset guard checks only inputTokens, cachedInputTokens, and outputTokens, so a regression in any of the other three counters is treated as a normal interval: the delta is clamped to zero and the lower cumulative value becomes the next baseline, silently dropping the reset usage and mis-deltaing later increments. Include all six cumulative counters in the reset condition.
const reset =
cumulative.inputTokens < baseline.inputTokens ||
cumulative.cachedInputTokens < baseline.cachedInputTokens ||
- cumulative.outputTokens < baseline.outputTokens;
+ cumulative.outputTokens < baseline.outputTokens ||
+ cumulative.cacheCreationTokens < baseline.cacheCreationTokens ||
+ cumulative.reasoningOutputTokens < baseline.reasoningOutputTokens ||
+ cumulative.costMicroUsd < baseline.costMicroUsd;🚀 Reply "fix it for me" or copy this AI Prompt for your agent:
In file @apps/server/src/orchestration/usageAccounting.ts around lines 44-47:
`usageDelta` fails to detect provider session resets when only `cacheCreationTokens`, `reasoningOutputTokens`, or `costMicroUsd` decrease. The `reset` guard checks only `inputTokens`, `cachedInputTokens`, and `outputTokens`, so a regression in any of the other three counters is treated as a normal interval: the delta is clamped to zero and the lower cumulative value becomes the next baseline, silently dropping the reset usage and mis-deltaing later increments. Include all six cumulative counters in the `reset` condition.
| /** Strip provider capacity suffixes (`claude-fable-5[1m]` → `claude-fable-5`) | ||
| * while keeping the raw id available for display and pricing overrides. */ | ||
| export function canonicalModelId(rawModel: string): string { | ||
| return rawModel.replace(/\[[^\]]+\]$/, ""); |
There was a problem hiding this comment.
🟡 Medium orchestration/usageAccounting.ts:85
canonicalModelId strips provider capacity suffixes, so raw keys like claude-fable-5 and claude-fable-5[1m] collapse to the same canonical ID. When both variants appear in the same modelUsage map, the second entry is deltaed against the first variant's already-recorded baseline counters and both facts get the same factId, producing incorrect deltas and one fact overwriting the other during projection. Consider aggregating all raw entries sharing a canonical ID before deltaing, or retaining the raw id in baseline and fact IDs.
🚀 Reply "fix it for me" or copy this AI Prompt for your agent:
In file @apps/server/src/orchestration/usageAccounting.ts around line 85:
`canonicalModelId` strips provider capacity suffixes, so raw keys like `claude-fable-5` and `claude-fable-5[1m]` collapse to the same canonical ID. When both variants appear in the same `modelUsage` map, the second entry is deltaed against the first variant's already-recorded baseline counters and both facts get the same `factId`, producing incorrect deltas and one fact overwriting the other during projection. Consider aggregating all raw entries sharing a canonical ID before deltaing, or retaining the raw id in baseline and fact IDs.
| [claude, CLAUDE_COLOR], | ||
| [codex, CODEX_COLOR], | ||
| ] as const) { | ||
| group.forEach((bucket, index) => { |
There was a problem hiding this comment.
🟡 Medium settings/UsageSettings.tsx:355
A UsageModelBucket with both exactCostMicroUsd > 0 and estimatedCostMicroUsd > 0 is rendered as a fully solid slice in the cost donut, because estimated is set only when exactCostMicroUsd === 0. The portion of that slice's value that comes from estimatedCostMicroUsd is not striped, so estimated spend appears provider-metered — contradicting the chart legend that says striped slices are list-price estimates. Consider splitting mixed buckets into separate exact and estimated slices so the striped overlay covers only the estimated portion.
🚀 Reply "fix it for me" or copy this AI Prompt for your agent:
In file @apps/web/src/components/settings/UsageSettings.tsx around line 355:
A `UsageModelBucket` with both `exactCostMicroUsd > 0` and `estimatedCostMicroUsd > 0` is rendered as a fully solid slice in the cost donut, because `estimated` is set only when `exactCostMicroUsd === 0`. The portion of that slice's value that comes from `estimatedCostMicroUsd` is not striped, so estimated spend appears provider-metered — contradicting the chart legend that says striped slices are list-price estimates. Consider splitting mixed buckets into separate exact and estimated slices so the striped overlay covers only the estimated portion.
| /** Exclusive ISO date-time upper bound; omit for now. */ | ||
| until: Schema.optional(IsoDateTime), | ||
| /** IANA timezone used for calendar bucketing (daily, hour-of-week). */ | ||
| timeZone: TrimmedNonEmptyString, |
There was a problem hiding this comment.
🟡 Medium src/usage.ts:92
UsageSummaryRequest.timeZone is typed as TrimmedNonEmptyString, so a request like { timeZone: "not-a-zone" } passes RPC decoding. The handler then passes that string to new Intl.DateTimeFormat, which throws RangeError; Effect.orDie converts the throw into a defect, so the summary RPC dies instead of returning a typed validation error. Consider validating timeZone as an IANA timezone (e.g. via a refinement that checks Intl.supportedValuesOf('timeZone')) so invalid input is rejected at the schema layer.
Also found in 1 other location(s)
packages/contracts/src/rpc.ts:309
WsUsageGetSummaryRpcacceptsUsageSummaryRequestwithout validatingtimeZoneas an IANA timezone (the field is onlyTrimmedNonEmptyString). A request such as{ timeZone: "not-a-zone" }passes RPC decoding, thensummarizeUsageFactsconstructsIntl.DateTimeFormatwith it, which throwsRangeError; the handler converts this to a defect viaEffect.orDie, so the new summary RPC fails instead of returning a typed client error.
🚀 Reply "fix it for me" or copy this AI Prompt for your agent:
In file @packages/contracts/src/usage.ts around line 92:
`UsageSummaryRequest.timeZone` is typed as `TrimmedNonEmptyString`, so a request like `{ timeZone: "not-a-zone" }` passes RPC decoding. The handler then passes that string to `new Intl.DateTimeFormat`, which throws `RangeError`; `Effect.orDie` converts the throw into a defect, so the summary RPC dies instead of returning a typed validation error. Consider validating `timeZone` as an IANA timezone (e.g. via a refinement that checks `Intl.supportedValuesOf('timeZone')`) so invalid input is rejected at the schema layer.
Also found in 1 other location(s):
- packages/contracts/src/rpc.ts:309 -- `WsUsageGetSummaryRpc` accepts `UsageSummaryRequest` without validating `timeZone` as an IANA timezone (the field is only `TrimmedNonEmptyString`). A request such as `{ timeZone: "not-a-zone" }` passes RPC decoding, then `summarizeUsageFacts` constructs `Intl.DateTimeFormat` with it, which throws `RangeError`; the handler converts this to a defect via `Effect.orDie`, so the new summary RPC fails instead of returning a typed client error.
![cursor[bot]](https://avatars.githubusercontent.com/in/1210556?s=60&v=4){kind=small}
There was a problem hiding this comment.
Cursor Bugbot has reviewed your changes using high effort and found 7 potential issues.
❌ Bugbot Autofix is OFF. To automatically fix reported issues with cloud agents, enable autofix in the Cursor dashboard.
Reviewed by Cursor Bugbot for commit ee46b61. Configure here.
| projectId: thread.projectId, | ||
| turnId: toTurnId(event.turnId) ?? null, | ||
| facts: [fact], | ||
| }); |
There was a problem hiding this comment.
Baseline updates before ledger commit
High Severity
The recordUsageForRuntimeEvent function updates in-memory usage baselines before successfully dispatching usage facts for persistence. If dispatch fails, the in-memory baseline becomes inconsistent with the persisted data. This causes subsequent usage to be under-counted, and after a service restart, re-seeding from the stale persisted data can lead to double-counting.
Reviewed by Cursor Bugbot for commit ee46b61. Configure here.
| return; | ||
| } | ||
| seededUsageSessions.add(providerSessionId); | ||
| const sums = yield* projectionUsageRepository.sumBySessionModel({ providerSessionId }); |
There was a problem hiding this comment.
Session seed marks before DB read
High Severity
The seedUsageBaselines function marks a session as seeded before successfully loading its baseline from the database. If the database query fails, the session's baseline remains uninitialized, causing subsequent usage deltas to be incorrectly calculated and potentially double-counted.
Reviewed by Cursor Bugbot for commit ee46b61. Configure here.
| reasoningOutputTokens: nonNegative( | ||
| cumulative.reasoningOutputTokens - baseline.reasoningOutputTokens, | ||
| ), | ||
| costMicroUsd: nonNegative(cumulative.costMicroUsd - baseline.costMicroUsd), |
There was a problem hiding this comment.
Partial counter reset detection
Medium Severity
The usageDelta function's reset condition is too narrow. It only checks for regressions in inputTokens, cachedInputTokens, and outputTokens. Regressions in cacheCreationTokens, reasoningOutputTokens, or costMicroUsd are not treated as a reset; instead, their negative deltas are clamped to zero by nonNegative, silently dropping usage and cost data.
Reviewed by Cursor Bugbot for commit ee46b61. Configure here.
| }; | ||
| } | ||
| return {}; | ||
| return sessionRef; |
There was a problem hiding this comment.
Claude completion missing session id
Medium Severity
When Claude emits turn.completed without an active turnState, providerRefs is {}, so usage ingestion falls back to the T3 thread.id as providerSessionId. Distinct SDK sessions on one thread share one baseline key and can conflate or mis-attribute cumulative modelUsage.
Additional Locations (1)
Reviewed by Cursor Bugbot for commit ee46b61. Configure here.
| } | ||
|
|
||
| function calendarParts(formatter: Intl.DateTimeFormat, iso: string): CalendarParts { | ||
| const parts = formatter.formatToParts(Date.parse(iso)); |
There was a problem hiding this comment.
Invalid ISO breaks calendar buckets
Low Severity
summarizeUsageFacts buckets facts with calendarParts, which uses Date.parse(iso) without validating the result. A malformed observedAt yields Invalid Date and can mis-assign daily stacks and heatmap cells instead of treating the timestamp as missing.
Triggered by learned rule: Use parseTimestampDate for ISO string parsing, return null for invalid
Reviewed by Cursor Bugbot for commit ee46b61. Configure here.
| /> | ||
| ); | ||
| })} | ||
| </> |
There was a problem hiding this comment.
Heatmap list missing React key
Low Severity
HourHeatmap maps weekday rows with a bare fragment (<>...</>) as the outer element, so React has no stable key per row. That triggers list warnings and can cause incorrect reconciliation when the grid re-renders.
Reviewed by Cursor Bugbot for commit ee46b61. Configure here.
| continue; | ||
| } | ||
| facts.push({ | ||
| factId: UsageFactId.make(`usage:${event.eventId}:${model}`), |
There was a problem hiding this comment.
Duplicate fact ids same model
Medium Severity
Claude turn.completed facts use factId derived from the canonical model id, not the raw map key. Two modelUsage entries that canonicalize to the same model (e.g. capacity suffix variants) collide on upsert and the last row wins, dropping tokens and cost from the other entry.
Reviewed by Cursor Bugbot for commit ee46b61. Configure here.
ApprovabilityVerdict: Needs human review 4 blocking correctness issues found. Diff is too large for automated approval analysis. A human reviewer should evaluate this PR. You can customize Macroscope's approvability policy. Learn more. |
1 participant
Note
WIP — intentionally not a draft so it gets real reviews. Core accounting is built and tested; the deferred list below is scoped follow-up work, not unfinished scaffolding.
What this is
Local, private usage analytics for T3 Code (think ccusage, but first-class): tokens, exact + estimated cost, time-of-day patterns, per-model/per-project breakdowns — recorded in each environment's own sqlite, displayed at Settings → Usage.
Plan doc (v2, independently reviewed by gpt-5.6-sol with findings folded in):
docs/t3-code-usage-plan.htmlin this PR. Approved UI mock built from ~1 month of real usage data: https://postplan.dev/d/5bx21h42g0b9/rawArchitecture (the part worth reviewing hardest)
Durable facts, replayable projector — not an in-memory fold.
ProviderRuntimeIngestionconverts provider-cumulative counters into interval usage facts and dispatches an internalthread.usage.recordcommand →thread.usage-recordedorchestration event (payload keeps provider-native session identity, per-model tokens, exact cost in integer micro-USD).projection.usageprojector (standardprojection_statecursor, replay-on-bootstrap) upserts facts by deterministicfact_idintoprojection_usage_facts(migration 033).usage.getSummaryRPC (read scope) aggregates facts into dashboard buckets, grouped in the caller's IANA timezone; list-price estimates are computed at query time from a versioned catalog (packages/shared/src/usagePricing.ts) and never stored or blended with exact cost.Accounting invariants encoded and tested:
providerSessionIdnow ridesproviderRefs): concurrent Codex subagent sessions on one T3 thread can't conflate cumulative counters (this inflated naive replay 12× in validation).sumBySessionModel), so a restart between samples never double counts.thread.reverted(deliberately no prune case — tokens were consumed); thread deletion redacts linkage but keeps accounting.stale=1), never dropped.modelUsagesettles per-modelfinalfacts with exactcostUSDdeltas; CodextokenUsage.totalflows asintervalfacts (new cumulativetotal*fields onThreadTokenUsageSnapshot).UI
Settings → Usage per the approved mock: twin harness-level donuts (tokens + cost; model shades within harness family; striped slices = estimated dollars), KPI tiles, tokens-per-day stacks, hour×weekday heatmap, models table (accessible fallback for every chart), project bars. Unpriced models surface as tokens-only, never $0.
Tests
usageAccounting.test.ts— delta/reset/normalization pure functions (9 tests)ProjectionPipeline.usage.test.ts— idempotent replay, revert retention, deletion redaction, session-model sums (6 tests)Explicitly deferred (per plan, not forgotten)
t3 usageCLI (thin client of the same RPC)Review focus requests
ProviderRuntimeIngestion.ts—usageBaselines/seedUsageBaselines): sound under concurrent sessions and the drainable-worker model?Math.roundon deltas — acceptable drift vs reconciling againstturn-totalfacts?usage.getSummaryloads all facts in range then aggregates in JS — fine at current volumes (~thousands of facts), but push down to SQL now or later?thread.usage-recordedvs a hypothetical dedicated aggregate.🤖 Generated with Claude Code
Note
Medium Risk
Touches billing-adjacent ingestion (delta baselines, restart reseed) and new durable ledger semantics; mistakes could under/over-count usage, though failures are logged and heavily tested.
Overview
Introduces local usage analytics: provider runtime events are turned into interval/final usage facts, persisted via orchestration (
thread.usage.record→thread.usage-recorded) and a newprojection.usageprojector intoprojection_usage_facts(migration 033).Ingestion in
ProviderRuntimeIngestiondeltas cumulative Codex token snapshots and Claude per-modelturn.completedusage against baselines keyed byproviderSessionId+ model, reseeding baselines from stored sums after restart; stale lifecycle-rejected completions are still recorded. Adapters now attachproviderSessionId(Claude resume session, Codex native thread id) and Codex snapshots includetotal*cumulative fields.Query path:
usage.getSummaryloads facts, resolves project titles, and aggregates inusageSummary.ts(timezone buckets, exact vs list-price estimates at read time). UI: Settings → Usage with donuts, daily bars, heatmap, and model/project tables.Facts survive thread revert; thread deletion redacts thread/project linkage only. Tests cover accounting helpers, projection behavior, and adapter fixtures.
Reviewed by Cursor Bugbot for commit ee46b61. Bugbot is set up for automated code reviews on this repo. Configure here.
Note
Add local usage analytics with ledger, RPC endpoint, and settings page
projection_usage_factstable (migration 033) andProjectionUsageRepositoryto persist per-fact token/cost data, with upsert, redact, list, and sum-by-session operations.thread.usage.recordcommand andthread.usage-recordedevent to the orchestration contract, handled by a new decider branch and projection pipeline projector.ProviderRuntimeIngestion) now derives usage deltas from cumulative Claude and Codex telemetry and dispatches usage facts, seeding baselines from persisted sums on restart to avoid double-counting; stale facts are flagged on rejected turn completions.usage.getSummaryWebSocket RPC that aggregates facts into totals, per-model, daily, hour-of-week, and per-project buckets, usingestimateCostMicroUsdfrom the new@t3tools/shared/usagePricingmodule for models lacking exact cost data./settings/usageroute renderingUsageSettingsPanelwith donut charts, daily bar stacks, a 7×24 hour heatmap, and per-model/project tables.📊 Macroscope summarized ee46b61. 24 files reviewed, 0 issues evaluated, 0 issues filtered, 0 comments posted
🗂️ Filtered Issues
No issues evaluated.