Skip to content

Phase 2 (Milestone 1): Durable analytics artifacts#13

Merged
kosminus merged 1 commit into
mainfrom
feat/phase2-durable-artifacts
Jun 8, 2026
Merged

Phase 2 (Milestone 1): Durable analytics artifacts#13
kosminus merged 1 commit into
mainfrom
feat/phase2-durable-artifacts

Conversation

@kosminus

@kosminus kosminus commented Jun 8, 2026

Copy link
Copy Markdown
Owner

Summary

Implements Milestone 1 of Phase 2 from planfull.md — turning one-shot answers into durable, owned, re-runnable objects.

  • Saved queries — named, owned NL question + pinned SQL with typed parameters ({{date_from}}, {{region}}), versioned, clone/fork.
  • Result snapshots + cache — results persisted in a Postgres table that doubles as a cache (keyed by sha256(final_sql + params + connection_id), TTL RESULT_CACHE_TTL_SECONDS default 300 + manual refresh), so re-runs don't re-hit the warehouse.
  • Charts — a persisted chart config per saved query, rendered with Recharts (table/line/bar/area/pie/scatter).
  • Export — client-side CSV/JSON on results; backend CSV/JSON/XLSX for saved queries.

Dashboards (Milestone 2) are intentionally out of scope here.

Backend

  • Models SavedQuery, Chart, ResultSnapshot — connection-scoped, organization_id-keyed, matching the Phase-1 metadata pattern (workspace isolation flows through the connection via require_connection_read/write).
  • Migration 005_durable_artifacts — 3 tables + (sql_hash, taken_at DESC) cache index.
  • saved_query_service.render_sql — type-safe parameter substitution (numbers/booleans inlined, strings/dates escaped + quoted); the rendered SQL still passes through check_sql_safety. Cache-first run_saved_query reuses query_service.execute_raw_sql.
  • Endpoints: CRUD + /run, /clone, /export?format=csv|json|xlsx, and a charts sub-resource. openpyxl added as the optional [export] extra.

Frontend (Recharts added)

  • SavedQueriesPage (list + Run/Edit/Clone/Delete), run drawer (param form + cache badge + chart panel), form modal with a params editor, ChartView, SaveQueryModal.
  • "Save query" + client-side CSV/JSON export on the Query result view; nav item + /saved-queries route.

Notable decisions

  • Result cache = Postgres table (keeps the default self-hosted deploy Redis-free).
  • Charts are managed inside the saved-query view, not a separate top-level page (a chart without a query is meaningless).
  • Artifacts are connection-scoped with organization_id rather than a direct workspace_id — matches the actual Phase-1 implementation rather than the idealized schema in planfull.md §5.

Verification

  • Backend: full pytest suite passes (incl. 15 new render_sql/sql_hash tests); ruff + format clean; mypy clean on new files.
  • Live (Docker + real Postgres + IFRS 9 sample DB): migration 004→005 ran on startup; create → run freshcachedrefresh-bypassdistinct cache key per param verified; chart create, CSV/JSON/XLSX export, and clone all work.
  • Security: a '; DROP TABLE counterparties; -- parameter value was neutralized (escaped to a literal and caught by the static blocklist); the table survived with 20 rows. Bad-type params → 422.
  • Frontend: ESLint clean; tsc -b && vite build succeeds.

🤖 Generated with Claude Code

Turns one-shot answers into durable, owned, re-runnable objects: saved
queries with typed parameters, a Postgres result cache/snapshots, charts,
and CSV/JSON/XLSX export.

Backend:
- Models SavedQuery, Chart, ResultSnapshot (connection-scoped, org-keyed,
  matching the Phase-1 metadata pattern); migration 005 with a
  (sql_hash, taken_at) cache index.
- saved_query_service: type-safe {{param}} rendering, sql_hash, and
  cache-first run_saved_query (reuses execute_raw_sql, so the SQL safety
  blocklist still applies). RESULT_CACHE_TTL_SECONDS config; openpyxl
  [export] extra.
- Endpoints: CRUD + run/clone/export + charts sub-resource.

Frontend (Recharts):
- Saved Queries page, run drawer (param form + cache badge + chart panel),
  form modal with params editor, ChartView, SaveQueryModal.
- Client-side CSV/JSON export + "Save query" on the Query result view;
  nav item + /saved-queries route.

Verified: 161 backend tests pass; ruff/mypy clean on new files; frontend
lint + build pass; live Docker run confirmed migration 004->5, the
fresh/cached/refresh cache lifecycle, charts, export, clone, and that
param injection is neutralized (table survived).

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
@kosminus kosminus merged commit 891b9bf into main Jun 8, 2026
2 checks passed
@kosminus kosminus deleted the feat/phase2-durable-artifacts branch June 8, 2026 09:36
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant