Skip to content

feat(etl): Wave 3A — hot-path routes migrate to marts (projects, dashboard-data, cost-data)#76

Merged
0bserver07 merged 4 commits into
mainfrom
feat/etl-routes-hot-path
May 5, 2026
Merged

feat(etl): Wave 3A — hot-path routes migrate to marts (projects, dashboard-data, cost-data)#76
0bserver07 merged 4 commits into
mainfrom
feat/etl-routes-hot-path

Conversation

@0bserver07
Copy link
Copy Markdown
Owner

Summary

Routes migrated

  • GET /api/projects?include_stats=true — reads project_mart for the per-project totals; bulk SQL helpers (PR perf(projects): bulk SQL replaces N+1 stats loop on /api/projects (26s → 3s cold) #65) stay as the empty-mart fallback.
  • GET /api/dashboard-dataproject_mart drives statistics.overview, daily_mart drives statistics.daily_stats + statistics.models. Tools/errors/hourly_pattern/sessions/user_interactions return shape-stable empties when the mart drives the response — those blocks already load lazily via /api/cost-data, /api/commands, /api/tool-distribution.
  • GET /api/cost-datatoken_composition.daily/totals come from daily_mart when materialised. Per-session / per-command / per-tool detail blocks (session_costs, command_costs, tool_costs, outliers, retry_signals, session_efficiency, error_cost, trends) stay aggregator-driven — they need lower-grain marts deferred to Wave 4.
  • GET /api/cost-data/by-provider — switches to provider_day_mart when populated; messages-table sweep stays as the empty-mart fallback.

Routes deferred to Wave 3B

  • /api/compare, /api/yield, /api/optimize, /api/context-budget — owned by Wave 3B per the spec.
  • /api/messages, /api/messages/summary, /api/sessions, /api/jsonl-files — need raw rows the marts don't carry.

What's new under the hood

  • stackunderflow/store/mart_queries.py — read helpers for project_mart, daily_mart, provider_day_mart. Empty-mart safe (returns []/None) so callers can gate on mart_has_project_row().
  • Three routes acquire a mart_has_project_row(...) gate: when the project is materialised the route serves the response from marts; when it isn't, the existing aggregator path runs unchanged.

Tests

  • 13 new tests in tests/stackunderflow/routes/test_*_uses_mart.py covering: mart-driven happy path, empty-mart fallback, multi-provider duplicate merge, provider/model filter pass-through, parity (empty store → mart populated), and < 100ms speed regression at 100K synthetic mart rows.
  • All existing route tests (test_dashboard_cache.py, test_cost.py, test_providers_filter.py, test_cost_by_provider.py, …) keep passing — they monkeypatch queries.get_project_stats which the empty-mart fallback path still calls, so the contract is preserved.
  • Suite total: 1472 passing, 2 skipped (was 1459 + 2 before this PR — +13 from the new mart tests).

Test plan

  • pytest tests/ -q clean (1472 passed, 2 skipped)
  • ruff check stackunderflow/routes/ stackunderflow/store/mart_queries.py clean
  • Speed regression: synthetic 100K mart rows → all three routes < 100ms warm
  • Existing route tests (test_dashboard_cache.py, test_cost.py, test_providers_filter.py, test_cost_by_provider.py) keep passing — JSON contract held
  • End-to-end smoke test on the user's real 28K-message project once Wave 4 backfill ships and populates the marts (manual, post-merge)

Constraints honoured

0bserver07 added 4 commits May 5, 2026 00:48
…aily_mart

Routes prefer materialised mart rows when present, falling back to the
existing aggregator path when project_mart is empty. Same JSON shape
either way — the data source swaps, the contract holds.

* projects.py: project_mart → ProjectStats UI shape; bulk SQL helpers
  stay as the fallback for projects not yet in the mart.
* data.py: project_mart + daily_mart → dashboard statistics block.
  Tools/errors/hourly_pattern/sessions/user_interactions return
  shape-stable empties when marts drive the response — those blocks
  already load lazily via /api/cost-data, /api/commands, etc.
* store/mart_queries.py: read helpers for project_mart, daily_mart,
  provider_day_mart with provider/model filter parity to the existing
  Annotated[list[str] | None, Query()] pattern.

Cost route migration arrives in the next commit.
…ds provider_day_mart

When project_mart has a row, /api/cost-data overlays the
token_composition.daily/totals blocks with daily_mart-derived values.
Per-session/per-command/per-tool detail blocks (session_costs,
command_costs, tool_costs, outliers, retry_signals, session_efficiency,
error_cost, trends) stay aggregator-driven — they need lower-grain marts
that ship in Wave 4.

/api/cost-data/by-provider switches to provider_day_mart when populated;
the messages-table rollup stays as the empty-mart fallback. Same JSON
contract either way.
…hetic rows

13 new tests covering:

* projects: mart-driven stats path, fallback, multi-provider duplicate
  merge, < 100ms speed at 100K daily_mart rows
* dashboard-data: overview from project_mart, daily_stats from
  daily_mart, models from daily_mart GROUP BY model, < 100ms speed
* cost-data: token_composition.daily/totals overlay,
  no-overlay-when-empty fallback parity
* cost-data/by-provider: mart fast-path, ?provider= filter narrowing,
  messages-table fallback when mart empty, < 100ms speed at 100K
  provider_day_mart rows
@0bserver07 0bserver07 merged commit 05b1959 into main May 5, 2026
9 checks passed
@0bserver07 0bserver07 deleted the feat/etl-routes-hot-path branch May 5, 2026 04:52
0bserver07 added a commit that referenced this pull request May 6, 2026
Bumps to 0.7.0. Consolidates the [Unreleased] CHANGELOG entries from
the 11 ETL PRs (#72, #73, #74, #75, #76, #79, #81, #80, #78, #77, #82)
into a single [0.7.0] section.

New: docs/HANDOFF.md — state-of-the-codebase walkthrough for incoming
agents. Architecture map, recent history, key gotchas, what's left,
files-to-read-first.

End-state on the maintainer's real store:
  150,337 usage_events
  Marts populated and watermarks in sync
  Dashboard cold-load 2.5s → <50ms warm
  Watcher 155ms end-to-end source-file-write → dashboard-data-fresh

1598 backend tests passing, 2 skipped, 11 deselected (slow suite).
Frontend typecheck + build clean.

Co-authored-by: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant