[SIP] Server-side Query Context Generation for Charts

[betodealmeida]

[SIP] Server-side Query Context Generation for Charts

Motivation

Today, it's impossible to create a chart programmatically via an API call and fetch chart data without first opening the chart in a browser. The reason is because new charts depend on query_context being generated from form_data by frontend JavaScript (buildQueryContext / chart plugin buildQuery logic). This creates a gap for backend-only and headless workflows:

• API-created charts may not have query_context persisted yet.
• Background jobs (eg alerts/reports) cannot always regenerate query_context without running browser JS.
• Integrations that use Superset as a pure backend API have an implicit browser dependency.

We need a backend-native way to generate/re-generate query_context from form_data without requiring a headless browser. In that the past we have implemented partial solutions to this problem, for example:

• feat: store query context when saving charts #15824
• feat: call screenshot to store query_context #15846
• fix: Update Query Context on Explore loading #15865
• fix: skip perms on query context update #16250
• fix: allow reports to update query_context #16303

Note that this would unblock #30816.

Proposed Change

Introduce an optional internal Node.js service that executes the same JS buildQueryContext logic used by Superset frontend plugins, and integrate it with Superset backend as a private dependency.

High-level behavior:

1. Private sidecar endpoint (not public)

   • A Node.js service exposes an internal endpoint:
     • POST /api/v1/build-query-context with { form_data: ... }
   • It returns generated query_context using the same chart plugin buildQuery logic.

2. Lazy server-side hydration of query_context

   • When Superset backend needs chart.query_context:
     • If present in DB: use it (current behavior).
     • If null/missing: check QUERY_CONTEXT_SIDECAR_URL.
       • If QUERY_CONTEXT_SIDECAR_URL is null/unset: raise an error (same behavior as today).
       • If QUERY_CONTEXT_SIDECAR_URL is set: call internal Node service, generate query_context, persist it in DB, then proceed.
   • This enables API-created charts to work without prior browser interaction.

3. Programmatic refresh support

   • Add (or leverage existing) chart API support for force=true semantics:
     • regenerate query_context from current form_data even if one already exists in DB.
   • This supports background refresh jobs and consistency after automated edits.

4. Security / exposure model

   • The Node service is not intended as a public API.
   • It should only be reachable by Superset backend over internal network.
   • No user-facing/public ingress route should be created for this endpoint in OSS default deployment.
   • Even if exposed, the sidecar simply implements logic that is purely functional and open-source.

5. Operational safeguards

   • Request body size limit.
   • Timeout and bounded error logging.
   • Optional origin checks / internal allowlist controls.

New or Changed Public Interfaces

Potential API/interface changes:

1. Chart-related API behavior

   • Existing chart data/read workflows may now auto-hydrate missing query_context server-side.
   • This is a behavior enhancement for API-only workflows.

2. Optional new parameter

   • Add force=true (or equivalent) to relevant chart API(s) to refresh stored query_context.
   • Exact endpoint(s) to be finalized in implementation discussion.

3. Configuration

   • New backend config for internal sidecar URL and timeout, eg:
     • QUERY_CONTEXT_SIDECAR_URL
     • request timeout/body limit settings
   • QUERY_CONTEXT_SIDECAR_URL is optional and defaults to null/unset.
   • Behavior with missing query_context:
     • sidecar URL unset: current error behavior
     • sidecar URL set: attempt server-side generation

4. No new public endpoint in OSS default

   • The Node endpoint is internal only and not documented as a public Superset API.

New dependencies

• Node.js runtime service to execute existing frontend query-context logic server-side.
• Reuse existing Superset frontend/plugin JS packages as build-time/runtime dependency of that service.
• No new product-facing browser dependency is introduced.

Migration Plan and Compatibility

• No required DB migration expected.
• Backward compatible:
  • Existing charts with persisted query_context continue to work unchanged.
  • Existing API behavior remains valid; this proposal fills missing-query-context cases.
• Rollout strategy:
  • No feature flag.
  • Optional configuration-only rollout through QUERY_CONTEXT_SIDECAR_URL.
  • Fallback behavior if sidecar unavailable should be explicit (error vs retry policy).
• Refresh semantics:
  • force=true should be opt-in and non-breaking.
• Stored dashboards/charts/bookmarks should remain compatible.

Rejected Alternatives

1. Require headless browser in backend jobs

   • Operationally heavy and brittle.
   • Hard to scale and maintain.
   • Couples backend jobs to frontend runtime environment.

2. Rewrite JS buildQuery logic in Python

   • Large duplication risk.
   • High long-term maintenance burden as chart plugins evolve.

3. Publicly expose a query-context build endpoint

   • Increases attack surface.
   • Not needed for intended OSS behavior; internal-only service is sufficient.

4. Do nothing (keep browser-first generation only)

   • Continues to block API-first and background-job workflows.
   • Keeps implicit frontend dependency in backend use cases.

[betodealmeida]

cc: @villebro