Viz migrations are not migrating the query object

[michael-s-molina]

Bug description

Viz migrations only migrate the form_data and not the query object. This is problematic as backend processes such as cache warmup or alerts & reports use the saved query object to generate the queries.

Screenshots/recordings

No response

Superset version

master / latest-dev

Python version

3.9

Node version

16

Browser

Chrome

Additional context

No response

Checklist

• I have searched Superset docs and Slack and didn't find a solution to my problem.
• I have searched the GitHub issue tracker and didn't find a similar bug report.
• I have checked Superset's logs for errors and if I found a relevant Python stacktrace, I included it here as text in the "additional context" section.

[sadpandajoe]

So is this something new with the latest group of visualizations we are deprecating or has this always happened? Just curious as to why we didn't run into this with 4.0.

[michael-s-molina]

So is this something new with the latest group of visualizations we are deprecating or has this always happened? Just curious as to why we didn't run into this with 4.0.

It always happened but it will be more problematic in 5.0 given the number of types migrated.

[mistercrunch]

So what's in the query_object that can't be inferred/determined again from the frontend as needed from the form_data?

[mistercrunch]

Here's a bit of guesswork on qualifying the issue and blast radius, this is important for us to understand whether this is a blocker for Preset migrating to 5.0:

Some things related to this:

• The logic taking a chart configuration and transforming it into a "dimensional query" (aka the query_object) lives on the frontend only. The backend doesn't know how a chart definition computes into a query, that's all in plugins/frontend code
• query_object was added to the chart model at some point to enable some "Alert & Reports"-related feature
  • The alerts & reports "force refresh" or "cache warmup" feature needs the query_object to operate
  • Some chart types that are backend-rendered (as opposed to headless-browser-rendered), like the Table and Pivot Table, where the data is fetched and put into an email as an html table (as opposed to a screenshot), require having the query_object as well
• the may be other features that build upon the assumption that the query_object exists on the backend (???)
• the query_object is schema-free JSON, not versioned nor schema-ed AFAIK
• while we have database migrations that migrate the form_data (chart definition/settings), we don't have some that migrate the query_object
• older instances of Superset may have different structures or straight out missing query_object field (?)

Some potential solutions:

• sandbox/execute the frontend code that provides the query_context from form_data directly in the backend (?)
• write database migrations for query_context
• identify alerts and reports affected, and "disable" them upon migration, have users reenable them manually on the frontend and regenerate the query_context there based on frontend logic
• hack: for out-of-sync query_object, fire up a headless browser async task to fix the broken charts prior to running the alerts and reports. Meaning if we can identify broken query_objects, we could have alert and reports identify them, and call an endpoint /api/v1/frontend/regenerate_query_object, wait for a result, and then fire up the alerts and reports
• ???

[mistercrunch]

Longer term solutions:

• schema and version-stamp query_context object
• implement a node subprocessor or service on the backend?
• for viz types that need a backend-initiated data access, force the definition of backend methods to generate that query
• ?

[mistercrunch]

GPT's summary of my thoughts with a proposal:

GPT Recommended Approach for Superset 5.0 Migration Issue with chart.query_context

Problem

• The chart.query_context field (also known historically as query_object) is generated entirely on the frontend from form_data.
• Backend features like Alerts & Reports, Cache Warmup, and backend-rendered charts (e.g., Table, Pivot Table) rely on this field.
• It is not versioned, not schema-ed, and not migrated across Superset upgrades.
• After upgrading to 5.0, stale or missing chart.query_context can break these features.
• The backend cannot validate or regenerate this field — only the frontend logic knows how.

---

Recommendations

✅ Treat chart.query_context as a frontend-generated cache

• The backend must not assume the field is valid.
• Only the frontend can:
  • Determine if it's out-of-date
  • Regenerate it from form_data
  • Update the database if needed

---

✅ Add a self-healing runtime layer

In backend:

ensure_query_context_and_run_report(
    refresh_cache_only=refresh_cache_only,
    generate_screenshot=generate_screenshot,
    fix_query_context=True,
)

safe_chart = load_chart_from_db_definition()
# At this point, chart.query_context is assumed fresh and correct

• This function should:
  • Launch a headless browser (or Node sandbox)
  • Trigger the frontend to validate and regenerate chart.query_context if needed
  • Save the updated field back to the database
  • Proceed with report, alert, or warmup logic

---

✅ Frontend (TypeScript) logic:

function validateAndMaybeRegenerate(formData, currentQueryContext) {
  const regenerated = buildQueryContext(formData);
  const isDifferent = JSON.stringify(regenerated) !== JSON.stringify(currentQueryContext);
  return isDifferent ? regenerated : currentQueryContext;
}

• This lives entirely in the frontend and is the source of truth for freshness.

---

🧠 Chart Model Enhancement

• Add a new is_query_context_trusted: boolean field to the chart model.
• Set it to false in any DB migration that might invalidate chart.query_context.
• Set it to true when it is regenerated and known to be in sync.
• Use this field to:
  • Identify which charts need regeneration
  • Skip unnecessary regeneration checks in runtime logic

---

🛠️ Trigger points

This regeneration logic should run automatically:

• At alert/report runtime
• During cache warmup jobs
• Only for chart types that need backend-rendered data (e.g., Table, Pivot Table)

---

🧰 Optional Tools

• CLI: superset regenerate-query-contexts --only-alerts
• UI button: “Fix query object” when editing charts
• Metadata: use is_query_context_trusted and optionally a hash or version field to track freshness

---

Net Result: Fully progressive, self-healing system with no disruption to users or need for manual upgrades.

[betodealmeida]

A node service would be useful for other things (like SIP-147). If we had an endpoint that takes form_data and returns query_context we could simply remove query_context from the database and have alerts/reports always use a fresh version.

[mistercrunch]

A node service would be useful

agreed, probably more elegant than my proposal above. It requires a clear entrypoint for each viz_type that may be affected though. I think there's a good convention around that currently in viz plugins

[betodealmeida]

Also, it looks like we just need to set the query_context to NULL, at least for alerts/reports: #15846

[mistercrunch]

Oh wow, so maybe simply a database migration to do something like

UPDATE chart SET query_context = NULL WHERE chart_id IN (SELECT DISTINCT chart_id FROM reports)

Probably as sqlalchemy in a database migration?

@michael-s-molina what do you think?

[michael-s-molina]

Thanks for writing down the problem statement in detail @mistercrunch.

UPDATE chart SET query_context = NULL WHERE chart_id IN (SELECT DISTINCT chart_id FROM reports)

This would not resolve one of the main problems which is cache warmup.

We're almost finishing the following option to unblock 5.0:

write database migrations for query_context

This is not a long term solution and we definitely need something better for the project. @villebro and I were planning on tackling this problem while transforming charts into extensions but this might take a while. We could introduce an endpoint to regenerate the query_context but I don't know how that would be invoked from migrations.

We tested the query_context migration with Python using 50k charts and compared with the Javascript counterpart achieving 100% match. This should be sufficient to unblock 5.0 and give us time to think about a long term solution.

[mistercrunch]

Oh! Great news, I thought your solution was local to your environment, but looks like the DRAFT PR is meant to be contributed back and should work for most/all cases?