Skip to content

Refactor(schema): cache find lookups in schema#7615

Merged
georgesittas merged 1 commit intomainfrom
jo/avoid_unnecessary_schema_lookups
May 6, 2026
Merged

Refactor(schema): cache find lookups in schema#7615
georgesittas merged 1 commit intomainfrom
jo/avoid_unnecessary_schema_lookups

Conversation

@georgesittas
Copy link
Copy Markdown
Collaborator

@georgesittas georgesittas commented May 6, 2026

Benched this using annotate_types on a query with 2,400 columns references across 8 tables (8x60 cols, each referenced 5x).

before (main): 35.9ms median, 33.7ms min
after (cache): 33.3ms median, 32.0ms min (~7% faster)

The gain is modest in wall-time because the trie walk itself is cheap. The main benefit is eliminating 2,400 redundant base find() calls down to the number of unique (table, ensure_data_types) pairs.

@georgesittas georgesittas force-pushed the jo/avoid_unnecessary_schema_lookups branch from 6b1abfc to d25b627 Compare May 6, 2026 16:44
@georgesittas georgesittas merged commit a2a9ba5 into main May 6, 2026
8 checks passed
@georgesittas georgesittas deleted the jo/avoid_unnecessary_schema_lookups branch May 6, 2026 16:46
@github-actions
Copy link
Copy Markdown
Contributor

github-actions Bot commented May 6, 2026

SQLGlot Integration Test Results

Comparing:

  • this branch (sqlglot:jo/avoid_unnecessary_schema_lookups, sqlglot version: jo/avoid_unnecessary_schema_lookups)
  • baseline (main, sqlglot version: 0.0.1.dev1)

By Dialect

dialect main sqlglot:jo/avoid_unnecessary_schema_lookups transitions links
bigquery -> bigquery 24647/24652 passed (100.0%) 23497/23497 passed (100.0%) No change full result / delta
bigquery -> duckdb 867/1154 passed (75.1%) 0/0 passed (0.0%) Results not found full result / delta
duckdb -> duckdb 5823/5823 passed (100.0%) 0/0 passed (0.0%) Results not found full result / delta
snowflake -> duckdb 1129/1935 passed (58.3%) 0/0 passed (0.0%) Results not found full result / delta
snowflake -> snowflake 65133/65133 passed (100.0%) 63027/63027 passed (100.0%) No change full result / delta
databricks -> databricks 1370/1370 passed (100.0%) 1370/1370 passed (100.0%) No change full result / delta
postgres -> postgres 6042/6042 passed (100.0%) 6042/6042 passed (100.0%) No change full result / delta
redshift -> redshift 7101/7101 passed (100.0%) 7101/7101 passed (100.0%) No change full result / delta

Overall

main: 113210 total, 112112 passed (pass rate: 99.0%), sqlglot version: 0.0.1.dev1

sqlglot:jo/avoid_unnecessary_schema_lookups: 101037 total, 101037 passed (pass rate: 100.0%), sqlglot version: jo/avoid_unnecessary_schema_lookups

Transitions:
No change

Dialect pair changes: 0 previous results not found, 3 current results not found

✅ 43 test(s) passed

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants