Recently, we added a dataform run --dry-run github action to all of our PRs on our repo containing all of our dataform sqlx files. For simple changes, this works nicely and provides a great CI check before merging to main.
However, recently we have found a few times where this check has failed when adding columns and then using them across 2 dependent views, e.g.
Existing on BigQuery:
SELECT viewA.foo, table2.bar FROM viewA INNER JOIN table2 USING ...
Then the PR adds baz to both views
SELECT foo, baz FROM table
(dry run action passes here)
SELECT viewA.foo, viewA.baz, table2.bar FROM viewA INNER JOIN table2 USING ...
(dry run action fails as viewA.baz does not exist)
Maybe it is our lack of understanding on how we would anticipate this to work, but it seems far from ideal that changes like this do not pass the check we have added.
Recently, we added a
dataform run --dry-rungithub action to all of our PRs on our repo containing all of our dataform sqlx files. For simple changes, this works nicely and provides a great CI check before merging to main.However, recently we have found a few times where this check has failed when adding columns and then using them across 2 dependent views, e.g.
Existing on BigQuery:
Then the PR adds baz to both views
(dry run action passes here)
(dry run action fails as
viewA.bazdoes not exist)Maybe it is our lack of understanding on how we would anticipate this to work, but it seems far from ideal that changes like this do not pass the check we have added.