Create a new Bruin template that demonstrates a bronze-to-silver data pipeline pattern using BigQuery as the destination. This template will use either frankfurter or chess as the data source (since they don't require credentials) and showcase a complete ingestr -> transformation workflow.
This template should be able to initialized with bruin init template-name
more about bruin init on : https://getbruin.com/docs/bruin/commands/init.html
Template Structure
Bronze Layer (Ingestr Asset):
- Raw data ingestion using ingestr from frankfurter or chess API
- No transformations, just data extraction to BigQuery
Silver Layer (SQL or Python Asset):
- Aggregates and cleans bronze data from BigQuery tables
- Demonstrates typical transformation patterns
Implementation
Create templates/bronze-silver-bigquery/ with:
.bruin.yml with BigQuery connection
README.md with setup instructions
pipeline.yml
- Bronze asset:
assets/bronze_raw_data.asset.yml (ingestr from frankfurter/chess)
- Silver asset:
assets/silver_aggregated.sql (BigQuery SQL transformation)
Acceptance Criteria
Reference Templates
Create a new Bruin template that demonstrates a bronze-to-silver data pipeline pattern using BigQuery as the destination. This template will use either frankfurter or chess as the data source (since they don't require credentials) and showcase a complete ingestr -> transformation workflow.
This template should be able to initialized with
bruin init template-namemore about bruin init on : https://getbruin.com/docs/bruin/commands/init.html
Template Structure
Bronze Layer (Ingestr Asset):
Silver Layer (SQL or Python Asset):
Implementation
Create
templates/bronze-silver-bigquery/with:.bruin.ymlwith BigQuery connectionREADME.mdwith setup instructionspipeline.ymlassets/bronze_raw_data.asset.yml(ingestr from frankfurter/chess)assets/silver_aggregated.sql(BigQuery SQL transformation)Acceptance Criteria
templates/bronze-silver-bigquery/bruin init bronze-silver-bigquerybruin validatepasses successfullybruin runexecutes the full pipeline successfullyReference Templates