Skip to content

Create Bronze-to-Silver Pipeline Template for BigQuery #1220

@terzioglub

Description

@terzioglub

Create a new Bruin template that demonstrates a bronze-to-silver data pipeline pattern using BigQuery as the destination. This template will use either frankfurter or chess as the data source (since they don't require credentials) and showcase a complete ingestr -> transformation workflow.
This template should be able to initialized with bruin init template-name
more about bruin init on : https://getbruin.com/docs/bruin/commands/init.html

Template Structure

Bronze Layer (Ingestr Asset):

  • Raw data ingestion using ingestr from frankfurter or chess API
  • No transformations, just data extraction to BigQuery

Silver Layer (SQL or Python Asset):

  • Aggregates and cleans bronze data from BigQuery tables
  • Demonstrates typical transformation patterns

Implementation

Create templates/bronze-silver-bigquery/ with:

  • .bruin.yml with BigQuery connection
  • README.md with setup instructions
  • pipeline.yml
  • Bronze asset: assets/bronze_raw_data.asset.yml (ingestr from frankfurter/chess)
  • Silver asset: assets/silver_aggregated.sql (BigQuery SQL transformation)

Acceptance Criteria

  • Template created in templates/bronze-silver-bigquery/
  • Bronze layer uses ingestr asset to extract data from frankfurter or chess source (no credentials needed)
  • Silver layer aggregates bronze data with BigQuery SQL transformations
  • Documented with clear setup and usage instructions in templates documentations
  • Can be initialized with bruin init bronze-silver-bigquery
  • bruin validate passes successfully
  • Assets have meaningful quality checks (column checks, custom checks)
  • bruin run executes the full pipeline successfully

Reference Templates

Metadata

Metadata

Assignees

No one assigned

    Labels

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions