🚀 Project : End-to-End Data Pipeline on GCP

🔗 GitHub: https://github.com/dhananjay93/data-engineering-end-to-end-demo

Tech Stack: PostgreSQL (Cloud SQL), Airbyte, dbt, Airflow, Tableau, GCP

🔧 Overview

Built a modern, cloud-native data pipeline using GCP and open-source tools to ingest, transform, orchestrate, and visualize retail data from the Sample Superstore dataset.

🛠️ Tools & Technologies

Layer	Tool Used	Description
Storage	GCP Cloud SQL (Postgres)	Hosted raw and transformed data
Ingestion	Airbyte (Cloud)	Moved data from raw DB to staging DB (ELT)
Modeling	dbt (Local)	SQL transformations: intermediate → destination
Orchestration	Apache Airflow (Local)	Scheduled & automated dbt runs
Visualization	Tableau	Built dashboards from transformed tables

🔄 Architecture Diagram

📥 Data Source

Used Tableau’s open-source Sample Superstore dataset, which contains sales, customer, region, and product information.

🔄 Pipeline Architecture

Excel (.csv)
   ↓
Cloud SQL (raw.orders, raw.returns)
   ↓             ← Airbyte →
Cloud SQL (intermediate.orders, intermediate.returns)
   ↓
dbt transformations
   ↓
Cloud SQL (destination.aggregated)
   ↓
Tableau dashboards

🧱 dbt Models Created

master_table.sql: Cleaned and formatted raw data
fct_sales_summary.sql: Aggregated metrics by region and category