Objective

Fetch and visualize live weather data (temperature, humidity, conditions) for major Indian cities using the Google Cloud Platform (GCP) ecosystem.

This pipeline:

• Calls OpenWeather API hourly
• Orchestrates ingestion with Cloud Composer (Airflow)
• Stores curated records in BigQuery (partitioned & clustered for cost efficiency)
• Archives raw JSON to Cloud Storage (optional, for audit/backfills)
• Draw insights using Google Collab
• Visualizes insights in Looker Studio

Components

• 🌐 OpenWeather API → Source of live weather data (15 Indian cities)
• 🔑 Secret Manager / Env Vars → Secure storage of API keys
• ☁️ Cloud Composer (Airflow) → Orchestration of ingestion and checks
• 🗄️ Cloud Storage (GCS) → Raw JSON archive (gs://weather-raw-archive/)
• 📊 BigQuery → Analytical warehouse
  • Partitioned by DATE(ts_utc)
  • Clustered by city
  • Unique key = (city, source_dt_utc)
• 📈 Looker Studio → Visualization dashboards

Architecture