Table of Contents

This notion page consists of the following:

• Technical Documentation
  • ETL Pipeline Design
  • Database Design
• Data Analytic Documentation

Purpose

The purpose of this section is to detail the design of the ETL Pipeline. It will go over the entire ETL process in detail.

Tech Stack

The Tech Stack used for the ETL Pipeline includes:

• Programming: Python 3.12
• Libraries: pandas, pdfplumber, google-api-python-client, google-auth-httplib2, google-auth-oauthlib, python-dotenv, pyYAML
• Database: Local file system
• Environment: VS Code, Windows

Requirements: https://github.com/JoshuaChua1106/data_pipeline_utility_bill/blob/main/requirements.txt

Architecture Overview

High-level data flow

The overview of the ETL pipeline can be found below.

Extract: