Overview

• The model's accuracy degrades over time and requires monitoring to retrain it.
• The ML model monitoring is different as it’s not only about service health (Uptime, Memory, Latency), but also data health and model health.

ML Monitoring

• We may add those metrics as starting points:
  1. Service Health
  2. Model Performance
  3. Data Quality and Integrity
  4. Data and Concept Drift
• We may add more metrics to look at:
  1. Performance by Segment
  2. Model Bias/Fairness
  3. Outliers
  4. Explainability

Batch vs Online Serving Models

• Monitoring:
  1. Add ML metrics to service health monitoring (Prometheus/Grafana)
  2. Build an ML-focused dashboard (MongoDB/Grafana, BI tools such as Tableau)

Environment Setup

• Create a new folder.

• Create a virtual environment

  conda create -n py11 python=3.11
  

• To activate this environment

  conda activate py11
  

• To deactivate an active environment

  conda deactivate
  

• Install the requirements through the requirements.txt file.

  pip install -r requirements.txt
  

• Build the Docker-compose file to operate the different services (PostgreSQL, Adminer, and Grafana)

  docker-compose up --build