Overview

The system is designed as a SaaS-style data migration platform where users can migrate data from a NoSQL database (MongoDB) to a SQL database (PostgreSQL).

The architecture follows a microservices approach and uses Kafka to handle large data migrations reliably.

The system is designed so that:

• Frontend triggers and monitors migration jobs
• Backend services orchestrate and execute migration steps
• Kafka decouples migration processing from orchestration
• Databases act as source, target, and state storage

High Level Data Flow Chart

User (MongoDB)
     │
     │1.Read-onlyAccess
     ▼
┌──────────────────┐
│Schema Discovery  │
│     Service      │
└────────┬─────────┘
         │
         │2.Schema Model
         ▼
┌──────────────────┐
│ Migration        │
│ Orchestrator     │
│ (Spring Boot)    │
└────────┬─────────┘
         │
         │3. Migration Tasks (BatchInfo)
         ▼
┌──────────────────┐
│ Kafka Topics     │
│ (Migration Pipe) │
└────────┬─────────┘
         │
         │4. Consume Tasks
         ▼
┌──────────────────┐
│ Migration Worker │
│ Service          │
└────────┬─────────┘
         │
         │5. TransformedRows
         ▼
┌──────────────────┐
│ PostgreSQL       │
│ (User Target DB) │
└────────┬─────────┘
         │
         │6. Validation Checks
         ▼
┌──────────────────┐
│ Validation       │
│ Engine           │
└────────┬─────────┘
         │
         │7. Status & Report
         ▼
┌──────────────────┐
│ Frontend (React) │
└──────────────────┘

Main Components

1. Frontend (React)

The frontend is responsible only for user interaction.

Responsibilities:

• Collect MongoDB & PostgreSQL connection details
• Start migration jobs
• Show live progress and job status
• Display final migration report