The current architecture provides reliable and scalable workflow execution using Kafka and the Outbox Pattern. The following improvements can further enhance reliability, observability, and feature depth.

1. Step-Level Execution Tracking

What to Improve

Currently, the system tracks workflow execution at a high level. A natural next step is to track each action (step) within a workflow.

How

• Introduce a zaprun_steps table
• Store status per step:
  • pending
  • running
  • completed
  • failed
• Link each step to a workflow execution (zaprun_id)

Benefits

• Clear visibility into which step failed
• Better debugging and observability
• Enables partial retries

2. Retry and Backoff Mechanism

What to Improve

Failed actions currently stop execution or require manual handling.

How

• Add retry metadata per action:
  • retry count
  • retry delay
  • max retries
• Use Kafka delayed topics or scheduled retries