This document enlists the various considerations, design and flow that will orchestrate sending of callback in phases post bulk batch processing.

This callback will be a part of Bulk Processor service and will be responsible to send confirmation on different levels of completion of batch processing

BPMN Design

Attached is the BPMN design

In new BPMN, if aggregate api retry count < x, then the execution moves to a parallel gateway where it uses individual exclusive gateway to check:

• Batch Status loop check: if success percentage is less than expected threshold so it calls aggregate api again
  • This logic keeps the loop to fetch batch status running till complete rate == complete threshold or retry count is exhausted
  • This has a default flow, which on loop exit converges back to an parallel gateway to resume workflow
• Callback initiation check: If success percentage is equal to a percentage for a phase callback then move to callback part
  • This logic works parallel to the above part.
  • Has own logic for retry and successful callback.
  • Both the path (retry loop failure and successful callback) converges back to the parallel gateway

Note: Parallel gateway cannot have condition in them like exclusive gateways

Design Considerations

• Current

  • The execution of phased callback will go on parallel to the existing logic for fetching completion rate for batch status.

  • Accepts the callback URL as a header

  • Processes callback after batch complete status is 100%

  • For non 2xx (unsuccessful callback), callback will be retried for a configurable number of times

  • Handles the error response in case of unsuccessful callback

  • Generates callback from Bulk Processor microservice.

  • Includes phased callbacks based on various completion rates of batch processing:

    • Callback for 50%, 75%, 100%
    • Callback for 20%, 40%, 80% etc

  • This callback will be designed as a phase array having different percentage that will be configurable.

    • Eg:

    phases:
      values:
        - 20
        - 40
        - 60
        - 80
        - 100
    

• Future

  • Aggregate API to be moved to a spark based logic.
  • Implement callback logic in connector common for scenario where bulk notification is disabled for a non bulk batch process.