potential problems to take note of when designing the system.
- Domain model system architecture,
- idempotency.
- Is achieved through 2 points, first from business center creating unique ID, and inserting into idempotent_key. With a new txnId if successful validation checks.
- Cross domain updates - distributed clusters / domains data synchronisation. As one domain in SG may try to send money to same database as another domain from US region. Hence we need to lock the domain
- distributed lock - lock the key, then perform, then unlock. Cache. Storing the key in a cache.
- concurrency
- Handled by combination of all concepts of distributed lock, listener and the transactional template with lock. Lock prevents race condition.
Interface
Create new account
sequenceDiagram
participant UserCenter
participant AccountCenter
participant account
UserCenter-->>AccountCenter: accountService.createAccount(CreateAccountRequest)
AccountCenter-->>account: accountServiceRepository.createAccount(CreateAccountRequest)
account-->>AccountCenter: result
AccountCenter-->>UserCenter: resu
Transfer service
- correct way to handle a distributed transactional event :
- Publish message broker event, and allow the transfer service to listen / consume to the event,
- then create an executor thread-pool, that will create multiple threads to concurrently process each transfer.
- In these thread pools, we need to implement a distributed lock to lock cross server / process from the two accounts.
- then after distributed lock we implement idempotency key check if its in INIT status, then update to PROCESSING, if its in FAILED, check retry counts, is less than 3, then retry, otherwise add to DLQ
- then we put solution 1 here : where we distributed lock the lockFirstAccount and lockSecondAccount.
- then only transactional template to ensure atomicity, either all succeed or all fail. either debit and credit or none.
- inside transaction template, we lock account payer and payee, check balance sufficient, the update amounts
- Then for each debit and credit, we need to use for update in sql update command, to ensure no race conditions.
- if success update idempotency to finished
- unlock both accounts
- if success then only send ACK, otherwise msgbroker retry limit to 3, also add count to the idempotency_keys retry count. so at top can check again. and update transaction status to FAILED and idempoetncy as well. no need distirbuted lock this again since already locked
- then unlock