We are working to build an exchange using Go. Following components are there in the system of now:
- Frontend
- Backend
- Matching Engine.
Here is how the system works:
- User creates an order which goes to Backend.
- Backend processes the new order in following manner :
- It validates the order request data & then a db transaction is created for updating users account balance in account table & new order is inserted in the order table.
- After db interaction is completed; backend produces a new order message in queue.
- Matching Engine processes the new order from the queue and pushes the engine response(i.e updated orders & trades) in another queue
Backend consumes the engine response as below:
- Multiple goroutines run in parallel to process engine response &store the updates received in engine response.
- For storing the the updates, a database transaction is created within each goroutine to update the users account balance, orders & store new trades.
Problem occurs during this stage when concurrent (parallel) transactions try to update the same users account at a time, a deadlock is created in DB as multiple transaction try to lock the same record. Also while a users previously placed orders are being matched and processed user can create new order as mentioned in step 2, so transaction created in step 2 also tries to update the same users account in database which is being accessed by backend to store Matching Engine response. This also adds to the chances of deadlock.
How to properly manage and prevent the database deadlock generated in above mentioned flow.
We have tried a solution to retry transaction by following code but the deadlock issue can still be reproduced
// Update accounts, orders, trades with max retry of 5
for i := 0; i < 5; i++ {
err = nil
time.Sleep(10 * time.Second)
if err = manager.saveEngineResponse(newEngineResponse, accountUpdatesToDoRef, &updatedOrders); err == nil {
break
}
logger.Debug("Error saving engine response", err)
}
if err != nil {
logger.Fatal(err)
return
}
Error : Error 1213: Deadlock found when trying to get lock; try restarting transaction
I have refered the link : How to avoid mysql 'Deadlock found when trying to get lock; try restarting transaction' which states that by keeping the operations in a specific order deadlocks can be avoided. And in the above scenario all transaction will first update account, then order & then trades if any. So the order of operations is consistent in different transactions
