I’m learning about microservices and I understand that in my scenario I want microservices to communicate with each other using message queues, with persistence. That means if the instance crashes and the message is not consummed, it will be consumed when the instance bounces back (or maybe by another instance).
Now, since these messages are critical, shouldn’t they be transactional and participate in the micro database transaction? I see examples using Kafka or RabbitMQ but these queue implementations don’t implement transactions. If they don’t do it there’s a clear chance some messages could be lost.
Are transactions really needed for microservices at all? Or should I design my microservices in a resilient way so they can lose information, and recover from this scenario?
Maybe my question doesn’t make sense. Maybe I’m still thinking in the “monolithic” way where isolation, locking, and transactions are king.
Advertisement
Answer
as Ivan Baranuk said in the comment. Most message broker has acknowledgement. and deadletter queue (DLQ) too.
As long as the crashes happened before the acknowledgement (could be handled by the library you are using or you could handle it manually), you would not lose your message, since it will got re-queued. If the crashes still happens after several times and it exceed maxRetryAttempt it will be passed to DLQ for you to handle it.
Just think like queue acknowledgement is equivalent to DB commit.
extra: I do think DB transactional is important when consuming message. To handle cases where you have multiple DB writes when consuming message and then on the middle of the process your app crash. When it crashed, the message would not be ack-ed and it will got re-queued and will get consumed again later on. It is to prevent double write on the second time the message is consumed.