Skip to content
Advertisement

Kafka consumer missing messages while consuming messages in loop

I am running my consumer code in loop due to memory constraints, committing my data and then loading into tables

Following is the code which will run in loop

JavaScript

But due to some reason I seem to be missing few messages. I believe this has to do something with consumer rebalancing/Committing.

How can I check if my consumer is ready to consume the next batch of messages from the start without missing any messages?

Advertisement

Answer

Update : I was able to figure out the issue myself. The messages are already downloaded in the records and while looping as I have put the following condition

JavaScript

Even before placing all the messages in list the loop is breaking and all the messages from records is not being inserted in the list. I have removed the break condition and it’s working fine

Advertisement