I’m trying to implement a Spring Batch job that is scheduled by a Spring scheduler. The current befavior is as follows: The job is launched by the scheduler using JarLauncher The job consumes the kafka records and processed them The next time the jobs ris launched by the scheduler, it consumes the kafka…
Tag: spring-kafka
Disable auto topic creation from Spring Kafka Consumer
I don’t want to auto create topics from my Consumer application when the topics are not present. I know it is a Kafka server level config to disable auto topic creation (auto.create.topics.enable = false), but I cannot do that change in my infrastructure. So I am looking for a way to disable auto topic …
kafka Offset commit failing org.apache.kafka.clients.consumer.CommitFailedException
I have written a kafka consumer using spring-kafka library (spring-boot-starter-parent 2.3.4.RELEASE). I have following consumer configuration in my code Following is my listener method : Here I am reading 1 message at a time, apply business logic and use ack.acknowledge() to commit offset, but what I have se…
Use Kakfa connection to dynamically Subscribe/Unsubscribe the Kafka Topics using Spring Boot
I’m developing a SpringBoot application which exposes the APIs to sub/unsub the Kafka topics. All we need to do is to pass the topic-name in the API call and the application will subscribe to it and consume messages. Subscribe topic API : Unsubscribe topic API : Dependency I have created KafkaConsumerCo…
Is it better to write integration or unit test for Kafka Spring?
I’ve been searching a while about the best way to test Kafka in Spring. As far as I know, @EmbeddedKafka is more for an unit test but isn’t it better to test Kafka in a real working environment by writing integration test for it? And what really am I testing by writing unit test for Kafka? A real …
Spring Kafka multiple topic for one class dynamically
I recently wanted to add a new behavior in my project that uses spring-kafka. The idea is really simple : App1 create a new scenario name “SCENARIO_1” and publish this string in the topic “NEW_SCENARIO” App1 publish some message on topic “APP2-SCENARIO_1” and “APP3-SC…
Spring Boot Kafka StreamsConfig or ConsumerConfig from application.yaml not applying
I have a very simple spring boot project with a KTable and I want to customize my configuration in application.yml, but the config seems to not be applied. This is my configuration file application.yml However, when starting the application the log outputs the following: ConsumerConfig: Below is the simple ap…
Incoming kafka metrics are not detected in Appdynamcis – springkafka
We are using springboot and springkafka to consume and process the messages, and Appdynamics for capturing the Performance metrics. Appdynamics is capturing the out going topics and metrics, but not detecting the incoming topic and the metrics.The solutions we tried Custom configured the topics name in the ba…
How to replace deprecated SeekToCurrentErrorHandler with DefaultErrorHandler (spring-kafka)?
I am trying to find a way to use the new DefaultErrorHandler instead of deprecated SeekToCurrentErrorHandler in spring-kafka 2.8.1, in order to override the retry default behavior in case of errors. I want to “stop” the retry process, so if an error occurs, no retry should be done. Now I have, in …
Spring Kafka Template – Connect to Kafka Topic on Spring Boot Startup
I have implemented a basic Spring Boot Application which uses Spring Kafka. I want my producer to connect to the Kafka Topic before the first .send() is called but I can’t find a way to do so. Is that possible? Logs to show that KafkaTemplate only connects to the Kafka Topic after I trigger the .send me…