I’m trying to implement a Spring Batch job that is scheduled by a Spring scheduler. The current befavior is as follows: The job is launched by the scheduler using JarLauncher The job consumes the kafka records and processed them The next time the jobs ris launched by the scheduler, it consumes the kafka records already processed The expected behavior is
Tag: spring-kafka
Disable auto topic creation from Spring Kafka Consumer
I don’t want to auto create topics from my Consumer application when the topics are not present. I know it is a Kafka server level config to disable auto topic creation (auto.create.topics.enable = false), but I cannot do that change in my infrastructure. So I am looking for a way to disable auto topic creation in my Consumer application (using
kafka Offset commit failing org.apache.kafka.clients.consumer.CommitFailedException
I have written a kafka consumer using spring-kafka library (spring-boot-starter-parent 2.3.4.RELEASE). I have following consumer configuration in my code Following is my listener method : Here I am reading 1 message at a time, apply business logic and use ack.acknowledge() to commit offset, but what I have seen, sometime offset commit succeed but many time I get org.apache.kafka.clients.consumer.CommitFailedException on line
Use Kakfa connection to dynamically Subscribe/Unsubscribe the Kafka Topics using Spring Boot
I’m developing a SpringBoot application which exposes the APIs to sub/unsub the Kafka topics. All we need to do is to pass the topic-name in the API call and the application will subscribe to it and consume messages. Subscribe topic API : Unsubscribe topic API : Dependency I have created KafkaConsumerConfiguration in which stated some beans (as follows). and I
Is it better to write integration or unit test for Kafka Spring?
I’ve been searching a while about the best way to test Kafka in Spring. As far as I know, @EmbeddedKafka is more for an unit test but isn’t it better to test Kafka in a real working environment by writing integration test for it? And what really am I testing by writing unit test for Kafka? A real working producer
Spring Kafka multiple topic for one class dynamically
I recently wanted to add a new behavior in my project that uses spring-kafka. The idea is really simple : App1 create a new scenario name “SCENARIO_1” and publish this string in the topic “NEW_SCENARIO” App1 publish some message on topic “APP2-SCENARIO_1” and “APP3-SCENARIO_1” App2 (group-id=app2) listens on NEW_SCENARIO and creates a new consumer<Object,String> listening on a new topic “APP2-SCENARIO_1”
Spring Boot Kafka StreamsConfig or ConsumerConfig from application.yaml not applying
I have a very simple spring boot project with a KTable and I want to customize my configuration in application.yml, but the config seems to not be applied. This is my configuration file application.yml However, when starting the application the log outputs the following: ConsumerConfig: Below is the simple application class I’m using: The values from my application.yml seems to
Incoming kafka metrics are not detected in Appdynamcis – springkafka
We are using springboot and springkafka to consume and process the messages, and Appdynamics for capturing the Performance metrics. Appdynamics is capturing the out going topics and metrics, but not detecting the incoming topic and the metrics.The solutions we tried Custom configured the topics name in the backend Set enable-kafka-consumer to true Custom-interceptors.xml mentioned below In any of the cases
How to replace deprecated SeekToCurrentErrorHandler with DefaultErrorHandler (spring-kafka)?
I am trying to find a way to use the new DefaultErrorHandler instead of deprecated SeekToCurrentErrorHandler in spring-kafka 2.8.1, in order to override the retry default behavior in case of errors. I want to “stop” the retry process, so if an error occurs, no retry should be done. Now I have, in a config class, the following bean that works
Spring Kafka Template – Connect to Kafka Topic on Spring Boot Startup
I have implemented a basic Spring Boot Application which uses Spring Kafka. I want my producer to connect to the Kafka Topic before the first .send() is called but I can’t find a way to do so. Is that possible? Logs to show that KafkaTemplate only connects to the Kafka Topic after I trigger the .send method at 16:12:44: Answer