I’m using docker with kafka and clickhouse. I want to connect ‘KsqlDB table’ and ‘clickhouse’ using ‘kafka connect’. So I referred to this document and modified ‘docker composite’. here is my docker-compose And This is Dockerfile-kafka-connect And I typed this command in ‘KsqlDB’. (‘S3_FINAL’ topic has AVRO format for both multi-key and values.) And It doesn’t work. So I read
Tag: apache-kafka-connect
Kafka Connect: Convert message from bytes to Json
I am trying to use a Google PubSub source connector to fetch data from my google cloud to kafka. I do get the data, but the message comes as bytes. I refered here and as mentioned, I have used a JSON convertor to change it. Here is my connector code part: And this what I get in my kafka: Even
Kafka Dependencies – ccs vs ce
To develop my Kafka connector I need to add a connect-API dependency. Which one I should use? For example mongodb connector use connect-api from maven central But links from dev guide go to https://packages.confluent.io/maven/org/apache/kafka/connect-api/5.5.0-ccs/ and beside 5.5.0-ccs there is also 5.5.0-ce version. So, at this moment last versions are: 2.5.0 from maven central 5.5.0-ccs from packages.confluent.io/maven 5.5.0-ce from packages.confluent.io/maven What
Debezium flush timeout and OutOfMemoryError errors with MySQL
Using Debezium 0.7 to read from MySQL but getting flush timeout and OutOfMemoryError errors in the initial snapshot phase. Looking at the logs below it seems like the connector is trying to write too many messages in one go: Wonder what the correct settings are http://debezium.io/docs/connectors/mysql/#connector-properties for sizeable databases (>50GB). I didn’t have this issue with smaller databases. Simply increasing