I have a talend job that i am trying to read the columns from the csv file sequentially row by row for each combination (if possible trying to collect unique combination) of COLUMN A & COLUMN B to get the values and store in tjava (context variables to reuse and query the tdbinput) CSV in the below format: OPTION 1:
Tag: talend
Kafka consumer missing messages while consuming messages in loop
I am running my consumer code in loop due to memory constraints, committing my data and then loading into tables Following is the code which will run in loop But due to some reason I seem to be missing few messages. I believe this has to do something with consumer rebalancing/Committing. How can I check if my consumer is ready
how to just extract the last 2 days recent files from tftpfilelist based on modified time without storing in a tbufferoutput component-talend job
As of now i am iterating through all the 5k files available in the folder and store them in a tbufferoutput and read through them by using tbufferinput and sorting them based on mtime desc(modified time in the ftp site) in the descending order and extract the top 10 files only. Since its iterating through all the 5k files at