Time efficient way to convert List of String Arrays into CSV file using StringBuilder?

Tags: , ,



I have a List of String arrays that I need to write into a CSV file efficiently. I’ve tried it two ways below.

Dataset is a file with at least 1 million records in size. ie. returnedList has at least 1 million String[] arrays.

My current code is terribly inefficient and takes far too long using the StringBuilder:

BufferedWriter br = new BufferedWriter(new FileWriter(filePath + "test.csv"));

for(String[] listEntry : returnedList) {
  // convert each array to a string first
  for(String s : listEntry) {
    builder.append(s);
  }
  String str = builder.toString();
  br.write(str);
  br.newLine();
}
br.close();
assertThat(newFirstRow).isNotEmpty();

My second attempt was with using the OpenCsv API:

Writer writer = Files.newBufferedWriter(Paths.get(filePath + "test.csv"));
StatefulBeanToCsv<String[]> beanToCsv = new StatefulBeanToCsvBuilder(writer)
        .withQuotechar(CSVWriter.NO_QUOTE_CHARACTER)
        .build();
beanToCsv.write(returnedList);

This method failed, where somehow, all the records that were inserted into the file was entirely empty.

Am I using the OpenCsv library wrongly?

Answer

Dont make use of StringBuilder. How BufferedWriter works is it stores data in main memory and flushes it on disk when asked for thus avoiding the clostly network trip. Can you try with below code.

for(String[] listEntry : returnedList) {
  for(String s : listEntry) {
    br.append(s);
  }
  br.flush(str);
  br.newLine();
}
//If memory constrain allows Flush everything in one go here and close instead of doing flush() in for loop
br.close();


Source: stackoverflow