Converting serially reading multiple files to reading them in parallel? [closed]

Following code in Java reads multiple files one after another serially and it works well till here. (The files are JSON and at this step they are stored in a String/Buffer without parsing.)

for (int fileIndex = 0; fileIndex < numberOfFiles; fileIndex++) {

        BufferedReader br = new BufferedReader(new FileReader("Files/file" + fileIndex + ".json"));
        try {
            StringBuilder sb = new StringBuilder();
            String line = br.readLine();

            while (line != null) {
                sb.append(line);
                sb.append(System.lineSeparator());
                line = br.readLine();
            }
            String contentJSON = sb.toString();
        } finally {
            br.close();
        }
    }

How to read those files in parallel by using Threads ?

I could not match Multithreading to above code and every time got errors.

Answer

I’ve not tested this code directly (as I don’t have a bunch of files to read), but the basic idea would be to do something like…

import java.io.BufferedReader;
import java.io.FileReader;
import java.util.ArrayList;
import java.util.List;
import java.util.concurrent.Callable;
import java.util.concurrent.ExecutionException;
import java.util.concurrent.ExecutorService;
import java.util.concurrent.Executors;
import java.util.concurrent.Future;
import java.util.logging.Level;
import java.util.logging.Logger;

public class Test {

    public static void main(String[] args) {
        new Test();
    }

    public Test() {
        try {
            int numberOfFiles = 10;
            ExecutorService service = Executors.newFixedThreadPool(10);
            List<ReadWorker> workers = new ArrayList<>(numberOfFiles);
            for (int fileIndex = 0; fileIndex < numberOfFiles; fileIndex++) {
                workers.add(new ReadWorker(fileIndex));
            }
            List<Future<String>> results = service.invokeAll(workers);
            for (Future<String> result : results) {
                try {
                    String value = result.get();
                } catch (ExecutionException ex) {
                    Logger.getLogger(Test.class.getName()).log(Level.SEVERE, null, ex);
                }
            }
        } catch (InterruptedException ex) {
            Logger.getLogger(Test.class.getName()).log(Level.SEVERE, null, ex);
        }
    }

    public class ReadWorker implements Callable<String> {

        private int fileIndex;

        public ReadWorker(int fileIndex) {
            this.fileIndex = fileIndex;
        }

        @Override
        public String call() throws Exception {
            try (BufferedReader br = new BufferedReader(new FileReader("Files/file" + fileIndex + ".json"))) {
                StringBuilder sb = new StringBuilder();
                String line = br.readLine();

                while (line != null) {
                    sb.append(line);
                    sb.append(System.lineSeparator());
                    line = br.readLine();
                }
                return sb.toString();
            }
        }

    }
}

This will basically execute a series of Callables and wait for them all to complete, at which time, you can then read the results (or errors)

See the Executors trail for more details

Tested and verified version…

So, I dumped a series of files into a the Files folder at the root of my working directory, modified the above example to list all the files in that directory and read them….

import java.io.BufferedReader;
import java.io.File;
import java.io.FileFilter;
import java.io.FileReader;
import java.util.ArrayList;
import java.util.List;
import java.util.concurrent.Callable;
import java.util.concurrent.ExecutionException;
import java.util.concurrent.ExecutorService;
import java.util.concurrent.Executors;
import java.util.concurrent.Future;
import java.util.logging.Level;
import java.util.logging.Logger;

public class Test {

    public static void main(String[] args) {
        new Test();
    }

    public Test() {

        File files[] = new File("Files").listFiles(new FileFilter() {
            @Override
            public boolean accept(File pathname) {
                return pathname.getName().toLowerCase().endsWith(".svg");
            }
        });

        try {
            int numberOfFiles = files.length;
            ExecutorService service = Executors.newFixedThreadPool(20);
            List<ReadWorker> workers = new ArrayList<>(numberOfFiles);
            for (File file : files) {
                workers.add(new ReadWorker(file));
            }
            System.out.println("Execute...");
            List<Future<String>> results = service.invokeAll(workers);
            System.out.println("Results...");
            for (Future<String> result : results) {
                try {
                    String value = result.get();
                    System.out.println(value);
                } catch (ExecutionException ex) {
                    Logger.getLogger(Test.class.getName()).log(Level.SEVERE, null, ex);
                }
            }
            service.shutdownNow();
        } catch (InterruptedException ex) {
            Logger.getLogger(Test.class.getName()).log(Level.SEVERE, null, ex);
        }
    }

    public class ReadWorker implements Callable<String> {

        private File file;

        public ReadWorker(File file) {
            this.file = file;
        }

        @Override
        public String call() throws Exception {
            System.out.println("Reading " + file);
            try (BufferedReader br = new BufferedReader(new FileReader(file))) {
                StringBuilder sb = new StringBuilder();
                String line = br.readLine();

                while (line != null) {
                    sb.append(line);
                    sb.append(System.lineSeparator());
                    line = br.readLine();
                }
                return sb.toString();
            }
        }

    }
}

And this works just fine and I have no issue.

java.io.FileNotFoundException: Filesfile0.json is a localised issue you are going to have to solve. Does file0.json actually exist? Does it exist in the Files directory? Is the Files directory in the root of the working directory when the program is executed?

None of these issues can be solved by us, as we don’t have access to your environment

Test #3

I then renamed all the files in my Files directory to file{x}.json using…

File files[] = new File("Files").listFiles(new FileFilter() {
    @Override
    public boolean accept(File pathname) {
        return pathname.getName().toLowerCase().endsWith(".svg");
    }
});

for (int index = 0; index < files.length; index++) {
    File source = files[index];
    File target = new File(source.getParent(), "file" + index + ".json");
    source.renameTo(target);
}

And the modified the example slightly to include a File#exists report…

import java.io.BufferedReader;
import java.io.File;
import java.io.FileReader;
import java.util.ArrayList;
import java.util.List;
import java.util.concurrent.Callable;
import java.util.concurrent.ExecutionException;
import java.util.concurrent.ExecutorService;
import java.util.concurrent.Executors;
import java.util.concurrent.Future;
import java.util.logging.Level;
import java.util.logging.Logger;

public class Test {

    public static void main(String[] args) {
        new Test();
    }

    public Test() {
        try {
            int numberOfFiles = 10;
            ExecutorService service = Executors.newFixedThreadPool(20);
            List<ReadWorker> workers = new ArrayList<>(numberOfFiles);
            for (int index = 0; index < numberOfFiles; index++) {
                workers.add(new ReadWorker(index));
            }
            System.out.println("Execute...");
            List<Future<String>> results = service.invokeAll(workers);
            System.out.println("Results...");
            for (Future<String> result : results) {
                try {
                    String value = result.get();
                    System.out.println(value);
                } catch (ExecutionException ex) {
                    Logger.getLogger(Test.class.getName()).log(Level.SEVERE, null, ex);
                }
            }
            service.shutdownNow();
        } catch (InterruptedException ex) {
            Logger.getLogger(Test.class.getName()).log(Level.SEVERE, null, ex);
        }
    }

    public class ReadWorker implements Callable<String> {

        private int fileIndex;

        public ReadWorker(int fileIndex) {
            this.fileIndex = fileIndex;
        }

        @Override
        public String call() throws Exception {
            System.out.println("Reading " + fileIndex);
            File file = new File("Files/file" + fileIndex + ".json");
            System.out.println("File " + fileIndex + " exists = " + file.exists());
            try (BufferedReader br = new BufferedReader(new FileReader(file))) {
                StringBuilder sb = new StringBuilder();
                String line = br.readLine();

                while (line != null) {
                    sb.append(line);
                    sb.append(System.lineSeparator());
                    line = br.readLine();
                }
                return sb.toString();
            } finally {
                System.out.println("All done here");
            }
        }

    }
}

Which prints

Execute...
Reading 8
Reading 1
Reading 2
Reading 4
Reading 6
Reading 9
Reading 3
Reading 7
Reading 0
Reading 5
File 8 exists = true
File 1 exists = true
File 5 exists = true
File 4 exists = true
File 9 exists = true
File 2 exists = true
File 0 exists = true
File 3 exists = true
File 7 exists = true
File 6 exists = true
All done here
All done here
All done here
All done here
All done here
All done here
All done here
All done here
All done here
All done here
Results...
// I won't bore you with the results, as it's a lot of pointless text

which all worked without issues