Fast and asynchronous way of making multiple http requests in JAVA

I have a program that should make really fast http requests. Requests should be made asynchronously so that it won’t block the main thread.

So I have created a queue which is observed by 10 separate threads that make http requests. If something is inserted in the queue then the first thread that gets the data will make the requests and process the result.

The queue gets filled with thousands of items so multithreading is really neccessary to get the response as fast as possible.

Since I have alot of code I’ll give a short example.

main class

package fasthttp;

import java.util.concurrent.ExecutorService;
import java.util.concurrent.Executors;
import java.util.concurrent.LinkedBlockingQueue;

public class FastHTTP {

    public static void main(String[] args) {
        ExecutorService executor = Executors.newFixedThreadPool(10);

        for (int i = 0; i < 10; i++) {
            LinkedBlockingQueue queue = new LinkedBlockingQueue();
            queue.add("http://www.lennar.eu/ip.php");//for example
            executor.execute(new HTTPworker(queue));
        }
    }

}

FastHTTP class

package fasthttp;

import java.io.BufferedReader;
import java.io.IOException;
import java.io.InputStreamReader;
import java.net.HttpURLConnection;
import java.net.URL;
import java.util.concurrent.LinkedBlockingQueue;

public class HTTPworker implements Runnable {

    private final LinkedBlockingQueue queue;

    public HTTPworker(LinkedBlockingQueue queue) {        
        this.queue = queue;
    }

    private String getResponse(String url) throws IOException {

        URL obj = new URL(url);
        HttpURLConnection con = (HttpURLConnection) obj.openConnection();

        StringBuilder response;
        try (BufferedReader in = new BufferedReader(
                new InputStreamReader(con.getInputStream()))) {
            String inputLine;
            response = new StringBuilder();
            while ((inputLine = in.readLine()) != null) {
                response.append(inputLine);
            }
        }
        return response.toString();
    }

    @Override
    public void run() {
        while (true) {
            try {
                String data = (String) queue.take();
                String response = getResponse(data);
                //Do something with response
                System.out.println(response);
            } catch (InterruptedException | IOException ex) {
                //Handle exception
            }
        }
    }
}

Is there a better or faster way to make thousands of http requests response processing asynchronously? Speed and performance is what I’m after.

Answer

Answering my own question. Tried Apaches asynchronous http client but after a while I started using Ning’s async client and I am happy with it.

Advertisement

Answer