Jersey Client download ZIP file and unpack efficiently

So, I have a server application that returns ZIP files and I’m working with huge files (>=5GB). I am then using the jersey client to do a GET request from this application after which I want to basically extract the ZIP and save it as a folder. This is the client configuration:

Client client = ClientBuilder.newClient();
client.register(JacksonJaxbJsonProvider.class);
client.register(MultiPartFeature.class);
return client;

And here’s the code fetching the response from the server:

client.target(subMediumResponseLocation).path("download?delete=true").request()
                                           .get().readEntity(InputStream.class)

My code then goes through a bunch of (unimportant for this question) steps and finally gets to the writing of data.

try (ZipInputStream zis = new ZipInputStream(inputStream)) {
   ZipEntry ze = zis.getNextEntry();
   while(ze != null){
     String fileName = ze.getName();
     if(fileName.contains(".")) {
       size += saveDataInDirectory(folder,zis,fileName);
     }
     is.closeEntry();
     ze = zis.getNextEntry();
  }
  zis.closeEntry();
} finally {
  inputStream.close();
}

Now the issue I’m getting is that the ZipInputStream refuses to work. I can debug the application and see that there are bytes in the InputStream but when it get to the while(ze != null) check, it returns null on the first entry, resulting in an empty directory.

I have also tried writing the InputStream from the client to a ByteArrayOutputStream using the transferTo method, but I get a java heap space error saying the array length is too big (even though my heap space settings are Xmx=16gb and Xms=12gb).

My thoughts were that maybe since the InputStream is lazy loaded by Jersey using the UrlConnector directly, this doesn’t react well with the ZipInputStream. Another possible issue is that I’m not using a ByteArrayInputStream for the ZipInputStream.

What would a proper solution for this be (keeping in mind the heap issues)?

Answer

Ok so I solved it, apparently my request was getting a 404 for adding the query param in the path… .path("download?delete=true")

Advertisement

Answer