Uncompressing base64 gzip on S3 and saving to local file system

Question

I am a Java bonehead/newbie so please be gentle. I have two functions which I realize are somewhat incompatible:

saveS3toFilesystem - takes a InputStream from AWS S3 and saves it to the local filesystem as a file
decompress - takes a string and decodes the base64 encoding and the decompresses the gzip compression.

I really want these two to work in concert to achieve the result of the file saved to the filesystem being the uncompressed file but I realize that my "decompress" function should probably be changed to receive a stream rather than a string but sadly I'm just a "cutter and paster" in the world of Java these days.

Here are my two functions as they are now:

private void saveS3toFilesystem(String filename, String bucketName, String localFilename) {
  S3Object obj = s3.getObject(bucketName, filename);
  InputStream in = obj.getObjectContent();
  try {
    Files.createDirectories(Paths.get(localFilename.replace(this.FILE_EXTENSION, "")));
    Files.copy(in, Paths.get(localFilename));
    this.logger.log("Input file has been placed in local filesystem for ITMS to pick up: " + localFilename + "\n");
  } catch (IOException err) {
    this.logger.log("There was a problem saving the file to " + localFilename);
    err.printStackTrace();
  } finally {
    try {
      in.close();
    } catch (IOException err) {
      err.printStackTrace();
    }
  }
  return;
}

and then ...

private String decompress(String compressedZip) {
  byte[] decodedBytes = Base64.getDecoder().decode(compressedZip);
  String result = null;
  GZIPInputStream zip = null;
  try {
    zip = new GZIPInputStream(new ByteArrayInputStream(decodedBytes));
    result = IOUtils.toString(zip);
  } catch (IOException e) {
    e.printStackTrace();
  } finally {
    IOUtils.closeQuietly(zip);
  }
  return result;
}

Can anyone please help me to achieve the dream? Happy to do it with streams, strings, or any method that will work. Sadly I can't afford atm to up my Java skills enough to grok the solution myself.

Many thanks in advance.

In order to refactor your decompress function from a String input to an InpuStream input: Do you mean something like this ? https://stackoverflow.com/questions/247161/how-do-i-turn-a-string-into-a-inputstreamreader-in-java — Rann Lifshitz, Mar 23 '18 at 04:59
By the way, did you consider a more piped approach - that is, getting the file from S3 to save its contents in memory instead of on the hard drive, then immediatly decompressing it, and only then saving it on the file system? — Rann Lifshitz, Mar 23 '18 at 05:05
@JimGarrison I think ken was just using a figure of speach. A SO user with a 3K rating knows how to ask questions properly..... — Rann Lifshitz, Mar 23 '18 at 05:06
I think _"...I can't afford atm to up my Java skills enough to grok the solution myself"_ is pretty explicitly "write the code for me". — Jim Garrison, Mar 23 '18 at 05:09
@Jim Or perhaps the idea is to get a few ideas on relevant APIs and work out a solution from there — Rann Lifshitz, Mar 23 '18 at 05:27
@RannLifshitz Still off-topic as that would be asking for recommendations. — Jim Garrison, Mar 23 '18 at 05:33
Hi folks I'm awake again. I'm happy to take suggestions on how best to proceed or be pointed toward code examples. My problem is that while I was certified in Java 0.9 in the mid 1990's I haven't touched Java since and so the API's and language present a large learning curve. Not that I'm not technically astute though. In principle, I'd expect the problem domain to be pretty common so I was secretly hoping that code snippets might be available. @RannLifshitz, as to your question of piping ... yeah that seems more elegant to me than serial process of loading, converting, then saving. — ken, Mar 23 '18 at 17:01
@ken should i post an answer with my take on a feasible approach to your problem ? Regardless, I'd suggest you use the try with resources approach as well : https://docs.oracle.com/javase/tutorial/essential/exceptions/tryResourceClose.html — Rann Lifshitz, Mar 23 '18 at 17:14
@RannLifshitz that would be super helpful. I looked at your oracle link and it takes the assumption of a file being a starting point for the compressed input, in my case the starting point is an S3 Object on AWS which would be streamed via the AWS SDK. Sadly even this kind of variance make my Java newbie heart jump. — ken, Mar 23 '18 at 17:51

Rann Lifshitz · Answer 1 · 2018-03-24T06:07:35.093

1

Based on the following APIs : Base64.Decoder and GZIPInputStream (look at the wrap method on the former and the constructors on the latter), the decompress method can be overloaded as follows:

private String decompress(InputStream compressedStream) {
  InputStream decodingStream = Base64.getDecoder().wrap(compressedStream);
  String result = null;
  GZIPInputStream zip = null;
  try {
    zip = new GZIPInputStream(decodingStream);
    result = IOUtils.toString(zip);
  } catch (IOException e) {
    e.printStackTrace();
  } finally {
    IOUtils.closeQuietly(zip);
  }
  return result;
}

And finally, the changes to saveS3toFilesystem are as follows :

private void saveS3toFilesystem(String filename, String bucketName, String localFilename) {
  S3Object obj = s3.getObject(bucketName, filename);
  InputStream in = obj.getObjectContent();
  // decoding the inputstream via decode into a string, which is then
  // used in order to create an inputstream of decoded data
  InputStream decodedStream = 
     new ByteArrayInputStream(decompress(in).getBytes(StandardCharsets.UTF_8));
  try {
    Files.createDirectories(Paths.get(localFilename.replace(this.FILE_EXTENSION, "")));
    Files.copy(decodedStream, Paths.get(localFilename));
    this.logger.log("Input file has been placed in local filesystem for ITMS to pick up: " + localFilename + "\n");
  } catch (IOException err) {
    this.logger.log("There was a problem saving the file to " + localFilename);
    err.printStackTrace();
  } finally {
    try {
      in.close();
    } catch (IOException err) {
      err.printStackTrace();
    }
  }
  return;
}

edited Mar 24 '18 at 06:07

answered Mar 23 '18 at 18:15

Rann Lifshitz

4,040
4
22
42

Within the **saveToFilesystem** method the `Base64.getDecoder().decode()` seems to complain about getting an InputStream: `[Java] The method decode(byte[]) in the type Base64.Decoder is not applicable for the arguments (InputStream)` – ken Mar 23 '18 at 19:31
@ken : you should use Base64.getDecoder().wrap() and not Base64.getDecoder().decode(). The former is used for InputStream parameters, the latter is used for byte array parameters, as seen in your linter error message. – Rann Lifshitz Mar 23 '18 at 23:05
I was just referring to the `decode(in)` line in your **saveS3toFilesystem**. Maybe that's also meant to be wrap? – ken Mar 24 '18 at 03:39
@ken A thousand apologies. I wrote decode(.....) instead of decompress(.....). This is where our decompress method should be used. – Rann Lifshitz Mar 24 '18 at 06:08
I did compile in the latest changes and am now getting a `java.io.EOFException` error. Here's a screenshot of the function with annotation on where the exception occurs. https://www.screencast.com/t/YP1mRqC8Y ... I'm struggling with how to debug it. Not sure if you have any insights. Thanks for stick with me @rann – ken Mar 24 '18 at 20:50
@ken : Have a look at this link about handling EOFExceptions : https://stackoverflow.com/questions/18451232/eofexception-how-to-handle – Rann Lifshitz Mar 25 '18 at 03:39

Uncompressing base64 gzip on S3 and saving to local file system

1 Answers1