Json Obj to String to Bytes to ByteArrayInputStream throws OutOfMemory Error

Question

I have a JsonString that needs to be written in file to proceed with further processing. I've written the below code for file writing.

val json = "{}"  // sample Json.
val jsonString = new StringBuilder()

val formatjson = PrettyParams.nospace.copy(preserveOrder = true).pretty(json)
jsonString.append(formatjson)

val tmpFileIS = new ByteArrayInputStream(
  jsonString.toString().getBytes()
)
try {
  IOUtils.copyLarge(tmpFileIS, tmpBufferedOutputStream)
  tmpBufferedOutputStream.flush()
} catch {
  case e: Exception => {
    e.printStackTrace()
    throw e
  }
} finally {
  tmpFileIS.close()
}

I'm getting below OOM issue, exactly at the line of where I've written jsonString.toString().getBytes()

java.lang.OutOfMemoryError: Java heap space at java.util.Arrays.copyOfRange(Arrays.java:3664)

It looks like the JSON content's been generated is over the heap memory and throwing an error.

I should not be increasing a heap memory this time nor allocate more heap memory all the time for this kind of rare instance.

So, do I need to split the JSON contents into pieces and write it a file? Hope it could solve the issue and be memory efficient. but does anyone have a better solution for reading a String byte to byte or byte[] to feed ByteArrayInputStream?

@cchantep Why did you remove the `scala` tag from a question about Scala code? — Tim, Jun 16 '22 at 07:12
I guess you find the answer in this thread : https://stackoverflow.com/questions/18508836/java-outofmemoryerror-in-reading-a-large-text-file — Yusuf Erdoğan, Jun 16 '22 at 07:13
There is no need for the `StringBuilder` or the `ByteArrayInputStream` or `IOUtils.copyLarge()` here. Just write the original String directly to the file. — user207421, Jun 16 '22 at 07:29

score 0 · Answer 1 · answered Jun 16 '22 at 08:36

The easiest way to avoid OutOfMemoryError is to use streams directly when building large data structures. Your code copies the json twice. See whether you can pretty format the json directly to an OutputStream for the file, not via String + byte[].

Alternatively since JDK11 you can use Files.writeString which eliminates most of your code (- though this assumes you haven't got OOME already when creating the pretty formatted json). Change charset / open options to suit so that the file content has your preferred encoding:

Files.writeString(formatjson, path, StandardCharsets.UTF_8, StandardOpenOption.WRITE, StandardOpenOption.TRUNCATE_EXISTING);
// or platform default encoding:
Files.writeString(formatjson, path, StandardOpenOption.WRITE, StandardOpenOption.TRUNCATE_EXISTING);

Json Obj to String to Bytes to ByteArrayInputStream throws OutOfMemory Error

1 Answers1