Can I make the size of a byte array dynamic?

Question

I'm using a byte array with the write() method of OutputStream to store the data into a file. For that, I'm initiating the byte array with a predefined size. But I don't want it to take up so much space for every InputStream whose size could be much lower than the size of the array. Is it possible to somehow make the array dynamic so that it adjusts its size depending on the size of the InputStream? I have something like this in my code:

byte[] b = new byte[20000];
int length;                 
while ((length = in.read(b))!= -1) {
    //writing it to a file
    op.write(b, 0, length);
    k++;
}

in.close();
op.close();

Here, in is the InputStream object and op is the FileOutputStream object. I've tried using ArrayLists, but there is no direct method of FileOutputStream to write data into an ArrayList, it only accepts byte[].

I just used that as dummy size. Actually, I'm using it to save images from the web. So, 1024 would do. But still, most of the images are <<1024. I don't want that extra space for nothing. If it can dynamically adjust its size somehow, that'd be great. — postmalloc, Oct 09 '13 at 06:20
Yes, It works. See the code I'm posted and I'm able to upload the files >200MB without any problem with that little buffer. — Suresh Atta, Oct 09 '13 at 06:30
Alternative approach: [Java NIO](http://stackoverflow.com/questions/921262/how-to-download-and-save-a-file-from-internet-using-java) — tom, Oct 09 '13 at 06:31
@tom NIO really worked! Using byte channel instead of a byte array works well! — postmalloc, Oct 10 '13 at 04:19

score 2 · Accepted Answer · answered Oct 09 '13 at 06:18

2

The problem isn't on the output side at all - it's just that you don't know how big the input is.

It's not clear where the InputStream is coming from - if the input is being received over the network for example, you may not have any indication of the size beforehand. InputStream itself doesn't have any simple way of checking whether the stream has a known length.

Bear in mind that the 20K array here is only important while you're copying the data. After that, it can be garbage collected. Do you have any indication that this is actually an issue?

You could just reduce the size of the array to (say) 4K or 8K. You'd still be able to handle any size of input, as you're looping over it and copying it in chunks. It just wouldn't be quite as efficient in cases where you can read more than that buffer size in one go. The extent of that efficiency difference will be very context-sensitive - only you can measure whether how that code will perform in your real situation - or how important that performance is in terms of time, or how useful it is to save a few thousand bytes which can be garbage collected immediately afterwards anyway.

answered Oct 09 '13 at 06:18

Jon Skeet

1,421,763
867
9,128
9,194

That above code works with a little buffer `byte[] bytes = new byte[1024] //`? Right ?? Am I missing something? – Suresh Atta Oct 09 '13 at 06:20
@sᴜʀᴇsʜᴀᴛᴛᴀ: Yes, which is why I suggested reducing it to 4 or 8K. 1K would clearly work too... it's just another arbitrary size. The cost is in efficiency, with more reads and writes. If you're suggesting 1K is better than 4K, why not 256 bytes instead? Without benchmarking the particular situation, we can't really know what the efficiency difference would be. – Jon Skeet Oct 09 '13 at 06:21
Yes Jon, I'm also suggesting some smaller size that's it. Got it. – Suresh Atta Oct 09 '13 at 06:22
I was just wondering if there was an efficient way to do the same. Why waste any space at all, no matter how small it is. What if the size of the inputstream exceeds the the size of the byte array? – postmalloc Oct 09 '13 at 06:33
@BreakingBenjamin The code works with any non-zero size buffer, because of the `while` loop. Wasting a few bytes out of 1024 isn't even worth thinking about, but if it really bugs you, make it 512, 256, 128, ... It's just a space-time tradeoff. – user207421 Oct 09 '13 at 06:58
1

@BreakingBenjamin: As I wrote in my answer: "You'd still be able to handle any size of input, as you're looping over it and copying it in chunks." I'm surprised you're even asking though - what did you *expect* the loop to be for? If you really don't want to waste a single byte of memory though, just call the overload of `read()` which reads a single byte at a time. Then you don't even have the overhead of allocating an array. Of course, that overhead is almost certainly insignficant, and I think you're optimizing inappropriately, but if you don't want to waste *any space at all*... – Jon Skeet Oct 09 '13 at 07:15

score 0 · Answer 2 · answered Oct 09 '13 at 06:18

0

It doesn't matter what size the buffer is from the point of view of the code. A fixed size of 8192 is sufficient for most purposes, regardless of the file length. Your code will work with any size greater than zero.

answered Oct 09 '13 at 06:18

user207421

305,947
44
307
483

score 0 · Answer 3 · edited May 23 '17 at 12:17

Simply take some small amount of buffer. It works.

Long back stumbled with this sort of situation and the below piece of code working fine for me.

 byte[] b = new byte[1024]; //a little buffer
            int length;                 
            while ((length = in.read(b))!= -1) {
                //writing it to a file
                op.write(b, 0, length);
                k++;
            }

            in.close();
            op.close();

Asked and got clarification here long back :How the buffer byte array is continuously filling while streaming?

Can I make the size of a byte array dynamic?

3 Answers3