How to set the buffer size on a BufferedWriter over a FileWriter

Question

I met a problem with BufferedWriter when I write data to a single file with some threads.

I set the buffer size of the BufferedWriter, but no matter what number I set, it flushes the data to disk when the buffer is 8192 (the default buffer size), not the size I set (here is 16384). Is there a problem with my code?

This is how I'm constructing the BufferedWriter:

new BufferedWriter(new FileWriter(fileName, true), 16384);

This is the full code:

import java.io.BufferedWriter;
import java.io.File;
import java.io.FileWriter;
import java.io.IOException;

public class Test1 {
    public static void main(String[] args) throws IOException {
        for(int i =0;i<10;i++){
            MyThread r = new MyThread();
            Thread t = new Thread(r);
            t.start();
        }
    }
}

class MyThread implements Runnable {
    public void run() {
        String s = "{addffffffkkkljlkj2015dd}\n";
        BufferedWriter bw = null;
        try {
            bw = new BufferedWriter(new FileWriter(
                    "/Users/liaoliuqing/Downloads/1.txt", true),16384);
        } catch (IOException e) {
            e.printStackTrace();
        }
        for(int i =0 ; i<1000; i++){
            try {
                bw.write(String.format("%03d", i)+s);
                //bw.flush();
            } catch (IOException e) {
                e.printStackTrace();
            }
        }
    }
}

Answer 1

Is there a problem with my code?

A few. Mainly: potential IO and concurrency errors. File buffer size might be a lesser concern (and one you can't effectively deal with).

• Trying to open already opened file. All your threads are trying to write into the same file (1.txt). That might be an issue. FileWriter documentation says:

  Some platforms, in particular, allow a file to be opened for writing by only one FileWriter (or other file-writing object) at a time. In such situations the constructors in this class will fail if the file involved is already open.

• Lines might be cut and mixed. If you have several threads with their respective buffers flushing at some point into the same output, you might not even need weird race-conditions or threads stopped right of the middle or a write operation to see your output corrupted.

  As I solution (If your threads must share the same output) you can use a shared object with synchronized access to take care of actual writing. I implemented SafeAppender in my example, but probably there are better alternatives out there.

• No flushing and closing buffers will mean (the tail of) your data will be lost (like tears in the rain). A finally block is usually good to take care of that.

• Also, as stated by other users, BufferedWriter buffer size does not affect the buffer size in FileOutputStream (and so FileWriter). And it looks the java.io and java.nio APIs dont offer any way to mess with that. If you look at the Java library sources you might notice BufferedWriter buffer size just means the amount of chars you store before actually writing into the delegate output. The default size (8192) is optimal for most cases, and increasing it might mean more trouble (potentially losing more data) than benefits.

This is my code, if it serves you:

// http://stackoverflow.com/questions/32451526/how-to-set-the-buffer-size-on-a-bufferedwriter-over-a-filewriter
public class TestWriter {

public static class SafeAppender {
    private BufferedWriter bw;
    private int users = 0;
    public SafeAppender(File f) throws IOException {
        bw = new BufferedWriter(new FileWriter(f));
    }

    public synchronized void append(String s) throws IOException {
        bw.write(s);
    }
    public synchronized void incrUsers() { 
        users ++; 
    }
    public synchronized void decrUsers() {
        if (--users <= 0) {
            try {
                bw.flush();
                System.err.println("INFO-appender-flush()");
            } catch (Throwable whatever) { /* log-if-you-care*/}
        }
    }
    // Might be called by GC, or not
    @Override protected void finalize() throws Throwable {
        try {
            bw.close();
            System.err.println("INFO-appender-close()");
        } catch (Throwable whatever) { /* log-if-you-care */}
        super.finalize();
    }
}

private static class MyRunnable implements Runnable {
    final static String S = "{addffffffkkkljlkj2015dd}";
    SafeAppender appender;
    String threadId;
    public MyRunnable (SafeAppender a, String tid) {
        appender = a; threadId = tid;
    }

    public void run() {
        appender.incrUsers();
        try {
            for(int i =0 ; i<1000; i++){
                // NOTE: Not a good idea to printStackTrace if each line fails. Let thread fail
                String line = String.format("%s-%03d-%s\n", threadId, i, S);
                appender.append(line);
            }
        } catch (IOException e) {
            System.err.printf("ERROR-%s-%s\n", threadId, e.toString());
        } finally {
            appender.decrUsers();
        }
    }
}

public static void main(String[] args) {
    try {
        File f = File.createTempFile("TestWriter", ".txt");
        System.err.printf("INFO-main-Writing into %s\n", f.getCanonicalPath());
        SafeAppender appender = new SafeAppender (f);
        for(int i =0;i<10;i++){
            MyRunnable r = new MyRunnable(appender, ""+i);
            Thread t = new Thread(r);
            t.start();
        }
    } catch (Throwable e) {
        e.printStackTrace(System.err);
    }
}

}

Answer 2

FileWriter actually uses its own fixed-size 1024 byte buffer. The BufferedWriter on the other hand, show that it uses and 8192 byte buffer size (default), which can be configured by the user to any other desired size.

And to further muddy the waters, the Java 6 implementation of OutputStreamWriter actually delegates to a StreamEncoder, which uses its own buffer with a default size of 8192 bytes. And the StreamEncoder buffer is user-configurable, although there is no way to access it directly through the enclosing OutputStreamWriter.

Answer 3

I solve the problem by using OutputStream, not writer, here is the code:

bw = new BufferedOutputStream(
                new FileOutputStream(new File("/Users/liaoliuqing/Downloads/1.txt"),true),165537);

Answer 4

What you are seeing is not the size of the buffer BufferedWriter, but the size of the buffer used internally by FileWriter. Quoting from the Java Documentation (http://docs.oracle.com/javase/7/docs/api/java/io/FileWriter.html)

The constructors of this class assume that the default character encoding and the default byte-buffer size are acceptable. To specify these values yourself, construct an OutputStreamWriter on a FileOutputStream.

So if you wanted to have a fine grain control on when the data is actually written to the disk you should instantiate your BufferedWriter as

bw = new BufferedWriter(new OutputStreamWriter(new FileOutputStream(new File('my_file.txt),true)));