java.io.IOException: Stream closed. what is the best way to write stream data to multiple files?

Question

My java code receives stream data like twitter. I need to store the data e.g. 10000 records for each file. So, I need to recreate the file writer and buffered writer to create a new file then write data on it.

    // global variables
    String stat;
    long counter = 0;
    boolean first = true;
    Date date;
    SimpleDateFormat format;
    String currentTime;
    String fileName;
    BufferedWriter bw = null;
    FileWriter fw = null;

    public static void main(String[] args) {
        String dirToSave = args[0];
        String fileIdentifier = args[1];

        createFile(dirToSave, fileIdentifier);

        StatusListener listener = new StatusListener() {
            @Override
            public void onStatus(Status status) {
                stat = TwitterObjectFactory.getRawJSON(status);

                try {
                    if(bw!=null){
                        bw.write(stat + "\n");
                    }
                } catch (IOException ex) {
                    System.out.println(ex.getMessage());
                }
                counter++;

                if (counter == 10000) {
                    createFile(dirToSave, fileIdentifier);
                    try {
                        TimeUnit.SECONDS.sleep(5);
                    } catch (InterruptedException ex) {
                       System.out.println(ex.getMessage());
                    }
                    counter = 0;
                }
            }
        };

TwitterStream twitterStream = new TwitterStreamFactory(confBuild.build()).getInstance();

    twitterStream.addListener(listener);

    // twitterStream.filter(filQuery);
    }

public static void createFile(String path, String fileIdentifier) {
        date = new Date();
        format = new SimpleDateFormat("yyyyMMddHHmm");
        currentTime = format.format(date);
        fileName = path + "/" + fileIdentifier + currentTime + ".json";

// if there was buffer before, flush & close it first before creating new file
        if (!first) {
            try {
                bw.flush();
                bw.close();
                fw.close();
            } catch (IOException ex) {
                Logger.getLogger(LocalFile_All_en.class
                        .getName()).log(Level.SEVERE, null, ex);
            }
        } else {
            first = false;
        }

        // create a new file
        try {
            fw = new FileWriter(fileName);
            bw = new BufferedWriter(fw);
        } catch (IOException ex) {
            Logger.getLogger(Stack.class
                    .getName()).log(Level.SEVERE, null, ex);
        }
    }

However, i always get error after some hours.

SEVERE: null
java.io.IOException: Stream closed

EDIT: The error message says that, these codes throw the error

if (counter == 10000) {
                    createFile(dirToSave, fileIdentifier);
...

and

bw.flush();

What is the problem of my code? or is there a better way to write stream data like this?

@Alex no, it is only one thread. Receiving stream data and storing to files. — Sugimiyanto, Oct 19 '17 at 07:32
how is the method `onStatus` being called? and which line throws the exception? — Alex, Oct 19 '17 at 07:34
1) Don't call `bw.flush()`, since that will be done by `bw.close()` anyway. --- 2) Stop writing horrendous error handling code. Catching an exception, logging the error, and continuing execution as-if nothing bad happened is BAD, REALLY REALLY BAD! Don't do it. What do you think value of `bw` is if `new FileWriter(fileName)` throws exception? *Yikes!!* — Andreas, Oct 19 '17 at 07:57
and try to avoid all this `static` global stuff. it is the best way to run into problems like this. — pirho, Oct 19 '17 at 07:59
*"i always get error after some hours"* How can this code run even once??? `createFile()` is called, `first` is true, so `if (!first)` skips and sets `first = false`, then `bw.close()` is called, but **`bw` is null**!!! Instant `NullPointerException` and `bw` will never ever be assigned and will *always* be null. Since call to `createFile()` is the first thing done in `main()`, it simply cannot be hours before it fails. — Andreas, Oct 19 '17 at 08:00
Even your program is not multithreaded, the underlying `TwitterStream` does. It will call your `onStatus` method in threads. Therefore, you need to think about it. — Alex, Oct 19 '17 at 08:07
@Andreas thanks for your comment. sorry, the `bw.close` is by mistake — Sugimiyanto, Oct 19 '17 at 08:07

pirho · Answer 1 · 2017-10-20T07:07:32.160

If this error comes every now and then and writing after this error is ok again i think it can happen that bw is closed and not yet reopened while onStatus() tries to write of flush it.

So bw can be be not null but closed. You need to synchronize the closing/opening somehow.

For example make this stuff in onStatus() like so that you do not just write directly to bw but with some callbacks that handle the close/reopen new file.

Update: assuming here that this twitterStream can call onStatus() without waiting previous call finished. The first call has just closed the stream and the second is right after that writing to. Rare, but will happen in a long period of time.

Update2: this applies also to the flush() part.

I added this also as a short comment already but people often tell to get rid of static and especially global statics in java argumenting that it will cause big problems later which are hard to resolve/debug. This might be good case of it.

java.io.IOException: Stream closed. what is the best way to write stream data to multiple files?

1 Answers1