GZIPInputStream reading line by line

Question

I have a file in .gz format. The java class for reading this file is GZIPInputStream. However, this class doesn't extend the BufferedReader class of java. As a result, I am not able to read the file line by line. I need something like this

reader  = new MyGZInputStream( some constructor of GZInputStream) 
reader.readLine()...

I though of creating my class which extends the Reader or BufferedReader class of java and use GZIPInputStream as one of its variable.

import java.io.BufferedReader;
import java.io.FileInputStream;
import java.io.FileNotFoundException;
import java.io.IOException;
import java.io.Reader;
import java.util.zip.GZIPInputStream;

public class MyGZFilReader extends Reader {

    private GZIPInputStream gzipInputStream = null;
    char[] buf = new char[1024];

    @Override
    public void close() throws IOException {
        gzipInputStream.close();
    }

    public MyGZFilReader(String filename)
               throws FileNotFoundException, IOException {
        gzipInputStream = new GZIPInputStream(new FileInputStream(filename));
    }

    @Override
    public int read(char[] cbuf, int off, int len) throws IOException {
        // TODO Auto-generated method stub
        return gzipInputStream.read((byte[])buf, off, len);
    }

}

But, this doesn't work when I use

BufferedReader in = new BufferedReader(
    new MyGZFilReader("F:/gawiki-20090614-stub-meta-history.xml.gz"));
System.out.println(in.readLine());

Can someone advice how to proceed ..

look at this link http://stackoverflow.com/q/6717165/779408. A compress and decompress method is represented there. — Bob, Jan 15 '13 at 10:41
For the love of all that is good and right in this world and for the sanity of any developers who write even remotely worthwhile code.....BE AWARE OF ENCODING AS @erickson POINTS OUT! He is the only answer that points this out, which makes me want to cry. — James, Feb 07 '13 at 20:59

score 155 · Accepted Answer · edited Sep 20 '13 at 20:10

155

The basic setup of decorators is like this:

InputStream fileStream = new FileInputStream(filename);
InputStream gzipStream = new GZIPInputStream(fileStream);
Reader decoder = new InputStreamReader(gzipStream, encoding);
BufferedReader buffered = new BufferedReader(decoder);

The key issue in this snippet is the value of encoding. This is the character encoding of the text in the file. Is it "US-ASCII", "UTF-8", "SHIFT-JIS", "ISO-8859-9", …? there are hundreds of possibilities, and the correct choice usually cannot be determined from the file itself. It must be specified through some out-of-band channel.

For example, maybe it's the platform default. In a networked environment, however, this is extremely fragile. The machine that wrote the file might sit in the neighboring cubicle, but have a different default file encoding.

Most network protocols use a header or other metadata to explicitly note the character encoding.

In this case, it appears from the file extension that the content is XML. XML includes the "encoding" attribute in the XML declaration for this purpose. Furthermore, XML should really be processed with an XML parser, not as text. Reading XML line-by-line seems like a fragile, special case.

Failing to explicitly specify the encoding is against the second commandment. Use the default encoding at your peril!

edited Sep 20 '13 at 20:10

Derek Mahar

27,608
43
124
174

answered Jul 03 '09 at 18:24

erickson

265,237
58
395
493

1

thanks it worked... However, there is no need for reader step .. we can also write it as GZIPInputStream gzip = new GZIPInputStream(new FileInputStream("F:/gawiki-20090614-stub-meta-history.xml.gz")); BufferedReader br = new BufferedReader(new InputStreamReader(gzip)); – Kapil D Jul 04 '09 at 15:55
13

@KapilD it makes me sad that you completely missed his point about the encoding...as shown by your comment and the example in your comment. Re-read erickson's answer....maybe 30 times over. – James Feb 07 '13 at 20:53
How does gzip command know the encoding? I want to read a lot of files from a lot of linux/unix servers from all over the world... so I want to make sure I do this right... The post mentions encoding usually can not be determined by the file itself... but the gzip -d command seems to work on any file without separate input... (its what I use now but want to circumvent) so I figure if I can just figure out what gzip does to know the encoding, I can do the same. Any thoughts/suggestions can anyone point me in the right direction? – glyphx Dec 18 '13 at 15:45
@glyphx Your question is not clear. Do you mean how can you recognize a gzip file in the absence of some external assertion about content type? One hint is the file extension, another is the presence of the magic number 0x1F8B in the file header. However, you can't know a file is a valid gzip file until you actually process the whole thing. – erickson Dec 18 '13 at 18:48
1

To be clear I know these files are gzip files. And the gzipped files are all text based files, like csv and pipe delim files. I just want to be able to read these files directly with java line by line. I can gzip -d them and then read them line by line no problem. I was just confused in your comments about having to specify the encoding... I would think most of the files are ASCII... but some might have asian characters so maybe UTF-8? I just want to make sure I do this correctly... Is that any clearer? Thanks! – glyphx Dec 20 '13 at 20:47
@glyphx I see. The `gzip` command doesn't know the text encoding; it stops after decompressing the file as a string of bytes. It's the other tools on your system that are making assumptions about the character encoding when they render the content as text. In my example, I use an "encoding" parameter that has to be specified somehow. There are some heuristics you can use to guess at the encoding based on content, but they aren't guaranteed. – erickson Dec 20 '13 at 20:52
I see - So the encoding is not needed to unzip but to read it afterwards? Currently I gzip -d the file, and then open it in java and read through line by line with no problem without specifying encoding. I am using a filereader (instead of an inputstreamreader) in my bufferedreader.. is that filereader doing the heuristics for me then? Can the inputstreamreader do the same heuristics? Or is there already built code I can use to do the heuristics? I would prefer to use something well tested, than write my own. Can you point me in the right direction? Thanks! – glyphx Dec 23 '13 at 17:39
Against the Second Commandment then means .. what .. we are idolizing /making an idol of the default encoding? – WestCoastProjects Feb 19 '14 at 21:09

score 45 · Answer 2 · edited Oct 03 '12 at 09:10

45

GZIPInputStream gzip = new GZIPInputStream(new FileInputStream("F:/gawiki-20090614-stub-meta-history.xml.gz"));
BufferedReader br = new BufferedReader(new InputStreamReader(gzip));
br.readLine();

edited Oct 03 '12 at 09:10

aviad

8,229
9
50
98

answered Jul 03 '09 at 18:23

ChssPly76

99,456
24
206
195

Your answer is great. Short and concise .. However, erickson's answer is more detailed. – Kapil D Jul 04 '09 at 15:54

score 4 · Answer 3 · edited Aug 22 '12 at 13:13

4

BufferedReader in = new BufferedReader(new InputStreamReader(
        new GZIPInputStream(new FileInputStream("F:/gawiki-20090614-stub-meta-history.xml.gz"))));

String content;

while ((content = in.readLine()) != null)

   System.out.println(content);

edited Aug 22 '12 at 13:13

j0k

22,600
28
79
90

answered Aug 22 '12 at 13:06

Arumugam Mathiazhagan

69
9

score 3 · Answer 4 · answered Jul 23 '18 at 12:10

You can use the following method in a util class, and use it whenever necessary...

public static List<String> readLinesFromGZ(String filePath) {
    List<String> lines = new ArrayList<>();
    File file = new File(filePath);

    try (GZIPInputStream gzip = new GZIPInputStream(new FileInputStream(file));
            BufferedReader br = new BufferedReader(new InputStreamReader(gzip));) {
        String line = null;
        while ((line = br.readLine()) != null) {
            lines.add(line);
        }
    } catch (FileNotFoundException e) {
        e.printStackTrace(System.err);
    } catch (IOException e) {
        e.printStackTrace(System.err);
    }
    return lines;
}

score 3 · Answer 5 · edited Jun 07 '19 at 12:23

3

here is with one line

try (BufferedReader br = new BufferedReader(
        new InputStreamReader(
           new GZIPInputStream(
              new FileInputStream(
                 "F:/gawiki-20090614-stub-meta-history.xml.gz"))))) 
     {br.readLine();}

edited Jun 07 '19 at 12:23

Chris

5,109
3
19
40

answered Nov 09 '18 at 10:58

Tamer

91
2
10

GZIPInputStream reading line by line

5 Answers5

Linked

Related