read the url content

Question

I want to read the url content by bytes. I have to read the 64 kb from the content of url.

public void readUrlBytes(String address) {
    StringBuilder builder = null;
    BufferedInputStream input = null;
    byte[] buffer = new byte[1024];
    int i = 0;
    try {
        URL url = new URL(address);
        URLConnection urlc = url.openConnection();
        input = new BufferedInputStream(urlc.getInputStream());
        int bytesRead;
        while ((bytesRead = input.read(buffer)) != -1) {
            builder.append(bytesRead);
            if (i==64) {
                break;
            }
            i++;
        }
        System.out.println(builder.toString());
    } catch (IOException l_exception) {
        //handle or throw this
    } finally {
        if (input != null) {
            try {
                input.close();
            } catch(IOException igored) {}
        }
    }

}

The above coding is for read character wise.

I need to read bytes.

What are you asking? The method `java.io.InputStream.read()` is reading a `byte` from the stream. — Nick Holt, Jun 18 '10 at 14:55
Ok, after looking over the new code you pasted in, there are a few other problems. You never instantiate the StrinngBuilder, you will get a NullPointerException when you first try to use it. Secondly, you can't call append(byte[]) and expect anything useful to happen. You say you want to read in bytes, but it appears you are trying to get a String after everything is said and done. Is this binary or character data you are trying to read in? — Greg Case, Jun 18 '10 at 15:56

score 1 · Answer 1 · answered Jun 18 '10 at 15:02

Like Bozho said, you already are reading in bytes. However, it's probably more efficient to read everything into a byte array rather than doing it one byte at a time.

BufferedInputStream input = null;
  byte[] buffer = new byte[4096];
  try {
     URLConnection urlc = url.openConnection();
     input=  new BufferedInputStream( urlc.getInputStream() );
     int bytesRead;
     while( ( bytesRead = input.read(buffer) ) != -1 )
     {
       //do something with the bytes, array has data 0 to bytesRead (exclusive)
     }
  }
  catch( IOException l_exception ) {
       //handle or throw this
  }
  finally {
     if (input != null) {
        try {
          input.close();
        }
        catch(IOException igored) {}
     }
  }

thanks for replay, ya, i know this, but i want to read only 64kb from the content. is there any possible? — zahir hussain, Jun 18 '10 at 15:12
I can take a wild guess why you are getting an error (sunspot activity), but if you let us know what the exact error you are getting it'd be easier to diagnose. — Greg Case, Jun 18 '10 at 15:43

Bozho · Answer 2 · 2011-07-28T05:41:14.937

If you remove the cast to char, you have a byte.

If you're going to store the whole content into memory, you can use ByteArrayOutputStream and write each byte to it. Finally call toByteArray() to obtain the array of bytes:

ByteArrayOutputStream baos = new ByteArrayOutputStream();
while ((byteRead = buffer.read()) != -1) {
    baos.write(byteRead);
}

byte[] result = baos.toByteArray();

Update: you mentioned you want only 64 kb. To achieve that just check whether baos.size() has reached 64*1024 and break

score 0 · Answer 3 · answered Jun 18 '10 at 14:57

0

You can simply read directly from the InputStream object returned:

  InputStream istream = urlc.getInputStream(); 

  int byteRead; 
  while ((byteRead = istream.read()) != -1) 
    builder.append(byteRead); 

  istream.close();

answered Jun 18 '10 at 14:57

JTeagle

2,196
14
15

I'd assume the builder is a StringBuilder, so writing an int to it would just append it to a string sequence of numbers. – Bozho Jun 18 '10 at 14:59
i just update my question program. but i am getting error when run the program. – zahir hussain Jun 18 '10 at 15:22
@Bozho - good point, but the question I was answering was how to read bytes, not chars - however, I fluffed anyway as InputStream and BufferedInputStream both read bytes. I think I know now what the OP wanted - see separate answer below. – JTeagle Jun 18 '10 at 19:41

score 0 · Answer 4 · answered Jun 18 '10 at 15:01

This is how I did it,

                    input = urlc.getInputStream();
                    byte[] buffer = new byte[4096];
                    int n = - 1;

                    ByteArrayOutputStream baos = new ByteArrayOutputStream(4096);

                    while ( (n = input.read(buffer)) != -1)
                    {
                            if (n > 0)
                            {
                                    baos.write(buffer, 0, n);
                            }
                    }
                    byte[] bytes = baos.toByteArray();

score 0 · Answer 5 · answered Jun 18 '10 at 19:49

I'm adding a separate answer as I suddenly realised another way the question could be interpreted: I think the OP wants to convert a stream of bytes representing the internal format of characters in a specific character set into the corresponding characters. For example, converting ASCII codes into ASCII characters.

This isn't a complete answer, but hopefully will put the OP on the right track if I've understood correctly. I'm using utf-8 as an example here:

BufferedInputStream istream = new BufferedInputStream(urlc.getInputStream() ); 
int numBytesAvailable = istream.available(); 
byte[] buffer = new byte[numBytesAvailable]; 
istream.read(buffer); 

ByteBuffer tempBuffer = ByteBuffer.wrap(buffer); 
Charset utf8Chars = Charset.forName("UTF-8"); 
CharBuffer chars = utf8Chars.decode(tempBuffer);

Now you have a buffer of chars as Java sees them (you can use chars.array() to get a char[] out of it), so they can be printed as a string.

WARNING: You will need to get the entire stream into a byte buffer before trying to decode; decoding a buffer when you don't know the correct end of the character's internal byte sequence will result in corrupt characters!

score 0 · Answer 6 · edited May 23 '17 at 12:30

You want to get the first 64KB from an URL into a byte[]?

That's easy:

public byte[] getFirst64KbFromUrl(String address) throws IOException {
    InputStream input = null;
    byte[] first64kb = new byte[64 * 1024];
    try {
        input = new URL(address).openStream();
        input.read(first64kb);
    } finally {
        if (input != null) try { input.close(); } catch(IOException ignore) {}
    }
    return first64kb;
}

If you actually have a problem with converting those bytes to String, here's how you could do it:

String string = new String(first64kb);

This however takes the platform default encoding into account. You'd like to use the server-side specified encoding for this which is available in the Content-Type response header.

URLConnection connection = new URL(address).openConnection();
// ...
String contentType = connection.getHeaderField("Content-Type");
String charset = "UTF-8"; // Let's default it to UTF-8.
for (String param : contentType.replace(" ", "").split(";")) {
    if (param.startsWith("charset=")) {
        charset = param.split("=", 2)[1];
        break;
    }
}
// ...
String string = new String(first64kb, charset);

See also:

read the url content

6 Answers6