5

I have an InputStream and the size of the data that is going to come out (Response of HTTP request). For space complexity reasons I can't read the whole thing. What I want is to send the data directly into a new request body. I've tried doing this with OkHttp but I can't get it to work. I'm not aware of any other HTTP clients that can do this.

If at all possible I'd like to avoid messing around with Socket. Any advice?

Edit: Added restriction is that the solution must work with Java 8

Jurgen Voorneveld
  • 229
  • 4
  • 13

3 Answers3

2

I believe the new HttpClient standardized in Java 11 should let you do this. It uses the Flow API (reactive streams), and you can provide a BodyHandler/BodySubscriber that will request/receive the bytes as they come. The HttpClient also lets you specifies a BodyPublisher when you send out a request. So it should only be a matter of tying the subscription forwarded by the request publisher to its subscribers, to the subscription handed out to BodySubscriber by the Http stack, and then having the BodySubscriber's onNext (etc..) invoke the Publisher's subscriber corresponding methods. Note that this is an academic description: I haven't actually tried to implement it. It might require some thinking and some tricks to set up the subscription link but I believe it should work.

Make sure that your implementation of BodySubscriber/BodyPublisher adhere to the reactive streams semantics however - and that they do not block in the callbacks.

https://docs.oracle.com/en/java/javase/11/docs/api/java.net.http/java/net/http/HttpClient.html https://openjdk.java.net/groups/net/httpclient/intro.html

On second thought maybe that's not what you're asking: if you already have an InputStream it's even simpler: just use BodyPublishers.ofInputStream when you send your request.

https://docs.oracle.com/en/java/javase/11/docs/api/java.net.http/java/net/http/HttpRequest.BodyPublishers.html#ofInputStream(java.util.function.Supplier)

daniel
  • 2,665
  • 1
  • 8
  • 18
  • Thats really cool. Alas the corporate environment I'm in is stuck on Java 8 and I'm not allowed to upgrade. – Jurgen Voorneveld Feb 06 '19 at 14:18
  • Oh - too bad. can you use an executor to pump the InputStream and transfer it to the request? The old legacy http stack (HttpURLConnection) exposes the request body as an OutputStream - so if you can use an executor thread for the transfer... – daniel Feb 06 '19 at 14:34
  • I considered it but the HttpURLConnection classes are so broken its probably easier to just use a regular socket and write the request manually. – Jurgen Voorneveld Feb 06 '19 at 14:46
  • 1
    Well - there's a reason for having a new API in 11... As to use a plain socket directly you might be surprised at all the little dark corner cases that lurk in any HTTP implementation. – daniel Feb 06 '19 at 15:03
0

I would add that if the request is of sufficient size that you are wanting to implement streaming it, it is likely that the receiving service would also want to implement streaming as well.

It also true that while the older existing HttpUrlConnection class, or anything that provides access to the Input and Output streams associated with a given Url connection could support streaming, it is also likely that you would have to code an inordinate amount of support for newer security concerns such as OPTIONS.

Rodney P. Barbati
  • 1,883
  • 24
  • 18
0

You can do something like this:

    File file = new File("path_to_file"); // specify path to the large file you want to upload

    InputStream inputStream = new FileInputStream(file); // Create input stream out of the file. This will be used to stream data directly to request body

    HttpURLConnection connection = (HttpURLConnection) new URL("remote_url").openConnection(); // Open connection. This will not send eny request until explicitly asked. 
    connection.setDoOutput(true); // This will set http method to POST
    connection.setFixedLengthStreamingMode(file.length()); // Define the length of the request body size

    OutputStream output = connection.getOutputStream(); // This is the output stream of the request body. It will be used to stream file bytes into.

    var bytes = inputStream.readNBytes(1024); // Read the first 1024 bytes from the file

    // Following while loop will read chunks of 1024 bytes from the file and write them into the request body output stream.
    // Note the use of the flush function. It will send the current content of the output stream to the remote server. 
    while (bytes.length > 0) {
        output.write(bytes);
        output.flush();
        bytes = inputStream.readNBytes(1024);
    }


    connection.connect(); // This finishes request and returns response

I used this for integration with Flussonic and it does the job.

You can use this answer for further research: https://stackoverflow.com/a/2793153/11676066

Djuradj Gvozdenovic
  • 3
  • 3