Asynchronous File.copy in Java

Question

Is there a way in Java to copy one file into another in an asynchrnous way? Something similar to Stream.CopyToAsync in C# is what I'm trying to find.

What I'm trying to achieve is to download a series of ~40 files from the Internet, and this is the best I've come up with for each file:

CompletableFuture.allOf(myFiles.stream()
        .map(file -> CompletableFuture.supplyAsync(() -> syncDownloadFile(file)))
        .toArray(CompletableFuture[]::class))
    .then(ignored -> doSomethingAfterAllDownloadsAreComplete());

Where syncDownloadFile is:

private void syncDownloadFile(MyFile file) {
    try (InputStream is = file.mySourceUrl.openStream()) {
        long actualSize = Files.copy(is, file.myDestinationNIOPath);
        // size validation here
    } catch (IOException e) {
        throw new RuntimeException(e);
    }
}

But that means I have some blocking calls inside of the task executors, and I'd like to avoid that so I don't block too many executors at once.

I'm not sure if the C# method internally does the same (I mean, something has to be downloading that file right?).

Is there a better way to accomplish this?

Shouldn't you be passing an array of CompletableFutures to allOf and not an array of MyFile? Once you call toArray it's going to block until it populates all the MyFiles — Novaterata, Nov 09 '17 at 18:39
@Novaterata yeah, thanks. I changed the code a bit to remove my company's private stuff and something got messed in the process. — Daniel, Nov 09 '17 at 18:46
So you should factor out the final CompletableFuture into a variable. Then you can do whatever else you need to do and call completableFuture.get() when you need the final results. It won't block until you call get() or complete() or similar, but it will have already started on the work. — Novaterata, Nov 09 '17 at 18:51
But in the process of doing that work, all my workers would have been blocked, isn't it? That's the part I'm not sure of: is it ok to put long running synchronous tasks on the workers? or for this case of copying a file to another there is a better way? — Daniel, Nov 09 '17 at 18:53
Because each one of them is executing `Files.copy`, which is a blocking call. — Daniel, Nov 09 '17 at 19:53

score 2 · Accepted Answer · answered May 08 '20 at 11:17

AsynchronousFileChannel (AFC for short) is the right way to manage Files in Java with non-blocking IO. Unfortunately it does not provide a promises based (aka as Task in .net) API such as the CopyToAsync(Stream) of .Net.

The alternative RxIo library is built on top of the AFC and provides the AsyncFiles asynchronous API with different calling idioms: callbacks based, CompletableFuture (equivalent to .net Task) and also reactive streams.

For instance, copying from one file to another asynchronously can be done though:

Path in = Paths.get("input.txt");
Path out = Paths.get("output.txt");
AsyncFiles
      .readAllBytes(in)
      .thenCompose(bytes -> AsyncFiles.writeBytes(out, bytes))
      .thenAccept(index -> /* invoked on completion */)

Note that continuations are invoked by a thread from the background AsynchronousChannelGroup.

Thus you may solve your problem using a non-blocking HTTP client, with ComplableFuture based API chained with the AsyncFiles use. For instance, AHC is valid choice. See usage here: https://github.com/AsyncHttpClient/async-http-client#using-continuations

Asynchronous File.copy in Java

1 Answers1