7

here's another question about splice(). I'm hoping to use it to copy files, and am trying to use two splice calls joined by a pipe like the example on splice's Wikipedia page. I wrote a simple test case which only tries to read the first 32K bytes from one file and write them to another:

#define _GNU_SOURCE
#include <fcntl.h>
#include <stdio.h>
#include <unistd.h>
#include <errno.h>
#include <string.h>

int main(int argc, char **argv) {
    int pipefd[2];
    int result;
    FILE *in_file;
    FILE *out_file;

    result = pipe(pipefd);

    in_file = fopen(argv[1], "rb");
    out_file = fopen(argv[2], "wb");

    result = splice(fileno(in_file), 0, pipefd[1], NULL, 32768, SPLICE_F_MORE | SPLICE_F_MOVE);
    printf("%d\n", result);

    result = splice(pipefd[0], NULL, fileno(out_file), 0, 32768, SPLICE_F_MORE | SPLICE_F_MOVE);
    printf("%d\n", result);

    if (result == -1)
        printf("%d - %s\n", errno, strerror(errno));

    close(pipefd[0]);
    close(pipefd[1]);
    fclose(in_file);
    fclose(out_file);

    return 0;
}

When I run this, the input file seems to be read properly, but the second splice call fails with EINVAL. Anybody know what I'm doing wrong here?

Thanks!

Community
  • 1
  • 1
  • To anyone reading this, the second `splice` call should only try to read the number of bytes from the pipe as the first `splice` call returned. On todays Linux's, the default pipe size is `65535`. – Jite May 15 '14 at 16:45

3 Answers3

4

From the splice manpage:

   EINVAL Target  file  system  doesn't  support  splicing; target file is
          opened in append mode; neither of the descriptors  refers  to  a
          pipe; or offset given for non-seekable device.

We know one of the descriptors is a pipe, and the file's not open in append mode. We also know no offset is given (0 is equivalent to NULL - did you mean to pass in a pointer to a zero offset?), so that's not the problem. Therefore, the filesystem you're using doesn't support splicing to files.

bdonlan
  • 224,562
  • 31
  • 268
  • 324
  • That was the problem. Thanks! I should have read the man page all the way through, and didn't realize that splice was filesystem-dependent. In my case I was copying to an NFS filer. I'm trying to find the fastest way to copy many files (which average at about 10MB) from one NFS filesystem to another. mmap-ing the source file and using write() aren't performing spectacularly. Thanks for the pointers! –  Oct 19 '09 at 17:29
3

What kind of file system(s) are you copying to/from?

Your example runs on my system when both files are on ext3 but fails when I use an external drive (I forget offhand if it is DOS or NTFS). My guess is that one or both of your files are on a file system that splice does not support.

Duck
  • 26,924
  • 5
  • 64
  • 92
  • NTFS would make sense - that's implemented via FUSE, and the actual filesystem driver runs as a userspace process. With other filesystems, otoh, the filesystem can futz around with the page cache directly. It's a shame `splice()` doesn't have a clean automatic fallback to a copy loop though... – bdonlan Oct 17 '09 at 01:43
  • It is NTFS though splice doesn't appear to work on DOS either. Agree about the fallback. I'm tempted to do some simple benchmarks as it seems impressive to the naked eye test. – Duck Oct 17 '09 at 02:11
2

The splice(2) system call is for copying between files and pipes and not between files, so it can not be used to copy between files, as has been pointed out by the other answers.

As of Linux 4.5 however a new copy_file_range(2) system call is available that can copy between files. In the case of NFS it can even cause server side copying.

The linked man page contains a full example program.

Sebastian Schrader
  • 1,453
  • 15
  • 19