Copy file or directories recursively in Python

Question

Python seems to have functions for copying files (e.g. shutil.copy) and functions for copying directories (e.g. shutil.copytree) but I haven't found any function that handles both. Sure, it's trivial to check whether you want to copy a file or a directory, but it seems like a strange omission.

Is there really no standard function that works like the unix cp -r command, i.e. supports both directories and files and copies recursively? What would be the most elegant way to work around this problem in Python?

Yes, this is a mess. One of the places where, by trying to reflect the underlying system calls, Python makes the visible interface worse. Although it's not difficult to switch between copy-file and copy-tree, it shouldn't have been necessary. Maybe file an enhancement request on the Python bug tracker to allow `copytree` to copy a single file? — bobince, Jan 03 '10 at 12:35

tzot · Accepted Answer · 2021-08-20T15:52:56.070

188

I suggest you first call shutil.copytree, and if an exception is thrown, then retry with shutil.copy.

import shutil, errno

def copyanything(src, dst):
    try:
        shutil.copytree(src, dst)
    except OSError as exc: # python >2.5
        if exc.errno in (errno.ENOTDIR, errno.EINVAL):
            shutil.copy(src, dst)
        else: raise

edited Aug 20 '21 at 15:52

answered Jan 03 '10 at 12:35

tzot

92,761
29
141
204

30

I think it would be much cleaner to simply check if src is a directory using os.path.isdir(src) instead of catching an exception like this. Or is there some special reason one should use an exception here instead? – pafcu Jan 03 '10 at 13:09
43

1) Because in the Python world EAFP (it's easier to ask forgiveness than permission) is preferred to LBYL (look before you leap). I can provide you with links about that, if it sounds new to you. 2) The library function already indirectly checks for that, so why replicate the check? 3) nothing stops the `shutil.copytree` function from improving and managing both cases in the future. 4) Exceptions aren't that exceptional in Python; e.g. an iteration stops by throwing a StopIteration exception. – tzot Jan 03 '10 at 20:27
3

Well, in this case handling the exception takes 6 lines, while checking the type takes 4 lines. Not much, but it adds up in the end. Also, as you say, copytree might someday very well support files as well. But it's impossible to tell what that implementation will be like. Maybe it throws an exception under some circumstance where copy works? In that case my code would suddenly stop working just because of the added functionality. But you are probably right, exceptions are pretty commong in Python, something which I find very annoying, but it's probably because I never seem to get used to it – pafcu Jan 04 '10 at 14:18
Since you mentioned it, the exception-checking code above can be written in 4 lines (just like the dir-or-file-checking code can be written in 2). In any case, exceptions are part of the "norm" in Python, and (forgive me if I'm coming through as patronizing) you probably have to get used to them (or drop Python! :). – tzot Jan 05 '10 at 00:08
5

Actually exceptions do have one clear objective advantage in this case: it is entirely possible (although highly unlikely) that the type changes between the check and the call to the correct function. – pafcu Jan 05 '10 at 14:00
1

I didn't mention this since the shutil.copytree is not an atomic operation, so it's still possible that the type changes while copytree runs. – tzot Jan 05 '10 at 18:38
That's just an implementation detail/bug though. Future versions might fix this (in a way that doesn't break code, so the argument I gave above doesn't apply to this case) – pafcu Jan 06 '10 at 09:23
9

in my personal opinion, adding core functionality in an except is bad practice, doesn't matter which language you are using. it puts functionality into a place, where a lot of developers won't search for. further, if you do not write a comment, a less experienced python developer would not really understand what the purpose of this retry is. and if you need to add a comment for such a trivial thing like here, something on your code style is wrong. finally, writing an if / else will result in a much easier to read code. – this.myself Jan 22 '15 at 09:26
On my Windows 8.1 machine with Python 2.7, trying `copytree` on files raise a different error: [`errno.EINVAL`](https://docs.python.org/2.7/library/errno.html#module-errno). So this EAFP approach isn't robust against seemingly trivial changes in error numbers. – Jean-François Corbett Feb 15 '17 at 15:43
@Jean-FrançoisCorbett You can instead simply do `except Exception as exc:`. The exception type has changed again in Python 3. Better to be more general. – Chris Collett Sep 17 '21 at 17:16
This assumes that exception is thrown only if the destination folder exists. There can be lots of other scenarios for sure. The EAFP thing clearly lead to a bad design decision in this case – Sergei Dec 02 '22 at 10:37
No, this *handles* only the case where the destination folder exist, hence the `raise` for the rest of the cases. Clearly assumptions about assumptions were bad in this case. – tzot Dec 02 '22 at 22:35

score 13 · Answer 2 · edited May 23 '17 at 12:26

To add on Tzot's and gns answers, here's an alternative way of copying files and folders recursively. (Python 3.X)

import os, shutil

root_src_dir = r'C:\MyMusic'    #Path/Location of the source directory
root_dst_dir = 'D:MusicBackUp'  #Path to the destination folder

for src_dir, dirs, files in os.walk(root_src_dir):
    dst_dir = src_dir.replace(root_src_dir, root_dst_dir, 1)
    if not os.path.exists(dst_dir):
        os.makedirs(dst_dir)
    for file_ in files:
        src_file = os.path.join(src_dir, file_)
        dst_file = os.path.join(dst_dir, file_)
        if os.path.exists(dst_file):
            os.remove(dst_file)
        shutil.copy(src_file, dst_dir)

Should it be your first time and you have no idea how to copy files and folders recursively, I hope this helps.

score 6 · Answer 3 · answered Feb 15 '17 at 12:37

6

shutil.copy and shutil.copy2 are copying files.

shutil.copytree copies a folder with all the files and all subfolders. shutil.copytree is using shutil.copy2 to copy the files.

So the analog to cp -r you are saying is the shutil.copytree because cp -r targets and copies a folder and its files/subfolders like shutil.copytree. Without the -r cp copies files like shutil.copy and shutil.copy2 do.

answered Feb 15 '17 at 12:37

gms

325
3
9

1

I don't think you understood the question. Try `shutil.copytree('C:\myfile.txt', 'C:\otherfile')`. It doesn't work. That's what the OP was asking about... 7 years ago. – Jean-François Corbett Feb 15 '17 at 15:09
Of course it doesn't work. Like cp doesn't work with folders. You need a special option. copy and copytree are the best way to handle copying. If copytree could target and files it would be easy to make mistakes. You must know what you are targeting both with Linux and Python. That hard. Plus someone else commented it here, but seeing the question and the replies couldn't resist to answer. The elegant way is to know what you want to do and not a universal copy without any control. – gms Feb 15 '17 at 16:15

score 4 · Answer 4 · edited Aug 18 '21 at 12:57

4

The fastest and most elegant way I've found until now is to use the copy_tree function of distutils.dir_util native package:

import distutils.dir_util
from_dir = "foo/bar"
to_dir = "truc/machin"
distutils.dir_util.copy_tree(from_dir, to_dir)

edited Aug 18 '21 at 12:57

Dharman

30,962
25
85
135

answered Aug 18 '21 at 12:52

Emmanuel

101
8

Does it support copying file if `from_dir = files.txt`? See the question. `Both src and dst must be directory names. If src is not a directory, raise DistutilsFileError.` – Muhammad Yasirroni Nov 05 '22 at 15:29

James Polley · Answer 5 · 2010-01-03T10:24:39.267

1

Unix cp doesn't 'support both directories and files':

betelgeuse:tmp james$ cp source/ dest/
cp: source/ is a directory (not copied).

To make cp copy a directory, you have to manually tell cp that it's a directory, by using the '-r' flag.

There is some disconnect here though - cp -r when passed a filename as the source will happily copy just the single file; copytree won't.

edited Jan 03 '10 at 10:24

answered Jan 03 '10 at 10:17

James Polley

7,977
2
29
33

2

http://docs.python.org/library/shutil.html includes the code for copytree() which demonstrates handling ordinary files, symlinks, and directories. – James Polley Jan 03 '10 at 10:28
1

This answer doesn't address the question. It should be a comment, not an answer. – Jean-François Corbett Feb 15 '17 at 11:51

score -2 · Answer 6 · answered Aug 01 '16 at 12:25

The python shutil.copytree method its a mess. I've done one that works correctly:

def copydirectorykut(src, dst):
    os.chdir(dst)
    list=os.listdir(src)
    nom= src+'.txt'
    fitx= open(nom, 'w')

    for item in list:
        fitx.write("%s\n" % item)
    fitx.close()

    f = open(nom,'r')
    for line in f.readlines():
        if "." in line:
            shutil.copy(src+'/'+line[:-1],dst+'/'+line[:-1])
        else:
            if not os.path.exists(dst+'/'+line[:-1]):
                os.makedirs(dst+'/'+line[:-1])
                copydirectorykut(src+'/'+line[:-1],dst+'/'+line[:-1])
            copydirectorykut(src+'/'+line[:-1],dst+'/'+line[:-1])
    f.close()
    os.remove(nom)
    os.chdir('..')

This code is nice for work individual file check (check overwriting issue), but will not work for binary files such as 'zip'. Why not using simple python file copy instead of line by line read/write? — notilas, Sep 28 '16 at 15:52

Copy file or directories recursively in Python

6 Answers6

Linked

Related