Transfer multiple files in a directory on a local server to different directories on remote server in Python

Question

There are some files inside a directory on an Informatica Server. I need to transfer these files onto another server based on their file names into different directories.

Say there are 2 files, a.dat_1 and b.dat_2 inside a directory called low on the Informatica server.

I need to transfer this low directory onto another server where the file a.dat_1 goes to a directory say, local and b.dat_2 goes to another directory called local2. This needs to be done in Python.

I have used Paramiko to do simple transfers but not directories. And not in separate directories like local and local2.

score 1 · Answer 1 · answered Oct 09 '12 at 14:15

You can make a tar archive out of your directory. For this use shell command

tar -cvf archive_name.tar directory_name

then transfer this archive to another machine and untar it there:

tar -xvf archive_name.tar

You could also uze gzip or other compressor on your tarred archive to make the transfer faster.

score 1 · Answer 2 · edited May 23 '17 at 10:27

1

Given this SO Directory transfers on paramiko; you won't be able to copy a directory as a bulk operation.

If it were me, I'd create a temp folder and start by copying the source files into a directory structure compatible with your destination requirements. Then compress them as yakxxx suggests, send the compressed file over the wire with Paramiko SFTP and uncompress on the other end with Paramiko SSH.

edited May 23 '17 at 10:27

Community

1
1

answered Oct 09 '12 at 15:15

marklap

471
4
15

The problem with this approach is that, the directories 'loacl' and 'local2' already exist on the second server. So if I make a temporary directory set up and send it over and then uncompress it won't I have a duplicate? – user1452759 Oct 10 '12 at 07:23
You won't have a duplicate directory. You may overwrite files in those directories. Which may or may not be a bad thing for you. You're biggest challenge at that point is coming up with an acceptable process of compression/decompression. Whereas, with your original approach, it seems to me that you'll have additional complexity to deal with. In my opinion its the difference between a process that works like this: "do this then this then this" and one that works like this: "do this and then if this do that else do other". The former is always more attractive to me than the latter. – marklap Oct 10 '12 at 10:05

score 1 · Answer 3 · edited Jun 20 '20 at 09:12

I do something similar for transferring files daily. As yakxxx suggests, I zip my files then transfer. Example of what I do (on Windows machines):

import zipfile
from glob import glob as gg

files = gg('path*.txt')
# open zip file (create it, or open if already exists)
zFile = zipfile.ZipFile('FileName.zip','w') 

# zip files on local machine
[zFile.write(r,r,zipfile.ZIP_DEFLATED) for r in files]

Alternately, if the zipfile already exists and you want to add new files.

zFile = zipfile.ZipFile('FileName.zip','a')

# List files already zipped.  
done = zipfile.ZipFile('FileName.zip','r').namelist()

# zip file into zip file.  
[zFile.write(r,r,zipfile.ZIP_DEFLATED) for r in files if r not in done]

Now push this zip file to your remote machine via paramiko.

---------------

EDIT

FYI, when writing to to zip files you need to be careful. I wrote this assuming you are running within the directory that has the files you want to zip. If you are not, you need to use:

import os
zFile.write(r,os.path.basename(r),zipfile.ZIP_DEFLATED)

Transfer multiple files in a directory on a local server to different directories on remote server in Python

3 Answers3

---------------