119

I want to use rsync to synchronize two directories in both directions.

I refer to synchronization in classical sense (not how it is meant in rsync manuals):
I want to update the directories in both directions, depending on which of them is newer.

Can this be done by rsync (preferable in a Linux-way)?
If not, what other solutions exist?

Braiam
  • 1
  • 11
  • 47
  • 78
java.is.for.desktop
  • 10,748
  • 12
  • 69
  • 103

8 Answers8

110

Just run it twice, with "newer" mode (-u or --update flag) plus -t (to copy file modified time), -r (for recursive folders), and -v (for verbose output to see what it is doing):

rsync -rtuv /path/to/dir_a/* /path/to/dir_b
rsync -rtuv /path/to/dir_b/* /path/to/dir_a

This won't handle deletes, but I'm not sure there is a good solution to that problem with only periodic sync'ing.

Octopus
  • 8,075
  • 5
  • 46
  • 66
jsight
  • 27,819
  • 25
  • 107
  • 140
  • 4
    Since the question was about rsync, I'll mark this answer as correct. – java.is.for.desktop Oct 26 '09 at 22:22
  • 2
    But you asked about other solutions also and Unison do a better job, otherwise I would never mentioned another tool. I understand that you perhaps have specific circumstances that favor the use of rsync. – MaD70 Oct 27 '09 at 17:43
  • 1
    Sheesh, you get 6 upvotes and you're still complaining? Unison is a really interesting solution, and I'm glad that you mentioned it. :) – jsight Oct 28 '09 at 15:05
  • 2
    Well I'm not complaining, I'm explaining. Just to check that I'm not missing something (not being a native English speaker). – MaD70 Oct 30 '09 at 02:27
  • 1
    MaD70 - No problem... I was just kidding. :) – jsight Nov 02 '09 at 04:14
  • 1
    Using -t (--times) is quite relevant when syncing both ways, otherwise it will have to re-check the files every time. – fiddur Mar 24 '13 at 05:56
  • 1
    how to convert this 'rsync -a -vv ..' | grep -v 'uptodate' ? rsync -rtu -vv .. | grep -v 'uptodate' ? – stackdave Jan 10 '18 at 04:22
  • I wish somebody could [make a Bash alias](https://stackoverflow.com/a/7131683/4999991) to do this in just one command. for example `rsync -b arg1 arg2` to sync in both directions with all the good parameters included in rsync. – Foad S. Farimani Jan 29 '18 at 15:23
  • FYI: you can delete using the `--delete` flag. There are several other delete variations as well. Check man for more details – bsheps Jul 24 '23 at 21:03
72

Do you know Unison File Synchronizer?

Unison is a file-synchronization tool for Unix and Windows. It allows two replicas of a collection of files and directories to be stored on different hosts (or different disks on the same host), modified separately, and then brought up to date by propagating the changes in each replica to the other. ...

Note also that it is resilient to failure:

Unison is resilient to failure. It is careful to leave the replicas and its own private structures in a sensible state at all times, even in case of abnormal termination or communication failures.

MaD70
  • 4,078
  • 1
  • 25
  • 20
  • 3
    +1. Unison is the best way to do this. Double-rsync is hard to get right and doesn't handle deletes at all. – Avdi Oct 21 '09 at 17:47
  • Yes, Unison is fault resilient. I'm updating my answer to emphasize this. – MaD70 Oct 21 '09 at 17:54
  • 7
    Hadn't heard of Unison. You are responsible of making me redo my backup scripts at home today. – Esteban Küber Oct 21 '09 at 17:56
  • 1
    I'm glad that my answer is so useful (this shows how could be practical a tool based on a sound theoretical foundation). – MaD70 Oct 21 '09 at 18:04
  • 1
    Here is a [short tutorial](http://www.linuxjournal.com/article/7712) to Unison at linuxjournal. – lepe Sep 05 '14 at 02:27
  • 2
    For the rsync answer, the actual command was given. It would be useful if for unison you provided some explanation about how to do the sync, apart from the copied and pasted description of the software. – Nike Mar 08 '20 at 20:23
38

You need to run rsync twice and I recommend to run it with -au:

rsync -au /local/source/* /remote/destination
rsync -au /remote/destination/* /local/source

-a (a for archive) is a shortcut for -rlptgoD:

  • -r Recurse into sub directories
  • -l Also sync symbolic links
  • -p Also sync file permissions
  • -t Also sync file modification times
  • -g Also sync file groups
  • -o Also sync file owner
  • -D Also sync special (not regular/meta) files

Basically whenever you want to create an identical one-to-one copy using rsync, you should always use -a as that's what most users expect to happen when they talk about "syncing". Other answers here seem to overlook that sometimes the content of a file stays unchanged but its owner may have changed or its access permissions may have changed and in that case rsync would not sync the file which could be fatal.

But you also require -u as that tells rsync to completely leave any file/folder alone, in case it exists already at the destination and has a newer last modification date. Without -u rsync would sync regardless if a file/folder is newer or not.

Please note that this solution cannot handle deleted files. Handling deletes is not easily possible as consider the following situation: A file has been deleted at the source, now how shall rsync know if that file once existed and has been deleted (in that case it must be deleted at the destination as well) or whether it never existed at the source (in that case it must be copied from the destination). These two situations look identical to rsync thus it cannot know how to react correctly. It won't help to sync the other way round as that can lead to the same situation: A file exists at the source but not at the destination. Why? Has it never existed at the destination or has it been deleted? Both cases look identical to rsync.

Sync tools that can reliably sync deleted files usually manage a sync log about all past sync operations. If that log reveals that there once was a file and has been synced but now it is missing, it's clear that it has been deleted. If there never was such a file according to the log, it must be synced. By storing all log entries with timestamps, it's even possible that a deleted file comes back and gets deleted multiple times yet the sync tool will always know what to do and the result is always correct. rsync has no such log, it only relies on the current file state of two sides of the operation.

You can however build yourself a sync command using rsync and a bit POSIX shell scripting which gets already very close to a sync tool as described above. As I needed such a tool myself, here is an answer on Stackoverflow that guides you through the creation of such a script.

Mecki
  • 125,244
  • 33
  • 244
  • 253
6

Thanks jsight

rsync -urv --progress dir_a dir_b && rsync -urv  --progress dir_b dir_a

This would result in the second sync happening immediately after 1st sync is over. In case the directory structure is huge, this will save time, as one does not need to sit before the pc. If the structure is huge, remove the verbose and progress stuff

rsync -ur dir_a dir_b && rsync -ur dir_b dir_a
Mayank Agarwal
  • 885
  • 1
  • 9
  • 7
  • 1
    Just wondering if I should add -t to copy file modified time. Is there reason I should leave -t out? – chrips Oct 15 '18 at 07:44
  • @Chrips No, I don't think so. It seems that the purpose of this answer was just to show that you can use `&&` between the two commands to automatically run the 2nd command immediately after the 1st command (if the 1st command is successful), as opposed to manually running one after the other as in @jsight's answer. Also, I think @jsight's answer was edited to include the `-t` option after this answer was posted. – Ruben9922 Mar 08 '19 at 11:10
1

Use rsync <OPTIONS> [hostname:]source-dir [hostname:]dest-dir

for example:

rsync -pogtEtvr --progress --bwlimit=2000 xxx-files different-stuff

Will sync xxx-files to different-stuff/xxx-files .If different-stuff/xxx-files did not exist, it will create it - i.e. copy it.

-pogtEtv - just bunch of options to preserve file metadata, plus v - verbose and r - recursive

--progress - show progress of syncing in real time - super useful if you copy big files

--bwlimit=2000 - sets maximum speed of copying/syncing (bw = bandwidth)

P.S. rsync is critically important when you work over network in case of local machine you can use commands like cp.

Good Luck!

Bohdan
  • 16,531
  • 16
  • 74
  • 68
1

What you need is Rclone. Rclone ("rsync for cloud storage") is a command line Linux program to sync files and directories to and from different cloud storage providers (box,dropbox,ftp etc) and local filesystems. Rlone supports mirror syncing only.

Another more graphical solution which includes real-time syncing would be to use FreeFileSync, which includes the program RealTimeSync. FreefileSync support 2-way bidirectional syncing which includes handling deletes.

FreFileSync gui

user1461607
  • 2,416
  • 1
  • 25
  • 23
1

I was having the same question and end up using git. It might not fit your situation, but if anyone find this topic and have the same question, you may consider a version control system.

sgon00
  • 4,879
  • 1
  • 43
  • 59
0

I'm using rsync with inotifywait. When you change any file, rsync will be executed.

inotifywait -m --exclude "$_LOG_FILE" -r -e create,delete,delete_self,modify,moved_to --format "%w%f" "$folder"

You need run inotifywait on both host. Please check example inotifywait

iadd
  • 93
  • 5