Distributed, decentralized data sharing and backup

Question

Consider the following scenario: you have a team of a few people scattered around the globe that needs to work on the same data. Each member of the team has a copy of the data he needs. When someone make changes to some of the data, he needs to have this data synced to the other members that are working on the same data. Also, some of the data the team members have is "private", in the sense that is useful only to a single memeber (and therefore there's only one copy of it), yet it may be useful to other members in the future and therefore a backup copy would be useful.

Given this situation, I'm looking for a way to have a decentralized share (i.e. some kind of storage drive, be it local or via network) that all users having access rights can read/write to. (We are talking about 3-4 users for now, but it may grow in a near future)

The "decentralized" part means that we don't have a server that can run a traditional share, so it has to run right on our computers (and they are often switched off). This means that each local share should be able to work even if at the time no other node is available.

We need this for a bunch of reasons, but mostly because:

1. we have a lot of data we work on (read/modify/write) daily and we'd like to have it synced
2. we have a lot of data (both shared between multiple users or private) we'd like to have a backup copy of, possibly remotely
3. we don't have connections fast enough to just pull/push everything from/to a server or another node each time we need/modify it (transparent local caching, differential compression)

Most of us are working on windows machines, so a windows compatible solution would be preferable. Any ideas?

Answer 1

What a great question! This subject area can be confusing because there are so many different technologies and options. As you point out, so many of them fail your particular requirements.

Local Distributed Storage - Continuous File Synchronization

As I understand your needs, you have several people each working on their own local hard drive containing TBs of data. But you want it to work AS IF there is just one hard drive that is always the same for everyone and is local. If one person adds or changes a file, all users have that new file or change (almost) immediately on their local hard drives.

ViceVersa Software - $49.95

I believe ViceVersa software from TGRMN Software will do this for you: http://www.tgrmn.com/web/file_synchronization.htm At $59.95 per user ($49.95 each for 2-5 users) it certainly is affordable. It is limited to MS Windows. (I have no financial interest in TGRMN, but a strong interest in over-the-Web synchronization and backup.)

It is truly decentralized. You set up your WAN, presumably a VPN. (But see a simpler option with PogoPlug, below.) On each PC you set up which directories you want to bi-directionally synchronize. I believe you want to set it up so that the newest file overwrites its older, unchanged counterparts. ViceVersa is smart enough that if a file changes on both Source and Target, you reconcile the conflict manually.

File Conflicts with Synchronization

File conflicts are potentially a big issue in a decentralized, synchronized world. You need to make smart choices about how to handle deletes. If you delete a file, do you want it deleted from all other PCs? If someone else was working on a document and you delete it before they save it to disk, what do you want to happen? I believe ViceVersa gives you all the options, but you need to understand their effect and choose wisely.

I am curious about this situation. Assume you have a Docs folder on each of three PCs on your WAN: A, B and C. You have continuous bi-directional synchronization set up between Docs on A and Docs on B. The same sync is set up between B and C. And the same for C and A.

Now you add a file to Docs on A. Let's say it syncs first to B and slightly later, due to a slower connection, to C. So now B tries to sync the file to C but let's say C already got the file and is about to sync it to B. It is really important that B and C each recognize that they each have the same file and DON'T need to sync it.

You can avoid any potential issue arising from these three-way connections by using a PogoPlug or other device as a hub.

Using ViceVersa with PogoPlug or NAS as a Hub

Here is an option that may be of interest if you want to avoid the complexities of setting up a VPN and setting up synchronization between each pair of computers. You can make a large USB drive and even multiple USB drives plugged into a PogoPlug appear as drive letters on all of your individual PCs. (Amazon has good pricing, around $80, on PogoPlug.)

See: www.pogoplug.com

Set up ViceVersa on each PC so that it synchronizes with the P: drive, the USB drive on the PogoPlug. The drive just acts as a go-between. You could do the same thing with a NAS device, but the PogoPlug is amazingly simply to set up and connect to. Any one of your users could host the PogoPlug by connecting it to a port on their cable modem or switch. You do not need to have a PC up and running for the PogoPlug to be available. It does not connect to a PC but straight to the Internet.

Aspera has a good image showing this configuration, but instead of a big server at the center, imagine a $79 PogoPlug and multiple TB USB drives at the center:

www.asperastaging.info/images/connect_deployment_thb.png

Potential Alternatives

I came across a few potential alternatives, but none as promising, simple nor as affordable as ViceVersa and PogoPlug.

FileReplicationPro Terrible website, at least in FireFox $495.00

www.filereplicationpro.com/_cart/products.cfm

Aspera

www.asperastaging.info/en/products/client_software_2/aspera_connect_8

Peer Software

www.peersoftware.com/purchase/registration.aspx

By the way, this is my first post on Super User. I hope others find it useful and vote-worthy despite its length!

Answer 2

Another possibility when you have a vast amount of data or don't want to put your data on 3rd party servers is to build your own Dropbox:

http://fak3r.com/2009/09/14/howto-build-your-own-open-source-dropbox-clone/

Answer 3

For that I use A NAS device.

NAS

The best (and most expensive) seem to be Synology or QNAP but there are many vendors. There are NAS devices in a wide range of prices and capacities. Many of them support multiple disks with at least RAID-0 or RAID-1 and can backup to other NAS or file-servers.

Personally I prefer a NAS that works without needing special client software installed on the PC and which supports Windows, MAC and Linux clients.

For a handful of PCs something like a Synology DS210J might be worth considering, but this isn't a product recommendation - there are many good makes and many good products.

Cloud

Another possibility is to rent online internet-based cloud storage. E.g. Amazon S3, Google Storage, Mozy, DriveHQ, I2drive ...

Local Distributed Storage

A distributed filesystem that maintains consistent availability of data across intermittently active computers may exist, but I suspect is likely to be more difficult to set up than a NAS.

---

Surfnet has an interesting Overview

BitTorrent

See Wikipedia on BitTorrent and this article on how to share large files with your friends (privately).

Answer 4

The easiest way to do this would be to use Dropbox. It automatically synchronizes shared-folders and creates backups.