File transfer 1 Tera Byte Bim file

Question

So I have been tasked to look for or create a service that allows file transfers into the Tera Byte. That's 1 file that could be as large as a Tera Byte. At first I scoffed but from what I understand there are files that large. A bim file for instance which is involved with CAD design and such. We have been looking at services like box or dropbox but they do not seem to have this ability. I am now curious if any online service can do this ... which goes down to the basic question ... Can a file 1TB large even be uploaded downloaded over the internet. I find it difficult to believe even a copy from hard drive to hard drive would not be fraught with peril.

Answer 1

Depending on the speed of the connection between locations, it could take ~1 day on a 100mb connection end to end, to ~10 days on a 10mb connection.

Your best best for tranferring by Internet, as DavidPostill pointed out, would be a torrent. Torrents are very fault tolerant. Torrents break large files down into very small chunks and does hashes on those chunks. This means if there was an error during the transfer of that small chunk, it would be discarded and redownloaded. This would be mandatory, as you do not want to keep downloading a large file repeatedly over a poor connection. Another advantage of torrents is that the more nodes involved in the transfer, the faster it will go.

Many torrent programs will let you make a torrent. If you are looking for something more automated and/or comes with support, look at BitTorrent Sync.

Another thing to point out, this is a massive bandwidth drain. You might have to limit your torrent speed, otherwise your users will complain of slow internet.

You might be able to compress the file to reduce it's size. I believe CAD drawings would compress well, however the time it takes to compress a file that large might be a factor. Compression programs like 7-Zip could significantly reduce the file size. However, depending on the file and the overall power of the computer running the compression, you are looking at many hours, at a minimum, to compress the file.

Another issue to consider is your terms of service with both your ISP and the remote locations' ISPs. Do you have unlimited bandwidth? Or do you pay per megabyte? Will they throttle your transfer?

Again, as DavidPostill pointed out, it would be faster and easier to mail a hard drive. 1 or 2TB SSDs are cheap these days. They are also safer to mail, as they have no moving parts and are extremely unlikely to be damaged while transferrring.