1

I am trying to use rclone to upload millions of small files to a storage provider. Using the default settings (--transfers 4) works fine, but it's a waste of the server bandwidth. Each file is approx 1-2KB. My bandwidth is ~50MBps.

I tried using --transfers 1000, which got me the following error from the provider: enter image description here

Their support suggested making persistent connections, but I could not find such an option in rclone.

I looked at the tpslimit option (setting it to 1.6), but that would effectively cap me at max 100 files per minute. I want to cap my connections, not the files transferred.

Is this something I can accomplish either with an rclone option or with a system tweak?

pkExec
  • 300

1 Answers1

0

Create a Dropbox remote in rclone, create your own Dropbox ID and use this:

rclone sync /path/to/your/local/folder YourDropboxRemote: --dropbox-batch-mode async --transfers 64 --checkers=64 --fast-list --size-only --tpslimit 12 --tpslimit-burst 12 -Pv
living being
  • 1,106