Slow 5 kbps uploads to Google Cloud Storage from Android

Question

I'm writing an Android app and want to use Google Cloud Storage for storing potentially large-ish media files. I'm using the Java client library for the JSON API: com.google.api.services.storage.

My problem is performance. I'm sure I must be doing something wrong. I've got file uploads working but it's almost comically slow. It takes approximately 5 minutes to transfer a 1.5 MB image file, so that's something like 5 kbps, which is going to be unusable for my app. I've enabled billing for my app, but I am on the free tier. Surely this isn't the expected performance level? I'm testing this on a Galaxy S4 on broadband wifi. I am using a Service Account OAUth client key to access GCS.

I've tried with and without gzip encoding, and with and without direct (non-resumable) uploads, and with different chunk sizes - default, minimum, multiples of minimum, etc. I get similar results in all cases. Here's my upload function:

public void uploadFile(String bucketName, String filePath, String mimeType, String gcsFilename, IOProgress ioProgress)
        throws IOException {

    StorageObject object = new StorageObject();
    object.setBucket(bucketName);

    File file = new File(filePath);
    Long fileSize = file.length();
    Log.d(TAG, "uploadFile START: " + bucketName + ":" + gcsFilename + " -> " + filePath);

    try (InputStream stream = new FileInputStream(file)) {
        InputStreamContent content = new InputStreamContent(mimeType,
                stream);

        Storage.Objects.Insert insert = storage.objects().insert(
                bucketName, null, content);
        insert.setName(gcsFilename);
        insert.getMediaHttpUploader().setDisableGZipContent(true); // this seems to help to disable... at least when debugging
        // insert.getMediaHttpUploader().setDirectUploadEnabled(true);
        insert.getMediaHttpUploader().setChunkSize(MediaHttpUploader.MINIMUM_CHUNK_SIZE);
        if (ioProgress != null) {
            insert.getMediaHttpUploader().setProgressListener(new CloudUploadProgressListener(ioProgress, fileSize));
        }

        insert.execute();
        Log.d(TAG, "uploadFile FINISH: " + bucketName + ":" + gcsFilename + " -> " + filePath);
    }
}

For comparison, try running "gsutil -m cp SOME_FILE gs://SOME_BUCKET" from a computer to confirm that both your network and GCS can transfer data faster. I'm not familiar enough with the Java side to really troubleshoot... but I'm curious about the ".setChunkSize(MediaHttpUploader.MINIMUM_CHUNK_SIZE)" - if a *very* small chunk size was used, it would transfer very slowly. — Brian Dorsey, Nov 29 '13 at 06:26
Hi Brian, thanks for the response. gsutil reports 220 KBps for uploading from the same LAN, which is about what I would expect. MINIMUM_CHUNK_SIZE is defined as 256 KB, and DEFAULT_ is 10 MB. I don't see any appreciable difference with either one. I also found [this issue and associated tracker ticket](http://stackoverflow.com/questions/16764115/google-cloud-storage-rest-api-takes-too-much-time-when-uploading-a-file?rq=1). I'm planning to try JetS3t to see if that helps. — Preston Landers, Nov 29 '13 at 18:31

score 9 · Accepted Answer · edited May 23 '17 at 12:03

9

My problem was two-fold.

1) Instead of using com.google.api.client.http.javanet.NetHttpTransport, I used com.google.api.client.http.apache.ApacheHttpTransport and saw approximately a 5x speed increase.

This also fixed a separate problem where I got socket errors trying to using Google Cloud Endpoints (with a NetHttpTransport) immediately after doing a GCS transfer.

2) Trying to run the upload while debugging with ADB accounted for the rest of the difference. Running in non-debug mode with ApacheHttpTransport brought my transfer speeds up to about 1.4 Mbps, where my local network has about a 1.8 Mbps capacity.

By the way, I attempted to use JetS3t but ran into this exact problem: IncompatibleClassChangeError exception is thrown when using JetS3t on Android

edited May 23 '17 at 12:03

Community

1
1

answered Dec 01 '13 at 18:19

Preston Landers

460
7
18

The problem is present in Google APIs Java Client Library too. Replacing GoogleNetHttpTransport with ApacheHttpTransport solved the upload speed problem. – user2396466 Oct 12 '14 at 07:03
Even after switching to ApacheHttpTransport there is a comparable time difference... if i upload image direct google storage from chrome browser it take 3.5 sec on a average.. while through code 8 sec on average... any thoughts on that..i am using the same above code without progress listener – Bhuvan Nov 17 '15 at 17:32

Slow 5 kbps uploads to Google Cloud Storage from Android

1 Answers1