1

We are using jclouds with Rackspace and when uploading lots of files via cloudfile api (multi threaded)

Once in while we are getting an exception on objectApi.put line (see example code at bottom)

Exception

16-Jul-2015 11:58:00.811 SEVERE [threadsPool-1]      org.jclouds.logging.jdk.JDKLogger.logError error after writing 8192/streaming bytes to https://*****/****.jpg
 java.io.IOException: Error writing request body to server
        at sun.net.www.protocol.http.HttpURLConnection$StreamingOutputStream.checkError(HttpURLConnection.java:3478)
        at sun.net.www.protocol.http.HttpURLConnection$StreamingOutputStream.write(HttpURLConnection.java:3461)
        at com.google.common.io.CountingOutputStream.write(CountingOutputStream.java:53)
        at com.google.common.io.ByteStreams.copy(ByteStreams.java:74)
        at org.jclouds.http.internal.JavaUrlHttpCommandExecutorService.writePayloadToConnection(JavaUrlHttpCommandExecutorService.java:297)
        at org.jclouds.http.internal.JavaUrlHttpCommandExecutorService.convert(JavaUrlHttpCommandExecutorService.java:160)
        at org.jclouds.http.internal.JavaUrlHttpCommandExecutorService.convert(JavaUrlHttpCommandExecutorService.java:64)
        at org.jclouds.http.internal.BaseHttpCommandExecutorService.invoke(BaseHttpCommandExecutorService.java:91)
        at org.jclouds.rest.internal.InvokeHttpMethod.invoke(InvokeHttpMethod.java:90)
        at org.jclouds.rest.internal.InvokeHttpMethod.apply(InvokeHttpMethod.java:73)
        at org.jclouds.rest.internal.InvokeHttpMethod.apply(InvokeHttpMethod.java:44)
        at org.jclouds.reflect.FunctionalReflection$FunctionalInvocationHandler.handleInvocation(FunctionalReflection.java:117)
        at com.google.common.reflect.AbstractInvocationHandler.invoke(AbstractInvocationHandler.java:87)
        at com.sun.proxy.$Proxy176.put(Unknown Source)
        at

Similar issue with S3

can be found here

Example Code

ObjectApi objectApi = cloudFiles.getObjectApi(REGION, container);
ByteSource byteSource = Files.asByteSource(file);
Payload payload = Payloads.newByteSourcePayload(byteSource);
objectApi.put(hashedName, payload);

The question:

Any one has experience some behavior like that? maybe someone has workaround for that kind of issue?

Thanks

Alon

Community
  • 1
  • 1
oak
  • 2,898
  • 2
  • 32
  • 65

1 Answers1

1

Networks are unreliable, so expect some exceptions when using cloud services, especially when dealing with many files. Specifically for jclouds uploads, we have some example code here:

https://github.com/jclouds/jclouds-examples/tree/master/blobstore-uploader

Edit: I have also added a JIRA issue to make sure we add a test specifically for this situation in swift:

https://issues.apache.org/jira/browse/JCLOUDS-965

zacksh
  • 196
  • 5
  • jclouds retry mechanism should handle network errors for repeatable payloads, e.g., ByteSource. Users need not add their own retry logic unless they use InputStream payloads. – Andrew Gaul Jul 16 '15 at 15:16
  • I am investigating potentially missing retry logic. – zacksh Jul 17 '15 at 18:42
  • Thanks for answering. For now the retry logic does not work, so we warp it with a loop and try/catch in order to force the retry. – oak Jul 23 '15 at 17:02