3

How do I tell Storm to use the dependency included in a fat jar over the one in the Storm classpath?

Here is some background/details:

  • joda-time is the dependency in question with 2.0 included in the storm classpath and 2.7 included in the fat jar.
  • running the topology using mvn compile exec:java -Dstorm.topology=ClassName seems to use 2.7 just fine.
  • submitting the topology using storm jar target/filename-jar-with-dependencies.jar ClassName seems to use 2.0 instead of 2.7.
  • filename-jar-with-dependencies.jar was created using mvn package:
    • joda-time 2.7 is in the pom.xml under the dependencies section
    • the correct joda timestamps for 2.7 are found when I do jar tvf target/filename-jar-with-dependencies.jar | grep joda

The reason I even notice this is I see the following warning/error when submitting the topology via the storm command:

WARN  com.amazonaws.services.s3.internal.S3MetadataResponseHandler - Unable to parse last modified date: Mon, 25 May 2015 13:23:29 GMT
java.lang.IllegalStateException: Joda-time 2.2 or later version is required, but found version: 2.0
    at com.amazonaws.util.DateUtils.handleException(DateUtils.java:156) ~[filename-jar-with-dependencies.jar:na]
    at com.amazonaws.util.DateUtils.parseRFC822Date(DateUtils.java:204) ~[filename-jar-with-dependencies.jar:na]
    at com.amazonaws.services.s3.internal.ServiceUtils.parseRfc822Date(ServiceUtils.java:78) ~[filename-jar-with-dependencies.jar:na]
    at com.amazonaws.services.s3.internal.AbstractS3ResponseHandler.populateObjectMetadata(AbstractS3ResponseHandler.java:115) ~[filename-jar-with-dependencies.jar:na]
    at com.amazonaws.services.s3.internal.S3ObjectResponseHandler.handle(S3ObjectResponseHandler.java:52) [filename-jar-with-dependencies.jar:na]
    at com.amazonaws.services.s3.internal.S3ObjectResponseHandler.handle(S3ObjectResponseHandler.java:30) [filename-jar-with-dependencies.jar:na]
    at com.amazonaws.http.AmazonHttpClient.handleResponse(AmazonHttpClient.java:1050) [filename-jar-with-dependencies.jar:na]
    at com.amazonaws.http.AmazonHttpClient.executeOneRequest(AmazonHttpClient.java:724) [filename-jar-with-dependencies.jar:na]
    at com.amazonaws.http.AmazonHttpClient.executeHelper(AmazonHttpClient.java:467) [filename-jar-with-dependencies.jar:na]
    at com.amazonaws.http.AmazonHttpClient.execute(AmazonHttpClient.java:302) [filename-jar-with-dependencies.jar:na]
    at com.amazonaws.services.s3.AmazonS3Client.invoke(AmazonS3Client.java:3672) [filename-jar-with-dependencies.jar:na]
    at com.amazonaws.services.s3.AmazonS3Client.getObject(AmazonS3Client.java:1160) [filename-jar-with-dependencies.jar:na]
Caused by: java.lang.IllegalArgumentException: Invalid format: "Mon, 25 May 2015 13:23:29 GMT" is malformed at "GMT"
    at org.joda.time.format.DateTimeFormatter.parseMillis(DateTimeFormatter.java:747) ~[joda-time-2.0.jar:2.0]
    at com.amazonaws.util.DateUtils.parseRFC822Date(DateUtils.java:202) ~[filename-jar-with-dependencies.jar:na]
    ... 15 common frames omitted
Matthias J. Sax
  • 59,682
  • 7
  • 117
  • 137
Gloopy
  • 37,767
  • 15
  • 103
  • 71
  • Did you ever find a real answer to this question? I'm running into the same kind of issues where my fat jar's classes are ignored in favor of local classpath – Havnar Jun 03 '16 at 10:49

2 Answers2

1

One workaround I found after reading this post is to replace the library (in my case joda-time) in Storm's /lib directory with the newer version since all .jar files in that directory show up in the storm classpath by default.

I'm hoping there is a better answer in case the two versions of a library need to co-exist for one reason or another.

Community
  • 1
  • 1
Gloopy
  • 37,767
  • 15
  • 103
  • 71
1

In my case with the Maven 3.3.3 and 'Wildfly 9.0.0-RC2', i need add in pom.xml

<dependency>
   <groupId>joda-time</groupId>
   <artifactId>joda-time</artifactId>
   <version>2.8.1</version>
</dependency>

This solved my problem.

If you look at the source code of the 'aws-sdk-java' you will see the following validation. https://github.com/aws/aws-sdk-java/blob/1.10.1/aws-java-sdk-core/src/main/java/com/amazonaws/util/DateUtils.java

/**
 * Returns the original runtime exception iff the joda-time being used
 * at runtime behaves as expected.
 *
 * @throws IllegalStateException if the joda-time being used at runtime
 * doens't appear to be of the right version.
 */
private static <E extends RuntimeException> E handleException(E ex) {
    if (JodaTime.hasExpectedBehavior())
        return ex;
    throw new IllegalStateException("Joda-time 2.2 or later version is required, but found version: " + JodaTime.getVersion(), ex);
}
frekele
  • 644
  • 6
  • 13