Using AutoValueSchema in Apache Beam PCollection gives `RuntimeException: Creator parameter arg0 Doesn't correspond to a schema field`

Question

I am trying to have a PCollection of AutoValue-defined objects that I have created, and I've added the appropriate annotations to infer the Schema via DefaultSchema(AutoValueSchema.class). Like so:

@DefaultSchema(AutoValueSchema.class)
@AutoValue
public abstract class MyAutoClas {
  public abstract String getMyStr();
  public abstract Integer getMyInt();

  @CreateSchema
  public static MyAutoClass create(String myStr, Integer myInt) {
    return new AutoValue_MyAutoClass(myStr, myInt);
  }
}

I have a small test case that looks like this:

PCollection<KV<String, MyAutoClass>> result = pipeline
    .apply(Create.of(MyAutoClass.create("abc", 1)))
    .apply(WithKeys.of(in -> in.getMyStr()));

PAssert.that(result).containsInAnyOrder(KV.of("abc", MyAutoClass.create("abc", 1)));
pipeline.run().waitUntilFinish();

When I try to run this, I am seeing the following errors:

[ERROR] testMyAutoValueClass(.....)  Time elapsed: 1.891 s  <<< ERROR!
java.lang.RuntimeException: Creator parameter arg0 Doesn't correspond to a schema field
    at org.apache.beam.sdk.schemas.utils.ByteBuddyUtils$InvokeUserCreateInstruction.<init>(ByteBuddyUtils.java:717)
    at org.apache.beam.sdk.schemas.utils.ByteBuddyUtils$StaticFactoryMethodInstruction.<init>(ByteBuddyUtils.java:660)
    at org.apache.beam.sdk.schemas.utils.JavaBeanUtils.createStaticCreator(JavaBeanUtils.java:284)
    at org.apache.beam.sdk.schemas.utils.JavaBeanUtils.lambda$getStaticCreator$4(JavaBeanUtils.java:273)
    at java.util.concurrent.ConcurrentHashMap.computeIfAbsent(ConcurrentHashMap.java:1664)
    at org.apache.beam.sdk.schemas.utils.JavaBeanUtils.getStaticCreator(JavaBeanUtils.java:269)
    at org.apache.beam.sdk.schemas.AutoValueSchema.lambda$schemaTypeCreatorFactory$673bce5b$1(AutoValueSchema.java:80)
    at org.apache.beam.sdk.schemas.UserTypeCreatorFactory.create(UserTypeCreatorFactory.java:21)
  ....... [ETCETERA] ......

Pablo · Accepted Answer · 2023-04-10T15:44:28.627

This error occurs because ByteBuddy and the Java reflection utilities are not able to infer the parameter names of your @SchemaCreate method (thus complaining about some unknown parameter arg0 - this is the default name).

To ensure Java reflection can find the parameter names, you need to instruct the compiler to include this information. If you are building with Maven, you can do it this way:

<properties>
    <!-- PLUGIN VERSIONS -->
    <maven-compiler-plugin.version>3.1</maven-compiler-plugin.version>

    <!-- OTHER PROPERTIES -->
    <java.version>1.8</java.version>
</properties>

<build>
    <plugins>
        <plugin>
            <groupId>org.apache.maven.plugins</groupId>
            <artifactId>maven-compiler-plugin</artifactId>
            <version>${maven-compiler-plugin.version}</version>
            <configuration>
                <!-- Original answer -->
                <compilerArgument>-parameters</compilerArgument>
                <!-- Or, if you use the plugin version >= 3.6.2 -->
                <parameters>true</parameters>
                <testCompilerArgument>-parameters</testCompilerArgument>
                <source>${java.version}</source>
                <target>${java.version}</target>
            </configuration>
        </plugin>
    </plugins>
</build>

Note: To accomplish the same thing with Gradle, use this answer.

For how to set this in Intellij: https://stackoverflow.com/questions/39217830/how-to-use-parameters-javac-option-in-intellij For gradle users: https://stackoverflow.com/questions/37463902/how-to-pass-parameters-javac-flag-to-java-compiler-via-gradle — Valentyn, May 29 '20 at 00:41
I didn't require further settings in IntelliJ. Simply adding the above to Maven worked a treat. Do make sure you comment out the config that you don't need for your plugin version. — DataMacGyver, Dec 03 '20 at 09:04

Using AutoValueSchema in Apache Beam PCollection gives `RuntimeException: Creator parameter arg0 Doesn't correspond to a schema field`

1 Answers1

Linked