3

I'm using Apache Beam with Scala. To create new ParquetIO.Sink instances with the respective schemas for the different types of streaming data, I'm trying to call .via(Contextful, Contextful) on FileIO.Write. However, IntelliJ cannot find the overloaded method that I'm trying to use and raises this error: Cannot resolve overloaded method 'via'.

FileIO
  .writeDynamic[String, DeserializedEvent]()
  .by(new UDFs.PartitionByEventName())
  .withDestinationCoder(StringUtf8Coder.of())
  .withNumShards(numShards)
  .withNaming(new UDFs.NameFiles())
  .via(
    Contextful.fn[DeserializedEvent, GenericRecord](
      new UDFs.EventToGenericRecord() // SerializableFunction[DeserializedEvent, String]
    ),
    Contextful.fn[String, ParquetIO.Sink](
      new UDFs.SinkParquet() // SerializableFunction[String, ParquetIO.Sink]
    )
  )
  .to(path)

What's the issue here?

Thanks

Berkay Öztürk
  • 393
  • 2
  • 11

1 Answers1

2

Scala's type inference system and overload resolution system don't work together quite perfectly. In this case, the overloads of the via() function are preventing type inference from working properly. Why does scala fail to compile when method is overloaded in a seemingly unrelated way? has a good explanation of the problem.

In your case, you should just need to change the type params of the second Contextful.fn to [String, FileIO.Sink] to make it match the signature of the desired overload.

danielm
  • 3,000
  • 10
  • 15
  • Before your answer, I fixed the problem by changing `ParquetIO.Sink` to `FileIO.Sink[GenericRecord]`, but I did not understand the reason behind the problem. Thanks for the link, it clarified the situation a bit. – Berkay Öztürk Mar 26 '21 at 06:09