1

I have Map in distributed mode step in a state machine, which iterates over large number of entries in S3 (>300,000 items).

Steps in the map all succeed, but the Map step itself then fails with The state/task 'process-s3-file' returned a result with a size exceeding the maximum number of bytes service limit. error.

I've followed advice in this stackoverflow question and set OutputPath to null, which means that every single step returns empty object now {}, but that didn't help for some reason. I've also set ResultPath of the Map step to Discard result and keep original input, but that also didn't help. I can confirm that each individual interation inside map is returning empty object ({}) now.

Am I missing something?

Tofig Hasanov
  • 3,303
  • 10
  • 51
  • 81

1 Answers1

0

I was able to fix this by enabling S3 export of results. Apparently, that overwrites the output entirely and no longer triggers output size error.

Tofig Hasanov
  • 3,303
  • 10
  • 51
  • 81