Overwrite parquet files from dynamic frame in AWS Glue

Question

I use dynamic frames to write a parquet file in S3 but if a file already exists my program append a new file instead of replace it. The sentence that I use is this:

glueContext.write_dynamic_frame.from_options(frame = table,
                                         connection_type = "s3",
                                         connection_options = {"path": output_dir,
                                                               "partitionKeys": ["var1","var2"]},
                                         format = "parquet")

Is there anything like "mode":"overwrite" that replace my parquet files?

score 30 · Accepted Answer · answered Aug 25 '18 at 01:15

30

Currently AWS Glue doesn't support 'overwrite' mode but they are working on this feature.

As a workaround you can convert DynamicFrame object to spark's DataFrame and write it using spark instead of Glue:

table.toDF()
  .write
  .mode("overwrite")
  .format("parquet")
  .partitionBy("var_1", "var_2")
  .save(output_dir)

answered Aug 25 '18 at 01:15

Yuriy Bondaruk

4,512
2
33
49

2

Thanks for your answer! – Mateo Rod Aug 27 '18 at 09:11
1

Would this replace all existing files in a partition? Or just those with conflicting names? – Frans May 06 '19 at 11:24
5

It will overwrite all files – Yuriy Bondaruk May 06 '19 at 11:48
@YuriyBondaruk: Does AWS Glue support this overwrite mode now? particularly for write to S3 or DynamoDB. – Thinhbk Nov 02 '22 at 15:50
1

Is the overwrite available now? Couldn't find anything on AWS Glue documentation. @YuriyBondaruk – Rahul Kumar Jan 19 '23 at 21:41
@RahulKumar did you find something? – Yefet Mar 17 '23 at 02:09
1

Did not find anything. I ended up using spark data frames instead as mentioned in the above answer. @Yefet – Rahul Kumar Mar 19 '23 at 12:32

score 8 · Answer 2 · edited Mar 18 '23 at 11:26

As mentioned earlier, AWS Glue doesn't support mode="overwrite" mode. But converting Glue Dynamic Frame back to PySpark data frame can cause lot of issues with big data.

You just need to add signle command i.e. purge_s3_path() before writing dynamic_dataFrame to S3.

glueContext.purge_s3_path(s3_path,  {"retentionPeriod": 0})
glueContext.write_dynamic_frame.from_options(frame = table,
                                     connection_type = "s3",
                                     connection_options = {"path": s3_path,
                                                           "partitionKeys": ["var1","var2"]},
                                     format = "parquet")

Please refer : AWS Documentation

score 5 · Answer 3 · answered Aug 23 '19 at 20:09

If you don't want your process to overwrite everything under "s3://bucket/table_name", you could use

spark.conf.set("spark.sql.sources.partitionOverwriteMode","dynamic")
data.toDF()
    .write
    .mode("overwrite")
    .format("parquet")
    .partitionBy("date", "name")
    .save("s3://folder/<table_name>")

This will only update the "selected" partitions in that S3 location. In my case, I have 30 date-partitions in my DynamicFrame "data".

I'm using Glue 1.0 - Spark 2.4 - Python 2.

Thank you, I couldn't get my job working right until I saw your spark.conf.set usage. — sevencontinents, Nov 01 '20 at 14:01

Overwrite parquet files from dynamic frame in AWS Glue

3 Answers3

Linked