0

I'm trying to find the most efficient way to update or create a bunch of rows (around 10K to 30k) from a json in my database.

I'm grabbing a json with the data I want to put in my database.

I tried Django atomic transactions but it is still very slow. I'm currently using this library but it is very slow: it takes around 1 hour to update 15k rows. But this is the shortest way I've found today.

I read a lot of questions and articles but I can't find a fastest way to do it.

This is how my update methods look like (using the library):

def update_data_in_db(data):
    objs = []
    for item in data:
        try:
            objs.append(Item(
                id=item['id'],
                name=item.get('name'),
                slug=item.get('slug'),
        except Exception as err:
            logger.exception(err)
            print(err)
    try:
        Currency.objects.bulk_update_or_create(objs, ['name', 'slug'], match_field='id')
    except Exception as err:
        print(err)
    return [obj.id for obj in objs]
Paul Bénéteau
  • 775
  • 2
  • 12
  • 36
  • Does this answer your question? [Accelerate bulk insert using Django's ORM?](https://stackoverflow.com/questions/4294088/accelerate-bulk-insert-using-djangos-orm) – ramganesh Nov 12 '21 at 16:54
  • no because it is update and note update or create, remove the duplicate flag please. – Paul Bénéteau Nov 12 '21 at 18:03
  • Does this answer your question? [How to 'bulk update' with Django?](https://stackoverflow.com/questions/12661253/how-to-bulk-update-with-django) – Michał Darowny Nov 15 '21 at 00:12

0 Answers0