0

I'm trying to understand if there is a better way to save only part of a list (only the objects I don't have in the database). With my current solution I'm doing it in O(n^2) complexity and I'm also holding a lot of hashes of the database objects in memory.

my class :

class Product(Base):

    __tablename__ = 'products'
    id = Column('id',BIGINT, primary_key=True)
    barcode = Column('barcode' ,BIGINT)
    productName = Column('name', TEXT,nullable=False)
    objectHash=Column('objectHash',TEXT,unique=True,nullable=False)



    def __init__(self, productData,picture=None):
        self.barcode = productData[ProductTagsEnum.barcode.value]
        self.productName = productData[ProductTagsEnum.productName.value]
        self.objectHash = md5((str(self.barcode)+self.produtName).encode('utf-8')).hexdigest()

my solution :

def saveNewProducts(self,products):
    Session = sessionmaker()
    session=Session()
    productsHashes=[ product.objectHash for product in products]
    query = session.query(Product.objectHash).filter(Product.objectHash.in_(productsHashes))
    existedHashes=query.all()
    newProducts = [ product for product in products if product.objectHash not in productsHashes]
    session.bulk_save_objects(newProducts)
halfer
  • 19,824
  • 17
  • 99
  • 186
JeyJ
  • 3,582
  • 4
  • 35
  • 83
  • Postgresql version 11 – JeyJ Sep 07 '19 at 12:11
  • if you need only records with unique `barcode` values - then set `barcode` column as unique index column – RomanPerekhrest Sep 07 '19 at 12:18
  • @RomanPerekhrest pretty sure u didnt understand my question. I have a list of objects and a table in my db that contains objects. I want to save into the db only the objects in the list that doesnt exist in the db.. – JeyJ Sep 07 '19 at 12:34
  • check this answer out: https://stackoverflow.com/a/44395983/4201810 – georgexsh Sep 07 '19 at 13:01
  • The solution there talks about update in case of a conflict or insert if it doesnt exist. I dont need to update the record in case it exists.. However, I can just use onconflict do none. Is there any way to do it with the session obj and not with an insert like the solution suggested – JeyJ Sep 07 '19 at 13:12
  • @georgexsh - in addition, in that solution I must import insert from postgresql module. I dont want to use a dedicated module per db .. I want to be as much as general as possible and not specific per one db.. – JeyJ Sep 07 '19 at 13:45

0 Answers0