How to accelerate a nested loop Jupyter Notebook Python with big tables

Question

this is my code to iterate two tables simultanous

for i in df_MailBox_SentService.index :
    val_s = df_MailBox_SentService.at[i,'MailId']
    for j in df_TBL_SentMail.index :
        if val_s == df_TBL_SentMail.at[j,'ID']:
            df_MailBox_SentService.at[i,'UnifiedMailNumber']=df_TBL_SentMail.at[j,'MailNbr']

you are assigning values to a position determined exclusively by `i` `df_MailBox_SentService.at[i,'UnifiedMailNumber']` but then you have an inner loop that will keep overwriting on that position every time the conditional is triggered (since `j` is not used to determine the position to write to). The loops don't make much sense to me, unless the conditional is guaranteed to trigger only once, in that case you should break the inner loop to improve performance — Sembei Norimaki, Feb 01 '23 at 15:15
It looks like you are using pandas. In pandas you can join two dataframes, so you don't need to loop over them with for loops. If you need help with that, provide a full code example - including code to create the dataframes - so you can get help. Also useful, say how big the big tables are. — 576i, Feb 01 '23 at 15:21
are you looking to https://stackoverflow.com/questions/37697195/how-to-merge-two-data-frames-based-on-particular-column-in-pandas-python — JonSG, Feb 01 '23 at 16:03

score 1 · Accepted Answer · answered Feb 01 '23 at 15:34

1

Try this:

mask = df_MailBox_SentService['MailId'] == df_TBL_SentMail['ID']
df_MailBox_SentService.loc[mask, 'UnifiedMailNumber'] = df_TBL_SentMail.loc[mask, 'MailNbr']

answered Feb 01 '23 at 15:34

jjislam

557
3
8

this is the error : Can only compare identically-labeled Series objects – Aymen Ragguem Feb 01 '23 at 15:40

How to accelerate a nested loop Jupyter Notebook Python with big tables

1 Answers1