I have created a function which calls our ERP API to recall customer data to place it in a database for further analysis & integration with external tools (ie MailChimp). The API returns 100 records per page and I need help with creating an iterative loop that allows the function to run x number of times as desired. I have two functions. The first one finds out the number of pages in the data call:
def pages():
df = Erply.getCustomers(createdUnixTimeFrom=start,
createdUnixTimeTo=end,
recordsOnPage=100)
count = df.total
pages = int(round(count / 100) + 1)
return pages
The second is the call itself:
def customers(page):
df = Erply.getCustomers(createdUnixTimeFrom=start,
createdUnixTimeTo=end,
recordsOnPage=100,
pageNo=page)
df = df.records
df = df[0]
table = pd.DataFrame.from_dict(df)
return table
I have then created a rather crude if statement which does the job where the number of records is < 6000 but if I want to grab the full dataset (65000) then this loop will be really inefficient plus it isn't scaleable for future demands.
pages = pages()
if pages == 1:
df = customers(1)
elif pages == 2:
df1 = customers(1)
df2 = customers(2)
dfs = [df1, df2]
df = pd.concat(dfs)
elif pages == 3:
df1 = customers(1)
df2 = customers(2)
df3 = customers(3)
dfs = [df1, df2, df3]
df = pd.concat(dfs)
elif pages == 4:
df1 = customers(1)
df2 = customers(2)
df3 = customers(3)
df4 = customers(4)
dfs = [df1, df2, df3, df4]
df = pd.concat(dfs)
elif pages == 5:
df1 = customers(1)
df2 = customers(2)
df3 = customers(3)
df4 = customers(4)
df5 = customers(5)
dfs = [df1, df2, df3, df4, df5]
df = pd.concat(dfs)
df = df.reset_index()
return df
How can I make this loop smart / iterative so that it runs x number of times and returns the results as appended single dataframe? I am relatively new with Python - really loving the scope of it but I struggle with loops a bit!
