append dataframe in nested loop

Question

I have the following code that does some calculation in a function depending on 2 variables (bh and frequency). when i run the code without loop with a fixed value of bh i get a proper output and data frame saved in a .csv file.:

    bh   frequency                Re                  Im
0  1e-05         1  5.86848609615851  -0.999374346845734
1  1e-05        11  4.34298196390882  -0.994875549720418
2  1e-05        21  3.93236459069042   -0.99112086235206
3  1e-05        31  3.68545733552675  -0.987695572513367
4  1e-05        41  3.50849758486341  -0.984487932588323

however, i would like to loop on a list of bh values and frequency when i code in a loop on bh i get the same output as before meaning it doesn't loop. would anyone has a solution to amend the dataframe or save eaxh bh loop output in a new .csv in order to plot the data later on.

from mpmath import *
import numpy as np
import cmath
import math
import pandas as pd

mp.dps = 15; mp.pretty = True
a = mpf(0.25)
b = mpf(0.25)
z = mpf(0.75)
frequency = np.arange(1, 50, 10)  # frequency range
bh = np.arange(10e-6, 30e-6, 10e-6) #10e-6 # width
print(bh)
D = 1e-6 #7.8e-4  # diffusivity
gamma = 0.5772 # Euler constant
v = []
w =[]
i = []
def q(frequency):
  for i in bh:
    # for f in frequency:
      omega = (((i ** 2) * 2 * math.pi * frequency) / D)  # depends on bh and frequency
      u = ((-j/(math.pi * omega))*meijerg([[1, 3/2], []], [[1, 1], [0.5, 0]], j*omega))
      v = np.real(u)
      w = np.imag(u)
      return i, frequency, v, w
#transpose arrays
T = np.vectorize(q)
print(T(frequency))
df = np.array(T(frequency)).T
print(df)
# create DataFrame
df1 = pd.DataFrame(data=df, columns=['bh', 'frequency','Re', 'Im'])
print(df1)
#save in .csv
df1.to_csv('C:\\Users\\calculations\\T.csv')

Answer 1

I am not entirely sure if I understand your problem domain, but you seem to be returning the first iteration of the loop. By having return inside the loop you essentially terminate the for loop early. This should be outside of the loop. Also, you do not save your values in the arrays v, w, and i. You are overwriting the variable.

I have done some modifications (maybe not correct according to your problem domain), but it should do what you want to accomplish.

from mpmath import *
import numpy as np
import cmath
import math
import pandas as pd

mp.dps = 15; mp.pretty = True
a = mpf(0.25)
b = mpf(0.25)
z = mpf(0.75)
frequencies = np.arange(1, 50, 10)  # frequency range
bh = np.arange(10e-6, 30e-6, 10e-6) #10e-6 # width
print(bh)
D = 1e-6 #7.8e-4  # diffusivity
gamma = 0.5772 # Euler constant
v = []
w = []
i = []
bhs = []
freqs = []
def q(frequencies):
  for frequency in frequencies:
    for i in bh:
      # for f in frequency:
        omega = (((i ** 2) * 2 * math.pi * frequency) / D)  # depends on bh and frequency
        u = ((-j/(math.pi * omega))*meijerg([[1, 3/2], []], [[1, 1], [0.5, 0]], j*omega))
        v.append(np.real(u))
        w.append(np.imag(u))
        bhs.append(i)
        freqs.append(frequency)
  return bhs, freqs, v, w

data = np.array(q(frequencies)).T
# create DataFrame
df1 = pd.DataFrame(data=data, columns=['bh', 'frequency','Re', 'Im'])
df1

output:

    bh  frequency             Re                 Im
0   1e-05   1   5.86848609615851    -0.999374346845734
1   2e-05   1   4.98625732244539    -0.99786698700645
2   1e-05   11  4.34298196390882    -0.994875549720418
3   2e-05   11  3.46384911041305    -0.983559190454865
4   1e-05   21  3.93236459069042    -0.99112086235206
5   2e-05   21  3.05626898509369    -0.972212391507732
6   1e-05   31  3.68545733552675    -0.987695572513367
7   2e-05   31  2.81234917403506    -0.962167989599812
8   1e-05   41  3.50849758486341    -0.984487932588323
9   2e-05   41  2.63833200578647    -0.952979213441469
10  1e-05   1   5.86848609615851    -0.999374346845734
11  2e-05   1   4.98625732244539    -0.99786698700645
12  1e-05   11  4.34298196390882    -0.994875549720418
13  2e-05   11  3.46384911041305    -0.983559190454865
14  1e-05   21  3.93236459069042    -0.99112086235206
15  2e-05   21  3.05626898509369    -0.972212391507732
16  1e-05   31  3.68545733552675    -0.987695572513367
17  2e-05   31  2.81234917403506    -0.962167989599812
18  1e-05   41  3.50849758486341    -0.984487932588323
19  2e-05   41  2.63833200578647    -0.952979213441469
20  1e-05   1   5.86848609615851    -0.999374346845734
21  2e-05   1   4.98625732244539    -0.99786698700645
22  1e-05   11  4.34298196390882    -0.994875549720418
23  2e-05   11  3.46384911041305    -0.983559190454865
24  1e-05   21  3.93236459069042    -0.99112086235206
25  2e-05   21  3.05626898509369    -0.972212391507732
26  1e-05   31  3.68545733552675    -0.987695572513367
27  2e-05   31  2.81234917403506    -0.962167989599812
28  1e-05   41  3.50849758486341    -0.984487932588323
29  2e-05   41  2.63833200578647    -0.952979213441469

Answer 2

I'd suggest (1) generating Cartesian product of bh and frequency in advance and (2) vectorize only the part you really need as np.vectorization is known to be costly (i.e., meijerg() which is not a vectorized function). The Cartesian product can be done by pd.MultiIndex.from_product (see this answer).

# run your code until gamma = 0.5772

# Cartesian product of input variables
idx = pd.MultiIndex.from_product([bh, frequency], names=["bh", "frequency"])
df = pd.DataFrame(index=idx).reset_index()

# Omega is vectorized naturally.
omega = (df["bh"].values**2 * df["frequency"].values) * (2 * math.pi / D)

# vectorize meijerg() only, so other operations won't interrupt with this
def f_u(omega_elem):
    return (-j/(math.pi * omega_elem)) * meijerg([[1, 3/2], []], [[1, 1], [0.5, 0]], j*omega_elem)

f_u_vec = np.vectorize(f_u, otypes=[np.complex128]) # output complex

u = f_u_vec(omega)  # np.complex128
df["Re"] = np.real(u)
df["Im"] = np.imag(u)

# output (please make sure your arange was set correctly)
df
Out[35]: 
        bh  frequency        Re        Im
0  0.00001          1  5.868486 -0.999374
1  0.00001         11  4.342982 -0.994876
2  0.00001         21  3.932365 -0.991121
3  0.00001         31  3.685457 -0.987696
4  0.00001         41  3.508498 -0.984488
5  0.00002          1  4.986257 -0.997867
6  0.00002         11  3.463849 -0.983559
7  0.00002         21  3.056269 -0.972212
8  0.00002         31  2.812349 -0.962168
9  0.00002         41  2.638332 -0.952979

If you want to save separate csv files, you can do something like this:

for bh_elem in bh:
    fname = f"bh={bh_elem:.4e}.csv"
    df_save = df[(df["bh"]==bh_elem)]
    df_save.to_csv(fname)

N.B. tested on pandas 1.1.3 and python 3.7, debian 10 64-bit