1

This is my code:

data = pd.read_csv(annotated_labelled_csv)

l = int(floor(len(data['#filename'])/num_process))
print(l)

for i in range(0,num_process,1):
    with open("./Pfiles/"+str(i)+".pkl", 'wb') as handle:
        print(len(data.iloc[:, l*i : (l*(i+1))-1]))
        pickle.dump(
            (data.iloc[:, l*i : (l*(i+1))-1].to_dict('dict')), 
            handle, 
            protocol=pickle.HIGHEST_PROTOCOL
        )

The length of dataframe is 8933 and I am dividing it by num_process = 19 that gives 470. Dataframe needs to be divided into 19 smaller dataframes. I have written the code above but it is not dividing the dataframe. The length it gives of the sub-dataframe is 8933.

prp
  • 914
  • 1
  • 9
  • 24
meu
  • 125
  • 2
  • 10
  • Does this answer your question? [Split a large pandas dataframe](https://stackoverflow.com/questions/17315737/split-a-large-pandas-dataframe) – Dan Feb 20 '20 at 09:10
  • Maybe you can add a small sample input and the desired output? – prp Feb 20 '20 at 09:10

0 Answers0