2

Every month I am downloading the whole ledger in csv format and saving the same to a folder. How I can read it in pandas as single dataframe

kmpillai1975
  • 23
  • 1
  • 1
  • 4
  • 1
    Does this answer your question? [Import multiple csv files into pandas and concatenate into one DataFrame](https://stackoverflow.com/questions/20906474/import-multiple-csv-files-into-pandas-and-concatenate-into-one-dataframe) – Ravi Dec 03 '20 at 19:51

3 Answers3

8

fthomson's answer but the last line should be

import pandas as pd
import glob
files = glob.glob("path/*.csv")

df = pd.DataFrame()
for f in files:
    csv = pd.read_csv(f)
    df = df.append(csv)
Sabzaliev Shukur
  • 361
  • 5
  • 12
  • 3
    This answer is NOT it. Never append DataFrames repeatedly in a loop as it repeatedly allocated memory! Every `df.append` copies the entire DataFrame. ni1o1's answer is to be used. – Make42 Jan 21 '22 at 15:02
7

This is a faster way. Append small DataFrame after large DataFrame will cost a lot. So a better way is to append all DataFrame into one list and use pd.concat to concat all DataFrame.

import pandas as pd
import glob
files = glob.glob("path/*.csv")
df = []
for f in files:
    csv = pd.read_csv(f)
    df.append(csv)
df = pd.concat(df)
ni1o1
  • 71
  • 1
  • 3
2

I'm doing this from memory but this is the general idea

import pandas as pd
import glob
files = glob.glob("path/*.csv")

df = pd.DataFrame()
for f in files:
    csv = pd.read_csv(f)
    df = df.append(csv)
fthomson
  • 773
  • 3
  • 9