What's wrong with my function to load multiple .csv files into single dataframe in R using rbind?

Question

I have written the following function to combine 300 .csv files. My directory name is "specdata". I have done the following steps for execution,

x <- function(directory) {     
    dir <- directory    
    data_dir <- paste(getwd(),dir,sep = "/")    
    files  <- list.files(data_dir,pattern = '\\.csv')    
    tables <- lapply(paste(data_dir,files,sep = "/"), read.csv, header = TRUE)    
    pollutantmean <- do.call(rbind , tables)         
}

# Step 2: call the function
x("specdata")

# Step 3: inspect results
head(pollutantmean)

Error in head(pollutantmean) : object 'pollutantmean' not found

What is my mistake? Can anyone please explain?

if Coursera indeed brought you here, read through the excellent practice assignment that they've provided first. (As of today, the link is: https://github.com/rdpeng/practice_assignment/blob/master/practice_assignment.rmd) — pugmarx, Jul 22 '15 at 05:46

hadley · Accepted Answer · 2016-10-31T13:39:44.017

50

There's a lot of unnecessary code in your function. You can simplify it to:

load_data <- function(path) { 
  files <- dir(path, pattern = '\\.csv', full.names = TRUE)
  tables <- lapply(files, read.csv)
  do.call(rbind, tables)
}

pollutantmean <- load_data("specdata")

Be aware that do.call + rbind is relatively slow. You might find dplyr::bind_rows or data.table::rbindlist to be substantially faster.

edited Oct 31 '16 at 13:39

answered Apr 21 '14 at 13:04

hadley

102,019
32
183
245

8

Or now `dplyr::bind_rows` instead of `dplyr::rbind_list` which has been deprecated. – Sam Firke Apr 17 '15 at 13:12
also, using `readr::read_csv` is much faster than `read.csv`. – amc Oct 28 '16 at 22:39

score 12 · Answer 2 · edited May 23 '17 at 12:17

To update Prof. Wickham's answer above with code from the more recent purrr library which he coauthored with Lionel Henry:

Tbl <-
    list.files(pattern="*.csv") %>% 
    map_df(~read_csv(.))

If the typecasting is being cheeky, you can force all the columns to be as characters with this.

Tbl <-
    list.files(pattern="*.csv") %>% 
    map_df(~read_csv(., col_types = cols(.default = "c")))

If you are wanting to dip into subdirectories to construct your list of files to eventually bind, then be sure to include the path name, as well as register the files with their full names in your list. This will allow the binding work to go on outside of the current directory. (Thinking of the full pathnames as operating like passports to allow movement back across directory 'borders'.)

Tbl <-
    list.files(path = "./subdirectory/",
               pattern="*.csv", 
               full.names = T) %>% 
    map_df(~read_csv(., col_types = cols(.default = "c")))

As Prof. Wickham describes here (about halfway down):

map_df(x, f) is effectively the same as do.call("rbind", lapply(x, f)) but under the hood is much more efficient.

and a thank you to Jake Kaupp for introducing me to map_df() here.

This is the best answer given the current state of the Tidyverse — HAVB, Jun 27 '17 at 23:04

score 5 · Answer 3 · answered May 31 '18 at 12:43

5

This can be done very succinctly with dplyr and purrr from the tidyverse. Where x is a list of the names of your csv files you can simply use:

bind_rows(map(x, read.csv))

Mapping read.csv to x produces a list of dfs that bind_rows then neatly combines!

answered May 31 '18 at 12:43

CClarke

503
7
18

Very elegant. Thanks! – Megatron Jan 13 '20 at 19:26

score 1 · Answer 4 · edited Oct 29 '16 at 07:15

```{r echo = FALSE, warning = FALSE, message = FALSE}

setwd("~/Data/R/BacklogReporting/data/PastDue/global/") ## where file are located

path = "~/Data/R/BacklogReporting/data/PastDue/global/"
out.file <- ""
file.names <- dir(path, pattern = ".csv")
for(i in 1:length(file.names)){
  file <- read.csv(file.names[i], header = TRUE, stringsAsFactors = FALSE)
  out.file <- rbind(out.file, file)
}

write.csv(out.file, file = "~/Data/R/BacklogReporting/data/PastDue/global/global_stacked/past_due_global_stacked.csv", row.names = FALSE) ## directory to write stacked file to

past_due_global_stacked <- read.csv("C:/Users/E550143/Documents/Data/R/BacklogReporting/data/PastDue/global/global_stacked/past_due_global_stacked.csv", stringsAsFactors = FALSE)

files <- list.files(pattern = "\\.csv$") %>%  t() %>% paste(collapse = ", ")
```

Code works - modified from Daniel Marcelino's code. Thanks for taking the time to comment. — Dave Headrick, Oct 17 '16 at 03:44

score 0 · Answer 5 · answered Mar 13 '19 at 18:15

If your csv files are into an other directory, you could use something like this:

readFilesInDirectory <- function(directory, pattern){
  files <- list.files(path = directory,pattern = pattern)
  for (f in files){
    file <- paste(directory,files, sep ="")
    temp <- lapply(file, fread, sep=",")
    data <- rbindlist( temp )
  }
  return(data)
}

CHP · Answer 6 · 2014-04-21T04:04:32.323

In your current function pollutantmean is available only in the scope of the function x. Modify your function to this

x <- function(directory) { 

    dir <- directory

    data_dir <- paste(getwd(),dir,sep = "/")

    files  <- list.files(data_dir,pattern = '\\.csv')

    tables <- lapply(paste(data_dir,files,sep = "/"), read.csv, header = TRUE)

    assign('pollutantmean',do.call(rbind , tables))

}

assign should put result of do.call(rbind, tables) into variable called pollutantmean in global environment.

What's wrong with my function to load multiple .csv files into single dataframe in R using rbind?

6 Answers6

Linked

Related