converting nested list with partially empty values to R data.frame

Question

For a single list of data.frames objects, I'd usually have no trouble converting that:

my_df <- do.call("rbind", lapply(my_list, data.frame))

However, the list object I currently have is nested. It is a list of lists of data.frames. A few points to note:

The elements of some child lists within the parent list are empty.
Among child lists with information, some lists have more than one data.frame object.
The number of data.frame objects can vary among child lists.

Here's a simplified example of what I'm dealing with:

List of 3
 $ 1 :List of 2
  ..$ : NULL
  ..$ : NULL
 $ 2 :List of 2
  ..$ :'data.frame':    3 obs. of  2 variables:
  .. ..$ name                   : chr [1:3] "jack" "jim" "joe" "jon"
  .. ..$ value                  : chr [1:3] "10" "12" "13" "14"
  ..$ :'data.frame':    4 obs. of  2 variables:
  .. ..$ name                   : chr [1:4] "jacky" "jane" "juanita" "julia"
  .. ..$ value                  : chr [1:4] "11" "9" "10" "14"
 $ 3 :List of 1
  ..$ :'data.frame':    5 obs. of  2 variables:
  .. ..$ name                   : chr [1:5] "adam" "ashley" "arnold" "avery" "arthur"
  .. ..$ value                  : chr [1:5] "6" "7" "11" "12" "11"

The do.call approach above reports an error that arguments imply differing number of rows, so it seems like my lists with data.frames with different row numbers is causing the issue?

I tried some strategies described in this post but each attempt had its own unique error.

The data.table "rbindlist" approach and dplyr "bind_rows" methods both reported:

fill=TRUE, but names of input list at position 1 is NULL

Thanks for any tips on how to deal with the situation.

See [this](https://stackoverflow.com/questions/2851327/convert-a-list-of-data-frames-into-one-data-frame) — Sonny, Mar 02 '19 at 14:56
I see `Error: Argument 1 must have names`. Any way to specify that what I want to bind are the child lists, not the list of lists? — Devon O'Rourke, Mar 02 '19 at 15:53

score 0 · Answer 1 · answered Mar 02 '19 at 17:11

Consider two do.call runs with Filter in between:

# APPEND ALL ITEMS TO SINGLE, FLAT LIST
df_list <- do.call("c", my_list)
# FILTER OUT NULL ITEMS
df_list <- Filter(NROW, df_list)

# CONCATENATE ALL DFs TO SINGLE DF
final_df <- do.call("rbind", df_list)

ekoam · Answer 2 · 2019-03-02T20:33:47.660

Firstly, the NULL values do not matter because rbind(NULL, NULL, ..., A, B, C, ...) is just the same as rbind(A, B, C, ...).

Secondly, the structure of your list matters. If your nested list is just as simple as your example, then the answer is also straightforward. The code below could solve this problem:

# This list is the same as your example
test <- list(
  list(NULL, NULL),
  list(data.frame(name = c("jack", "jim", "joe", "jon"), 
                  value = c("10", "12", "13", "14")),
       data.frame(name = c("jacky", "jane", "juanita", "julia"), 
                  value = c("11", "9", "10", "14"))),

  list(data.frame(name = c("adam", "ashley", "arnold", "avery", "arthur"),
                  value = c("6", "7", "11", "12", "11")))
)

# This function rbinds the dataframes inside a list
ls_rbind <- function(ls_of_df) do.call(rbind, ls_of_df)
# Applying "ls_rbind" to each of your child lists creates a list, each of whose elements is either a NULL or a dataframe
# Applying "ls_rbind" again to the result list gives you a dataframe you want
result_df <- ls_rbind(lapply(test, ls_rbind))

However, if your nested list is, in fact, more complex, then you may need a more general way to handle it. For example, each child list could be either of the following items:

A non-list item i.e. a dataframe or a NULL
A list that may also contain lists, dataframes or NULLs

In this case, recursions could be helpful. Consider the following code:

# These two lines complicate the list structure 
test[[4]] <- test
test[[1]][[3]] <- test

recr_ls_rbind <- function(ls_of_ls) {
  are_lists <- lapply(ls_of_ls, class) == "list"
  # Any child lists will be recursively evaluated. "part1" is always a list
  part1 <- lapply(ls_of_ls[are_lists], recr_ls_rbind)
  # Applying the above function "ls_rbind" to all non-list child items and then coerce the result into a list. 
  part2 <- list(ls_rbind(ls_of_ls[!are_lists]))
  # Put part1 and part2 into the same list and apply "ls_rbind" to it.
  ls_rbind(c(part1, part2))
}
result_df <- recr_ls_rbind(test)

converting nested list with partially empty values to R data.frame

2 Answers2