0

I have 7 different data frames that I would like to merge. When I use a basic merge function like the following, I get an error: 

 new <- list(A, B, C, D, E, F, G) %>% Reduce(function(df1, df2, df3, df4, dtf5, df6, df7) left_join(df1,df2,by="ID"), .)

 Error: cannot allocate vector of size 9.9 Gb

So I would like to approach this by choosing a select few columns in each to merge. The data sets would look like this but with more columns and rows. 

A           B        C       D       E       F          G
ID C1 C2    ID C3    ID C4   ID C5   ID C6   ID C7 C8   ID C9
1L 5  7     1L  3    2L  4   1L  10  2L  4   1L  5  9   1L  4
2L 9  3     2L  4    3L  7   2L  4           2L  0  10  2L  9
                             3L  0

Once merged: 

new
ID  C1  C2  C3  C4  C5  C6  C7  C8  C9
1L  5   7   3       10      5   9   4 
2L  9   3   4   4   4   4   0   10  9
3L              7   0

Something I've tried is this: 

ncombined <- merge(x = A, y = B[,c("C3")], by = "ID", all.x = TRUE)
Reduce(function(dtf1, dtf2) merge(dtf1, dtf2, by = "i", all.x = TRUE),
   list(A[,c("C1", 
"C2")],B[,c("C3")],C[,c("C4")],D[,c("C5")],E[,c("C6")],F[,c("C7", 
"C8")],G[,c("C9")]))

(Pulled from examples: Simultaneously merge multiple data.frames in a list merge only one or two columns from a different dataframe in R)

STL
  • 283
  • 1
  • 3
  • 13

1 Answers1

1

Probably not the most memory efficient way but you could try:

library(data.table)

data <- list(df1, df2, df3, df4, df5, df6, df7)
lapply(data, setDT)
for (df in data[-1]) df1 <- merge(df1, df, by = "ID", all = TRUE)

And that should join all your data frames with df1.

josemz
  • 1,283
  • 7
  • 15
  • Thanks for your suggestion. I was wondering if this would work with files not named df or if there were duplicates? – STL Oct 25 '18 at 18:32
  • You can fill the list `data` with all your different dataframes and the for loop iterates through each one, referencing them as `df`. The dataframes, however, can have any name you want. Notice how I remove the first entry in the list, which is the one that gets joined with the rest. Hope this helps. – josemz Oct 25 '18 at 21:53