Improve speed of 3 for loops in R

Question

I am working with a matrix set_onco of 206 rows x 196 cols and I have a vector, genes_100 (it's a matrix but I take only the first col), with 101 names. here's a snippet of how they look

> set_onco[1:10,1:10]
                             V2       V3        V4        V5      V6     V7     V8      V9     V10      V11
GLI1_UP.V1_DN             COPZ1 C10orf46 C20orf118   TMEM181   CCNL2  YIPF1  GTDC1    OPN3   RSAD2  SLC22A1
GLI1_UP.V1_UP            IGFBP6 HLA-DQB1     CCND2     PTH1R TXNDC12   M6PR   PPT2   STAU1     IGJ    TMOD3
E2F1_UP.V1_DN           TGFB1I1    CXCL5    POU5F1    SAMD10    KLF2  STAT6 ENTPD6    VCAN  HMGCS1    ANXA8
E2F1_UP.V1_UP             RRP1B     HES1     ADCY6    CHAF1B  VPS37B  GRSF1   TLX2  SSX2IP    DNA2     CMA1
EGFR_UP.V1_DN             NPY1R    PDZK1     GFRA1     GREB1    MSMB   DLC1    MYB SLC6A14   IFI44   IFI44L
EGFR_UP.V1_UP               FGG     GBP1 TNFRSF11B       FGB    GJA1  DUSP6 S100A9     ADM   ITGB6    DUSP4
ERB2_UP.V1_DN             NPY1R    PDZK1     ANXA3     GREB1   HSPB8   DLC1  NRIP1    FHL2    EGR3    IFI44
FAM18B1                                                                                                    
ERB2_UP.V1_UP            CYP1A1  CEACAM5   FAM129A TNFRSF11B   DUSP4 CYP1B1   UPK2    DAB2 CEACAM6 KIAA1199
GCNP_SHH_UP_EARLY.V1_DN   SRRM2 KIAA1217     DEFA1      DLK1   PITX2   CCL2  UPK3B    SEZ6   TAF15     EMP1

genes_100[1:10,1]
 [1] AL591845.1   B3GALT6      RAP1GAP      HSPG2        BX293535.1   RP1-159A19.1 IFI6         FAM76A       FAM176B      CSF3R       
101 Levels: 5_8S_rRNA AC018470.1 AC091179.2 AC103702.3 AC138972.1 ACVR1B AL049829.5 AL137797.2 AL139260.2 AL450326.2 AL591845.1 AL607122.2 B3GALT6 BX293535.1 ... ZNF678

what I want to do is to parse through the matrix and count the frequency at which each row contains the names in genes_100

to do that I created 3 for loops: the first one moves down one row at the time, the second one moves into the row and the third one loops over the list genes_100 checking for matches. at the end I save in a matrix how many times genes_100 matched with the terms in each row, saving also the row names from the matrix (so that I know which one is which)

the code works and gives me the correct output...but it's just really slow!!

a snippet of the output is:

head(result_matrix_100)

                    freq_100
[1,] "GLI1_UP.V1_DN" "0"     
[2,] "GLI1_UP.V1_UP" "0"     
[3,] "E2F1_UP.V1_DN" "0"     
[4,] "E2F1_UP.V1_UP" "0"     
[5,] "EGFR_UP.V1_DN" "0"     
[6,] "EGFR_UP.V1_UP" "0"

I used system.time() and I get:

  user  system elapsed 
 525.38    0.06  530.34

which is way too slow since I have even bigger matrices to parse, and in some cases I have to repeat this 10k times!!!

the code is:

result_matrix_100 <- matrix(nrow=0, ncol=2)

for (q in seq(1,nrow(set_onco),1)) {
  for (j in seq(1, length(set_onco[q,]),1)) {
    for (x in seq(1,101,1)) {
      if (as.character(genes_100[x,1]) == as.character(set_onco[q,j])) {
        freq_100 <- freq_100+1
      }
    }
  }
  result_matrix_100 <- rbind(result_matrix_100, cbind(row.names(set_onco)[q], freq_100))
}

what would you suggest?

thanks in advance :)

Surely the first step in a naive approach would be to simply call `table` on your matrix...? And `++freq_100` doesn't look like R code at all. — joran, Nov 01 '13 at 18:26
@David, I just added snippets from the data and the results, thanks — TriRook, Nov 01 '13 at 18:26
@joran, sorry, I corrected the `++freq_100` mistake. could you elaborate on the `table` suggestion? — TriRook, Nov 01 '13 at 18:30
@Seb, please read: http://stackoverflow.com/q/5963269/2167315 it's very helpful to have code so that anyone can simply copy+paste to create the sample data set. — David, Nov 01 '13 at 18:30
@Seb How about you look up `?table` first, read a little bit, and then see if you still need further explanation. — joran, Nov 01 '13 at 18:31

score 1 · Answer 1 · answered Nov 01 '13 at 18:37

1

Something like this will probably be quite fast:

#Sample data
m <- matrix(sample(letters,206*196,replace = TRUE),206,196)
genes_100 <- letters[1:5]

m1 <- matrix(m %in% genes_100,206,196)
rowSums(m1)

answered Nov 01 '13 at 18:37

joran

169,992
32
429
468

IRTFM · Accepted Answer · 2013-11-01T19:18:59.180

@joran's will possibly be faster although it may not be "factor-safe". Your set_onco values are probably encoded as factor variables (because your genes_100 object clearly is.) This will be safer:

set_onco[] <- lapply(set_onco, as.character)
# that converts a data.frame with factor columns to character valued
# at that point @joran's solution could be used safely
freq100 <- apply(set_onco, 1, function(x) sum(x %in% genes_100) )
# that does a row-by-row count of the number of matches to genes_100
freq100
          GLI1_UP.V1_DN           GLI1_UP.V1_UP           E2F1_UP.V1_DN 
                      0                       0                       0 
          E2F1_UP.V1_UP           EGFR_UP.V1_DN           EGFR_UP.V1_UP 
                      0                       0                       0 
          ERB2_UP.V1_DN           ERB2_UP.V1_UP GCNP_SHH_UP_EARLY.V1_DN 
                      0                       0                       0

The size of your dataset (206 rows x 196 cols) is quite small so this will be virtually immediate. These dput statements and output can be used to construct what I think your objects look like internally:

dput(set_onco)
structure(list(V2 = structure(c(1L, 4L, 8L, 6L, 5L, 3L, 5L, 2L, 
7L), .Label = c("COPZ1", "CYP1A1", "FGG", "IGFBP6", "NPY1R", 
"RRP1B", "SRRM2", "TGFB1I1"), class = "factor"), V3 = structure(c(1L, 
6L, 3L, 5L, 8L, 4L, 8L, 2L, 7L), .Label = c("C10orf46", "CEACAM5", 
"CXCL5", "GBP1", "HES1", "HLA-DQB1", "KIAA1217", "PDZK1"), class = "factor"), 
    V4 = structure(c(3L, 4L, 8L, 1L, 7L, 9L, 2L, 6L, 5L), .Label = c("ADCY6", 
    "ANXA3", "C20orf118", "CCND2", "DEFA1", "FAM129A", "GFRA1", 
    "POU5F1", "TNFRSF11B"), class = "factor"), V5 = structure(c(7L, 
    5L, 6L, 1L, 4L, 3L, 4L, 8L, 2L), .Label = c("CHAF1B", "DLK1", 
    "FGB", "GREB1", "PTH1R", "SAMD10", "TMEM181", "TNFRSF11B"
    ), class = "factor"), V6 = structure(c(1L, 8L, 5L, 9L, 6L, 
    3L, 4L, 2L, 7L), .Label = c("CCNL2", "DUSP4", "GJA1", "HSPB8", 
    "KLF2", "MSMB", "PITX2", "TXNDC12", "VPS37B"), class = "factor"), 
    V7 = structure(c(8L, 6L, 7L, 5L, 3L, 4L, 3L, 2L, 1L), .Label = c("CCL2", 
    "CYP1B1", "DLC1", "DUSP6", "GRSF1", "M6PR", "STAT6", "YIPF1"
    ), class = "factor"), V8 = structure(c(2L, 5L, 1L, 7L, 3L, 
    6L, 4L, 8L, 9L), .Label = c("ENTPD6", "GTDC1", "MYB", "NRIP1", 
    "PPT2", "S100A9", "TLX2", "UPK2", "UPK3B"), class = "factor"), 
    V9 = structure(c(4L, 8L, 9L, 7L, 6L, 1L, 3L, 2L, 5L), .Label = c("ADM", 
    "DAB2", "FHL2", "OPN3", "SEZ6", "SLC6A14", "SSX2IP", "STAU1", 
    "VCAN"), class = "factor"), V10 = structure(c(8L, 6L, 4L, 
    2L, 5L, 7L, 3L, 1L, 9L), .Label = c("CEACAM6", "DNA2", "EGR3", 
    "HMGCS1", "IFI44", "IGJ", "ITGB6", "RSAD2", "TAF15"), class = "factor"), 
    V11 = structure(c(8L, 9L, 1L, 2L, 6L, 3L, 5L, 7L, 4L), .Label = c("ANXA8", 
    "CMA1", "DUSP4", "EMP1", "IFI44", "IFI44L", "KIAA1199", "SLC22A1", 
    "TMOD3"), class = "factor")), .Names = c("V2", "V3", "V4", 
"V5", "V6", "V7", "V8", "V9", "V10", "V11"), class = "data.frame", row.names = c("GLI1_UP.V1_DN", 
"GLI1_UP.V1_UP", "E2F1_UP.V1_DN", "E2F1_UP.V1_UP", "EGFR_UP.V1_DN", 
"EGFR_UP.V1_UP", "ERB2_UP.V1_DN", "ERB2_UP.V1_UP", "GCNP_SHH_UP_EARLY.V1_DN"
))

dput(factor(genes_100) )
structure(c(1L, 2L, 9L, 7L, 3L, 10L, 8L, 6L, 5L, 4L), .Label = c("AL591845.1", 
"B3GALT6", "BX293535.1", "CSF3R", "FAM176B", "FAM76A", "HSPG2", 
"IFI6", "RAP1GAP", "RP1-159A19.1"), class = "factor")

Improve speed of 3 for loops in R

2 Answers2