How to avoid double for-loop in matrix calculation

Question

My function works, but it is very slow when I have large dataset.

What can I do to speed it up? I know we should avoid using double for-loop, but I don't know why.

Thanks so much!

    n <- 3
    wr <- c(0.9, 0.6, 0.5)
    mat <- matrix(1:9, nrow=3, byrow=TRUE)
    
    tmp    <- matrix(nrow = n, ncol = n)
    out   <- rep(0, n)
    
    colsum <- apply(mat, 2, sum)
    
    for (i in 1:n) {
      for (j in 1:n) {
        tmp[i, j] <- (mat[i, j]/ colsum[j])*(1-wr[j])
      }
    }
    
    for (i in 1:n) {
      out[i] <- 1-sum(tmp[1:n,i])
    }

`1 - rowSums(t(mat) / colSums(mat)* (1-wr))` ... although that transpose may be slow if your data is big — user20650, Jul 04 '20 at 13:18
Regarding why loops are slow (sometimes) you can check out https://stackoverflow.com/questions/7142767/why-are-loops-slow-in-r — kath, Jul 04 '20 at 13:21
.. and can get rid of the minus' `rowSums(t(mat) / colSums(mat)*wr)` — user20650, Jul 04 '20 at 13:40

score 3 · Answer 1 · answered Jul 04 '20 at 13:24

Try this:

n <- 3
wr <- c(0.9, 0.6, 0.5)
mat <- matrix(1:9, nrow=3, byrow=TRUE)

tmp    <- matrix(nrow = n, ncol = n)
out   <- rep(0, n)

colsum <- apply(mat, 2, sum)

for (i in 1:n) {
  for (j in 1:n) {
    tmp[i, j] <- (mat[i, j]/ colsum[j])*(1-wr[j])
  }
}

for (i in 1:n) {
  out[i] <- 1-sum(tmp[1:n,i])
}

# alternatively:
tmp2 = t(t(mat) / colsum * (1-wr))
out2 = 1 - colSums(tmp)

> tmp
            [,1]       [,2]       [,3]
[1,] 0.008333333 0.05333333 0.08333333
[2,] 0.033333333 0.13333333 0.16666667
[3,] 0.058333333 0.21333333 0.25000000
> out
[1] 0.9 0.6 0.5

> tmp2
            [,1]       [,2]       [,3]
[1,] 0.008333333 0.05333333 0.08333333
[2,] 0.033333333 0.13333333 0.16666667
[3,] 0.058333333 0.21333333 0.25000000
> out2
[1] 0.9 0.6 0.5

kath · Accepted Answer · 2020-07-04T13:42:40.247

Using apply can speed things up here:

colsum <- apply(mat, 2, sum)
1 - rowSums(apply(mat, 1, function(x) (x / colsum)*(1-wr)))

We can clearly see a difference in with the microbenchmark package and using a larger n:

n <- 1000
wr <- rep(c(0.9, 0.6, 0.5), length.out=n)
mat <- matrix(1:(n^2), nrow=n, byrow=TRUE)

tmp <- matrix(nrow = n, ncol = n)
out <- rep(0, n)

colsum <- apply(mat, 2, sum)
    
microbenchmark(
  for_loops = {
    for (i in 1:n) {
      for (j in 1:n) {
        tmp[i, j] <- (mat[i, j]/ colsum[j])*(1-wr[j])
      }
    }
    
    for (i in 1:n) {
      out[i] <- 1-sum(tmp[1:n,i])
    }}, 
  apply = {
    out = 1 - rowSums(apply(mat, 1, function(x) (x / colsum)*(1-wr)))
  }, 
  transpose = {
    tmp = t(t(mat) / colsum * (1-wr))
    out = 1 - colSums(tmp)
  }, 
  rowSums = {
    1 - rowSums(t(mat) / colsum * (1-wr))
  }
)

Interestingly the transpose approach by @BellmanEqn seems to be faster than using apply, but using rowSums as proposed by @user20650 instead of a second transpose even tops that on average.

# Unit: milliseconds
#      expr      min        lq      mean   median        uq      max neval cld
# for_loops 198.6269 211.68075 246.55071 220.3864 239.66485 476.6462   100   c
#     apply  21.7299  23.98720  39.97067  29.9156  33.85995 232.0723   100  b 
# transpose  11.1222  11.66100  23.86154  13.6034  19.52560 271.2242   100 a  
#   rowSums   8.6790   9.32655  14.09392  10.0072  15.18220 171.8077   100 a

hello_friend · Answer 3 · 2020-07-05T22:45:10.080

1

Base R one-liner:

1-colSums(t(t(prop.table(mat, 2)) * (1 - wr)))

Or using sweep:

1-rowSums(t(sweep(mat, 2, "/", STATS = colSums(mat))) * (1 - wr))

Data:

n <- 3
wr <- c(0.9, 0.6, 0.5)
mat <- matrix(1:9, nrow=3, byrow=TRUE)

edited Jul 05 '20 at 22:45

answered Jul 04 '20 at 13:38

hello_friend

5,682
1
11
15

How to avoid double for-loop in matrix calculation

3 Answers3