I'm using RStudio 2023.03.0 on my machine with Windows 10 Pro 22H2.
My machine has 8 cores and I'm running my code on a cluster of 7 cores with doParallel:
cores=detectCores()
cl <- makeCluster(min(cores[1]-1,7))
registerDoParallel(cl)
on.exit(stopCluster(cl))
output <- foreach(
j=1:100,
.export=c('myfunc'),
.packages=c('mypackages')
) %dopar% {
### do stuff in parallel
}
After initial ~10-30 seconds where each core uses ~12.5% of the available CPU - which brings the overall CPU usage at that moment to ~100% - the CPU usage of each core reduces drastically to only ~8.5% - which brings the total CPU usage to ~65%:
Any ideas where this comes from or how to increase the CPU usage again? Thanks!
Update / additional Info:
I tried the same with %dofuture% and clusterApply and with both packaged I have the same issue.
Sometimes the time where I have maximum CPU usage is larger (Minutes to a few hours) sometimes it directly starts with 8,6% per core.
For reference, my session_info():
─ Session info ──────────────────────────────────────────────────────────────────────────────────────────────────────────
setting value
version R version 4.2.3 (2023-03-15 ucrt)
os Windows 10 x64 (build 19045)
system x86_64, mingw32
ui RStudio
language (EN)
collate German_Germany.utf8
ctype German_Germany.utf8
tz Europe/Berlin
date 2023-08-11
rstudio 2023.03.0+386 Cherry Blossom (desktop)
pandoc 2.19.2 @ C:/Program Files/RStudio/resources/app/bin/quarto/bin/tools/ (via rmarkdown)
─ Packages ──────────────────────────────────────────────────────────────────────────────────────────────────────────────
package * version date (UTC) lib source
abind 1.4-5 2016-07-21 [1] CRAN (R 4.2.0)
askpass 1.1 2019-01-13 [1] CRAN (R 4.2.3)
backports 1.4.1 2021-12-13 [1] CRAN (R 4.2.0)
base64enc 0.1-3 2015-07-28 [1] CRAN (R 4.2.0)
cachem 1.0.8 2023-05-01 [1] CRAN (R 4.2.3)
callr 3.7.3 2022-11-02 [1] CRAN (R 4.2.3)
car 3.1-2 2023-03-30 [1] CRAN (R 4.2.3)
carData 3.0-5 2022-01-06 [1] CRAN (R 4.2.3)
checkmate 2.2.0 2023-04-27 [1] CRAN (R 4.2.3)
cli 3.6.1 2023-03-23 [1] CRAN (R 4.2.3)
cluster 2.1.4 2022-08-22 [2] CRAN (R 4.2.3)
codetools 0.2-19 2023-02-01 [2] CRAN (R 4.2.3)
colorspace 2.1-0 2023-01-23 [1] CRAN (R 4.2.3)
corrplot * 0.92 2021-11-18 [1] CRAN (R 4.2.3)
cowplot * 1.1.1 2020-12-30 [1] CRAN (R 4.2.3)
crayon 1.5.2 2022-09-29 [1] CRAN (R 4.2.3)
crul 1.4.0 2023-05-17 [1] CRAN (R 4.2.3)
curl 5.0.0 2023-01-12 [1] CRAN (R 4.2.3)
cvar 0.5 2022-11-03 [1] CRAN (R 4.2.3)
data.table * 1.14.8 2023-02-17 [1] CRAN (R 4.2.3)
devtools * 2.4.5 2022-10-11 [1] CRAN (R 4.2.3)
dgof * 1.4 2022-06-16 [1] CRAN (R 4.2.0)
digest 0.6.31 2022-12-11 [1] CRAN (R 4.2.3)
DistributionUtils 0.6-0 2018-11-27 [1] CRAN (R 4.2.3)
doFuture * 1.0.0 2023-03-24 [1] CRAN (R 4.2.3)
doParallel * 1.0.17 2022-02-07 [1] CRAN (R 4.2.3)
dplyr * 1.1.1 2023-03-22 [1] CRAN (R 4.2.3)
ellipsis 0.3.2 2021-04-29 [1] CRAN (R 4.2.3)
evaluate 0.21 2023-05-05 [1] CRAN (R 4.2.3)
fansi 1.0.4 2023-01-22 [1] CRAN (R 4.2.3)
fastmap 1.1.1 2023-02-24 [1] CRAN (R 4.2.3)
fBasics 4022.94 2023-03-04 [1] CRAN (R 4.2.3)
fGarch * 4022.89 2022-11-05 [1] CRAN (R 4.2.3)
fitdistrplus 1.1-11 2023-04-25 [1] CRAN (R 4.2.3)
flextable 0.9.1 2023-04-02 [1] CRAN (R 4.2.3)
fontBitstreamVera 0.1.1 2017-02-01 [1] CRAN (R 4.2.0)
fontLiberation 0.1.0 2016-10-15 [1] CRAN (R 4.2.0)
fontquiver 0.2.1 2017-02-01 [1] CRAN (R 4.2.3)
foreach * 1.5.2 2022-02-02 [1] CRAN (R 4.2.3)
foreign 0.8-84 2022-12-06 [2] CRAN (R 4.2.3)
Formula 1.2-5 2023-02-24 [1] CRAN (R 4.2.2)
fs 1.6.2 2023-04-25 [1] CRAN (R 4.2.3)
future * 1.32.0 2023-03-07 [1] CRAN (R 4.2.3)
future.apply 1.10.0 2022-11-05 [1] CRAN (R 4.2.3)
gbutils 0.5 2022-05-27 [1] CRAN (R 4.2.3)
gdtools 0.3.3 2023-03-27 [1] CRAN (R 4.2.3)
GeneralizedHyperbolic 0.8-4 2018-05-15 [1] CRAN (R 4.2.3)
generics 0.1.3 2022-07-05 [1] CRAN (R 4.2.3)
gfonts 0.2.0 2023-01-08 [1] CRAN (R 4.2.3)
ggplot2 * 3.4.2 2023-04-03 [1] CRAN (R 4.2.3)
ghyp 1.6.3 2022-11-02 [1] CRAN (R 4.2.3)
globals 0.16.2 2022-11-21 [1] CRAN (R 4.2.2)
glue 1.6.2 2022-02-24 [1] CRAN (R 4.2.3)
gmp * 0.7-1 2023-02-07 [1] CRAN (R 4.2.3)
gridExtra 2.3 2017-09-09 [1] CRAN (R 4.2.3)
gtable 0.3.3 2023-03-21 [1] CRAN (R 4.2.3)
hash * 2.2.6.2 2022-03-22 [1] CRAN (R 4.2.3)
Hmisc * 5.1-0 2023-05-08 [1] CRAN (R 4.2.3)
hms 1.1.3 2023-03-21 [1] CRAN (R 4.2.3)
htmlTable 2.4.1 2022-07-07 [1] CRAN (R 4.2.3)
htmltools 0.5.5 2023-03-23 [1] CRAN (R 4.2.3)
htmlwidgets 1.6.2 2023-03-17 [1] CRAN (R 4.2.3)
httpcode 0.3.0 2020-04-10 [1] CRAN (R 4.2.3)
httpuv 1.6.11 2023-05-11 [1] CRAN (R 4.2.3)
iterators * 1.0.14 2022-02-05 [1] CRAN (R 4.2.3)
jsonlite 1.8.4 2022-12-06 [1] CRAN (R 4.2.3)
KernSmooth 2.23-20 2021-05-03 [2] CRAN (R 4.2.3)
knitr 1.42 2023-01-25 [1] CRAN (R 4.2.3)
ks 1.14.0 2022-11-24 [1] CRAN (R 4.2.3)
later 1.3.1 2023-05-02 [1] CRAN (R 4.2.3)
lattice 0.20-45 2021-09-22 [2] CRAN (R 4.2.3)
lifecycle 1.0.3 2022-10-07 [1] CRAN (R 4.2.3)
listenv 0.9.0 2022-12-16 [1] CRAN (R 4.2.3)
magrittr 2.0.3 2022-03-30 [1] CRAN (R 4.2.3)
MASS * 7.3-58.2 2023-01-23 [2] CRAN (R 4.2.3)
Matrix 1.5-3 2022-11-11 [2] CRAN (R 4.2.3)
matrixStats * 0.63.0 2022-11-18 [1] CRAN (R 4.2.3)
mclust 6.0.0 2022-10-31 [1] CRAN (R 4.2.3)
memoise 2.0.1 2021-11-26 [1] CRAN (R 4.2.3)
mime 0.12 2021-09-28 [1] CRAN (R 4.2.0)
miniUI 0.1.1.1 2018-05-18 [1] CRAN (R 4.2.3)
moments * 0.14.1 2022-05-02 [1] CRAN (R 4.2.0)
munsell 0.5.0 2018-06-12 [1] CRAN (R 4.2.3)
mvtnorm 1.1-3 2021-10-08 [1] CRAN (R 4.2.0)
nnet 7.3-18 2022-09-28 [2] CRAN (R 4.2.3)
numDeriv 2016.8-1.1 2019-06-06 [1] CRAN (R 4.2.0)
officer 0.6.2 2023-03-28 [1] CRAN (R 4.2.3)
openssl 2.0.6 2023-03-09 [1] CRAN (R 4.2.3)
pacman 0.5.1 2019-03-11 [1] CRAN (R 4.2.3)
parallelly 1.35.0 2023-03-23 [1] CRAN (R 4.2.3)
PerformanceAnalytics * 2.0.4 2020-02-06 [1] CRAN (R 4.2.3)
pillar 1.9.0 2023-03-22 [1] CRAN (R 4.2.3)
pkgbuild 1.4.0 2022-11-27 [1] CRAN (R 4.2.3)
pkgconfig 2.0.3 2019-09-22 [1] CRAN (R 4.2.3)
pkgload 1.3.2 2022-11-16 [1] CRAN (R 4.2.3)
plyr 1.8.8 2022-11-11 [1] CRAN (R 4.2.3)
pracma 2.4.2 2022-09-22 [1] CRAN (R 4.2.3)
prettyunits 1.1.1 2020-01-24 [1] CRAN (R 4.2.3)
processx 3.8.1 2023-04-18 [1] CRAN (R 4.2.3)
profvis 0.3.8 2023-05-02 [1] CRAN (R 4.2.3)
promises 1.2.0.1 2021-02-11 [1] CRAN (R 4.2.3)
ps 1.7.5 2023-04-18 [1] CRAN (R 4.2.3)
purrr 1.0.1 2023-01-10 [1] CRAN (R 4.2.3)
quadprog 1.5-8 2019-11-20 [1] CRAN (R 4.2.0)
quantmod 0.4.22 2023-04-07 [1] CRAN (R 4.2.3)
R6 2.5.1 2021-08-19 [1] CRAN (R 4.2.3)
ragg 1.2.5 2023-01-12 [1] CRAN (R 4.2.3)
rbibutils 2.2.13 2023-01-13 [1] CRAN (R 4.2.3)
Rcpp 1.0.10 2023-01-22 [1] CRAN (R 4.2.3)
Rdpack 2.4 2022-07-20 [1] CRAN (R 4.2.3)
readr * 2.1.4 2023-02-10 [1] CRAN (R 4.2.3)
remotes 2.4.2 2021-11-30 [1] CRAN (R 4.2.3)
reshape2 * 1.4.4 2020-04-09 [1] CRAN (R 4.2.3)
rlang 1.1.0 2023-03-14 [1] CRAN (R 4.2.3)
rmarkdown 2.21 2023-03-26 [1] CRAN (R 4.2.3)
rpart 4.1.19 2022-10-21 [2] CRAN (R 4.2.3)
Rsolnp 1.16 2015-12-28 [1] CRAN (R 4.2.3)
rstudioapi * 0.14 2022-08-22 [1] CRAN (R 4.2.3)
rugarch * 1.4-9 2022-10-26 [1] CRAN (R 4.2.3)
RVAideMemoire * 0.9-83 2023-06-14 [1] CRAN (R 4.2.3)
scales 1.2.1 2022-08-20 [1] CRAN (R 4.2.3)
sessioninfo 1.2.2 2021-12-06 [1] CRAN (R 4.2.3)
shiny 1.7.4 2022-12-15 [1] CRAN (R 4.2.3)
SkewHyperbolic 0.4-0 2018-11-29 [1] CRAN (R 4.2.3)
spatial 7.3-16 2023-01-23 [2] CRAN (R 4.2.3)
spd 2.0-1 2015-07-03 [1] CRAN (R 4.2.3)
StockDistFit * 1.0.0 2023-05-09 [1] CRAN (R 4.2.3)
stringi 1.7.12 2023-01-11 [1] CRAN (R 4.2.2)
stringr * 1.5.0 2022-12-02 [1] CRAN (R 4.2.3)
survival 3.5-3 2023-02-12 [2] CRAN (R 4.2.3)
systemfonts 1.0.4 2022-02-11 [1] CRAN (R 4.2.3)
textshaping 0.3.6 2021-10-13 [1] CRAN (R 4.2.3)
tibble 3.2.1 2023-03-20 [1] CRAN (R 4.2.3)
tidyr * 1.3.0 2023-01-24 [1] CRAN (R 4.2.3)
tidyselect 1.2.0 2022-10-10 [1] CRAN (R 4.2.3)
timeDate 4022.108 2023-01-07 [1] CRAN (R 4.2.3)
timeSeries 4021.105 2022-10-16 [1] CRAN (R 4.2.3)
truncnorm 1.0-9 2023-03-20 [1] CRAN (R 4.2.3)
tseries * 0.10-54 2023-05-02 [1] CRAN (R 4.2.3)
tsmethods * 0.3.2 2023-05-19 [1] Github (tsmodels/tsmethods@3f88b94)
tstests * 0.1.0 2023-05-19 [1] Github (tsmodels/tstests@8085ab5)
TTR 0.24.3 2021-12-12 [1] CRAN (R 4.2.3)
tzdb 0.4.0 2023-05-12 [1] CRAN (R 4.2.3)
urlchecker 1.0.1 2021-11-30 [1] CRAN (R 4.2.3)
usethis * 2.1.6 2022-05-25 [1] CRAN (R 4.2.3)
utf8 1.2.3 2023-01-31 [1] CRAN (R 4.2.3)
uuid 1.1-0 2022-04-19 [1] CRAN (R 4.2.0)
vctrs 0.6.2 2023-04-19 [1] CRAN (R 4.2.3)
withr 2.5.0 2022-03-03 [1] CRAN (R 4.2.3)
xfun 0.39 2023-04-20 [1] CRAN (R 4.2.3)
xml2 1.3.4 2023-04-27 [1] CRAN (R 4.2.3)
xtable 1.8-4 2019-04-21 [1] CRAN (R 4.2.3)
xts * 0.13.1 2023-04-16 [1] CRAN (R 4.2.3)
zip 2.3.0 2023-04-17 [1] CRAN (R 4.2.3)
zoo * 1.8-12 2023-04-13 [1] CRAN (R 4.2.3)
[1] C:/Users/aylaa/AppData/Local/R/win-library/4.2
[2] C:/Program Files/R/R-4.2.3/library
