I've got a Mac and I'd like to be able to parallel process on it. Ideally, I'd like to use RStudio for development and then run scheduled jobs from the command line. As far as I can tell, Macs don't support Intel's MKL so none of the parallel packages work.
Any suggestions? I've tried this using Linux, but I'd prefer to run it on my Mac.
If you are on a Mac with ARM chip (e.g. M1/M2), Intel MKL will not work due to the architectural differences. So far so good,
But this in no way is correlated to the ability for parallel computing. MKL is "only" being used to speed-up LAPACK/BLAS operations (It contains an Intel chipset optimised LAPACK/BLAS implementation amongst many other things). Unless you are really doing large-scale matrix operations and the like, the performance gain from using Intel MKL compared to standard BLAS/LAPACK on a system with Intel/AMD (x86_64) CPUs is not very significant.
Parallel Computing using parallel, snow, or any of the doMC, doParallel, do... is unaffected by the choice which LAPACK/BLAS you are choosing.
Thanks, I'm using an Intel Mac (2019), and much of my work involves statistics with algorithms that likely could benefit from MKL, but the only time I see a notice about MKL being used is when I start up R in Linux.
I've tried replacing lapply() with mclapply() and it speeds things up from the command line on Linux, but not MacOS. I'll take a look at parallelization and other packages.
I am still not sure why you don't see any performance improvements on your Intel Mac.
Can you please run the below code and see what it gets back with ? I am mostly interested in mc.cores and the timings for the two sleep and two compute runs.
library(parallel)
library(palmerpenguins)
#detect available cores
mc.cores<-parallel::detectCores()
mc.cores
x <- as.data.frame(penguins[c(4, 1)])
trials <- 10000
sleep <- function(i) {
Sys.sleep(1) ## Do nothing for 1 seconds
}
# this should repoort 1 seconds elapsed time
system.time(mclapply(1:mc.cores, sleep, mc.cores = mc.cores))["elapsed"]
# this should repoort 2 seconds elapsed time
system.time(mclapply(1:mc.cores, sleep, mc.cores = mc.cores/2))["elapsed"]
# Now to something number-crunching
compute <- function(i,data) {
ind <- sample(344, 344, replace = TRUE)
result1 <- glm(data[ind, 2] ~ data[ind, 1], family = binomial(logit))
coefficients(result1)
}
# single core
system.time(res <- lapply(1:trials, compute, data=x))["elapsed"]
# mc.cores cores
system.time(res <- mclapply(1:trials, compute, data=x, mc.cores=mc.cores))["elapsed"]
Here's what happens when I run the code in Rstudio:
R version 4.0.5 (2021-03-31) -- "Shake and Throw"
Copyright (C) 2021 The R Foundation for Statistical Computing
Platform: x86_64-apple-darwin17.0 (64-bit)
R is free software and comes with ABSOLUTELY NO WARRANTY.
You are welcome to redistribute it under certain conditions.
Type 'license()' or 'licence()' for distribution details.
Natural language support but running in an English locale
R is a collaborative project with many contributors.
Type 'contributors()' for more information and
'citation()' on how to cite R or R packages in publications.
Type 'demo()' for some demos, 'help()' for on-line help, or
'help.start()' for an HTML browser interface to help.
Type 'q()' to quit R.
source("~/Dropbox/R/Parallel/MMayer Timings.R", echo=TRUE)
library(parallel)
library(palmerpenguins)
#detect available cores
mc.cores<-parallel::detectCores()
mc.cores
[1] 8
x <- as.data.frame(penguins[c(4, 1)])
trials <- 10000
sleep <- function(i) {
this should repoort 1 seconds elapsed time
system.time(mclapply(1:mc.cores, sleep, mc.cores = mc.cores))["elapsed"]
elapsed
1.017
this should repoort 2 seconds elapsed time
system.time(mclapply(1:mc.cores, sleep, mc.cores = mc.cores/2))["elapsed"]
elapsed
2.01
Now to something number-crunching
compute <- function(i,data) {
single core
system.time(res <- lapply(1:trials, compute, data=x))["elapsed"]
elapsed
16.894
mc.cores cores
system.time(res <- mclapply(1:trials, compute, data=x, mc.cores=mc.cores))["elapsed"]
elapsed
5.723
So your examples show the advantage of parallel processing.
My practical application is more complicated and when I run it on MacOS and RStudio using mclapply I get an error message (one for each core):
In mclapply(data, graph, graphs1, periods[iPeriod, ], mc.cores = 6) :
When I run it from the command line I get different error messages:
objc[52637]: +[NSPlaceholderMutableString initialize] may have been in progress in another thread when fork() was called.
objc[52637]: +[NSPlaceholderMutableString initialize] may have been in progress in another thread when fork() was called. We cannot safely call it or ignore it in the fork() child process. Crashing instead. Set a breakpoint on objc_initializeAfterForkError to debug.
In Ubuntu (RStudio and command line) it runs perfectly.
This is more a clutch of straw but I realized that there were some changes on MAC regarding forking that may not play well with R (they don't play well with python for sure) - As a matter of consequence, could you set
export OBJC_DISABLE_INITIALIZE_FORK_SAFETY=YES
in your bash or zsh profile (.bashrc or .zshrc). Once you set that, make sure that in your R session Sys.getenv("OBJC_DISABLE_INITIALIZE_FORK_SAFETY")will return YES. Once you established that, please re-run the code and see if that fixes things.
Python reference: Multiprocessing causes Python to crash and gives an error may have been in progress in another thread when fork() was called - Stack Overflow
Thanks for the suggestion. I tried it from a Z shell window and get the error messages:
1: In mclapply(data, graph, graphs1, periods[iPeriod, ], mc.cores = 6) :
scheduled cores 1, 2, 3, 4, 5, 6 did not deliver results, all values of the jobs will be affected
So I started an R session from the command line and got the YES from Sys.getenv. Then I sourced my R program and got different errors:
address 0x110, cause 'memory not mapped'
Traceback:
1: jpeg(filename = fn, width = 800, pointsize = 12, quality = 100)
2: FUN(X[[i]], ...)
3: lapply(X = S, FUN = FUN, ...)
4: doTryCatch(return(expr), name, parentenv, handler)
5: tryCatchOne(expr, names, parentenv, handlers[[1L]])
6: tryCatchList(expr, classes, parentenv, handlers)
7: tryCatch(expr, error = function(e) { call <- conditionCall(e) if (!is.null(call)) { if (identical(call[[1L]], quote(doTryCatch))) call <- sys.call(-4L) dcall <- deparse(call)[1L] prefix <- paste("Error in", dcall, ": ") LONG <- 75L sm <- strsplit(conditionMessage(e), "\n")[[1L]] w <- 14L + nchar(dcall, type = "w") + nchar(sm[1L], type = "w") if (is.na(w)) w <- 14L + nchar(dcall, type = "b") + nchar(sm[1L], type = "b") if (w > LONG) prefix <- paste0(prefix, "\n ") } else prefix <- "Error : " msg <- paste0(prefix, conditionMessage(e), "\n") .Internal(seterrmessage(msg[1L])) if (!silent && isTRUE(getOption("show.error.messages"))) { cat(msg, file = outFile) .Internal(printDeferredWarnings()) } invisible(structure(msg, class = "try-error", condition = e))})
8: try(lapply(X = S, FUN = FUN, ...), silent = TRUE)
9: sendMaster(try(lapply(X = S, FUN = FUN, ...), silent = TRUE))
10: FUN(X[[i]], ...)
11: lapply(seq_len(cores), inner.do)
12: mclapply(data, graph, graphs1, periods[iPeriod, ], mc.cores = 6)
13: bestCompiled(debugIt = FALSE)
14: eval(ei, envir)
15: eval(ei, envir)
16: withVisible(eval(ei, envir))
17: source("pBestETFs.R")
It makes me wonder if anyone has gotten mclapply to work on a Mac. Mine is a 2019 Intel version.
This topic was automatically closed 21 days after the last reply. New replies are no longer allowed.
If you have a query related to it or one of the replies, start a new topic and refer back with a link.