tl;dr: ctrl+shift+f10 cures multithreading issues, why is that, and can I do this using a function that does not actually restart R ?
Hello,
First thank you all for your nice work on this IDE.
I would like to report a funny and potentially useful phenomenon, and ask for directions to actually make use of it. I'm sorry, I am working on quite a large project, so I am unable to provide any practical minimal working example.
First a bit of context. I do MCMC on quite heavy objects. So I wrote myself a R code to do the MCMC, with a RCPP function that uses OMP to achieve a nice speedup at the critical points. However, OMP has decreasing returns with the number of threads, so I wanna run a couple chains in parallel say three. So I have two options.
The first option is to do a socket run with for example parallel::parLapply or plan("multisession"). It works, but it is nasty with respect to the additional used memory, who actually is more than my working environment size multiplied by the number of workers (I don't get why but whatever). So I have one working solution, but with an overuse of RAM.
The second option is to use a multithreaded R parallelization, for example through parallel::mclapply or plan("multicore"). However, OMP conflicts with multithreaded R, and the code just gets stuck when I use more than 1 thread with OMP. People did tell me that OMP and multithreaded R don't get along well. OK, but... I remarked that when I restart R from Rstudio using ctrl+shift+f10 or .rs.restartR, then reload my package, it works like a charm. Restarting R somehow solves the issue, but I don't know how.
Does that behavior ring a bell ? Could you please point some operations that are done when restarting R that could explain this behavior ? I obviously cannot tell package users "you need to restart R before using the function" but I would like to see if there is one simpler, less demanding function, that I could use into my package to fix mu OMP within multithreaded R issue.
Thank you for reading all this !