I'm analyzing a big set of data on a SWITCH-Engine research cloud. The operating system is Windows Server 2012. Because R did not use the full capacity of the cloud (used only one core), I started to use sparklyr in the local mode. However, whereas I can deploy more cores with config$sparklyr.cores.local and am very happy with the calculating performance, I somehow do not manage to deploy the full Working Memory capacity and often run into memory problems, despite the task manager showing that at max. only 20% of Working Memory is used. If I try deploy more memory with for example Sys.setenv("SPARK_MEM" = "32g") it will not connect. I read in the sparklyr cheatsheet that local mode should only be used for learning purposes. How can I fix the memory problem? Is the solution to use an other mode than the local mode?
Session Info:
