I am very new to machine learning (caret) and I am having trouble with RAM allocation in Rstudio, but not R.
I am trying to run a Random forest model using cross validation. After preparing my train data.table (149717 obs with a size of 5392816 bytes) I get a error message:
Error: cannot allocate vector of size 1.1 Gb
Meaning that I don't have enough RAM to load the object. So I decreased my train data.table systematically and eventually, with only 5% of the original training data, the model works in Rstudio without producing this error.
However, this is not ideal, and i need to run my model on a larger sample than 5% of the original size. I decided to test this in R (not Rstudio) and my model works just fine with no memory issues using the full training data.table.
When looking at my task manager during R and Rstudio processing, it seems that R is way more efficient with memory usage than Rstudio (Rstudio used 100% of RAM while R used 65% while ruing the SAME test with the SAME data!).
When I use the memory.limit() function in Rstudio i get this output: 1.759219e+13. And then I run the same function in R I get this output: 16291
Laptop specs: 256 GB SSD. 16 GB RAM. Windows 10 Pro 64bit. I7 processor with 8 logical cores.
My question is: Why is my memory allocated so differently in R vs Rstudio? The memory limit for R is fine and uses all my RAM I have, but this is not the case with Rstudio. Is this a bug or is my memory allocation just incorrectly set up within Rstudio? If the latter, can someone please advise how to set up my Rstudio memory allocation.
Also, any other ideas what can be causing this issue?
jcblum
July 31, 2019, 5:56pm
2
This is not a complete answer (sorry! memory.limit() result is a known bug, with work underway.
Some previous discussion here:
R on Windows running in RStudio produces a grossly incorrect result for memory.limit() (which should be in Mb) even though the command works perfect fine in the standard Windows RGui.
Here's what I get in a fresh session in RStudio on a machine with 16 Gb of RAM:
> memory.limit()
[1] 1.759219e+13
> sessionInfo()
R version 3.6.1 (2019-07-05)
Platform: x86_64-w64-mingw32/x64 (64-bit)
Running under: Windows 10 x64 (build 17763)
Matrix products: default
locale:
[1] LC_COLLATE=English_United States.1252 LC_CTYPE=English_United States.1252
[3] LC_MONETARY=English_United States.1252 LC_NUMERIC=C
[5] LC_TIME=English_United States.1252
attached base packages:
[…
And the Github issue is here:
opened 02:54PM - 20 Jun 19 UTC
closed 09:07PM - 15 Oct 19 UTC
bug
reproducible
On Windows 10 64bit
Base R's memory.limit() function reports wildly different… results in (a) R Console and (b) RStudio 1.3.2xx's console for R 3.6.0.
R3.5.3 reports 16279, as expected on my system, in both RConsole and RStudio.
R3.6.0 reports 16279 (R Console) and erroneously reports 17592186044416 (RStudio console)
memory.limit() results:
. | R 3.5.3 | R 3.6.0
---|---|---
R Console | 16279 | 16279
RStudio 1.3.2xx | 16279 | 17592186044416
```
######################################################################################
# R 3.5.3 (R console)
######################################################################################
> Sys.info()
sysname release version machine
"Windows" "10 x64" "build 16299" "x86-64"
> R.version
_
platform x86_64-w64-mingw32
arch x86_64
os mingw32
system x86_64, mingw32
status
major 3
minor 5.3
year 2019
month 03
day 11
svn rev 76217
language R
version.string R version 3.5.3 (2019-03-11)
nickname Great Truth
> rstudioapi::getVersion()
Error: RStudio not running
> memory.limit()
[1] 16279
> memory.size()
[1] 41.87
######################################################################################
# R 3.6.0 (R console)
######################################################################################
> Sys.info()
sysname release version machine
"Windows" "10 x64" "build 16299" "x86-64"
> R.version
_
platform x86_64-w64-mingw32
arch x86_64
os mingw32
system x86_64, mingw32
status
major 3
minor 6.0
year 2019
month 04
day 26
svn rev 76424
language R
version.string R version 3.6.0 (2019-04-26)
nickname Planting of a Tree
> rstudioapi::getVersion()
Error: RStudio not running
> memory.limit()
[1] 16279
> memory.size()
[1] 42.77
######################################################################################
# R 3.5.3 (RStudio)
######################################################################################
> Sys.info()
sysname release version machine
"Windows" "10 x64" "build 16299" "x86-64"
> R.version
_
platform x86_64-w64-mingw32
arch x86_64
os mingw32
system x86_64, mingw32
status
major 3
minor 5.3
year 2019
month 03
day 11
svn rev 76217
language R
version.string R version 3.5.3 (2019-03-11)
nickname Great Truth
> rstudioapi::getVersion()
[1] ‘1.3.236’
> memory.limit()
[1] 16279
> memory.size()
[1] 74.1
######################################################################################
# R 3.6.0 (RStudio)
######################################################################################
> Sys.info()
sysname release version machine
"Windows" "10 x64" "build 16299" "x86-64"
> R.version
_
platform x86_64-w64-mingw32
arch x86_64
os mingw32
system x86_64, mingw32
status
major 3
minor 6.0
year 2019
month 04
day 26
svn rev 76424
language R
version.string R version 3.6.0 (2019-04-26)
nickname Planting of a Tree
> rstudioapi::getVersion()
[1] ‘1.3.236’
> memory.limit()
[1] 1.759219e+13
> memory.size()
[1] 43.41
######################################################################################
# R 3.5.3 (RStudio)
######################################################################################
> Sys.info()
sysname release version machine
"Windows" "10 x64" "build 16299" "x86-64"
> R.version
_
platform x86_64-w64-mingw32
arch x86_64
os mingw32
system x86_64, mingw32
status
major 3
minor 5.3
year 2019
month 03
day 11
svn rev 76217
language R
version.string R version 3.5.3 (2019-03-11)
nickname Great Truth
> rstudioapi::getVersion()
[1] ‘1.3.260’
> memory.limit()
[1] 16279
> memory.size()
[1] 74.43
######################################################################################
# R 3.6.0 (RStudio)
######################################################################################
> Sys.info()
sysname release version machine
"Windows" "10 x64" "build 16299" "x86-64"
> R.version
_
platform x86_64-w64-mingw32
arch x86_64
os mingw32
system x86_64, mingw32
status
major 3
minor 6.0
year 2019
month 04
day 26
svn rev 76424
language R
version.string R version 3.6.0 (2019-04-26)
nickname Planting of a Tree
> rstudioapi::getVersion()
[1] ‘1.3.260’
> memory.limit()
[1] 1.759219e+13
> memory.size()
[1] 55.22
```
Thanks for your response. Waiting for the new Rstudio release.
system
August 9, 2019, 8:56am
4
This topic was automatically closed 7 days after the last reply. New replies are no longer allowed.
) but the wacky