Cannot allocate vector of size

[dbcuervo]

I'm getting the error "cannot allocate vector of size 111.3" (see uploaded photo).

The object referenced in my code, df, seems relatively small, 22.4 million rows and 38 columns, which comes out to 148.6 Mb when saved as an .RDS file.

I realize that there are similar topics in the Posit Community, but none has provided a solution so far:

Error: cannot allocate vector of size 76.4 Gb

i have already decreased the number of observation. the total number of records were 40596053 and i decrased it to 102269 number of records.

I cannot increase memory allocation to R in R studio.

Hi, I am trying to increase memory allocation to R. I run "memory.limit(size = NA) " or "memory.limit() ", but the memory doesn't increase when I check "memory.size(max = FALSE) ". Here is my computer software information. sessionInfo() R version 4.1.2 (2021-11-01) Platform: x86_64-w64-mingw32/x64 (64-bit) Running under: Windows 10 x64 (build 19044) My memory RAM is 128GB. When I run "memory.limit(size = NA)", it shows 1370781. I also tried "memory.limit(size = 1370781) ", but "memory.size(max = FALSE)" shows only 1421.84. Can anyone help me?

Error: cannot allocate vector of size 196.1 Mb In addition: Warning message:

Any suggestion to overcome this error Error: cannot allocate vector of size 196.1 Mb In addition: Warning message: In create2GDAL(dataset = dataset, drivername = drivername, type = type, : mvFlag=NA unsuitable for type Byte

Manage available memory in R when doing operations with large amount of big vectors. Avoid Error: vector memory exhausted (limit reached?)

When doing operations on a large amount of vectors (e.g., as part of the creation of a null distribution in a permutation test), I either get the error "vector memory exhausted (limit reached?)" or my RStudio session crashes. These problems occur with the code below. I want to carry out operations on vectors including subtracting one vector from all other vectors as well as computing the dot product between one vector and all other vectors. On my computer (R version 3.6.2, 64-bit on MAC OS Catalina) I can do this with 1000 0000 vectors; so to get more I tried to split the process up by creating an outer for loop; so to get 2 millions I basically do it in a loop twice and then merge the fina…

Some of the suggestions include using the memory.limit() function, but this is (apparently) no longer supported by R. I get this warning when I try to use on my desktop: "'memory.limit()' is no longer supported", and this warning on the Posit server: "'memory.limit()' is Windows-specific".

I've also tried system('free -m'), but the output on the Posit server is just this: [1] 127.

I'm at my wit's end. Any suggestions?

[nirgrahamuk]

Lets do some estimates. A double / numeric would be 8bytes. You have 22.4million of those uncompressed for a single column, so that would be 172Mb for a column, lets say you have 35. That would be 6gb uncompressed.
So perhaps you have highly compressible data, but in uncompressed representation it seems you do have large memory needs.
How much system memory do you have ?
Can you do with loading and processing the data in parts rather than as a whole?