RStudio - File Load

[tmmet]

Hi, Could anyone please help me?
I'm running into memory issues when I load huge files in RStudio? Are there any best practices or special packages that I need to use when loading huge data files? Do I need to use any special packages?
Thanks in advance!

[nick]

Any specific recommendations will depend on what kind of files they are and what you want to do with them. Also, check out these recent threads in case they will help:

Limitations of R

One of the limitations of R that I have read about is that it needs to load entire data into memory, so its not suitable for analyses of big data. Has anything been done to address this problem? Are there any workarounds?

Best practice to handle out-of-memory data

Hello, I am a data analyst using mainly R and SAS. While I generally prefer to use R and tidyverse tools for my data science and programming tasks, I miss SAS datasets whenever R data frames consume all the memory. I could use variety of R packages to handle large data (bigmemory, ff, dplyr interface to databases, etc.), but unified binary data format on disk as in SAS has several advantages: ease of data management, no need to learn additional syntax, being able to create many intermediate datasets without caring about RAM (which makes debugging easy). So, I want to ask R community: What are your ideas, techniques, workflows, and best practices to handle out-of-memory data in R ? (I …

[Tazinho]

How much ram do you have? What kind of data do you have? How big are the files on disk? (And...What do you want to do?)

• A first approach might be the useage of ls(), pryr::mem_used() and pryr::object_size() to determine, if there are any unnecessary objects in ram. If so, just use rm(). Be careful whenever you assign your data to another name and change the old or the new object afterwards (memory size might double).
• If you have independent objects and just want to process each of them, you might just be able to use some looping pattern involving for example readRDS() and saveRDS() to have less objects in memory at the same time.
• Otherwise, the other threads, mentioned above, might include more possibilities.

If you want to understand R's memory management in more detail, the memory chapter in Advanced R is a good start.