List Columns and Memory

[cderv]

You can be interested by this current discussion here

Best practice to handle out-of-memory data

Hello, I am a data analyst using mainly R and SAS. While I generally prefer to use R and tidyverse tools for my data science and programming tasks, I miss SAS datasets whenever R data frames consume all the memory. I could use variety of R packages to handle large data (bigmemory, ff, dplyr interface to databases, etc.), but unified binary data format on disk as in SAS has several advantages: ease of data management, no need to learn additional syntax, being able to create many intermediate datasets without caring about RAM (which makes debugging easy). So, I want to ask R community: What are your ideas, techniques, workflows, and best practices to handle out-of-memory data in R ? (I …

data.table was already mentionned by @martin.R - I can confirm that its memory efficiency is its strongest point. data.table has a special syntax and mechanism to work on data by reference, therefore limiting the copy in memory. Now, it is pretty different from dplyr in syntax.

For memory efficiency in the tidyverse, I think you can try using database for your data. dplyr works very well with database connection. see rstudio website about database Using a SQLlite data.table and dplyr verbs can help you deal with big dataset.