Large csv Census files in tidyverse

[cderv]

Could you use a database instead of writing your binded files in csv at the end ?
I think you could use SQLlite to create a database that you fill up which each csv you read.
You will have at the end a table in the database that you can request easily with dplyr after.

See this post on Memory if not already read by you, there are some good advices

Best practice to handle out-of-memory data

Hello, I am a data analyst using mainly R and SAS. While I generally prefer to use R and tidyverse tools for my data science and programming tasks, I miss SAS datasets whenever R data frames consume all the memory. I could use variety of R packages to handle large data (bigmemory, ff, dplyr interface to databases, etc.), but unified binary data format on disk as in SAS has several advantages: ease of data management, no need to learn additional syntax, being able to create many intermediate datasets without caring about RAM (which makes debugging easy). So, I want to ask R community: What are your ideas, techniques, workflows, and best practices to handle out-of-memory data in R ? (I …