read a bunch of RDS objects into a list with filename

General
1

I have a bunch of very heavy RDS files I want to manipullate.


customer_data.rds
product_data.rds
...

So I load into a list and then do some stuff. My issue is regarding the name as the list importing procedure is giving me a numbered list.

rds_list <- lapply(file_list,readRDS)

rds_list[[1]]
rds_list[[2]]
......

is there any way of reading the files setting the element name from the filename?

something like this:

rds_list$customer_data
rds_list$product_data
.....
2

Hi,

My first advice is regarding the size of your .Rds files. I recommend looking into the {qs} package, which saves R objects into ".qs" files (just like ".Rds"), which are lighter.

Now, regarding your question, the answer is YES! I am going to assume you use RStudio projects and that all your Rds files are inside a folder called data/.

# Make a vector of all your file paths
file_paths <- list.files(path = here::here("data/"), pattern = "\\.rds", full.names = TRUE)

# Make a vector of file names
file_names <-  gsub(pattern = "\\.rds$", replacement = "", x = basename(file_paths))

# Read all your data into a list
data_list <- lapply(file_paths, readRDS)

# Assign file names to list elements
names(data_list) <- file_names
3

Two alternatives:

data.table::rbindlist(lapply(setNames(rds_list, rds_list), readRDS), idcol = "file")

purrr::map_dfr(setNames(rds_list, rds_list), readRDS, .id = "file")
4

sadly, qs is not working. The object is saved but an error is raised on reading.

5

Providing as much details/information as possible will help us troubleshoot your problem. What code did you use? What error did you get?

6

not all objects are dataframes and map_dfr converted it all to data frames.

7

sorry I ended using RDS, but your approach assigning names after reading is ok. Thanks.

My error with qs is:


Error in c_qread(file, use_alt_rep, strict, nthreads) : 
  Failed to open file
8

If Rds works for you, that's fine. But how did you save your qs files?

9

just followed the instructions:

qs::qsave(object,path)

10

Did you make sure to add the .qs extension to the file name?

qs::qsave(mtcars, "data/mtcars.qs")
11

Yes. The extension was properly set. Don’t worry, I fall back to rds format. The matter was urgent. In the near future I will test qs format.
Thanks a lot

12

For the future, you never said what makes your rds files " heavy". Data, large models??? Kind of important to understand.

13

Everyone of them around 1 GB. More than 500 columns each