Loading files from multiple subfolders with in same folder

Hi
With in working directory I have a main folder folder called 'monthyr', with in main folder I have sub folders for each month ex: 2017-04,2017-05..2021-03 (40+ folders). Each monthly subfolder have 50 csv files. I would like to load all the csv files (50X40 = 200 files) from subfolders and combined them in to single file in R

Trying to use Library (Tidyrvese) . I set my working directory to Main folder, but when i use dir() It recognising only single subfolder; Unable to load from all subfolder at one go. May I know where I am doing wrong

Thanks
Psbio

Use purrr:map_dfr(). For example:

Trying to use Purrr:map_dft() but not working, pleaes see the screenshot

I did tried this too
readr::read_csv(myfiles) %>% map_dfr(read_csv)

The code pattern would be like this

library(tidyverse)

list_of_files <- list.files(path = "path/to/your/files",
                            recursive = TRUE,
                            pattern = "\\.csv$",
                            full.names = TRUE)
df <- list_of_files %>%
    set_names() %>% 
    map_df(read_csv, .id = "file_name")  
3 Likes

Just a heads up, with the new version of readr (2.0.0) this has become way easier

list_of_files <- list.files(path = "path/to/your/files",
                            recursive = TRUE,
                            pattern = "\\.csv$",
                            full.names = TRUE)

df <- readr::read_csv(list_of_files, id = "file_name")
2 Likes

Hi Andresrcs,
Many thanks for your kind response
Am still getting an error ; tried both ways
Readr::read_csv(list_of_files,id=null)
and
Readr::read_csv(list_of_files,id= "File_name")

Issue with id argument

trying to load all the files in the folders ( you can see list in attached snapshot) to do multivariate analysis

As I said this is a new feature for 2.0.0 version, you are still running 1.4,you need to update first

Thank you Andre..
It doesn't working for me. For some reason Tidyverse Readr unable to update to 2.0.
I found the solution this way
I copied all the files in the subfolders using CopyWhiz (Copy all files into a single folder without any folder structure in Windows using Copywhiz - YouTube) then used dt <-dir("E:/testing2", full.names = TRUE) %>% map_df(read_csv)
worked for me

Thanks for your help Andresrcs

This topic was automatically closed 21 days after the last reply. New replies are no longer allowed.

If you have a query related to it or one of the replies, start a new topic and refer back with a link.