Using rm() to remove list objects from memory

[liberry]

Having trouble with this. The driver of this problem is that I am doing some analysis of large data sets that I need to reshape using tidyr::pivot_longer(). However, it throws a memory allocation error if I try to pivot the entire dataset(~20m rows). I have tried to work around this by splitting the data into ten pieces and looping over each one. However, for this to work, I need to clear memory after each iteration of the loop.

The problem comes when I try to use rm() to remove a specific list object from memory. The format df_pivots[[i]] throws an error, but if I turn it into a string, it breaks the iteration of the loop. Reprex below:

library(dplyr)
library(tidyr)
library(data.table)

df_pivots <- list()

for(i in 1:10){

  df_pivots[[i]] <-   data.frame(a= rnorm(1000), b= rnorm(1000))
  
  df_pivots[[i]] <- pivot_longer(df_pivots[[i]], cols =   c(a,b),
                                 names_to = 'Original_Column',
                                 values_to = 'Value')

  fwrite(df_pivots[[i]], paste('df_pivots_',
         0, i, '.csv', sep = '' ))

  rm(df_pivots[[i]])
    gc()

}

#> Error in rm(df_pivots[[i]]): ... must contain names or character strings
Created on 2022-09-15 with reprex v2.0.2

[FactOREO]

To delete specific list elements you can just asign them to NULL:

mylist <- list(A = 1:3, B = 1:3, C = 1:3)
mylist
#> $A
#> [1] 1 2 3
#> 
#> $B
#> [1] 1 2 3
#> 
#> $C
#> [1] 1 2 3
mylist[[1]] <- NULL
mylist
#> $B
#> [1] 1 2 3
#> 
#> $C
#> [1] 1 2 3

^{Created on 2022-09-15 with reprex v2.0.2}

Kind regards

[liberry]

Assignment to NULL doesn't work, because deleting a list object changes the order of the remaining objects in the list. However, that did give me the idea to just rewrite over each list object with a single character after the fwrite() step, which preserves the original list structure and allows the loop to work as designed. This seems to have worked.

[FactOREO]

Another way would have been to just do it backwards, e.g. processing the last list item and then assign it to NULL. This way the order doesn't matter and your list shrinks together with the "countdown" of your iterator.

Glad you found a solution that's feasible for you!