Using parallel purrr with a memoised function

[mattwarkentin]

Hi,

Ignoring why I might want to do something weird like this, any idea why the fourth call to purrr::map(...) does not work with a memoised version of a function?

foo <- function(...) {
  Sys.sleep(1)
  rnorm(1)
}

baz <- memoise::memoise(foo)

purrr::map(1:10, \(x) foo())
#> [[1]]
#> [1] 0.5924674
#> 
#> [[2]]
#> [1] -1.08545
#> 
#> [[3]]
#> [1] -0.485124
#> 
#> [[4]]
#> [1] -0.2295291
#> 
#> [[5]]
#> [1] -0.5523497
#> 
#> [[6]]
#> [1] 0.4690967
#> 
#> [[7]]
#> [1] -0.4608877
#> 
#> [[8]]
#> [1] -0.7551163
#> 
#> [[9]]
#> [1] 1.32158
#> 
#> [[10]]
#> [1] 0.1743966

purrr::map(1:10, \(x) baz())
#> [[1]]
#> [1] 0.5768033
#> 
#> [[2]]
#> [1] 0.5768033
#> 
#> [[3]]
#> [1] 0.5768033
#> 
#> [[4]]
#> [1] 0.5768033
#> 
#> [[5]]
#> [1] 0.5768033
#> 
#> [[6]]
#> [1] 0.5768033
#> 
#> [[7]]
#> [1] 0.5768033
#> 
#> [[8]]
#> [1] 0.5768033
#> 
#> [[9]]
#> [1] 0.5768033
#> 
#> [[10]]
#> [1] 0.5768033

purrr::map(1:10, purrr::in_parallel(\(x) foo(), foo = foo))
#> [[1]]
#> [1] -0.1441313
#> 
#> [[2]]
#> [1] -0.3606063
#> 
#> [[3]]
#> [1] -0.7168853
#> 
#> [[4]]
#> [1] -0.7713734
#> 
#> [[5]]
#> [1] -1.070559
#> 
#> [[6]]
#> [1] -1.655758
#> 
#> [[7]]
#> [1] -0.2389709
#> 
#> [[8]]
#> [1] -0.1121482
#> 
#> [[9]]
#> [1] -0.7071783
#> 
#> [[10]]
#> [1] 0.0701082

purrr::map(1:10, purrr::in_parallel(\(x) baz(), baz = baz))
#> Error in `purrr::map()`:
#> ℹ In index: 1.
#> Caused by error in `baz()`:
#> ! attempt to apply non-function

[shikokuchuo]

In short, this is due to our internal use of carrier::crate() to prevent accidentally serializing large objects from the local environment of functions. In this case, it strips the closure from baz which contains what is needed to execute the memoized function.

An immediate workaround is to wrap baz in a list when it is passed in. Per the carrier documentation, "containers (such as lists) are not recursively analysed."

Instead of:

map(1:10, in_parallel(\(x) baz(), baz = baz))

Use:

map(1:10, in_parallel(\(x) fns$baz(), fns = list(baz = baz))

We'll see if it makes sense to cater for this usage, but the above should always work.

[mattwarkentin]

Thanks for the explanation, @shikokuchuo. Much appreciated.