replace `mclapply` with `future_lapply`

Luis · March 16, 2022, 2:44am

Hi all,

I'm looking for general advice relating to modifying an existing function that uses mclapply for parallel computing across many cores and changing it to use a relevant futureverse function that can distribute the work across nodes in a cluster.

The function in question has a lot of moving parts (linked below in case anyone is interested) because the function being applied to list elements calls unexported functions defined within other function definitions. Just swapping the parallelization funs didn't work and I'm still trying to understand why.

I tried to replicate the errors I keep getting but can't seem to break future_lapply with nested fn definitions, packages, and unexported functions. Like so:

library(future.apply)
#> Loading required package: future
library(dplyr)
#> 
#> Attaching package: 'dplyr'
#> The following objects are masked from 'package:stats':
#> 
#>     filter, lag
#> The following objects are masked from 'package:base':
#> 
#>     intersect, setdiff, setequal, union
library(usethis)

iris <- datasets::iris
chunks <- split(iris, seq(1, nrow(iris), length.out = 5L))


# define fns
mult_length <- function(vec){vec*3}
get_min1 <- function(z){min(z)+1}

# with nested definitions and unexported fns
sumlength <- function(x,constantval=3){
  drop_col3 <- function(dat){dplyr::select(dat,-1)} # nested fn
  newdat <- drop_col3(x)
  length_vec <- sum(newdat[,1])
  mult_length <- function(y){y*3} # another nested fn
  multiplied <- mult_length(length_vec)
  usethis:::base_and_recommended() # unexported fn
  get_min1(multiplied)+constantval
}

sumlength(iris)
#> [1] 1379.8
future_lapply(chunks, FUN = sumlength) # works OK
#> $`1`
#> [1] 287.2
#> 
#> $`38.25`
#> [1] 281.8
#> 
#> $`75.5`
#> [1] 276.4
#> 
#> $`112.75`
#> [1] 270.4
#> 
#> $`150`
#> [1] 280

^{Created on 2022-03-15 by the reprex package (v2.0.1)}
I will try to break down the process into smaller bits, but in the meantime, if someone here has successfully replaced mclapply with anything from future.apply please let me know, I'd appreciate any feedback.
Thanks!

The fns in question:

github.com

khabbazian/l1ou/blob/master/R/shift_configuration.R

# 
#' Detects evolutionary shifts under an OU model
#'
#'This function takes in one or multiple traits, and automatically detects the phylogenetic placement and 
#'the magnitude of shifts in the evolution of these traits. The model assumes an Ornstein-Uhlenbeck process
#'whose parameters are estimated (adaptation `strength' \eqn{\alpha}{alpha} and drift variance \eqn{\sigma^2}{sigma^2}).
#'Instantaneous shifts in the optimal trait value affect the traits over time.
#'
#'@param tree ultrametric tree of class phylo with branch lengths, and edges in postorder.
#'@param Y trait vector/matrix without missing entries. The row names of the data must be in the same order as the tip labels.
#'@param max.nShifts upper bound for the number of shifts. The default value is half the number of tips.
#'@param criterion information criterion for model selection (see Details in \code{\link{configuration_ic}}).
#'@param root.model ancestral state model at the root.
#'@param candid.edges a vector of indices of candidate edges where the shifts may occur. If provided, shifts will only be allowed on these edges; otherwise all edges will be considered.
#'@param quietly logical. If FALSE, a basic summary of the progress and results is printed.
#'@param alpha.starting.value optional starting value for the optimization of the phylogenetic adaptation rate. 
#'@param alpha.upper optional upper bound for the phylogenetic adaptation rate. The default value is log(2) over the minimum branch length connected to tips. 
#'@param alpha.lower optional lower bound for the phylogenetic adaptation rate.
#'@param lars.alg model selection algorithm for LARS in univariate case. 
#'@param nCores number of processes to be created for parallel computing. If nCores=1 then it will run sequentially. Otherwise, it creates nCores processes by using mclapply function. For parallel computing it, requires parallel package.

This file has been truncated. show original

system · April 6, 2022, 2:45am

This topic was automatically closed 21 days after the last reply. New replies are no longer allowed.

If you have a query related to it or one of the replies, start a new topic and refer back with a link.