How can speed up the running of pgmm models in R?

[Rashed_Ashqar]

Hello,

I'm using R to run (pgmm models) from package "plm", I have data with 3.5 million obs. It takes a long time (4 hours) to run the model with 4 thousand obs. The RAM of the device is 16 GB. Is there any way to speed up the process?

Thank you.

[technocrat]

Try running this example with 1K observations (just cut and paste; install tictoc if needed) and see how you compare. Then we can look at the specific model with example data and your specific model. See the FAQ: How to do a minimal reproducible example reprex for beginners.

suppressPackageStartupMessages({
  library(plm)
  library(tictoc)
})



# from documentation for pgmm
data("EmplUK", package = "plm")

tic()
## Arellano and Bond (1991), table 4 col. b 
z1 <- pgmm(log(emp) ~ lag(log(emp), 1:2) + lag(log(wage), 0:1)
           + log(capital) + lag(log(output), 0:1) | lag(log(emp), 2:99),
           data = EmplUK, effect = "twoways", model = "twosteps")
summary(z1, robust = FALSE)
#> Twoways effects Two steps model
#> 
#> Call:
#> pgmm(formula = log(emp) ~ lag(log(emp), 1:2) + lag(log(wage), 
#>     0:1) + log(capital) + lag(log(output), 0:1) | lag(log(emp), 
#>     2:99), data = EmplUK, effect = "twoways", model = "twosteps")
#> 
#> Unbalanced Panel: n = 140, T = 7-9, N = 1031
#> 
#> Number of Observations Used: 611
#> 
#> Residuals:
#>       Min.    1st Qu.     Median       Mean    3rd Qu.       Max. 
#> -0.6190677 -0.0255683  0.0000000 -0.0001339  0.0332013  0.6410272 
#> 
#> Coefficients:
#>                         Estimate Std. Error  z-value  Pr(>|z|)    
#> lag(log(emp), 1:2)1     0.474151   0.085303   5.5584 2.722e-08 ***
#> lag(log(emp), 1:2)2    -0.052967   0.027284  -1.9413 0.0522200 .  
#> lag(log(wage), 0:1)0   -0.513205   0.049345 -10.4003 < 2.2e-16 ***
#> lag(log(wage), 0:1)1    0.224640   0.080063   2.8058 0.0050192 ** 
#> log(capital)            0.292723   0.039463   7.4177 1.191e-13 ***
#> lag(log(output), 0:1)0  0.609775   0.108524   5.6188 1.923e-08 ***
#> lag(log(output), 0:1)1 -0.446373   0.124815  -3.5763 0.0003485 ***
#> ---
#> Signif. codes:  0 '***' 0.001 '**' 0.01 '*' 0.05 '.' 0.1 ' ' 1
#> 
#> Sargan test: chisq(25) = 30.11247 (p-value = 0.22011)
#> Autocorrelation test (1): normal = -2.427829 (p-value = 0.01519)
#> Autocorrelation test (2): normal = -0.3325401 (p-value = 0.73948)
#> Wald test for coefficients: chisq(7) = 371.9877 (p-value = < 2.22e-16)
#> Wald test for time dummies: chisq(6) = 26.9045 (p-value = 0.0001509)
toc()
#> 0.129 sec elapsed
sessionInfo()
#> R version 4.0.4 (2021-02-15)
#> Platform: x86_64-pc-linux-gnu (64-bit)
#> Running under: Pop!_OS 20.10
#> 
#> Matrix products: default
#> BLAS:   /usr/local/lib/R/lib/libRblas.so
#> LAPACK: /usr/local/lib/R/lib/libRlapack.so
#> 
#> locale:
#>  [1] LC_CTYPE=en_US.UTF-8       LC_NUMERIC=C              
#>  [3] LC_TIME=en_US.UTF-8        LC_COLLATE=en_US.UTF-8    
#>  [5] LC_MONETARY=en_US.UTF-8    LC_MESSAGES=en_US.UTF-8   
#>  [7] LC_PAPER=en_US.UTF-8       LC_NAME=C                 
#>  [9] LC_ADDRESS=C               LC_TELEPHONE=C            
#> [11] LC_MEASUREMENT=en_US.UTF-8 LC_IDENTIFICATION=C       
#> 
#> attached base packages:
#> [1] stats     graphics  grDevices utils     datasets  methods   base     
#> 
#> other attached packages:
#> [1] tictoc_1.0 plm_2.4-1 
#> 
#> loaded via a namespace (and not attached):
#>  [1] pillar_1.5.1      compiler_4.0.4    highr_0.8         miscTools_0.6-26 
#>  [5] tools_4.0.4       digest_0.6.27     nlme_3.1-152      evaluate_0.14    
#>  [9] lifecycle_1.0.0   tibble_3.1.0      debugme_1.1.0     lattice_0.20-41  
#> [13] pkgconfig_2.0.3   rlang_0.4.10      reprex_1.0.0      maxLik_1.4-6     
#> [17] yaml_2.2.1        xfun_0.22         styler_1.3.2      stringr_1.4.0    
#> [21] knitr_1.31        fs_1.5.0          vctrs_0.3.6       lmtest_0.9-38    
#> [25] grid_4.0.4        glue_1.4.2        bdsmatrix_1.3-4   fansi_0.4.2      
#> [29] Rdpack_2.1.1      rmarkdown_2.7     Formula_1.2-4     purrr_0.3.4      
#> [33] magrittr_2.0.1    backports_1.2.1   ellipsis_0.3.1    htmltools_0.5.1.1
#> [37] rbibutils_2.0     MASS_7.3-53.1     sandwich_3.0-0    utf8_1.2.1       
#> [41] stringi_1.5.3     crayon_1.4.1      zoo_1.8-9