Migrating from MRAN to PPPM: thousands of win.binary packages missing. What to do?

Hi all,

I know this is similar to these posts:
missing binary package "fst" for Windows (R4.1 and 4.2) - R Admins / Package Management - RStudio Community

No binaries found in packagemanager when using M1 Mac (arm64) - R Admins / Package Management - RStudio Community

As Windows users, we are currently evaluating how to migrate from MRAN to Posit Public Package Manager. We noticed that thousands of packages are missing in PPPM (for the same snapshot date) which endangers our project reproducibility in the future.

Short result of MRAN vs Posit PPM

Here's the the short result. Sources coverage is pretty decent, but there are lots of binaries missing.

SNAPSHOT_DATE TYPE NAME_MATCH VERSION_MATCH VERSION_MISMATCH MISSING_POSIT MISSING_MRAN
2021-12-24 source 18621 18618 1 2 0
2021-12-24 win.binary 18675 15206 102 3325 42
2023-01-25 source 19075 19075 0 0 0
2023-01-25 win.binary 19129 14893 100 4102 34

Question

  • Will the missing binary packages be added at some time?
  • Are you (Posit) striving to jump in for the retiring MRAN?

Thanks a lot for your answer!

source code and reprex

library(tidyverse)

snapshot_dates <- c("2023-01-25", "2021-12-24")

url_stubs_tbl <- tribble(~PM_NAME, ~URL_STUB,
                         "MRAN", "https://mran.microsoft.com/snapshot/",
                         "POSIT", "https://packagemanager.posit.co/cran/")
pkg_types <- c("win.binary", "source")


overview_tbl <- expand.grid(SNAPSHOT_DATE = snapshot_dates, 
                            TYPE = pkg_types,
                            PM_NAME = c("MRAN", "POSIT"),
                            stringsAsFactors = FALSE) |>
    as_tibble() |>
    left_join(url_stubs_tbl, by = "PM_NAME") |>
    mutate(URL = str_c(URL_STUB, SNAPSHOT_DATE))

overview_tbl <- overview_tbl |>
    mutate(AVAILABLE_PKGS = map2(URL, TYPE, ~available.packages(repos = .x, type = .y) |> as_tibble()))


# function to compare to available pkgs tables
compare_avail_pkgs <- function(tbl_1, tbl_2) {
    matching <- select(tbl_1, Package, Version_left = Version) |>
        full_join(select(tbl_2, Package, Version_right = Version),
                  by = "Package") |>
        mutate(NAME_MATCH = TRUE,
               VERSION_MATCH = Version_left == Version_right)
    
    matching <- matching |>
        mutate(MISSING_RIGHT = is.na(Version_right),
               MISSING_LEFT = is.na(Version_left))
    
    matching |>
        rename_with(str_to_upper)
}


wide_overview_tbl <- overview_tbl |> 
    select(SNAPSHOT_DATE, PM_NAME, TYPE, AVAILABLE_PKGS) |>
    pivot_wider(names_from = c(PM_NAME), values_from = AVAILABLE_PKGS) |>
    mutate(COMPARE_RESULT = map2(MRAN, POSIT, compare_avail_pkgs)) |>
    select(SNAPSHOT_DATE, TYPE, COMPARE_RESULT) |>
    unnest(COMPARE_RESULT) |>
    rename_with(~str_replace(.x, "LEFT", "MRAN")) |>
    rename_with(~str_replace(.x, "RIGHT", "POSIT"))
    
wide_overview_tbl |>
    group_by(SNAPSHOT_DATE, TYPE) |>
    mutate(VERSION_MISMATCH = !VERSION_MATCH) |>
    summarise(across(c(NAME_MATCH, VERSION_MATCH, VERSION_MISMATCH, 
                       MISSING_POSIT, MISSING_MRAN), sum, na.rm = TRUE))
#> `summarise()` has grouped output by 'SNAPSHOT_DATE'. You can override using the
#> `.groups` argument.
#> # A tibble: 4 × 7
#> # Groups:   SNAPSHOT_DATE [2]
#>   SNAPSHOT_DATE TYPE       NAME_MATCH VERSION_MATCH VERSION_MI…¹ MISSI…² MISSI…³
#>   <chr>         <chr>           <int>         <int>        <int>   <int>   <int>
#> 1 2021-12-24    source          18621         18618            1       2       0
#> 2 2021-12-24    win.binary      18675         15206          102    3325      42
#> 3 2023-01-25    source          19075         19075            0       0       0
#> 4 2023-01-25    win.binary      19129         14893          100    4102      34
#> # … with abbreviated variable names ¹​VERSION_MISMATCH, ²​MISSING_POSIT,
#> #   ³​MISSING_MRAN

Created on 2023-01-27 with reprex v2.0.2

1 Like

Hi @smichal,

Developer on the Posit Package Manager product here. To answer your questions:

Will the missing binary packages be added at some time?

Yes, we're actively reviewing the data you've posted here. We strive to build 90%+ (99% ideally) of the binary packages for Windows that CRAN provides and should have an update soon.

Are you (Posit) striving to jump in for the retiring MRAN?

While the news of MRAN shutting down is recent, we hope Public Posit Package Manager can act as a replacement for MRAN for the community. We're reviewing any missing functionality that MRAN provides and hoping to either replicate it or have a path forward for most cases by the shutdown date.

I hope that helps,

Tyler

2 Likes

Hi Tyler,

thanks for your swift reply. A solution to that is much appreciated :+1:

Hey @smichal, the recent large dip in Windows binary coverage should be resolved now. The current available Windows binary counts for R 4.2, 4.1, and source are:

> nrow(available.packages(contriburl = "https://packagemanager.posit.co/cran/latest/bin/windows/contrib/4.2", filters = list()))
[1] 17458

> nrow(available.packages(contriburl = "https://packagemanager.posit.co/cran/latest/bin/windows/contrib/4.1", filters = list()))
[1] 18207

> nrow(available.packages(contriburl = "https://packagemanager.posit.co/cran/latest/src/contrib", filters = list()))
[1] 19181

R 4.2 is above 90% but still a little low, but we're always working on improving coverage, especially for very highly used packages.

There are also a few other reasons why Package Manager's Windows binary coverage is lower than CRAN's:

  • We don't currently support Bioconductor binary packages, so any CRAN packages that have Bioconductor dependencies won't be available as binary.
  • Available binary package versions always match the latest source package versions. If a new package version fails to build, Package Manager won't include the previous working version in the binary PACKAGES like CRAN does. This may be changed in the future, but you can still install previous versions of binary packages using an older snapshot URL.
  • Package Manager maintains binary package builds for the latest and 4 previous R versions (R 3.5 through 4.2 currently), so as latest packages drop support for older versions of R over time, the binary coverage will drop for older versions of R.
2 Likes

Hello,

I am not sure if my message fits here, but I noticed that some packages are missing from the snapshots( for example from here https://packagemanager.posit.co/cran/2024-05-04/src/contrib/PACKAGES )

`‘symengine’, ‘Rglpk’, ‘parma’, ‘rxode2parse’, ‘rxode2random’, ‘rxode2et’, ‘rxode2’

They are available on CRAN. Are they supposed to be included in the posit snapshots? Or do they need to satisfy some requirements to be included?

I apologize if my questions do not fit here.

Thank you very much!

Welcome, @boaretti! Thanks for checking out Public Package Manager.

The packages you list all appear to have both source and binary packages available in that snapshot. What specific errors are you seeing and what platform and R versions are you installing them for? Hope we can help you out!

Joe

I was using linux with this command, R 4.3.3:
zz <- renv::dependencies(); Require::Require(zz[,"Package"], require = FALSE, repos = "https://packagemanager.posit.co/cran/2024-05-04/")

after a long output, I got this error towards the end:

#21 154.1 make: *** [Makefile:221: symengine.ts] Error 1
#21 163.0 make: *** [Makefile:256: Rglpk.ts] Error 1
#21 537.4 make: Target 'all' not remade because of errors.
#21 537.4 cat: parma.out: No such file or directory
#21 537.4 cat: rxode2parse.out: No such file or directory
#21 537.4 cat: rxode2random.out: No such file or directory
#21 537.4 cat: rxode2et.out: No such file or directory
#21 537.4 cat: rxode2.out: No such file or directory
#21 537.7 
#21 537.7 The downloaded source packages are in
#21 537.7 	‘/tmp/RtmpbyRyFa/downloaded_packages’
#21 537.8 Warning message:
#21 537.8 In (function (pkgs, lib, repos = getOption("repos"), contriburl = contrib.url(repos,  :
#21 537.8   installation of 7 packages failed:
#21 537.8   ‘symengine’, ‘Rglpk’, ‘parma’, ‘rxode2parse’, ‘rxode2random’, ‘rxode2et’, ‘rxode2’
#21 DONE 538.2s

Many other packages were installed except for those I listed above.

I also tried using pak to install the list of packages:

options(repos = "https://packagemanager.posit.co/cran/2024-05-04/"); zz <- renv::dependencies(); pak::pkg_install(unique(zz[,"Package"]))

#22 0.556 Finding R package dependencies ... [38/52] [39/52] [40/52] [41/52] [42/52] [43/52] [44/52] [45/52] [46/52] [47/52] [48/52] [49/52] [50/52] [51/52] [52/52] Done!
#22 2.934 
#22 7.253 ✔ Updated metadata database: 3.79 MB in 9 files.
#22 7.254 
#22 7.255 ℹ Updating metadata database
#22 13.28 ✔ Updating metadata database ... done
#22 13.28 

and it gets stuck in updating the metadata database

It looks like it's downloading the source package and then having trouble building it, rather than using our pre-built binaries. Can you try the following?

options(renv.download.trace = TRUE)
renv::install("symengine", repos = "https://packagemanager.posit.co/cran/2024-05-04")

Oh, and which distribution of Linux are you using?

This is the output, seems fine:

#21 [16/17] RUN R -e 'options(renv.download.trace = TRUE); renv::install("symengine", repos = "https://packagemanager.posit.co/cran/2024-05-04")'
#21 0.426 
#21 0.426 R version 4.3.3 (2024-02-29) -- "Angel Food Cake"
#21 0.426 Copyright (C) 2024 The R Foundation for Statistical Computing
#21 0.426 Platform: x86_64-pc-linux-gnu (64-bit)
#21 0.426
#21 0.426 R is free software and comes with ABSOLUTELY NO WARRANTY.
#21 0.426 You are welcome to redistribute it under certain conditions.
#21 0.426 Type 'license()' or 'licence()' for distribution details.
#21 0.426
#21 0.426   Natural language support but running in an English locale
#21 0.426
#21 0.426 R is a collaborative project with many contributors.
#21 0.426 Type 'contributors()' for more information and
#21 0.426 'citation()' on how to cite R or R packages in publications.
#21 0.426
#21 0.426 Type 'demo()' for some demos, 'help()' for on-line help, or
#21 0.426 'help.start()' for an HTML browser interface to help.
#21 0.426 Type 'q()' to quit R.
#21 0.426
#21 0.511 > options(renv.download.trace = TRUE); renv::install("symengine", repos = "https://packagemanager.posit.co/cran/2024-05-04")     
#21 0.920 List of 6
#21 0.920  $ url     : chr "https://packagemanager.posit.co/cran/__linux__/jammy/2024-05-04/src/contrib/PACKAGES.rds"
#21 0.920  $ destfile: chr "/tmp/RtmpcSky1n/renv-tempfile-82aee0312"
#21 0.920  $ method  : chr "auto"
#21 0.920  $ quiet   : logi TRUE
#21 0.921  $ mode    : chr "wb"
#21 0.922  $ headers : NULL
#21 13.37 # Downloading packages -------------------------------------------------------
#21 13.37 - Downloading symengine from CRAN ...
#21 13.37 # Downloading 'https://packagemanager.posit.co/cran/__linux__/jammy/2024-05-04/src/contrib/symengine_0.2.6.tar.gz' [auto] --------
#21 13.37
#21 13.37 List of 6
#21 13.37  $ url     : chr "https://packagemanager.posit.co/cran/__linux__/jammy/2024-05-04/src/contrib/symengine_0.2.6.tar.gz"
#21 13.37  $ destfile: chr "/root/.cache/R/renv/binary/R-4.3/x86_64-pc-linux-gnu/repository/symengine/renv-tempfile-843ff6fae"
#21 13.37  $ method  : chr "auto"
#21 13.37  $ quiet   : logi TRUE
#21 13.37  $ mode    : chr "wb"
#21 13.37  $ headers : NULL
#21 17.17 OK [2.1 Mb in 3.8s]
#21 17.27 Successfully downloaded 1 package in 17 seconds.
#21 17.27
#21 17.28 The following package(s) will be installed:
#21 17.28 - symengine [0.2.6]
#21 17.28 These packages will be installed into "/usr/local/lib/R/site-library".
#21 17.28
#21 17.28 # Installing packages --------------------------------------------------------
#21 17.33 - Installing symengine ...                      OK [installed binary and cached in 0.93s]
#21 18.28 Successfully installed 1 package in 1 second.
#21 18.28 >
#21 18.28 >
#21 DONE 18.4s

I am building a Docker from rocker/shiny-verse:4.3

Apparently I am limited to 3 posts as a new user; not sure how I can keep replying :confused:

It looks like that if I do this:

RUN R -e 'options(renv.download.trace = TRUE, renv.config.pak.enabled=TRUE); renv::install("rxode2", repos = "https://packagemanager.posit.co/cran/2024-01-01")'

I get a the version of rxode2 2.1.2 and this seems wrong because on the snapshot it is 2.1.1:

#21 38.92 → Will download 11 packages with unknown size.
#21 38.93 + BH                     1.84.0-0   [bld][dl] (14.02 MB)
#21 38.93 + cachem        1.0.8  → 1.1.0      [bld][cmp][dl] (26.51 kB)
#21 38.93 + checkmate              2.3.1      [dl]
#21 38.93 + dparser                1.3.1-11   [dl]
#21 38.93 + farver        2.1.1  → 2.1.2      [bld][cmp][dl] (1.27 MB)
#21 38.93 + fastmap       1.1.1  → 1.2.0      [bld][cmp][dl] (46.41 kB)
#21 38.93 + ggplot2       3.5.0  → 3.5.1      [bld][dl] (3.56 MB)
#21 38.93 + inline                 0.3.19     [dl]
#21 38.93 + lattice       0.22-5 → 0.22-6     [bld][cmp][dl] (598.58 kB)
#21 38.93 + lazyeval               0.2.2      [dl]
#21 38.93 + lotri                  0.4.3      [dl]
#21 38.93 + PreciseSums            0.6        [dl]
#21 38.93 + qs                     0.26.3     [bld][cmp][dl] (2.15 MB)
#21 38.93 + RApiSerialize          0.1.3      [bld][cmp][dl] (8.41 kB)
#21 38.93 + RcppArmadillo          0.12.8.3.0 [bld][cmp][dl] (1.41 MB)
#21 38.93 + RcppEigen              0.3.4.0.0  [bld][cmp][dl] (1.77 MB)
#21 38.93 + RcppParallel           5.1.7      [dl] + ✔ make
#21 38.93 + rex                    1.2.1      [dl]
#21 38.93 + rxode2                 2.1.2      [bld][cmp][dl] (3.25 MB)
#21 38.93 + rxode2et               2.0.12     [bld][cmp][dl] (72.73 kB)
#21 38.93 + rxode2ll               2.0.11     [dl]
#21 38.93 + rxode2parse            2.0.18     [bld][cmp][dl] (550.82 kB)
#21 38.93 + rxode2random           2.0.13     [bld][cmp][dl] (68.14 kB)
#21 38.93 + sitmo                  2.0.2      [dl]
#21 38.93 + StanHeaders            2.32.7     [bld][cmp][dl] (2.40 MB) + ✔ make, ✔ pandoc
#21 38.93 + stringfish             0.16.0     [dl] + ✔ make
#21 38.93 + symengine              0.2.6      [bld][cmp][dl] (810.91 kB) + ✔ cmake, ✖ libgmp3-dev, ✔ make, ✖ libmpfr-dev
#21 38.93 + xfun          0.43   → 0.44       [bld][cmp][dl] (144.50 kB)

while if I set renv.config.pak.enabled=FALSE (and the rest of the code above unchanged):

#20 83.27 The following package(s) will be installed:
#20 83.27 - BH            [1.81.0-1]
#20 83.27 - checkmate     [2.3.1]
#20 83.27 - dparser       [1.3.1-11]
#20 83.27 - inline        [0.3.19]
#20 83.27 - lazyeval      [0.2.2]
#20 83.27 - lotri         [0.4.3]
#20 83.27 - PreciseSums   [0.6]
#20 83.27 - qs            [0.25.7]
#20 83.27 - RApiSerialize [0.1.2]
#20 83.27 - RcppArmadillo [0.12.6.6.1]
#20 83.27 - RcppEigen     [0.3.3.9.4]
#20 83.27 - RcppParallel  [5.1.7]
#20 83.27 - rex           [1.2.1]
#20 83.27 - rxode2        [2.1.1]
#20 83.27 - rxode2et      [2.0.11]
#20 83.27 - rxode2ll      [2.0.11]
#20 83.27 - rxode2parse   [2.0.17]
#20 83.27 - rxode2random  [2.0.12]
#20 83.27 - sitmo         [2.0.2]
#20 83.27 - StanHeaders   [2.26.28]
#20 83.27 - stringfish    [0.16.0]
#20 83.27 - symengine     [0.2.4]

I apologize if this is not the right place to post this.