Pak's handling of dependency resolution

General
1

Hello @Gabor :wave: and pak development team!

We've been using renv to handle dependency management and installs, but recently given pak a go to see how the grass is on the other side.

I have a package that I have to support for a legacy system running R3 (ugh). And to make this possible some package versions in the DESCRIPTION file have hard pins == or upper bounds <= on the version.

renv has been able to handle installing this on the system, selecting the right versions of the dependencies to satisfy the archaic environment. When I tried this with pak it reported errors attempting to resolve the dependencies, that looked like this:

  * Can't install dependency cpp11 (== 0.5.0) (== 1.1.0) (<= 1.6.1) (== 1.0.1) (== 1.0.0)
  * Can't install dependency hardhat (== 1.1.0) (<= 1.6.1) (== 1.0.1) (== 1.0.0)
  * Can't install dependency httpuv (<= 1.6.1) (== 1.0.1) (== 1.0.0)
  * Can't install dependency purrr (== 1.0.1) (== 1.0.0)
  * Can't install dependency yardstick (== 1.0.0)
* cpp11: Needs R >= 4.0.0
* hardhat: Can't install dependency sparsevctrs (>= 0.2.0)
* sparsevctrs: Needs R >= 4.0.0
* purrr: Needs R >= 4.0
* yardstick: Can't install dependency hardhat (>= 1.3.0)

The reason for the upper bounds / pins is because the later versions of some of these packages require R4. These errors look like pak is trying to fetch the latest versions of these packages and correctly identifying that it can't install them in the current R3 environment.

Am I doing something wrong? Does pak support == and <= notation in the DESCRIPTION file?

I had hoped that it might even be able to automatically resolve the possible dependencies and find the latest version that satisfied the R3 constraint :innocent: Although I know that this kind of dependency search can be extremely expensive. Though hoped that doing it on the package repo meta data might allow it.

Project looks super cool and promising, hoping to get a better understanding of it to see if it meets our needs. Thank you very much!

2

pak will select the right package version, if it is included in the package metadata. However the package metadata typically (e.g. on CRAN) only includes the latest version for each package, so there isn't much to choose from.

Btw. I am not sure how the following could work at all, you obviously cannot install multiple versions of cpp11, so there is no way to satisfy all these requirements, as far as I can tell.

3

Thank you very much for the information :pray:

So pak is limited to selecting dependencies from the packages listed in the Packages.rds file in the repo? I have seen before while exploring that CRAN-like repos also provide a /src/contrib/Meta/archive.rds with information about packages in the archive. Is this something you've come across? Does pak also search the Archive for older versions?

I'm also not sure how all of those cpp11 versions come about. Thanks for confirming that this looks completely messed up. I am only pinning this version to 0.5.0, so I'm not sure where the other ==1.1.0 is coming from. Am I right in thinking that each constraint in parens is a version constraint from a different dependency, or transitive dependency of the package?

Thanks for your help!

4

CRAN-like repos don't need to have archive.rds. CRAN does, others might not.

CRAN's archive.rds does not contain dependency information, so pak cannot use that for dependency resolution. The only way to extract the dependencies of an old package is to download the package file and extract it. That's obviously not feasible.

But even we had information about all dependencies, solving them is an NP-complete integer programming problem, and if every package had several possible candidates, that would make it unsolvable in practice.

5

Thank you, understood. I didn't know that the problem was NP-complete, but easy to imagine it given the number of combinations!

I will have another go with pak soon to understand why I'm seeing so many version conflicts for my installation. Thank you for your help :pray:

6

If you have an example that I can look at then I can probably help better.

7

Thank you very much for the offer. I have been able to reproduce the error messages that I'm seeing from just this DESCRIPTION file. I built it into a package and used pak::local_install to install it. I see the same error messages that I shared at the start of the thread.

Things that confuse me are:

  • Why are there so many version constraints listed for some of the packages, e.g. cpp11 - Can't install dependency cpp11 (== 0.5.0) (== 1.1.0) (<= 1.6.1) (== 1.0.1) (== 1.0.0), when I have only specified one (== 0.5.0) in the DESCRIPTION file?
  • Some of these constraints, e.g. (<= 1.6.1)correspond to constraints on other packages, e.g. httpuv rather than cpp11. Are they getting combined for some reason?
  • The constraint on yardstick (== 1.0.0), is only one constraint, so why can't it be satisfied?
Package: example
Version: 1.0.0
Depends:
    R (>= 3.6.0)
Imports:
    assertr (>= 2.8),
    aws.s3 (>= 0.3.21),
    base64enc (>= 0.1-3),
    cpp11 (== 0.5.0),
    data.table (>= 1.12.8),
    dplyr (>= 0.8.3),
    ellipsis (>= 0.3.2),
    fastmap (>= 1.1.0),
    glue (>= 1.4.2),
    gtools (>= 3.8.1),
    hardhat (== 1.1.0),
    here (>= 1.0.1),
    httpuv (>= 1.5.1),
    httpuv (<= 1.6.1),
    httr (>= 1.4.1),
    jsonlite (>= 1.6.1),
    lifecycle (>= 1.0.0),
    lubridate (>= 1.7.4),
    magrittr (>= 1.5.0),
    memoise (>= 1.1.0),
    moments (>= 0.14),
    purrr (== 1.0.1),
    R6 (>= 2.4.0),
    readr (>= 1.3.1),
    readxl (>= 1.2.0),
    reshape2 (>= 1.4.3),
    rlang (>= 1.0.2),
    stringi (>= 1.4.3),
    stringr (>= 1.4.0),
    tibble (>= 3.1.7),
    tidyr (>= 1.0.2),
    tidyselect (>= 1.0.0),
    timeDate (>= 3043.102),
    uuid (>= 1.1-0),
    xlsx (>= 0.6.1),
    yaml (>= 2.2.0),
    yardstick (== 1.0.0),
    zoo (>= 1.8-6)
Suggests:
    assertive (>= 0.3-6),
    devtools,
    git2r (>= 0.26.1),
    here,
    knitr (>= 1.27),
    mockery,
    networkD3 (>= 0.4),
    rmarkdown (>= 1.15),
    testthat

Any further light you can shed is greatly appreciated :pray:

8

Yes, that seems like a bug in the printout.

yardstick cannot be installed (on my system with the default repos), because that version is not available in the package metadata:

āÆ pak::meta_list("yardstick")
# A data frame: 2 Ɨ 33
  package   version depends   suggests license md5sum sha256sum needscompilation
* <chr>     <chr>   <chr>     <chr>    <chr>   <chr>  <chr>     <chr>
1 yardstick 1.3.2   R (>= 3.ā€¦ "covr, ā€¦ MIT + ā€¦ 0e63dā€¦ "\n     ā€¦ yes
2 yardstick 1.3.2   R (>= 3.ā€¦ "covr, ā€¦ MIT + ā€¦ b40c0ā€¦  NA       yes
# ā„¹ 25 more variables: imports <chr>, linkingto <chr>, archs <chr>,
[...]
9

Thank you for confirming that there might be errors in the output - I'll take it with a pinch of salt.

Re yardstick - I believe what renv does when an explicit version is provided is to try and pull the specific version from the pacakage Archive (yardstick 1.0.0 is available on CRAN, but in the Archive). It sounds like this is not how pak is designed, it only considers available versions as those explicitly defined in the package metadata.

I can do pak::pkg_install("yardstick@1.0.0") though. In this case pak correctly finds and installs the 1.0.0 version. How is this implemented? Is it similar to renv, checking the Archive URL for the specific version? And why doesn't this work when this specific version is requested as a package dependency instead?