github-actions with Remote url in DESCRIPTION

Package development
1

I am trying to get the github actions CI from usethis working on a new project -- I haven't made any modifications to the scripts yet. They may be found here.

In the project DESCRIPTION, I am using the Remote: section to install xgboost precompiled for linux with the GPU functionality enabled. Here is the snippet from the DESCRIPTION which is in question:

...
    xgboost (>= 1.7.2.1)
Config/testthat/edition: 3
Remotes: 
    url::https://github.com/dmlc/xgboost/releases/download/v1.7.2/xgboost_r_gpu_linux_1.7.2.tar.gz
URL: https://cmatkhan.github.io/brentlabXGBoost/

This is the error I am getting:

 Error: 
  ! error in pak subprocess
  Caused by error: 
  ! Could not solve package dependencies:
  * deps::.: ! pkgdepends resolution error for deps::..
  Caused by error: 
  ! Cannot determine package name for 1 package:
  "url::https://github.com/dmlc/xgboost/releases/download/v1.7.2/xgboost_r_gpu_linux_1.7.2.tar.gz".
  ℹ Maybe you need to add a `<packagename>‌=` prefix?
  * local::.: ! pkgdepends resolution error for local::..
  Caused by error: 
  ! Cannot determine package name for 1 package:
  "url::https://github.com/dmlc/xgboost/releases/download/v1.7.2/xgboost_r_gpu_linux_1.7.2.tar.gz".
  ℹ Maybe you need to add a `<packagename>‌=` prefix?

I see that it is suggesting that I add <packagename>‌= prefix, but I don't know what that means, where to put it, or what syntax to use.

If this suggestion from the Error message is clear to someone else, I would greatly appreciate either getting pointed in the right direction re: documentation (couldn't find any that helped me) or just an example of what this message is trying to tell me to do.

Otherwise, any suggestions on what to try to debug this? The package does install through the usual means (outside of the CI and possibly specifically pak or pkgdown...not sure yet).

2

should be

Remotes:
  xgboost=url::github.com/...

See 'Package names' here: Install packages from CRAN, Bioconductor, GitHub, URLs, etc. — Package sources • pak

1 Like
3

Ugh. I didn't look in the pak docs, though that is where the error was clearly happening -- thank you.

Adding the package appropriately did change the error, which is good. Now it is:

  ✖ Failed to build xgboost 1.7.2.1
  Error: 
  ! error in pak subprocess
  Caused by error in `stop_task_build(state, worker)`:
  ! Failed to build source package 'xgboost'
  Full installation output:
  * installing *source* package ‘xgboost’ ...
  staged installation is only possible with locking
  ** using non-staged installation
  ** libs
  make: Nothing to be done for 'all'.
  ** R
  ** data
  ** demo
  ** inst
  ** byte-compile and prepare package for lazy loading
  ** help
  *** installing help indices
  ** building package indices
  ** installing vignettes
  ** testing if installed package can be loaded
  Error: Error: package or namespace load failed for ‘xgboost’ in library.dynam(lib, package, package.lib):
   shared object ‘xgboost.so’ not found
  Error: Error: loading failed
  Execution halted
  ERROR: loading failed
  * removing ‘/tmp/RtmpFmOJ01/pkg-lib133776673483/xgboost’
  ---
  Backtrace:
  1. pak::lockfile_install(".github/pkg.lock")
  2. pak:::remote(function(...) { …
  3. err$throw(res$error)
  ---
  Subprocess backtrace:
   1. base::withCallingHandlers(cli_message = function(msg) { …
   2. get("lockfile_install_internal", asNamespace("pak"))(...)
   3. plan$install()
   4. pkgdepends::install_package_plan(plan, lib = private$library, num_workers = nw, …
   5. base::withCallingHandlers({ …
   6. pkgdepends:::handle_events(state, events)
   7. pkgdepends:::handle_event(state, i)
   8. pkgdepends:::stop_task(state, worker)
   9. pkgdepends:::stop_task_build(state, worker)
  10. base::throw(new_pkg_build_error("Failed to build source package {pkg}", …
  11. | base::signalCondition(cond)
  12. global (function (e) …
  Execution halted
  Error: Process completed with exit code 1.
4

This package is not really a source R package it seems, as it contains an x86_64 shared lib:

> curl -LO https://github.com/dmlc/xgboost/releases/download/v1.7.2/xgboost_r_gpu_linux_1.7.2.tar.gz
> tar xzf xgboost_r_gpu_linux_1.7.2.tar.gz
> ldd xgboost/src/xgboost.so
	libpthread.so.0 => /lib/x86_64-linux-gnu/libpthread.so.0 (0x00000040081e5000)
	librt.so.1 => /lib/x86_64-linux-gnu/librt.so.1 (0x00000040081ea000)
	libR.so => not found
	libgomp.so.1 => not found
	libdl.so.2 => /lib/x86_64-linux-gnu/libdl.so.2 (0x00000040081ef000)
	libstdc++.so.6 => /lib/x86_64-linux-gnu/libstdc++.so.6 (0x00000040081f4000)
	libm.so.6 => /lib/x86_64-linux-gnu/libm.so.6 (0x0000004008420000)
	libgcc_s.so.1 => /lib/x86_64-linux-gnu/libgcc_s.so.1 (0x0000004008507000)
	libc.so.6 => /lib/x86_64-linux-gnu/libc.so.6 (0x0000004008527000)
	/lib64/ld-linux-x86-64.so.2 (0x0000004000000000)

Once I install the appropriate system package, and the dependent R packages, I can install xgboost:

> apt-cache search libgomp
libgomp1 - GCC OpenMP (GOMP) support library
...
> apt-get install libgomp1
...

> R -q -e 'pak::pkg_install("deps::xgboost")'
...
> R CMD INSTALL xgboost_r_gpu_linux_1.7.2.tar.gz

So I guess you probably need to install libgomp1 before the setup-r-dependencies step, because it is not installed automatically by pak, as it is not declared as a system dependency by the xgboost R package.

1 Like
5

I'm not very savvy with github actions, and I appreciate your help with this.

I added this:

      - name: install xgboost deps
        run: sudo apt-get install -y libgomp1

above setup-r-dependencies, and it runs successfully. However, I still get the same error.

In thinking about this, I possibly need to set an environmental variable, since the error is that it can't find xgboost.so.

edit:

The output of the apt install libgomp1 is this:

Run sudo apt-get install -y libgomp1
Reading package lists...
Building dependency tree...
Reading state information...
libgomp1 is already the newest version (12.1.0-2ubuntu1~22.04).
libgomp1 set to manually installed.
0 upgraded, 0 newly installed, 0 to remove and 32 not upgraded.

which makes me think that it isn't necessary?

This issue makes it look like this has to do with the gcc version.

6

You are right, libgomp1 was a red herring. I think what happens is that pak calls R CMD build on the "package", because it does not look like an R source package.

But the package is not prepared for this, and the shared library will not be included in the built package, so when pak tried to call R CMD INSTALL on it, that fails because of the missing .so file.

Unfortunately I don't think that you can install this package with pak, as it is neither a source, nor a binary R package. OTOH install.packages() happens to work on it, so I suggest you download and install it with install.packages(), and also its dependencies.

1 Like
7

Adding this build xgboost step successfully builds xgboost

      - uses: r-lib/actions/setup-r@v2
        with:
          r-version: ${{ matrix.config.r }}
          http-user-agent: ${{ matrix.config.http-user-agent }}
          use-public-rspm: true

      - name: Build xgboost
        run: install.packages(c('Matrix','data.table', 'jsonlite'), dependencies=TRUE);install.packages("https://github.com/dmlc/xgboost/releases/download/v1.7.2/xgboost_r_gpu_linux_1.7.2.tar.gz", type="source", repo=NULL, dependencies=TRUE)
        shell: Rscript {0}

      - uses: r-lib/actions/setup-r-dependencies@v2
        with:
          extra-packages: any::rcmdcheck
          needs: check

image

But, then in the next stage where pak tries to install the packages, it doesn't see that xgboost is already built, tries to build it itself, and I get the same error again.

the install.packages step is installing into the renv environment:

...
installing to /home/runner/.cache/R/renv/library/brentlabXGBoost-eac63f19/R-4.3-r83810/x86_64-pc-linux-gnu/00LOCK-xgboost/00new/xgboost/libs
** R
** data
** demo
** inst
** byte-compile and prepare package for lazy loading
** help
*** installing help indices
** building package indices
** installing vignettes
** testing if installed package can be loaded from temporary location
** checking absolute paths in shared objects and dynamic libraries
** testing if installed package can be loaded from final location
** testing if installed package keeps a record of temporary installation path
* DONE (xgboost)

But then pak sets its libpath like this:

Run r-lib/actions/setup-r-dependencies@v2
Run # Set site library path
ℹ Using R 4.3.0 (lockfile was generated with R 4.2.2)
Set site library path
Run # Install pak
Install pak
  ℹ Using R 4.3.0 (lockfile was generated with R 4.2.2)
  * Project '~/work/brentlabXGBoost/brentlabXGBoost' loaded. [renv 0.16.0]
  > dir.create(Sys.getenv("R_LIB_FOR_PAK"), recursive = TRUE, showWarnings = FALSE)

Is the solution to set R_LIB_FOR_PAK to point at the renv environment? Is there a good way of doing this in the r-lib/actions/setup-r-dependencies@v2 step -- some way of saying Sys.setenv(R_LIB_FOR_PAK = .libPaths()[1]) maybe?

8

Unfortunately setup-r-dependencies does not work well with renv, there is a separate setup-renv action for that.

1 Like
9

OK! @Gabor, thank you so much for your time on this. This is now solved.

I'm going to summarize what worked here -- there are a lot of subtle details.

tl;dr -- i need to use a specific preocompiled tarball from xgboost, which is compiled on a linux machine with the gpu features enabled. Doing this via pak in the github-actions CI doesn't work.

Solution:

Using usethis::use_dev_package with the remote argument set according to the remotes specifications for the appropriate source, add the trouble-making dependency. This adds something like the following to the DESCRIPTION:

Remotes: 
    url::https://github.com/dmlc/xgboost/releases/download/v1.7.2/xgboost_r_gpu_linux_1.7.2.tar.gz
URL: https://cmatkhan.github.io/brentlabXGBoost/

But, per the pak documentation, this can be further specified. This may be unnecessary, since we aren't going to use pak in this solution, but i like the clarity:

Remotes: 
    xgboost=url::https://github.com/dmlc/xgboost/releases/download/v1.7.2/xgboost_r_gpu_linux_1.7.2.tar.gz
URL: https://cmatkhan.github.io/brentlabXGBoost/

Now, assuming we're using renv, we can renv::install() and renv will respect what is in the DESCRIPTION and in particular the Remotes section. My renv.lock file for this package looks like this:

    "xgboost": {
      "Package": "xgboost",
      "Version": "1.7.2.1",
      "Source": "URL",
      "RemoteType": "url",
      "RemoteUrl": "https://github.com/dmlc/xgboost/releases/download/v1.7.2/xgboost_r_gpu_linux_1.7.2.tar.gz",
      "Hash": "d6328ccd7dbb29c1c1c285b045083e02",
      "Requirements": [
        "Matrix",
        "data.table",
        "jsonlite"
      ]
    },

to get this to work with the usethis CI, this worked for me:

# Workflow derived from https://github.com/r-lib/actions/tree/v2/examples
# Need help debugging build failures? Start at https://github.com/r-lib/actions#where-to-find-help
on:
  push:
    branches: [main, master]
  pull_request:
    branches: [main, master]

name: R-CMD-check

jobs:
  R-CMD-check:
    runs-on: ${{ matrix.config.os }}

    name: ${{ matrix.config.os }} (${{ matrix.config.r }})

    strategy:
      fail-fast: false
      matrix:
        config:
          - {os: ubuntu-latest,   r: 'devel', http-user-agent: 'release', gcc: '9'}
          - {os: ubuntu-latest,   r: 'release', gcc: '9'}
          - {os: ubuntu-latest,   r: 'oldrel-1', gcc: '9'}

    env:
      GITHUB_PAT: ${{ secrets.GITHUB_TOKEN }}
      R_KEEP_PKG_SOURCE: yes
      # this needs to be added 
      RENV_PATHS_ROOT: ~/.local/share/renv

    steps:

      - uses: actions/checkout@v3

      - uses: r-lib/actions/setup-pandoc@v2

      - uses: r-lib/actions/setup-r@v2
        with:
          r-version: ${{ matrix.config.r }}
          http-user-agent: ${{ matrix.config.http-user-agent }}
          use-public-rspm: true
     
     # setup-r-dependencies is removed !
     # instead, we're handling dependencies with renv
      - name: Cache packages
        uses: actions/cache@v1
        with:
          path: ${{ env.RENV_PATHS_ROOT }}
          key: ${{ runner.os }}-renv-${{ hashFiles('**/renv.lock') }}
          restore-keys: |
            ${{ runner.os }}-renv-
     
     # note! have to add the rcmdcheck install
      - name: Restore packages
        shell: Rscript {0}
        run: |
          if (!requireNamespace("renv", quietly = TRUE)) install.packages("renv")
          renv::restore()
          renv::install('rcmdcheck')

      - uses: r-lib/actions/check-r-package@v2
        with:
          upload-snapshots: true
1 Like
10

This topic was automatically closed 7 days after the last reply. New replies are no longer allowed.

If you have a query related to it or one of the replies, start a new topic and refer back with a link.