How to handle testing that requires the internet for a CRAN submission?

[mattwarkentin]

Hi,

It had been a while since I had submitted a package to CRAN and I seem to be doing something wrong with how I am handling documentation examples and my testing suite. I am using roxygen2 and testthat for both of these tasks. The package largely just wraps API endpoints to provide an R interface to some API. The API needs to be running locally after installing some software (available by download).

In our testing suite, we have some code that downloads and installs the software and runs the API locally so tests can run each time, but we wanted to skip this on CRAN (we thought this was how we were supposed to do it). So all of our tests contain testthat::skip_on_cran().

Similarly, all of our examples in our documentation are wrapped in \dontrun{} for the same reason. We want to avoid running examples which would will fail on CRAN machines which do not have the necessary software and API running.

The tests are being skipped (as desired) on all machines except Debian (for some reason?). So the CRAN submission failed. I emailed the CRAN maintainer to let them know my testthat tests should be skipping on CRAN and they provided the following response:

We do not know testthat and how they can detect that a check is executed on CRAN. At least we do not know how to detect that and ideally this would not be detectable, as people should not hide issue from CRAN. For the case with internet access: You can run on CRAN but define a timeout of few seconds and exit nicely in case it takes too long.

Any suggestions for how I should handle the testing suite on a CRAN submission with this in mind? I was always under the impression that you should skip tests that require internet when running on a CRAN machine. Is this wrong?

[edavidaja]

Not sure how to diagnose the substantive problem you're having, but FYI: the skip on CRAN check does not know whether something is being run on CRAN; instead, it checks for environment variables asserting that the test is not being run on CRAN.

r-lib/testthat/blob/main/R/skip.R#L257-L263

[mattwarkentin]

Thanks for sharing that. Either way, whether the tests know it IS being run on CRAN or rather knows it is NOT on CRAN, shouldn't the tests be skipped on CRAN? Unless the CRAN machines are setting the "NOT_CRAN" env var.

I'd be happy to provide more specifics about my package, but I am mostly just looking for guidance on how others handle potentially long-running tests requiring the internet on CRAN.

[Gabor]

If you show us your package, then there is a chance that we can tell you why your test cases are not skipped on CRAN.

[mattwarkentin]

The package can be found here: GitHub - mattwarkentin/openmpp: Programmatic interface to the OpenM++ microsimulation platform

I think I have implemented a better solution for checking whether the API connection is available for testing and failing gracefully if not, rather than outright trying to skip tests simply because it's on CRAN.

I think with the way the test suite is setup now (in particular setup.R), the tests should skip if the software cannot be downloaded+installed or the API connection is not available (for any reason), regardless of testing environment.

[Gabor]

Seems like in your CRAN submission you tried to download something in the setup.R file, and that failed:

  > test_check("openmpp")
  Error in `FUN()`:
  ! In path: "/srv/hornik/tmp/CRAN_pretest/openmpp.Rcheck/tests/testthat/setup.R"
  Caused by error in `download.file()`:
  ! download from 'https://github.com/openmpp/main/releases/download/v1.17.6-patch-1/openmpp_debian-11_20250128.tar.gz' failed
  Backtrace:
       ▆
    1. ├─testthat::test_check("openmpp")
    2. │ └─testthat::test_dir(...)
    3. │   └─testthat:::test_files(...)
    4. │     └─testthat:::test_files_serial(...)
    5. │       └─testthat:::test_files_setup_state(...)
    6. │         └─testthat::source_test_setup(".", env)
    7. │           └─testthat::source_dir(path, "^setup.*\\.[rR]$", env = env, wrap = FALSE)
    8. │             └─base::lapply(...)
    9. │               └─testthat (local) FUN(X[[i]], ...)
   10. │                 └─testthat::source_file(path, env = env, chdir = chdir, wrap = wrap)
   11. │                   ├─base::withCallingHandlers(...)
   12. │                   └─base::eval(exprs, env)
   13. │                     └─base::eval(exprs, env)
   14. │                       └─openmpp:::local_install_openmpp() at tests/testthat/setup.R:72:1
   15. │                         └─utils::download.file(url, path, quiet = TRUE) at tests/testthat/setup.R:22:3
   16. └─base::.handleSimpleError(...)
   17.   └─testthat (local) h(simpleError(msg, call))
   18.     └─rlang::abort(...)

(from openmpp/tests/testthat/setup.R at eb7e405209b69213e1eb6a578ab5f4948b7e36e5 · mattwarkentin/openmpp · GitHub)

skip_on_cran() works from test_that() blocks, if your setup.R needs the internet, it does not help with that.

[mattwarkentin]

I think you're exactly right. The testing never got to skip_on_cran() because it errored in the setup code.

I think the new approach avoids this by catching errors in setup.R and then skipping all subsequent test due to the skip_if_not(test_connection()) when the API is not running.