How to test functions that operate on projects without cluttering test output?

[lewinfox]

I'm writing a package that will operate on other projects (specifically, it will read, write and delete files within project folders). I need to write some tests for it which means creating temporary dummy project folders, testing my functions in them, and cleaning up afterwards. Basically I'm after some kind of withr::with_temporary_project() function.

I've found a couple of approaches that work but the test output is full of messages about switching directories and I can't figure out how to get rid of them.

Attempt 1

I've come across a helper function in usethis, create_local_thing(), which seems to do exactly what I want. I've pinched the source code and dropped it into my tests (only modifying it to prefix functions with usethis:: where necessary).

Here's an example test:

# mypackage/tests/testthat/test-foo.R

test_that("foo is created", {
  proj <- create_local_thing(thing = "project")
  file.create("foo") # This should be in the root folder of my temp project
  expect_true(file.exists("foo"))
})

And here's the output of running that test file:

==> Testing R file using 'testthat'

ℹ Loading mypackage

══ Testing test-foo.R ══════════════════════════════════════════════════════════
[ FAIL 0 | WARN 0 | SKIP 0 | PASS 0 ]v Setting active project to '/home/lewin/git/mypackage'
v Setting active project to '/tmp/RtmpLjJnjV/file28d06f4f1bc5'
[ FAIL 0 | WARN 0 | SKIP 0 | PASS 1 ]v Restoring original working directory: '/home/lewin/git/mypackage/tests/testthat/'
v Setting active project to '/home/lewin/git/mypackage'
v Deleting temporary project: '/tmp/RtmpLjJnjV/file28d06f4f1bc5/'
 Done!

Test complete

The output is full of messages about switching directories, which I believe come from the usethis::proj_*() family.

I found an example use of this function in usethis/tests/testthat/test-helpers.R, and running the tests for that file gives me this:

==> Testing R file using 'testthat'

ℹ Loading usethis

══ Testing test-helpers.R ══════════════════════════════════════════════════════
[ FAIL 0 | WARN 0 | SKIP 0 | PASS 46 ] Done!

Test complete

I.e. nice clean output with no messages about changing project directories. How can I achieve the same result? Is there some kind of verbosity option?

I have tried wrapping the code in suppressMessages() and usethis::ui_silence() with no success. I assume I'm just using it wrong...

Attempt 2

I also experimented with making my own version:

with_temp_project <- function(code, quiet = TRUE) {
  # Create a temporary directory
  temp_proj <- tempdir()
  on.exit(unlink(temp_proj, recursive = TRUE, force = TRUE))

  # This makes it look like a project to `usethis`
  file.create(file.path(temp_proj, ".here"))

  # Run the code, quietly
  usethis::ui_silence(usethis::with_project(temp_proj, code, quiet = quiet))
}

But ran into similar issues.

I guess I would like to know how other people writing packages-that-operate-on-packages have approached this. Any input gratefully received!

Thanks, Lewin

> sessionInfo()
R version 4.1.1 (2021-08-10)
Platform: x86_64-pc-linux-gnu (64-bit)
Running under: Manjaro Linux

Matrix products: default
BLAS:   /usr/lib/libblas.so.3.10.0
LAPACK: /usr/lib/liblapack.so.3.10.0

locale:
 [1] LC_CTYPE=en_NZ.UTF-8       LC_NUMERIC=C               LC_TIME=en_NZ.UTF-8        LC_COLLATE=en_NZ.UTF-8     LC_MONETARY=en_NZ.UTF-8    LC_MESSAGES=en_NZ.UTF-8   
 [7] LC_PAPER=en_NZ.UTF-8       LC_NAME=C                  LC_ADDRESS=C               LC_TELEPHONE=C             LC_MEASUREMENT=en_NZ.UTF-8 LC_IDENTIFICATION=C       

attached base packages:
[1] stats     graphics  grDevices utils     datasets  methods   base     

other attached packages:
[1] mypackage_0.1.0.9005 testthat_3.0.4    

loaded via a namespace (and not attached):
 [1] Rcpp_1.0.7        rstudioapi_0.13   xml2_1.3.2        knitr_1.33        roxygen2_7.1.1    magrittr_2.0.1    usethis_2.0.1     devtools_2.4.2    pkgload_1.2.1    
[10] R6_2.5.0          rlang_0.4.11      fastmap_1.1.0     stringr_1.4.0     httr_1.4.2        tools_4.1.1       pkgbuild_1.2.0    xfun_0.25         sessioninfo_1.1.1
[19] cli_3.0.1         withr_2.4.2       ellipsis_0.3.2    remotes_2.4.0     yaml_2.2.1        rprojroot_2.0.2   lifecycle_1.0.0   crayon_1.4.1      processx_3.5.2   
[28] purrr_0.3.4       callr_3.7.0       fs_1.5.0          ps_1.6.0          memoise_2.0.0     glue_1.4.2        cachem_1.0.5      stringi_1.7.3     compiler_4.1.1   
[37] desc_1.3.0        prettyunits_1.1.1 renv_0.14.0   

Thanks, Lewin

[maelle]

@lewinfox

I.e. nice clean output with no messages about changing project directories. How can I achieve the same result? Is there some kind of verbosity option?

Yes! There's options(usethis.quiet = TRUE) but from reading that same page I'm not sure why wrapping the code in usethis::ui_silence() does not work. In usethis tests the option is used cf setup file.

Looking into usethis testing infrastructure is also what I would have done so I have no further tip except reading the testthat "Test fixtures" vignette if you haven't yet -- you'll have encountered most of it in usethis tests. (reg setup vs helper files for testthat see summary table in a blog post of mine if helpful)

[lewinfox]

Hi @maelle, thanks very much for the reply. Adding options(usethis.quiet = TRUE) in tests/testthat/setup.R worked like a charm

I'd previously hacked a solution together by wrapping any usethis-related code in ui_silence() until all the messages disappeared, but this is so much neater. Thanks for the other links as well, super helpful. I didn't know about test helpers!

I'm hazarding a guess that the reason usethis::ui_silence() wasn't the solution is because the code I'm trying to silence uses withr::defer() to restore the old project at the end of the function (which triggers the messages). Maybe something in the execution order means that the temporary option set by ui_silence() expires before the deferred code is called (? ?). I was silencing some - but not all - messages and it was driving me crazy.

Either way my test output looks super now and I can get back to fixing the actual bugs

Many thanks!

[lewinfox]

In case anyone else is working on the same sort of project and finds this useful, here's my tests/testthat/setup.R including the message-suppressing code and a slightly modified version of the create_local_thing() function from usethis. I ran into an issue when using R CMD check (via devtools::check()) which is fixed and commented below.

# tests/testthat/setup.R

# We don't want usethis printing messages during testing because it clutters the console. See
# https://forum.posit.co/t/how-to-test-functions-that-operate-on-projects-without-cluttering-test-output/113283/2?u=lewinfox
# for the background
options(usethis.quiet = TRUE)


# Pinched from https://github.com/jimhester/usethis/blob/de8aa116820a8e54f2f952b341039985d78d0352/tests/testthat/helper.R#L28-L68
# See https://github.com/r-lib/devtools/blob/78b4cabcba47e328e255fffa39e9edcad302f8a0/tests/testthat/test-build-readme.R
# for an example of how to use.

create_local_thing <- function(dir = fs::file_temp(),
                               env = parent.frame(),
                               rstudio = FALSE,
                               thing = c("package", "project")) {

  if (!requireNamespace("fs", quietly = TRUE)) {
    rlang::abort("Package `fs` is required but not installed.")
  }

  thing <- match.arg(thing)
  if (dir.exists(dir)) {
    usethis::ui_stop("Target {ui_code('dir')} {usethis::ui_path(dir)} already exists.")
  }

  # I ran into an issue with R CMD check where the old_project (tests/testthat/ in the temporary
  # folder) was not a project so this `proj_get()` call failed with an error. Wrapping this in a
  # `try()` allows us to A: stop those tests failing and B: check for the existence of an old
  # project before trying to switch back to it.
  old_project <- try(usethis::proj_get(), silent = TRUE)
  has_old_project <- !inherits(old_project, "try-error")
  old_wd <- getwd()          # not necessarily same as `old_project`


  withr::defer(
    {
      fs::dir_delete(dir)
    },
    envir = env
  )

  switch(
    thing,
    package = usethis::create_package(dir, rstudio = rstudio, open = FALSE, check_name = FALSE),
    project = usethis::create_project(dir, rstudio = rstudio, open = FALSE)
  )

  if (has_old_project) {
    withr::defer(usethis::proj_set(old_project, force = TRUE), envir = env)
  }
  usethis::proj_set(dir)

  withr::defer(
    {
      setwd(old_wd)
    },
    envir = env
  )

  setwd(usethis::proj_get())
  invisible(usethis::proj_get())
}

# Helper function to create a local project, run tests and clean up afterwards
# e.g.
#
# test_that("foo is barred correctly", {
#   with_local_project({
#     bar(foo)
#     expect_true(is.barred(foo))
#   })
# })
#
with_local_project <- function(code) {
    create_local_thing(thing = "project")
    # If you need to do anything else to set up your test project, do it here
    force(code)
  })
}