Capturing error messages from `callr::r()` in snapshot tests

Package development
1

Consider the following function that's meant to be ran in an R subprocess:

foo <- function() stop("foo error")

I'd like to capture the error message in a snapshot test.

testthat::test_that("foo", {
  testthat::expect_snapshot(callr::r(foo), error = TRUE)
})

The above will produce the following snapshot, usually with all the backtrace in more complex cases which makes the snapshot quite messy:

# foo

    Code
      callr::r(foo)
    Condition
      Error:
      ! in callr subprocess.
      Caused by error:
      ! foo error

I'm trying to simplify the snapshot to only print the error thrown by foo(). This is what I ideally want:

Code
  executed code
Output
  <simpleError: foo error.>

I've tried to capture the error in a stderr file or tryCatch() but the best I could get was to remove the backtrace only:

err <- callr::r(foo, stderr = withr::local_tempfile())
err$stderr

tmp <- withr::local_tempfile()
callr::r(foo, stderr = tmp)
readLines(tmp)

tryCatch(callr::r(foo), error = function(e) e)

How could I only output the foo error only?

Also, is there a way to use callr::r() inside expect_error()? I usually like to structure error snapshots as follows when testing for multiple errors:

test_that("blah", {
  expect_snapshot({
    (expect_error(
      expr_1
    ))
    (expect_error(
      expr_2
    ))
    (expect_error(
      expr_n
    ))
  })
})

But I couldn't make it work at all with callr::r(). I have to create a specific snapshot for each error.

2

You can catch the error and print it. callr wraps the original error into a callr_error, so it you don't want to snapshot that, then take the original error from e$parent:

❯ expect_snapshot(tryCatch(callr::r(foo), error = function(e) e$parent))

Current value:
Code
  tryCatch(callr::r(foo), error = function(e) e$parent)
Output
  <simpleError in c("(function () ", "stop(\"foo error\"))()"): foo error>
1 Like
3

Thanks for this. It seems like my example was too simplistic and doesn't replicate the behavior of my actual tests.

I made a temporary repo on GitHub that reproduces the context of the test.

Using tryCatch(error = \(e) e$parent) doesn't generate an error in my case.

Here's the error returned by just using callr::r().

Code
  callr::r(tmppkg:::local_files)
Condition
  Error:
  ! in callr subprocess.
  Caused by error:
  ! in callr subprocess.
  Caused by error:
  ! package or namespace load failed for 'tmppkg':
   .onLoad failed in loadNamespace() for 'tmppkg', details:
    call: check_bibliography()
    error: No bibliography found.

I'd like to output error: No bibliography found..

Edit:
I've realized that I'm calling callr::r() twice (and unnecessarily in the actual tested expression) so maybe I'm simply not capturing the error in the right place.

4

I am sorry, I am not sure what your question is this time.

5

I'm a little confused by the use of snapshot in this case. My understanding of snapshots is to capture lots of output that would be awkward to manually type out. If you want to test that the code 1) throws an error, and 2) includes the string "No bibliography found.", then why isn't expect_error() sufficient for this purpose? What is gained by using expect_snapshot()?

I downloaded your example package and added the following simple test. It confirms that the code fails and produces the desired error message.

test_that("test 3", {
  expect_error(
    callr::r(tmppkg:::local_files),
    "No bibliography found."
  )
})
1 Like
6

It's essentially the same question as in the original post. I used a bad example in the first post because it doesn't replicate the full context of the test.

I'm trying to test for an error that can be thrown when loading a package in an R Markdown/Quarto document during the rendering process. The error can only be triggered when .onLoad() is executed.

It seems like using tryCatch(callr::r(foo), error = function(e) e$parent) returns NULL in this particular case and I don't manage to capture just my error in the example repository I made.

I've to admit in this particular case the benefit of using a snapshot test is minimal and comes down to not having to write the error message twice (and possibly having to edit it in two different places). I also personally find snapshot tests more convenient and readable when you have many errors to test. When testing for a couple of errors, writing (part of) them is certainly fine. When you have tens to write, it can be a bit of a pain to be honest.

This is not the case here but when using e.g. rlang::abort(), snapshot tests allow you to ensure the error also includes the original call where it occurred (as well as the formatting of bulleted errors). This is what they use most of the time in dplyr. It's the repo I originally used to learn unit tests in R so I'm reusing that structure out of "habit" in a way.

You're certainly right that it might be a simpler solution here. I'd still be curious to make it work with a snapshot test for the sake of learning.

1 Like
7

Because you stack two callr calls together, so the original error is wrapped twice. So you need to unwrap twice:

expect_snapshot({
  tryCatch(
    callr::r(tmppkg:::local_files),
    error = function(e) e$parent$parent
  )
})

Current value:
Code
  tryCatch(callr::r(tmppkg:::local_files), error = function(e) e$parent$parent)
Output
  <error/rlang_error>
  Error:
  ! package or namespace load failed for 'tmppkg':
   .onLoad failed in loadNamespace() for 'tmppkg', details:
    call: check_bibliography()
    error: No bibliography found.
1 Like
8

Thanks. Yep, I realized the double wrapping yesterday and fixed it in the example. It doesn't work if I wrap it in expect_error() but I'm happy enough to leave it in its own snapshot.

9

You need to re-throw the err$parent$parent with stop() if you want to still generate an error.

But the tryCatch() also checks that there is an error.

1 Like
10

Ah yes that makes sense.

That's what confused me. Since I was expecting an error I originally was using tryCatch() with expect_snapshot(error = TRUE) or inside expect_error() but tryCatch() doesn't throw an error, it simply returns the error message caught.

11

This topic was automatically closed 7 days after the last reply. New replies are no longer allowed.

If you have a query related to it or one of the replies, start a new topic and refer back with a link.