Managing testthat output and Reporters?

[njtierney]

How are folks managing results from testthat?

Tests can sometimes take a long time to run, and I want to cache/store the results of the tests somewhere.

This feels like the sort of ergonomics that usethis would help smooth over.

It sounds like the functionality is already there, when I asked at Helper to extract summary of previously run tests? · Issue #1494 · r-lib/testthat · GitHub

It sounds like there are options with

option(testthat.output_file)

It also sounds like there are ways to manage this with custom Reporters/Multireporters

But reading Manage test reporting — Reporter • testthat this feels a bit beyond me.

What I want is:

1. A function to easily capture the test output
2. Ways to extract out key test files that are failing
3. Ways to extract out key test files that are warning
4. Extract out notes

The functionality I am imagining is something like

write_tests_to("file/path")
# or
turn_on_test_logging()
# provides informative message about how to access test results
devtools::test()
# results are in console or Build tab in RStudio
test_results <- extract_test_results("file/path")

# notes, warnings, errors are stored as data
test_results$note
test_results$warning
test_results$error

# summary/top of note/warning/error are stored in summary list
test_results$summary$note
test_results$summary$warning
test_results$summary$error

This has the feeling of something that already exists, but does it?

I'm just finding it a bit tricky to spend 5-10 minutes running tests locally, fixing something, then trying to remember what the other remaining things were that needed fixing - sometimes 10 minutes ago, sometimes as of a certain date. I know that Github CI could help with this, but sometimes I need to solve things locally first as GH runners create tricky issues with installation/sometimes just don't work.

Any thoughts, much appreciated!

[Gabor]

Some tips:

1. Only run the tests that you are fixing. See the filter argument. RStudio has a shortcut to run the tests only from the "current test file".
2. Try to keep your tests snappy.
3. It is usually worth the time investment to get your CI in order.
4. You can run tests in parallel. This may create new issues, especially if your tests assume a certain order!
5. Kirill has an experimental package that only re-runs the failing tests: GitHub - cynkra/lazytest: Runs only failing tests

[njtierney]

Thanks, Gabor! I'll reply to your points below.

But overall, is my request possible? Is capturing output something that reporters can help with? I understand that overall doing so is kind of going against part of the grain of what unit testing is about, but is it possible?

1. Only run the tests that you are fixing. See the filter argument.

Thanks - didn't know about the filter argument. It seems that lazytest will help!

2. Try to keep your tests snappy.

Agreed, this is definitely something that could be improved with the greta project, but it's hard because it involves running MCMC, which isn't always fast.

3. It is usually worth the time investment to get your CI in order.

Also agreed!

4. You can run tests in parallel. This may create new issues, especially if your tests assume a certain order!

This unfortunately seems to fall over when we use future in greta, which we do every time we run chains in parallel.

5. Kirill has an experimental package that only re-runs the failing tests: GitHub - cynkra/lazytest: Runs only failing tests

[Gabor]

Sure, testthat functions return the results in a list in general. E.g.

results <- testthat::test_local()
...
❯ results[[1]]
$file
[1] "test-cleanup-reporter.R"

$context
[1] "cleanup-reporter"

$test
[1] "unit: test, mode: cleanup-fail"

$user
[1] 0.237

$system
[1] 0.243

$real
[1] 1.208

$results
$results[[1]]
<expectation_success/expectation/condition>
As expected
...

So you can extract the files with failures and re-run them.