I'd like travis to fail my build if warnings are generated within tests. I thought this was the default, but it doesn't seem to be happening for me. Is there a way to set this up? Here's a build that passes with a warning from this test:
test_that("warnings cause travis to fail", {
odd <- 1:2 * 1:5
expect_equal(length(odd), 5)
})
Try something like this to express that you expect no warning (I repeat your code then a variant):
library(testthat)
test_that("warnings cause travis to fail", {
odd <- 1:2 * 1:5
expect_equal(length(odd), 5)
})
test_that("warnings cause travis to fail", {
expect_warning(odd <- 1:2 * 1:5, NA)
expect_equal(length(odd), 5)
})
#> Error: Test failed: 'warnings cause travis to fail'
#> * `odd <- 1:2 * 1:5` generated warnings:
#> * longer object length is not a multiple of shorter object length
Created on 2018-03-15 by the reprex package (v0.2.0).
But, yes, I acknowledge this does not scale!
Travis has the default option warnings_are_errors: true. However the warnings referred to here are the WARNING notes from R CMD check, not the r level warnings you get from warning().
As jenny said if there is a particular piece of code you know should not have warnings you can use expect_warning(regexp = NA). Note you can nest expectations, so you could theoretically have one of these expectations around all the exceptions in the test_that() block if you wanted.
If you wanted to fail the tests on any warning in any file what you would have to do is write a custom testthat reporter which caused any warning condition to fail.
Was thinking that maybe a slight variation to @jennybryan reply could be simpler to implement:
library(testthat)
test_that("warnings cause travis to fail", {
expect_warning({
odd <- 1:2 * 1:5
expect_equal(length(odd), 5)
}, NA)
})
#> Error: Test failed: 'warnings cause travis to fail'
#> * `{ ... }` generated warnings:
#> * longer object length is not a multiple of shorter object length
This way you would not have to wrap any possibly "offending" line within a expect_warning call: just wrap the whole test within it (though you would lose reference to the line which triggered the warning). Any other caveats?
Created on 2018-03-16 by the reprex package (v0.2.0).
What about using options(warn = 2)? Is there any kind of on.exit functionality in testthat to ensure that it gets reset at the end of testing?
testthat uses condition handlers which supercede options(warn=2), so the option has no effect in testthat blocks.
What you would likely want to do is modify https://github.com/r-lib/testthat/blob/master/R/reporter-check.R to additionally tally expectation_warning and then stop for warnings in a similar way to how it stops for errors.
But maybe at this point it would be better to open an issue in the testthat repository and we can consider adding a stop_for_warning option to the check reporter.