shinytest passing on devtools::test() but failing on devtools::check()

[t.bowling]

Hi,

I'm having an issue with a project that is a shiny app, developed around the golem framework, with some unit-testing using shinytest as well as testthat.

Essentially, there are 2 tests that fail when using devtools::check, but pass when using devtools::test. I've read through a few similar posts, but haven't been able to work out exactly what is causing this, I've seen some suggestion that the location of files needs to be slightly different, but I've tried some configurations and haven't got anywhere with it.

The tests themselves are pretty basic, for the shinytest, there is a copy of app.R in the inst folder, and alongside that file in the inst folder is a tests subfolder with the following simply comparing one snapshot of the loaded app

app <- ShinyDriver$new("../", loadTimeout = 1e5)
app$snapshotInit("mytest")

app$snapshot()

With this corresponding code in the main directories tests/testthat subfolder

context("shinytest")

library(shinytest)

test_that("Application works", {
  appdir <- system.file(package = "Myapp", "inst")
  expect_pass(testApp(appdir, compareImages = TRUE))
})

Then the other test that fails in devtools::check is something that is set out as standard under the golem framework to test successful loading of the app (also saved in standard tests/testthat subfolder)

context("golem tests")

library(golem)

test_that("app server", {
  server <- app_server
  expect_is(server, "function")
})

# Configure this test to fit your need
test_that(
  "app launches", {
    skip_on_cran()
    skip_on_travis()
    skip_on_appveyor()
    x <- processx::process$new(
      "R",
      c(
        "-e",
        "setwd('../../'); pkgload::load_all();run_app()"
      )
    )
    Sys.sleep(5)
    expect_true(x$is_alive())
    x$kill()
  }
)

The results of devtools::test as follows:
Testing Myapp
√ | OK F W S | Context
√ | 1 | shinytest [11.8 s]
√ | 2 | golem tests [5.2 s]
√ | 4 | repeat-model
√ | 5 | trial-model
√ | 3 | wrapper-function-output
√ | 3 | wrapper-function-value

== Results =====================================================================
Duration: 17.2 s

OK: 18
Failed: 0
Warnings: 0
Skipped: 0

Results of devtools:check (truncated):

Running the tests in 'tests/testthat.R' failed.
Last 13 lines of output:
Backtrace:
1. shinytest::expect_pass(testApp(appdir, compareImages = TRUE))
2. shinytest::testApp(appdir, compareImages = TRUE)
3. shinytest:::is_rmd(appDir)

 -- 2. Failure: app launches (@test-golem-recommended.R#24)  --------------------------------------------------
 x$is_alive() isn't true.
 
 == testthat results  =========================================================================================
 [ OK: 16 | SKIPPED: 0 | WARNINGS: 0 | FAILED: 2 ]
 1. Error: Application works (@test-appload.R#8) 
 2. Failure: app launches (@test-golem-recommended.R#24) 
 
 Error: testthat unit tests failed
 Execution halted

Any help would be greatly appreciated

[purell]

Hi,
I'm also seeing the same: devtools::test works but devtools::check fails. Did you manage to resolve this problem? Any advice?
Cheers,
Purell

[julianstanley]

@t.bowling

x$is_alive() isn't true.

See this issue page.

If I understand this post correctly, this is a known golem issue. You can fix it by just only running that test if interactive().

[t.bowling]

Thanks! That has helped fix the error I was getting from the golem test.

Unfortunately I'm still hitting the issue with the shinytest, where it's passing devtools::test, but failing devtools::check with this issue

-- 1. Error: Application works (@test-appload.R#8) ----------------------------------------------------------
Unknown whether app is a regular Shiny app or .Rmd:
Backtrace:

1. shinytest::expect_pass(testApp(appdir, compareImages = TRUE))
2. shinytest::testApp(appdir, compareImages = TRUE)
3. shinytest:::is_rmd(appDir)

When I run the is_rmd function outside of the check it works fine as well, very odd

[t.bowling]

In fact, I can apply the same skip_if_not(interactive()) to that second test, and then check runs fine, guess I can simply add a check on devtools::test in addition to the one I have on devtools::check in to my CI and should have coverage