Automating testing of students' code

[clausp]

Hello all,

I will be teaching a new quantitative methods class the next academic year. One will be a regular class and one online. In both cases, the plan is to use R/RStudio and have students submit their code and assignments through Canvas.

Does anybody have some pointers for how to automate checking students' submitted scripts? I am mainly looking for a way to catch runtime errors and fundamental style problems (like having absolute paths, etc.) in the code.

If this is easier outside of Canvas, for example through students' public GitHub repositories, then I am okay with that.

Any suggestions would be much appreciated.

Claus

[mishabalyasin]

I'm not sure if following resources answer your use-case to the full extent, but they might be helpful:

1. learnr package from RStudio: "The learnr package makes it easy to turn any R Markdown document into an interactive tutorial. Tutorials consist of content along with interactive components for checking and reinforcing understanding"
2. R/exams project that primarily is used to generate tests/exams/etc., but might be useful for you.
3. Datacamp allows you to create materials and automate certain parts of feedback.

[hoelk]

You are looking for lintr if you want to check style. the default settings are pretty nice (tidyverse style, pretty sure it also checks for absolute paths)

I'm not sure what you mean by checking for runtime errors. I guess the script will abort when the first error is encountered. If you want to check several scripts in a row, you should be able to do something like

tryCatch(
  source(...pathtoscript...), 
  error = function(e) ...printsomeinfoabouttheerror...
)

[Pakillo]

Hi Claus,

That sounds like a perfect job for a continuous integration service like Travis-CI. Basically, you can run any kind of check (including lintr) upon students submissions/commits.

I know nothing about Canvas, but this can definitely be implemented through GitHub. Here are a couple of links to get started:

Real-time feedback for students using continuous integration tools

Rapid feedback on code with lintr

Hope this helps,

Paco

[torvaney]

I'd second the use of a CI tool like travis-CI. I haven't any experience with Canvas, but travis has good integration with GitHub and there are lots of resources/tutorials/templates and the like to make getting started relatively painless.

It might also be worth taking a look at the testthat package (GitHub - r-lib/testthat: An R 📦 to make testing 😀), which although usually used for package development could perhaps be used for your needs here. I had a similar(?) question recently and this was recommended to me:

Testing R scripts/code outside of package development

You can run testthat tests without package structure, but it is less clean/convenient. I would put your test files in the same location as if you had already done the package conversion stuff. That means put test files into folder "tests/testthat". When you run tests on a package in Rstudio, you can just Ctrl-Shift-T, which automatically loads the package and then runs the tests in the appropriate place. If not a package you can do these steps manually: Prior to testing, load the functions you want to test and their dependencies. Source the function files and load packages, either manually or at the top of the test file. Then you can run testthat::test_dir("tests/testthat") which s…