running pytest in RMarkdown knitr pipeline?

[sfinnie]

Hi all. I realise this might be a bit niche/obscure, but hoping someone can give me some pointers.

Background

I'm looking into running tests using pytest from RMarkdown documents. The wider goal is to use RMarkdown documents as a form of executable specifications. The idea is to interleave narrative describing/defining requirements with executable tests that verify the implementation.

Something like this in the context of calculating tax:

Personal Allowance

An individual earning up to the Personal Tax Allowance pays no tax

import pytest
from tax import calculate_net_income

def test_personal_allowance():
    """There's no tax to pay for income in the Personal Allowance band"""
    gross = 12500
    assert calculate_net_income(gross) == gross

pytest.main(args=['-v'])

Whilst this runs, pytest doesn't find any tests to run:

## ============================= test session starts =============================
## platform win32 -- Python 3.8.0, pytest-6.2.2, py-1.10.0, pluggy-0.13.1 -- C:\sf\bin\R\R-40~1.3\bin\x64\Rterm.exe
## cachedir: .pytest_cache
## rootdir: C:\Users\sfinnie\projects\executable_specifications\RMarkdown
## plugins: jupyter-pytest-2-1.0.1
## collecting ... collected 0 items
## 
## ============================ no tests ran in 0.01s ============================
## <ExitCode.NO_TESTS_COLLECTED: 5>

I'm assuming that's because there's no actual test module for pytest to discover (e.g. test_tax.py). I imagine it might be possible to add a custom step into the knitr pipeline, for example to extract all python snippets and put them into a file that pytest can find and run. But I've never looked into the pipeline, so no idea if that's viable / if there's a better way.

Any pointers appreciated.

Thanks.

[cderv]

@sfinnie

I imagine it might be possible to add a custom step into the knitr pipeline, for example to extract all python snippets and put them into a file that pytest can find and run

I don't really know much about pytest and how it works but I know how rmarkdown and knitr feature that could enable you idea.

There is way to extract all code from previous chunk into one chunk. See this examples

4.19 Put together all code in the appendix (*) | R Markdown Cookbook

This means you could have a chunk filed with all the code except from the one in your test chunk.

You could couple that with the cat engine that allow to write the content of a chunk to a file

15.6 Write the chunk content to a file via the cat engine | R Markdown Cookbook

Here is an example of combining these two feature

---
title: "Untitled"
output:
  html_document: default
---

These are chunks of python code

```{python}
#First chunk
1 + 1
```

```{python}
#Second chunk
2+2
```

```{r get-labels, echo = FALSE}
# This will extract the label of all python chunk 
labs = knitr::all_labels(engine == "python")
# We remove the chunk with a label starting by pytest
labs = grep("^pytest", labs, invert = TRUE)
```

```{cat, ref.label=labs, engine.opts=list(file = "test-chunks.py")}
# we write the content of this chunk, which will be filled by ref.label, to a py file.
```

This is the chunk containing pytest code, which has a label starting with _pytest_.
It can access the file written containing all other code

```{python, pytest-1, comment = ""}
f = open("test-chunks.py", "r")
print(f.read())
```

Hope it helps going through with you idea.

You can do a lot of custom stuff, even using new engine.
The R Markdown cookbook contains lots of information about the tools that would enable this.

[sfinnie]

@cderv wow, that's excellent. Many thanks for that - plenty to go explore there...!

Very grateful.