Solution:
Create virtual environment.
bash$ virtualenv .venv
This creates a .venv directory in .
Activate .venv.
bash$ source .venv/Scripts/activate
Then to RStudio.
Setting the in .Rprofile to ".venv/python". The .venv directory doesn't contain .venv/bin/python, instead the python.exe is at .venv/python.
library(reticulate)
Sys.setenv(RETICULATE_PYTHON = here::here(".venv/Scripts/python.exe"))