Rconnect Python virtual environment not found on publish

[nwheatley]

I have a plumber API that calls python scripts using reticulate package. It works until I try to use a virtual environment.

I followed these instruction Installing-and-Configuring-Python-with-RStudio to create a virtual environment on RStudio Server Pro. Running the API locally works (the venv is found). But, upon deploying to Rconnect, I get errors such as:

----- Bundle upload started at  2021-01-15 20:39:32  -----
[2021-01-15 20:39:32] Inferring App mode and parameters 
[2021-01-15 20:39:32] Bundling app dir 
[2021-01-15 20:39:32] Generate manifest.json 
Error: 
* Error detecting python for reticulate: Error during pip freeze: Traceback (most recent call last): File
   "/opt/python/3.7.7/lib/python3.7/runpy.py", line 183, in _run_module_as_main mod_name, mod_spec, code =
   _get_module_details(mod_name, _Error) File "/opt/python/3.7.7/lib/python3.7/runpy.py", line 142, in _get_module_details
   return _get_module_details(pkg_main_name, error) File "/opt/python/3.7.7/lib/python3.7/runpy.py", line 109, in
   _get_module_details __import__(pkg_name) File
   "/home/me/plumber_reticulate_test/python/lib/python3.7/site-packages/pip/__init__.py", line 1, in <module> from
   pip._internal.utils.typing import MYPY_CHECK_RUNNING ModuleNotFoundError: No module named 'pip._internal'
----- Deployment error -----
Error: 
* Error detecting python for reticulate: Error during pip freeze: Traceback (most recent call last): File
   "/opt/python/3.7.7/lib/python3.7/runpy.py", line 183, in _run_module_as_main mod_name, mod_spec, code =
   _get_module_details(mod_name, _Error) File "/opt/python/3.7.7/lib/python3.7/runpy.py", line 142, in _get_module_details
   return _get_module_details(pkg_main_name, error) File "/opt/python/3.7.7/lib/python3.7/runpy.py", line 109, in
   _get_module_details __import__(pkg_name) File
   "/home/me/plumber_reticulate_test/python/lib/python3.7/site-packages/pip/__init__.py", line 1, in <module> from
   pip._internal.utils.typing import MYPY_CHECK_RUNNING ModuleNotFoundError: No module named 'pip._internal'
 
----- Error stack trace -----
Error during wrapup: length(max.lines) <= 1 is not TRUE
Error: no more error handlers available (recursive errors?); invoking 'abort' restart
> 

I created the venv in the terminal - and placed it in my project directory according to RStudio recommendation.

virtualenv --python=/opt/python/3.7.7/bin/python3 python

I'm using virtualenv version 20.2.2
virtualenv 20.2.2 from /opt/python/3.7.7/lib/python3.7/site-packages/virtualenv/__init__.py

I've tried using R v4.0.2 and v3.6.3 in RStudio.

.Rprofile file contains one line to set RETICULATE_PYTHON
Sys.setenv(RETICULATE_PYTHON = "python/bin/python")
(I restart my Rstudio session, as suggested, when I made the .Rprofile)

plumber.R

library(reticulate)
library(plumber)

print(py_discover_config())
reticulate::source_python("say_hello.py")

#* @param name Your name
#* @get /say_hello
function(name="Nicole"){
  say_hello(name)
}

When I run the App on RStudio (which works), this is the output of py_discover_config()

> plumb(file='plumber.R')$run()
python:         /home/me/plumber_reticulate_test/python/bin/python
libpython:      /opt/python/3.7.7/lib/libpython3.7m.so
pythonhome:     /home/me/plumber_reticulate_test/python:/home/me/plumber_reticulate_test/python
virtualenv:     /home/me/plumber_reticulate_test/python/bin/activate_this.py
version:        3.7.7 (default, May  7 2020, 21:25:33)  [GCC 7.3.0]
numpy:          /home/me/plumber_reticulate_test/python/lib/python3.7/site-packages/numpy
numpy_version:  1.19.5

NOTE: Python version was forced by RETICULATE_PYTHON

I have tried 'click' publishing, and rsconnect::deployApp() function, to publish to Rconnect, but both fail due to not finding python.

Our admin set up the python on RStudio and R-connect following these suggestions:

Install Python - RStudio Documentation

However, I didn't install/configure python myself so I don't have many details. However, without the virutal environments, publishing to Rconnect does work for normal python functions. So, Rconnect python is technically working, just not when I am introducing a virutal environment.

Anyone see what is wrong? Or have any troubleshooting suggestions?

[oisin]

Hi Nicole,
I have tried recreating your error with a project that should be similar to yours.
I think your issue may be that you are publishing the .Rprofile file as part of your bundle.

In the best-practices article that you reference, it warns the following:

When deploying the app using the publish wizard in RStudio do not add .Rprofile to the bundle as RStudio connects recreates the environment and manages this for you on the deployment environment.

I have tried publishing with and without the .Rprofile. The latter publishes successfully for me while the former throws a Packrat error in the RStudio Connect deployment logs. If you have not tried this, could you please have a go?