Problem deploying app. Using a virtual env with reticulate to run python code in app. Error — virtual environment: permission denied

[esulli]

I am creating an app in Shiny using R. I have a model in python that I am using in the app so I use the reticulate package to run it and a virtual environment in the same folder as the app to access python3. It works great locally, but once I deploy it I get an error in the logs that says

venv/bin/python: Permission denied

(venv is my virtual env).

I have tried adding an .Rprofile file (to the folder with the app) that includes source venv/bin/activate . Also, in case it is helpful, the Python component uses the keras package. I have also downloaded all of the necessary packages into my virtual environment.

I also ran use_python("venv/bin/python", required = TRUE) instead of reticulate::use_virtualenv("venv", required = TRUE) which also works locally, but I get the same error described above once I deploy it.

[andresrcs]

It's not clear, where are you trying to deploy your app? , is your virtual environment available in the server you are deploying to?

[esulli]

I'm trying to deploy to the shinyapps.io site. I'm not sure how to check or ensure that the virtual environment is available there except including it in my upload which I am doing.

[andresrcs]

Take a look at this related thread

Error in value[[3L]](cond): Python virtual environment Directory can't be recognized after ShinyApp deployed

Uploading your virtualenv is not likely to result in success either, since there is likely compiled code that works on whatever you are running locally, e.g. Windows, that will not work on shinyapps.io which runs Linux. There is work underway in the rsconnect package to capture Python packages. Once that is released, there will need to be work on shinyapps.io to install them. In the meantime, you will need to create the virtualenv and install the packages in the code that you deploy to shinyapps.io.

[esulli]

Thank you! That helps with why it isn't working, but I still don't quite have a solution. It recommended that I

create the virtualenv from within the code you deploy, so that it is recreated in shinyapps.io.

So I tried to do that by running

virtualenv_create(envname = "python_environment",python= "venv/bin/python")
reticulate::use_virtualenv("python_environment", required = TRUE)

It works locally, but when I try to deploy it I get the following error:

I then tried running it with

virtualenv_create(envname = "python_environment",python= python3)

and when I do that it deploys but then comes up with this error

[andresrcs]

@esulli

python= "venv/bin/python"

This path doesn't exist on the shinyapps.io server, and after creating your python environment (which is empty) I suspect you have to explicitley install all the python libraries that you want to use e.g. py_install(c('pandas', 'matplotlib'))

[esulli]

I thought that might be the issue as well which is why I tried

virtualenv_create(envname = "python_environment",python= python3)

though it does make sense to need to install all of the libraries I want. I was working on that using virtualenv_install(envname, packages, ignore_installed = FALSE) but will try py_install(c('pandas', 'matplotlib')). Thanks!

[andresrcs]

You are probably right with this virtualenv_install(envname, packages, ignore_installed = FALSE) I have never had the need to use python virtual environments before.

[esulli]

I tried it and py_install() works too! Now the problem is that even though it recognizes that I want it to use python3 it seems like it's still using python2 which my code won't work with

[Yarnabrina]

I suppose you can use use_python function.

[andresrcs]

Please share this setup section of your code to have a better idea of what is going on.
Up until now you have created the virtual environment and installed your libraries but you are not telling reticulate to use the virtual environment, try with something like this use_virtualenv("python_environment")

[esulli]

Yes, the three relevant lines are here

virtualenv_create(envname = "python_environment", python= "python3")
py_install(packages = c('keras', 'pandas','numpy','scipy','scikit-learn', 'tensorflow'), envname =  "python_environment")
reticulate::use_virtualenv("python_environment", required = TRUE)

so I unfortunately do already have a use_virtualenv() line.

@Yarnabrina excellent suggestion but I think use_python requires a path to something stored on my computer which has been part of the problem so far. Let me know if you know of any way to just specify the python version though. Thank you!

[andresrcs]

I think you actually need to use virtualenv_install(), take a look at this
https://rstudio.github.io/reticulate/articles/python_packages.html

[Yarnabrina]

Yes, it's my fault. I haven't noticed that you've already used this function. Sorry.

I know nothing about shiny, so can't really help it it's specific for that problem.

However, I just checked the following on a RStudio Cloud project. You may be able to use it by modifying suitably.

reticulate::py_config()
#> python:         /cloud/project/r-reticulate/bin/python
#> libpython:      /usr/lib/python2.7/config-x86_64-linux-gnu/libpython2.7.so
#> pythonhome:     /usr:/usr
#> virtualenv:     /cloud/project/r-reticulate/bin/activate_this.py
#> version:        2.7.12 (default, Nov 12 2018, 14:36:49)  [GCC 5.4.0 20160609]
#> numpy:          /cloud/project/r-reticulate/local/lib/python2.7/site-packages/numpy
#> numpy_version:  1.16.1
#> 
#> python versions found: 
#>  /cloud/project/r-reticulate/bin/python
#>  /usr/bin/python
#>  /usr/bin/python3
#>  /cloud/project/r-tensorflow/bin/python
reticulate::py_config()[13][[1]][3]
#> [1] "/usr/bin/python3"

FWIW, you may check this thread regarding the use of python 3.

[esulli]

I tried using virtualenv_install() instead of py_install() and now it works! Thank you both so much for your help @andresrcs and @Yarnabrina!

If others come across this with the same problem, here is the relevant code to set up the python virtual environment to be able to deploy to shinyapps.io:

virtualenv_create(envname = "python_environment", python= "python3")
virtualenv_install("python_environment", packages = c('keras', 'pandas','numpy','scipy','scikit-learn', 'tensorflow'))
reticulate::use_virtualenv("python_environment", required = TRUE)

For clarity, the specific packages are just what my python code needed.

Also, you may need to run (and then comment out for deployment) the following line to use an older version of rsconnect if you get an error that says: reticulate is in use, but python was not specified

devtools::install_github("rstudio/rsconnect", ref='737cd48')