Shiny deployment with reticulate fails to build python environment: ERROR: Could not install packages due to an EnvironmentError: [Errno 2] No such file or directory

Posit Professional & Hosted Products Posit Connect
, ,
1

I am working with a shiny deployment that uses reticulate for a scikit-learn pipeline. It had been working, but I attempted to re-deploy a minor tweak to the interface, and now the deployment fails.

The error is: ERROR: Could not install packages due to an EnvironmentError: [Errno 2] No such file or directory: '/tmp/build/80754af9/joblib_1594236160679/work'.

I have a virtual environment in the home directory of the project, and the app is stored in a folder called shiny with a few other files. The app runs locally on RStudio Server Pro with no issues.

Any advice? I should not that a toy deployment modeled after ranikay/shiny-reticulate-app (github.com) works. Another aside: relative file paths with .Rprofile RETICULATE_PYTHON do not seem to work with the shiny app stored in the shiny folder.'

Package Versions:
{shiny} 1.6.0
{reticulate} 1.16
{rsconnect} 0.8.16

Here's a more complete error:
[Connect] Completed packrat build against R version: '4.0.3' [Connect] Bundle requested Python version 3.8.3; using /opt/python/3.8.3/bin/python3.8 which has version 3.8.3 GET /__api__/tasks/XlpMgeYwxnxFYQ8C?first_status=521 16ms [Connect] 2021/01/29 07:29:27.465865207 Running with python "3.8.3 (default, May 19 2020, 18:47:26) [GCC 7.3.0]" at /opt/python/3.8.3/bin/python3.8 [Connect] 2021/01/29 07:29:27.466663638 Running as user: rstudio-connect [Connect] 2021/01/29 07:29:27.469007305 Skipped packages: mkl-fft==1.1.0, mkl-random==1.1.1, mkl-service==2.3.0 [Connect] 2021/01/29 07:29:27.469030715 Creating environment: gkbz_aZBfpT3dpdzhKtHag GET /__api__/tasks/XlpMgeYwxnxFYQ8C?first_status=525 9ms [Connect] 2021/01/29 07:29:28.439914124 created virtual environment CPython3.8.3.final.0-64 in 720ms [Connect] 2021/01/29 07:29:28.439944325 creator CPython3Posix(dest=/opt/rstudio-connect/mnt/python-environments/pip/3.8.3/gkbz_aZBfpT3dpdzhKtHag, clear=False, global=True) [Connect] 2021/01/29 07:29:28.440056316 seeder FromAppData(download=False, pip=bundle, setuptools=bundle, wheel=bundle, via=copy, app_data_dir=/home/rstudio-connect/.local/share/virtualenv) [Connect] 2021/01/29 07:29:28.440073490 added seed packages: pip==20.2.4, setuptools==51.0.0, wheel==0.36.2 [Connect] 2021/01/29 07:29:28.440096217 activators BashActivator,CShellActivator,FishActivator,PowerShellActivator,PythonActivator,XonshActivator GET /__api__/tasks/XlpMgeYwxnxFYQ8C?first_status=530 11ms GET /__api__/tasks/XlpMgeYwxnxFYQ8C?first_status=530 9ms [Connect] 2021/01/29 07:29:30.527244486 Collecting certifi==2020.6.20 [Connect] 2021/01/29 07:29:30.532795552 Using cached certifi-2020.6.20-py2.py3-none-any.whl (156 kB) [Connect] 2021/01/29 07:29:30.559015888 Processing /tmp/build/80754af9/joblib_1594236160679/work [Connect] 2021/01/29 07:29:30.567061423 ERROR: Could not install packages due to an EnvironmentError: [Errno 2] No such file or directory: '/tmp/build/80754af9/joblib_1594236160679/work' [Connect] 2021/01/29 07:29:30.567122100 [Connect] 2021/01/29 07:29:30.668911069 Error during installation: pip install failed with exit code 1; removing environment from cache: /opt/rstudio-connect/mnt/python-environments/pip/3.8.3/gkbz_aZBfpT3dpdzhKtHag [Connect] 2021/01/29 07:29:30.722159947 Error while removing /opt/rstudio-connect/mnt/python-environments/pip/3.8.3/gkbz_aZBfpT3dpdzhKtHag/rsc-lock: [Errno 2] No such file or directory: '/opt/rstudio-connect/mnt/python-environments/pip/3.8.3/gkbz_aZBfpT3dpdzhKtHag/rsc-lock' [Connect] 2021/01/29 07:29:30.722217565 pip install failed with exit code 1 [Connect] Build error: exit status 1

2

So sorry for the delay in response here. Are you still encountering this issue?

It definitely does look like something is going awry here! This might be worth opening a support ticket at support@rstudio.com so that our team can dig in a bit further.

Do you mind sharing a bit more about your Connect installation - the version of Connect you have installed, etc.?

Looking forward to getting this issue resolved for you!

3

It was not an urgent deployment issue, so no worries on the delay. I thought having the community benefit from my mistakes might be useful.

I have resolved it, but the solution was a bit unsatisfying.

I had a python virtual environment saved at the top level of the project and called it .venv. I built it with reticulate::virtualenv_create and installed necessary packages with reticulate::py_install. I set the environment with RETICULATE_PYTHON in my .Rprofile. If I use relative paths (.venv/bin/python), the shiny app fails locally but the deployment works. This is what I'm using now.

If I use an absolute path (~/Projects/PROJECT_NAME/.venv/bin/python ), the deployment failed but the app worked locally. I created the shiny app from the build panel stored in a shiny directory. If I try to deploy with the virtual environment in the same folder as the shiny app, I get a warning saying that I have more than the permitted number of files selected. This happens even when I deselect .venv.

1 Like
4

Thanks so much for sharing this! Sharing issues on the community definitely does benefit others, though neither you nor I will likely ever know when / whom / how many :smile: It's an awesome way to contribute though!

This is a tricky pattern that hopefully we can find a way to improve long term.

The .Rprofile itself is generally something that you do not want to publish to RStudio Connect, because Connect wants to build and manage its own virtual environment. As a result, you should exclude the .Rprofile (and any definition of RETICULATE_PYTHON) from your deployment.

The basic idea is:

  • use whatever RETICULATE_PYTHON definition locally that makes your app work
  • when you publish, either include a requirements.txt file or one will be generated for you based on your active python installation / environment
  • exclude all reference to a specific RETICULATE_PYTHON when you publish
  • Connect will rebuild a virtualenv for you on deployment (matching your requirements.txt) and associate it with your app (i.e. setting RETICULATE_PYTHON for you)

As for the "too many files," I wonder if that is an error in rsconnect not excluding the .venv path properly. If it's possible to make a simple reproducible example, we would love to get that fixed!

If you want to give this workflow a try, definitely let us know how it goes!! We would love to get you on a "happier path" here :smile: