Python/Connect setup on Windows to deploy to RSConnect

[slodge]

I'm trying to set up our RStudio Connect server with Python at present.

I think I've got the server setup OK using the admin documnets.

I've now got a sample notebook - based off of https://github.com/sol-eng/python-examples/tree/master/rmarkdown-notebook.

I'm running this locally using a python miniconda environment configured using:

Sys.setenv(RETICULATE_PYTHON = "C:/Users/s.lodge/AppData/Local/r-miniconda/envs/r-reticulate/python.exe")

This notebook runs fine on my local machine.

When I try to deploy from RStudio 1.4 (still running the preview), I get some problems with numpy not installing.

The error message looks like it contains a distinctly "Windows" flavour path /D:/bld/numpy_1610324703282/work - although on my local machine I don't have a D drive...

When I look in the manifest.json and requirements.txt files that rsconnect::write_manifest produces I see:

  "python": {
    "version": "3.6.13",
    "package_manager": {
      "name": "pip",
      "version": "21.0.1",
      "package_file": "requirements.txt"
    }
  },

and

certifi==2020.12.5
numpy @ file:///D:/bld/numpy_1610324703282/work
pandas @ file:///D:/bld/pandas_1607423921518/work
python-dateutil==2.8.1
pytz @ file:///home/conda/feedstock_root/build_artifacts/pytz_1612179539967/work
six @ file:///home/conda/feedstock_root/build_artifacts/six_1590081179328/work
wincertstore==0.2

Full error message in Connect is:

... lots of R installation lines
[Connect] Completed packrat build against R version: '4.0.2'
[Connect] Bundle requested Python version 3.6.13; using /usr/bin/python3.6 which has version 3.6.9
[Connect] 2021/03/25 13:12:20.175712048 Running with python "3.6.9 (default, Oct  8 2020, 12:12:24)  [GCC 8.4.0]" at /usr/bin/python3.6
[Connect] 2021/03/25 13:12:20.176883245 Running as user: rstudio-connect
[Connect] 2021/03/25 13:12:20.178630183 Skipped packages: wincertstore==0.2
[Connect] 2021/03/25 13:12:20.178645272 Creating environment: 1JAFM8dkKh_4HrMBGNArjg
[Connect] 2021/03/25 13:12:21.303884503 created virtual environment CPython3.6.9.final.0-64 in 869ms
[Connect] 2021/03/25 13:12:21.303919496   creator CPython3Posix(dest=/opt/rstudio-connect/mnt/python-environments/pip/3.6.9/1JAFM8dkKh_4HrMBGNArjg, clear=False, no_vcs_ignore=False, global=True)
[Connect] 2021/03/25 13:12:21.303969815   seeder FromAppData(download=False, pip=bundle, setuptools=bundle, wheel=bundle, via=copy, app_data_dir=/opt/rstudio-connect/mnt/tmp/tmpn92rjv2p)
[Connect] 2021/03/25 13:12:21.303971949     added seed packages: pip==21.0.1, setuptools==54.1.2, wheel==0.36.2
[Connect] 2021/03/25 13:12:21.303992338   activators BashActivator,CShellActivator,FishActivator,PowerShellActivator,PythonActivator,XonshActivator
[Connect] 2021/03/25 13:12:22.735061975 Processing /D:/bld/numpy_1610324703282/work
[Connect] 2021/03/25 13:12:22.738428226 ERROR: Could not install packages due to an OSError: [Errno 2] No such file or directory: '/D:/bld/numpy_1610324703282/work'
[Connect] 2021/03/25 13:12:22.738484562 
[Connect] 2021/03/25 13:12:22.781323886 Error during installation: pip install failed with exit code 1; removing environment from cache: /opt/rstudio-connect/mnt/python-environments/pip/3.6.9/1JAFM8dkKh_4HrMBGNArjg
[Connect] 2021/03/25 13:12:22.823898235 Error while removing /opt/rstudio-connect/mnt/python-environments/pip/3.6.9/1JAFM8dkKh_4HrMBGNArjg/rsc-lock: [Errno 2] No such file or directory: '/opt/rstudio-connect/mnt/python-environments/pip/3.6.9/1JAFM8dkKh_4HrMBGNArjg/rsc-lock'
[Connect] 2021/03/25 13:12:22.823929538 pip install failed with exit code 1
[Connect] Build error: exit status 1

Any ideas what's wrong in my setup? I'm assuming this is a Windows config issue? (And that I'll have other issues along the way too!)

[slodge]

An update on this...

I can manually update the requirements.txt to remove the Python package locations and then my upload works:

ie changing requirements.txt to:

certifi==2020.12.5
numpy
pandas
python-dateutil==2.8.1
pytz
six
wincertstore==0.2

but it feels like rsconnect shouldn't generate requirements.txt with these options in the first place?

[edavidaja]

There is a more narrowly specified requirements.txt file in the repo. I think rsconnect is capturing things installed in your conda environment unrelated to the execution of the notebook content, like packages for accessing the Windows certificate store (which would not be used on connect).

[slodge]

Thanks

I'm just looking at how to generate these files ourselves - including in our CI/CD pipelines... so ideally am hoping I can tease the rsconnect writeManifest method into generating "good" requirements files.