rgee App Deployment: Virtual Environment from requirements.txt Not Being Used

auricht · March 5, 2025, 12:20pm

Hi All,

I’m encountering an issue deploying my rgee-based Shiny app. Locally, everything works perfectly, but once deployed, the app seems to default to the system Python (/usr/bin/python3) instead of using a virtual environment built from my requirements.txt.
As a result, essential packages like numpy and earthengine-api aren’t available, and Earth Engine fails to initialize non?interactively.

Can get app to deploy fine if EE aspects are removed but that limits required functionality.

Details:

Environment:
    rgee version: 1.1.7 (CRAN release)
    reticulate, Shiny, and other packages are up-to-date.
    I’m deploying on [ShinyApps.io/Posit Connect]

requirements.txt (in the project root):

numpy
earthengine-api
google-auth
google-api-python-client

Credentials:
I’m using a service account key (a valid JSON file with all required fields) located in my project under a credentials folder. I set the environment variable with:

Sys.setenv(GOOGLE_APPLICATION_CREDENTIALS = "credentials/credentials.json") but also tried google cloud console credentials file.

App Initialization Code:

Clear the console

cat("\014")

Load required packages for app - that runs locally

library(rgee)
library(reticulate)
library(shiny)
library(leaflet)
library(DT)
library(dplyr)
library(writexl)
library(ausplotsR)
library(shinyjs)
library(shinycssloaders)
library(zip)
library(ggplot2)
library(plotly)
library(httr)
library(zoo)
library(forecast)
library(tidyr)

Set service account credentials (relative path)

Sys.setenv(GOOGLE_APPLICATION_CREDENTIALS = "credentials/credentials.json")

(Optional) Print debugging info

print(reticulate::py_config())
print(reticulate::py_module_available("numpy"))
print(reticulate::py_list_packages())

Initialize Earth Engine (should use credentials and the virtualenv built from requirements.txt)

ee_Initialize(drive = FALSE, gcs = FALSE)

Logs:
The logs show that the Python environment used is still:

python:         /usr/bin/python3
...
numpy:           [NOT FOUND]
ee:             [NOT FOUND]
NOTE: Python version was forced by RETICULATE_PYTHON

I’ve removed any forced RETICULATE_PYTHON settings in the code and relied solely on the requirements.txt file for dependency management. Despite this the build logs indicate that the system Python is being used, so my app never picks up numpy or earthengine-api.

Has anyone encountered this issue? Any advice on ensuring that the virtual environment built from requirements.txt is used (or on how to force the correct environment) would be greatly appreciated.

Many thanks.

Kind regards,
Chris

Simon_Smart · March 5, 2025, 4:12pm

I've got zero experience doing this, but according to r - Use python 3 in reticulate on shinyapps.io - Stack Overflow you need to use various {reticulate} functions to manage your python dependencies, instead of requirements.txt.

auricht · March 6, 2025, 2:08am

Title:
rgee on ShinyApps.io: Reticulate Virtualenv Works, But Non-Interactive Authentication Fails

Hi All

Thanks Simon for the message.

Update - managed to have success in managing Python dependencies entirely via reticulate (without using requirements.txt).
As such virtual environment (named “python3_env”) is set up correctly—logs confirm that numpy, earthengine-api, and related packages are installed in the virtualenv:

Python: /home/shiny/.virtualenvs/python3_env/bin/python
numpy: Installed at /home/shiny/.virtualenvs/python3_env/lib/python3.10/site-packages/numpy
ee (earthengine-api): Installed at /home/shiny/.virtualenvs/python3_env/lib/python3.10/site-packages/ee

Code then forces the use of the virtual environment with:

reticulate::use_virtualenv("python3_env", required = TRUE)

and debugging output confirms the correct environment is in use.

Issue now is authentication.

Have tried setting service account credentials using:

Sys.setenv(GOOGLE_APPLICATION_CREDENTIALS = "credentials/credentials.json")

where "credentials/credentials.json" is a properly formatted service account key (the same one that works locally).

The Issue:
initialize Earth Engine with:

ee_Initialize(drive = FALSE, gcs = FALSE)

the app still triggers an interactive authentication flow by prompting for a verification code via a URL, which then results in an EOFError.

Have also attempted:

ee_Initialize(
service_account = "auricht@able-device-311600.iam.gserviceaccount.com",
drive = FALSE,
gcs = FALSE
)

and switching the credentials file to:

Sys.setenv(GOOGLE_APPLICATION_CREDENTIALS = "credentials/able-device-311600-593b5d52bc9c.json")

Unfortunately, all of these variations still result in the same authentication issues on ShinyApps.io.

So in summary

Managing Dependencies via reticulate Only:
I.e. not using requirements.txt; instead, installing packages using reticulate and forcing the virtual environment, which shows the correct packages (including numpy and earthengine-api) are installed.

Service Account Authentication Attempts:

Setting GOOGLE_APPLICATION_CREDENTIALS to point tservice account key (plus alternative filenames) fails as does attempting to pass the service_account parameter in ee_Initialize(), but none of these methods prevent the interactive authentication prompt.

Questions:

1. Has anyone managed to bypass the interactive authentication requirement on ShinyApps.io using a reticulate-only dependency management approach (without relying on requirements.txt)?
2. Is there a recommended workaround (perhaps using the development version of rgee) that supports non-interactive authentication via a service account key?

How to ensure that non-interactive authentication works on ShinyApps.io in an rgee app if cached credentials aren’t available?

Any insights, alternative approaches, or sample code would be greatly appreciated.

Kind regards,

Chris

Simon_Smart · March 6, 2025, 9:17am

Glad you've made progress. The normal way to handle credentials is via a .Renviron file and that should work the same locally as on shinyapps. I'm not sure of the specifics relating to rgee, but I'd first try moving whatever is in credentials.json into there and see if that makes a difference.

auricht · March 7, 2025, 12:26am

Many thanks Simon.
Update

Based on latest troubleshooting, can confirm that the .Renviron file is being read correctly on ShinyApps.io. When printing the environment variable in the app:

print(Sys.getenv("GOOGLE_APPLICATION_CREDENTIALS"))

it correctly returns:

"credentials/credentials.json"

Additionally, the virtual environment is being correctly forced via:

reticulate::use_virtualenv("python3_env", required = TRUE)

and logs show that numpy and earthengine-api are installed in that environment:

Python path: /home/shiny/.virtualenvs/python3_env/bin/python
numpy: Installed at /home/shiny/.virtualenvs/python3_env/lib/python3.10/site-packages/numpy
ee: Installed at /home/shiny/.virtualenvs/python3_env/lib/python3.10/site-packages/ee

However, despite these correct settings, when calling ee_Initialize(drive = FALSE, gcs = FALSE), rgee 1.1.7 still falls back to an interactive authentication flow—prompting for a verification code. This interactive prompt causes an EOFError as I understand ShinyApps.io is non‑interactive.

It appears that rgee 1.1.7 does not fully support non‑interactive authentication using a service account key or cached credentials, even though the .Renviron file and virtual environment are correctly configured?. I've tried both pre‑authenticating locally (to generate cached credentials) and passing a service_account parameter (which rgee 1.1.7 doesn’t support), but neither approach prevents the interactive prompt.

Has anyone successfully achieved non‑interactive authentication on ShinyApps.io with rgee? I'm open to using the development version (rgee2) if it provides a solution, but so far I haven’t been able to get it installed due to encoding issues.

Any guidance or suggestions to work around this limitation would be greatly appreciated.

Many thanks,
Chris

Simon_Smart · March 7, 2025, 8:26pm

Sorry I realise I wasn't very clear - .Renviron needs to contain the actual credentials, not a link to the json.

auricht · March 8, 2025, 1:23am

Subject:
Using Embedded Credentials in .Renviron for rgee on shinyapps.io

Message:

Hi all,

I’ve been testing different ways to deploy an rgee-based Shiny app on shinyapps.io.
Have set the environment variable GOOGLE_APPLICATION_CREDENTIALS to a file path (e.g., "credentials/credentials.json"), but based on info from Simon have tried to embed the actual JSON contents directly in the .Renviron file. For example, current .Renviron now looks like this:

SHINY_ACC_NAME="xx"
TOKEN="xx"
SECRET="xx"
MASTERNAME="xx"
GOOGLE_APPLICATION_CREDENTIALS={xx

However, when checking logs, the GOOGLE_APPLICATION_CREDENTIALS variable appears empty before ee_Initialize(), and authentication still ends up being interactive.

Based on the documentation and various examples (including those in the shiny_rgee_template)
https://github.com/csaybar/shiny_rgee_template?tab=readme-ov-file
the expected usage is to point GOOGLE_APPLICATION_CREDENTIALS to a file path rather than embedding the JSON content directly. In other words, I understand it’s generally recommended to have a file (e.g., credentials.json) that contains the JSON and then set:

GOOGLE_APPLICATION_CREDENTIALS="credentials/credentials.json"

Using embedded JSON in the .Renviron file may not be parsed or passed correctly by the rgee initialization routines. (The .Renviron file is read as plain text, and rgee expects a file path so that it can read and parse the JSON file.)

Current approach:

Running ee_Initialize() interactively on local machine to cache credentials (which are stored in a folder like ~/.config/earthengine/credentials).
Then copy the cached credentials into my project folder (in a subfolder called, say, credentials/) and update my .Renviron to set:

GOOGLE_APPLICATION_CREDENTIALS="credentials/credentials.json"

Deploy the app using a deploy script that reads .Renviron and calls ee_Initialize() non-interactively.

Given that the deployment still prompts for interactive authentication (and the variable appears empty in the logs), it seems that embedding the JSON directly in .Renviron isn’t working as intended and neither is sue of credentials.json

Questions:

Has anyone successfully embedded the full JSON credentials in .Renviron (instead of a file path) for non-interactive rgee authentication on shinyapps.io?
Is the recommended approach still to include a credentials file and set GOOGLE_APPLICATION_CREDENTIALS to the file path? 
Are there any known issues with reading .Renviron on shinyapps.io that might be causing the variable to appear empty at runtime?

Any guidance or workarounds would be greatly appreciated. Thanks in advance!

Rgds, C