I am reaching out regarding a challenging yet exciting task ahead in my project: scaling a Shiny R application to support a national-scale web map service for InSAR data in Sweden. The application, which I have been developing, utilizes a PostgreSQL database (on a linux server) with approximately 10 TB of data as its backend and a Shiny R frontend. Kindly see the attached picture for an overview of the application.
I am now at the critical stage of deploying the service (on web-facing separate linux server) and scaling it to support multiple users effectively. Given the unprecedented nature of this project - a first-of-its-kind national-scale web map service written in R in Sweden, hosted by a governmental agency - I find myself in need of guidance and expertise. I am wondering if anybody in Shiny community could offer advice, insights, or support to help ensure the success of this deployment (and even maybe code review).
I am prepared to acknowledge any contributions listing everybody involved in a separate pop-up window that appears when the user hits the "Guide" button.
I would greatly appreciate any feedback. Please reach out if you are interested!
Thank you for your time, and I look forward to hearing from you!
Posit has offerings like ShinyApps.io and Posit Connect that would probably meet your needs, but come with recurring costs. I've deployed free apps to ShinyApps.io, but nothing that required persistent data storage like with a PostgresSQL database so I'm not sure what additional cost considerations there might be for that. In any case, it might be worth consulting with someone from Posit if SaaS is a viable option.
An alternative option, that would require self-hosted infrastructure and some know-how with networking, would be to use something like Shiny Proxy which can (help) manage containerization, site deployment, and creation of new sessions for users. For starters, you'll probably want to mount the database to a shared volume that the containers can read from.
Thanks for reaching out. Yeah, I myself deployed a bunch of apps at shinyapps.io but this case is totally different. I asked Posit and they recommended to post a question here.
I also deployed one Shiny app (with PostgreSQL connection) on the linux server via nginx proxy before. But this one is a small scale, just up to 10 users. Works ok.
In case of "InSAR Sweden", this will be a national scale web application, so I really need to think differently. Containerization is something that I considered, but have not started testing yet - thought I will reach out to community first.
The whole set up so far is 2 linux servers: one hosting PostgreSQL db with all data and another linux server as a separate web-facing one that hosts R code. So the architecture is prepared, it is more the scaling (optimization, load balancing and containerization) that I need advice for.
Appreciate any help really. The more help - the better.
I think containerization will be necessary in your case then. To the best of my knowledge, there's not really another good way to scale many R processes across a machine otherwise. I would imagine set-up is roughly something like this:
Create a Dockerfile listing all of the dependencies and application code for the Docker image. You could use ShinyProxy as the base image, or do something entirely custom (but ShinyProxy will probably make your life much easier).
Create a Docker volume for the PostgresSQL data base that can be networked to the containers.
Use something like Docker Swarm or Kubernetes to manage scaling of containers. I think ShinyProxy actively recommends the use of Kubernetes.
Depending on your expected user-load, I would consider preemptively investing in purchasing a lot of RAM and SSDs for the persistent storage (if you aren't already using.)
Thank you very much, this is very helpful. Then we were on the right track with our thoughts, gonna test what you suggested.
Regarding RAM and SSD, that part is taken care of
How skilled are you with Shiny R? Do you want to contribute code-wise? I mean any code can be improved and optimized... As I mentioned, I am happy to list all contributors, no problem with that.
