Usually, it is R that needs to restart for a package install (not RStudio). The reason is that the package has already been attached, and "detaching" the package cannot always be done cleanly. So restarting the session allows you to install the package into an environment where the package is not in use (This is not unusual in software. Upgrading / overwriting while things are running can often be cause for trouble).
As for upgrading with R in production, I echo @mishabalyasin's sentiment that it would be helpful to have more clarification about what you mean for "R in production" and why you are looking for a way to install packages within an RServe session?
I am not super familiar with RServe, but RServe is a "server" responding to client requests. I.e. you might think about a node.js server responding to clients. Usually, in "production," you would seek to reduce package installations and similar types of time consuming, potentially fault intolerant processes. This would definitely mean not installing packages within an RServe session.
Rather, you would usually be aiming for reproducible processes, and therefore have an environment set up and stable to serve RServe client requests. Another package you might think about for an API framework is plumber, which sets up a REST API.
These threads might also be interesting to peruse:
And the following article on reproducible package management (disclaimer: I'm the author):