You should be able to do this by modifying the systemd service config for your rstudio-server service.
This topic shows an example of how to set the umask for rstudio-server:
RStudio Server Linux - file permissions
The ulimit could be set in a similar fashion by using the appropriate key in place of 'UMask', though off hand I don't know what that would be for your specific case .
This stackexchange thread has a nice compact list of the possibilities (but check the docs for your system):
https://unix.stackexchange.com/questions/345595/how-to-set-ulimits-on-service-with-systemd