How to start rstudio-server (free), as service with an ulimit-flag?

You should be able to do this by modifying the systemd service config for your rstudio-server service.

This topic shows an example of how to set the umask for rstudio-server:

RStudio Server Linux - file permissions

The ulimit could be set in a similar fashion by using the appropriate key in place of 'UMask', though off hand I don't know what that would be for your specific case .

This stackexchange thread has a nice compact list of the possibilities (but check the docs for your system):