I just installed RStudio Server 1.0.153 on Ubuntu 17.10. The server is up and listening on port 8787. Whenever a user tries to log on the following error is shown: "Error occurred during transmission" and after that there is only a blank page. In /var/logs/syslog there are no messages from RStudio Server.
Did you ever solve this? I'm getting similar symptoms. I'm just looking into it. I suspect my libR.so (which I'm getting from conda MRO) is incompatible with how rsession has been compiled. But I haven't got to the bottom of it yet.