This issue happens also without a reverse proxy on ec2/localhost so I do not think the connection/proxy has anything to do with it.
The problem is definitely on the RStudio side and can be reproduced using the public docker/rocker container (eg rocker/rstudio) and the AWS AMI by louis aslett.
On the browser side I tested everything with Chrome (v. 74.0.3729.169) on a Mac.
Edit: The following bug posted on Github seems to be related: https://github.com/rstudio/rstudio/issues/3299