Unexpected empty response from server - curious..

Posit Professional & Hosted Products
1

Hi All,

I've installed R-Studio V1.2.1335 as a docker container from rocker - rstudio. It sits behind an nginx proxy. It all seems to work fine but when user logs in and tries to upload anything from local hard disk to the R-Studio server, I get the error message "Unexpected empty response from server" as shown.

[https://imgur.com/gqm63LI](Image of error message from imgur.com)

Everything else seems OK.

I followed the nginx proxy configuration instructions and they work fine. I have tried adjusting the "client_max_body_size 1000M;" directive inside nginx to allow larger file uploads, set the permissions on user folders to 777 to eliminate file access problems, added users to the staff and R Studio groups.

There are two articles about this problem but they do not apply in this case. I would link them but new users are not allowed to link articles.

I've then run tcpdump on the R Studio server container and seen what is being transmitted. When the OK button is clicked in the Upload Files dialogue box, about the first 300 of 1000 records are transmitted, but it stops It seems, when trying to upload a simple 70k CSV file.

However, if you check the /tmp folder on the R-Studio container, R Server creates a temporary directory called /tmp/RtmpKljPDe (for example) and the complete dataset is in there in a file called upload3f26d1e8b30.bin, in its entirety, which means the data was successfully uploaded to the server. So far so good.

The server just has to move the upload3f26d1e8b30.bin file from /tmp/RtmpKljPDe to /home/someuser.

So, the solution feels like it should be some kind of settings in session.conf or rserver.conf with something like

multiuser=true
rootdirectory=/home

Which the server then uses to park the downloaded data.

Not sure what to do. As a novice to R Server, I feel I've touched most of the bases, but could have missed some insider info that every admin learns after a while working with R Server.

Any help you can provide would be gratefully received.

Thanks for reading!

2 Likes
2

Very interesting! And thanks for the thorough walk-through! That does seem very curious!

Have you noticed any warnings / errors in the nginx proxy that suggest a terminated connection? It seems to me like the nginx proxy is interfering. One way to confirm whether this is the case would be to tunnel to the RStudio server (i.e. somehow take the nginx proxy out of the picture) and see if the same behavior occurs.

Would you mind giving that a shot? It would at least narrow down whether the nginx proxy is interfering. If you can share any information about or configuration of your nginx proxy, that would be helpful, too! I'm inclined to see if this is reproducible given an nginx proxy and rocker container.

3

Hi Cole,

So's you know, I wrote a long reply to your post (and thanks for it!) and I'm still waiting for it to be modded, hence the delay. This is shorter so hopefully this goes up straight away.

Cheers,

Morgan

4

Hi Cole,

Thanks for your prompt reply. I will try to get replies in sooner - yesterday was super busy and I just couldn't get back to it.

Anyhoo, the nginx container log just gives me this one html message when I click the OK button to upload the files.

172.30.1.102 - - [25/Apr/2019:07:00:57 +0000] "POST /rstudio/upload HTTP/2.0" 200 167 "https://host.fully.qualified.domain.com/rstudio/" "Mozilla/5.0 (X11; Ubuntu; Linux x86_64; rv:66.0) Gecko/20100101 Firefox/66.0"

So a straight 200 message, HTTP OK.

I installed syslog-ng on the nginx proxy and the rstudio container to see if there was anything else I was missing, but no deal.

Some Notes

When you log in to R-Studio server, that is when the randomly named /tmp/RtmpRqSHM8 folder is created.

That folder is not deleted, even if you log out. It will only be deleted if you click the orange 'Quit the current R session' button in the top right corner, no matter how many times you log in and out.

New session, new folder. Quit session, folder is deleted.

If you upload from the desktop, your file is in that folder and will look like this.. upload2bff404d992e.bin upload2bff43b2eadb.bin upload2bffee808fa.bin. In these cases, the last 8 chars appear random, so possibly a hash of a timestamp or something. Anyhoo, despite the .bin extension, everything is plain text.

So now, all that's needed is for R Studio to transfer /tmp/RtmpRqSHM8/upload2bff404d992e.bin to /home/userid/test-location

So, this feels like a file monitor/file mover process is not called or does not run or is not running and that is what prompts the message displayed.

You can see the files in appear using watch, thus...

root@d166712ea00c:/tmp# watch -d -n0.1 ls -las

Is there a list of processes that should be running? Here's mine..

root@d166712ea00c:/# ps aux
USER       PID %CPU %MEM    VSZ   RSS TTY      STAT START   TIME COMMAND
root         1  0.0  0.0    204     4 ?        Ss   Apr24   0:00 s6-svscan -t0 /var/run/s6/services
root        35  0.0  0.0    204     4 ?        S    Apr24   0:00 s6-supervise s6-fdholderd
root       209  0.0  0.0    204     4 ?        S    Apr24   0:00 s6-supervise rstudio
rstudio+   212  0.0  0.1 275272 53768 ?        Ssl  Apr24   1:00 /usr/lib/rstudio-server/bin/rserver --server-daemonize 0
root      2574  0.0  0.0 180480  2080 ?        Ssl  09:08   0:02 /usr/sbin/rsyslogd
root      3451  0.0  0.0  50984   592 ?        S    09:18   0:00 supervising syslog-ng
root      3452  0.0  0.0 226244  7632 ?        Ss   09:18   0:00 /usr/sbin/syslog-ng -p /var/run/syslog-ng.pid
userid   11263  0.6  0.3 922136 102004 ?       Sl   09:37   0:08 /usr/lib/rstudio-server/bin/rsession -u userid --launcher-token B2FFDB0
root     11853  2.0  0.0  19912  3544 pts/0    Ss   10:02   0:00 /bin/bash
root     11859  0.0  0.0  38308  3212 pts/0    R+   10:02   0:00 ps aux

Thanks for all your help and ideas so far.

Best,

Morgan

1 Like
5

Thanks for the response here! I haven't had a whole lot of time to dig in, but one of our IDE devs did, and had the following feedback to try:

  1. Are there any error logs from RSP when this occurs? If the process is failing, we should generally see an error logged.
  2. The upload is processed in two steps. First, a call to /upload, and then once the .bin file is written, the front-end then calls /complete_upload. It looks like the second call is not being made, but from the nginx logs it looks like the upload call succeeded, so it looks like the proxy is having issues here forwarding the response properly.
  3. If possible, try the operation without the nginx proxy altogether
  4. Does a previous version of RStudio work?

On #1, I know the logging of RStudio Server in a docker container can be somewhat limited. One way around this is to run interactively. I.e. start the container with docker run -it rocker/rstudio bash and then start the server process manually i.e. something like:

/usr/lib/rstudio-server/bin/rserver --server-daemonize 0

Logging should dump to stdout/stderr in your interactive terminal in this case.

If you can try those bullets, it may help narrow down whether this is a bug in the proxy or in the latest release of RStudio! Thanks!

6

This topic was automatically closed 21 days after the last reply. New replies are no longer allowed.

If you have a query related to it or one of the replies, start a new topic and refer back with a link.

7

Making a note here that this issue has a solution posted by @liammoran in a follow-on thread here:

In my case, I had the good fortune of having a simple, working configuration that I'd put together for a test environment before setting up the production server that didn't work at first. By default, my production nginx servers all include the anti-clickjacking add-header directive for: X-Frame-Options: DENY;

R-Studio server uses iframes in the interpreter UI: relaxing that directive to add the header X-Frame-Options:SAMEORIGIN rather than DENY got R-Studio uploads working well.

Hope that may help someone someday.