rserver[1692]: ERROR system error 13 (Permission denied)

ArchosaurRise · December 10, 2019, 9:11pm

I am running rstudio-server on Google Cloud Platform VM instance. My OS is CentOS 8. I try to access the rstudio-server using my Firefox browser at the IP address: http://34.94.188.231:8787/ I create a test user called "testuser" and according password called "testpassword".

Then I got a "Rstudio Initialization Error" "Unable to connect to service".

I check the status of rstudio-server by the command: "sudo rstudio-server status", I got the following message:

● rstudio-server.service - RStudio Server
Loaded: loaded (/etc/systemd/system/rstudio-server.service; enabled; vendor preset: disabled)
Active: active (running) since Tue 2019-12-10 19:29:06 UTC; 13min ago
Process: 1097 ExecStart=/usr/lib/rstudio-server/bin/rserver (code=exited, status=0/SUCCESS)
Main PID: 1133 (rserver)
Tasks: 4 (limit: 22408)
Memory: 62.8M
CGroup: /system.slice/rstudio-server.service
├─1133 /usr/lib/rstudio-server/bin/rserver
└─1692 /usr/lib/rstudio-server/bin/rserver

Dec 10 19:29:05 third-test systemd[1]: Starting RStudio Server...
Dec 10 19:29:06 third-test systemd[1]: Started RStudio Server.
Dec 10 19:30:28 third-test rserver[1692]: ERROR system error 13 (Permission denied); OCCURRED AT: rstudio::core::Error rstudio::core::system::launchChildProcess(std::__cxx11::string, std::__cxx11::string, rstudio::core::system::ProcessConfig, rstudio::core::system::ProcessConfigFilter, PidType*) /var/lib/jenkins/workspace/IDE/open-source-pipeline/v1.2-patch/src/cpp/core/system/PosixSystem.cpp:2152; LOGGED FROM: rstudio::core::Error rstudio::core::system::launchChildProcess(std::__cxx11::string, std::__cxx11::string, rstudio::core::system::ProcessConfig, rstudio::core::system::ProcessConfigFilter, PidType*) /var/lib/jenkins/workspace/IDE/open-source-pipeline/v1.2-patch/src/cpp/core/system/PosixSystem.cpp:2153

Can any one help on that? I searched a lot using Google, and tried many suggested suggestions, and it doesn't work.

cole · December 11, 2019, 3:21pm

Hello! Apologies for the trouble! Typically the "ERROR system error 13 (Permission denied); " error suggests that the user you are using either:

does not have a home directory
does not have write access to the home directory

Can you verify the home directory for your user by SSHing into the server?

getent passwd myusername

And then look at the permissions on that directory?

namei -l /path/to/my/home/directory
ls -lha /path/to/my/home/directory

ArchosaurRise · December 11, 2019, 5:53pm

Hi, Cole,

Thanks for the suggestions. Here is the test results:

[testuser@third-test ~]$ getent passwd testuser
testuser:x:1002:1003::/home/testuser:/bin/bash

[testuser@third-test ~]$ namei -l /home/testuser/
f: /home/testuser/
dr-xr-xr-x root root /
drwxr-xr-x root root home
drwx------ testuser testuser testuser

[testuser@third-test ~]$ ls -lha /home/testuser/
total 16K
drwx------. 3 testuser testuser 95 Dec 10 19:27 .
drwxr-xr-x. 5 root root 58 Dec 9 23:22 ..
-rw-------. 1 testuser testuser 392 Dec 11 00:11 .bash_history
-rw-r--r--. 1 testuser testuser 18 Oct 1 13:26 .bash_logout
-rw-r--r--. 1 testuser testuser 141 Oct 1 13:26 .bash_profile
-rw-r--r--. 1 testuser testuser 312 Oct 1 13:26 .bashrc
drwx------. 2 testuser testuser 29 Dec 10 19:13 .ssh

cole · December 11, 2019, 6:13pm

Thanks for sharing! That's interesting! Permissions look fine there. It might be worth clearing the cookies in your browser and then authenticating to RStudio Server again to be sure the issue is the same.

Meanwhile, you should tail -f /var/log/syslog or tail -f /var/log/messages (depending on which exists) to see if you get the same error message.

Are you using the open source version of RStudio Server? You could also try creating a new user to be sure things failing are not because of strange state with the testuser:

useradd -m -s /bin/bash testuser2
passwd testuser2

ArchosaurRise · December 11, 2019, 6:37pm

Hi, Cole,
Thanks for your quick response.

I cleared the cookies from my firefox browser on Windows 7, and tried to log into the Rstudio server again. It popped up the same error again.

Yes. I am using the open source version of Rstudio server. I followed the instructions here:

Here is the test result following your suggestions:

[testuser@third-test ~]$ tail -f /var/log/syslog
tail: cannot open '/var/log/syslog' for reading: No such file or directory
tail: no files remaining
[testuser@third-test ~]$ tail -f /var/log/messages
tail: cannot open '/var/log/messages' for reading: Permission denied
tail: no files remaining
[testuser@third-test ~]$ useradd -m -s /bin/bash testuser2
useradd: Permission denied.
useradd: cannot lock /etc/passwd; try again later.
[testuser@third-test ~]$ sudo useradd -m -s /bin/bash testuser2
[testuser@third-test ~]$ sudo passwd testuser2
Changing password for user testuser2.
New password:
BAD PASSWORD: The password fails the dictionary check - it is based on a dictionary word
Retype new password:
passwd: all authentication tokens updated successfully.

As you can see, I created a new user called "testuser2" with a password "testpassword2", however, when I logged into rstudio server using my firefox web browser, it popped the same error.

cole · December 12, 2019, 2:48am

Ahh sad! Sorry to hear about that! At least the problem is reproducible

On the log file, can you try sudo cat /var/log/messages? It should give you a bunch of output.

Can you also share the output of:

namei -l /var/lib/rstudio-server/shared-storage/sessions
ls -lha /var/lib/rstudio-server/shared-storage/sessions

It is very curious that this error is appearing for a user who has a home directory with proper privileges!

ArchosaurRise · December 12, 2019, 6:31pm

Hi, Cole,

Thanks for problem-solving.

Here is the log file generated by sudo cat /var/log/messages:
https://drive.google.com/file/d/1AqKZsqpT771xkTAUfH14KSzOoaxs_d7z/view?usp=sharing

[testuser@third-test ~]$ namei -l /var/lib/rstudio-server/shared-storage/sessions
f: /var/lib/rstudio-server/shared-storage/sessions
dr-xr-xr-x root root /
drwxr-xr-x root root var
drwxr-xr-x root root lib
drwxrwxrwx root root rstudio-server
                     shared-storage - No such file or directory
[testuser@third-test ~]$ ls -lha /var/lib/rstudio-server/shared-storage/sessions
ls: cannot access '/var/lib/rstudio-server/shared-storage/sessions': No such file or directory

kevinushey · December 12, 2019, 7:28pm

This error:

is being emitted here:

github.com

rstudio/rstudio/blob/330255ddec489e7a147ace3e8a9a3e4157d8d5ad/src/cpp/core/system/PosixSystem.cpp#L2148-L2155


      
          // obtain a new process group (using our own process id) so our
          // lifetime isn't tied to our parent's lifetime
          if (::setpgid(0,0) == -1)
          {
             Error error = systemError(errno, ERROR_LOCATION);
             LOG_ERROR(error);
             ::exit(EXIT_FAILURE);
          }

which means the call to ::setpgid(0, 0) is failing, for some reason.

cole · December 12, 2019, 7:57pm

I noticed some SELinux stuff in your messages file, although little else seemed interesting. Is it possible that SELinux is causing trouble? Can you turn off SELinux and see if that resolves the issue?

I have no idea why the ::setpgid(0, 0) call would be failing (as @kevinushey noted)

ArchosaurRise · December 12, 2019, 10:15pm

Hi, Cole and kevinushey,

Thanks for providing suggestions. I disabled the SELinux and rebooted the operating system, now the rstudio-server can be accessed through web browser! Here is the screenshot of success!

Since the SELinux is turned on by default on RedHat/CentOS/Fedora or several derivatives, perhaps you guys can try to make rstudio-server compatible with SELinux, or give a warning in the installation instructions. That would be very helpful for users.

Here is the test process. I refer to the web page here: https://www.tecmint.com/disable-selinux-in-centos-rhel-fedora/ , I tested the hypothesis about SELinux.

[testuser@third-test ~]$ sestatus
SELinux status:                 enabled
SELinuxfs mount:                /sys/fs/selinux
SELinux root directory:         /etc/selinux
Loaded policy name:             targeted
Current mode:                   enforcing
Mode from config file:          enforcing
Policy MLS status:              enabled
Policy deny_unknown status:     allowed
Memory protection checking:     actual (secure)
Max kernel policy version:      31

I changed the directive SELinux=enforcing to SELinux=disabled

[testuser@third-test ~]$ sudo vi /etc/sysconfig/selinux

Then I reboot my system.

[testuser@third-test ~]$ sudo shutdown -r now

Now the SELinux is disabled.

[testuser@third-test ~]$ sestatus
SELinux status:                 disabled

Then the rstudio-server status looks good.

[testuser@third-test ~]$ sudo rstudio-server status
● rstudio-server.service - RStudio Server
   Loaded: loaded (/etc/systemd/system/rstudio-server.service; enabled; vendor preset: disabled)
   Active: active (running) since Thu 2019-12-12 21:42:54 UTC; 2min 0s ago
  Process: 1091 ExecStart=/usr/lib/rstudio-server/bin/rserver (code=exited, status=0/SUCCESS)
 Main PID: 1135 (rserver)
    Tasks: 3 (limit: 22408)
   Memory: 58.7M
   CGroup: /system.slice/rstudio-server.service
           └─1135 /usr/lib/rstudio-server/bin/rserver

Dec 12 21:42:54 third-test systemd[1]: Starting RStudio Server...
Dec 12 21:42:54 third-test systemd[1]: Started RStudio Server.
[testuser@third-test ~]$ sudo rstudio-server status
● rstudio-server.service - RStudio Server
   Loaded: loaded (/etc/systemd/system/rstudio-server.service; enabled; vendor preset: disabled)
   Active: active (running) since Thu 2019-12-12 21:42:54 UTC; 3min 37s ago
  Process: 1091 ExecStart=/usr/lib/rstudio-server/bin/rserver (code=exited, status=0/SUCCESS)
 Main PID: 1135 (rserver)
    Tasks: 8 (limit: 22408)
   Memory: 141.0M
   CGroup: /system.slice/rstudio-server.service
           ├─1135 /usr/lib/rstudio-server/bin/rserver
           └─1662 /usr/lib/rstudio-server/bin/rsession -u testuser2 --launcher-token A8F380C6

Dec 12 21:42:54 third-test systemd[1]: Starting RStudio Server...
Dec 12 21:42:54 third-test systemd[1]: Started RStudio Server.

cole · December 13, 2019, 4:42pm

Woohoo!! So glad that you got things working! This is definitely on our list of things to tackle, we just have not done so yet:

There is an open github issue here:

github.com/rstudio/rstudio

Support SELinux via configurations and documentation

opened 07:33PM - 12 Jun 19 UTC

adatum

enhancement

### System details RStudio Edition : Server RStudio Version : 1.2.13…35 OS Version : Fedora 30 R Version : 3.5.3 ### Steps to reproduce the problem Install `rstudio-server` and attempt to log in at `localhost:8787` with SELinux in enforcing mode (default on Fedora and others). Login will fail due to SELinux denial. ### Describe the problem in detail SELinux prevents RStudio Server from functioning. Setting SELinux in permissive mode works fine, albeit with dozens of SELinux alerts. ### Describe the behavior you expected RStudio Server allowing user login and working normally with SELinux enabled in enforcing mode.

And a community thread on the topic here, that I have unfortunately not had much time to dig further into:

ArchosaurRise · December 14, 2019, 11:20pm

Thanks for the help. Hopefully, the feature could be fixed soon.

system · December 21, 2019, 11:20pm

This topic was automatically closed 7 days after the last reply. New replies are no longer allowed.