On our RStudio server users are unable to generate SSH RSA keys, either using Tools > Global Options > Git/SVN > Create RSA Key... or within the Terminal or using something like system("ssh-keygen -f id_rsa -t rsa -N ''") within the R console. It will always say Permission Denied. I've tested this with network/ldap users with NFS home directories and with local accounts with local homes. If I login to the server directly I'm able to run the same bash commands to generate the keys successfully. Is this a bug? Any troubleshooting tips?
That is very strange!! Maybe try some of the following for more information (in the terminal)?
id
env
which ssh-keygen
ls -lha /path/to/ssh-keygen
ssh-keygen --help
If you could execute similarly from a bash shell, that would be helpful too! Also, if you can provide the version of RStudio Server, I think that would be helpful as well. (Help -> About RStudio)
Version 1.1.453
bash-4.2$ id
uid=285233(mah60) gid=1000(access) groups=1000(access),188(staff.up.cis),<REMOVED ADDITIONAL LDAP GROUPS> context=system_u:system_r:initrc_t:s0
bash-4.2$ env
SED=/usr/bin/sed
LN_S=ln -s
R_TEXI2DVICMD=/usr/bin/texi2dvi
RSTUDIO_PANDOC=/usr/lib/rstudio-server/bin/pandoc
XDG_SESSION_ID=c2
R_PDFVIEWER=/usr/bin/xdg-open
R_INCLUDE_DIR=/opt/R/3.5.0/lib64/R/include
HOSTNAME=<REMOVED HOSTNAME>
RSTUDIO_PROJ_NAME=
RSTUDIO_SESSION_STREAM=mah60-ds/cfc78a31/107764e8
R_PRINTCMD=lpr
TERM=xterm-256color
RSTUDIO_MULTI_SESSION=1
HISTSIZE=1000
R_SYSTEM_ABI=linux,gcc,gxx,gfortran,?
R_RD4PDF=times,inconsolata,hyper
R_PAPERSIZE=a4
RSTUDIO=1
RSTUDIO_HTTP_REFERER=https://<REMOVED>/s/da271cfc78a31107764e8/
RSTUDIO_CONSOLE_WIDTH=111
RSESSION_PROFILE_OPTIONS=
USER=mah60
LS_COLORS=rs=0:di=38;5;27:ln=38;5;51:mh=44;38;5;15:pi=40;38;5;11:so=38;5;13:do=38;5;5:bd=48;5;232;38;5;11:cd=48;5;232;38;5;3:or=48;5;232;38;5;9:mi=05;48;5;232;38;5;15:su=48;5;196;38;5;15:sg=48;5;11;38;5;16:ca=48;5;196;38;5;226:tw=48;5;10;38;5;16:ow=48;5;10;38;5;21:st=48;5;21;38;5;15:ex=38;5;34:*.tar=38;5;9:*.tgz=38;5;9:*.arc=38;5;9:*.arj=38;5;9:*.taz=38;5;9:*.lha=38;5;9:*.lz4=38;5;9:*.lzh=38;5;9:*.lzma=38;5;9:*.tlz=38;5;9:*.txz=38;5;9:*.tzo=38;5;9:*.t7z=38;5;9:*.zip=38;5;9:*.z=38;5;9:*.Z=38;5;9:*.dz=38;5;9:*.gz=38;5;9:*.lrz=38;5;9:*.lz=38;5;9:*.lzo=38;5;9:*.xz=38;5;9:*.bz2=38;5;9:*.bz=38;5;9:*.tbz=38;5;9:*.tbz2=38;5;9:*.tz=38;5;9:*.deb=38;5;9:*.rpm=38;5;9:*.jar=38;5;9:*.war=38;5;9:*.ear=38;5;9:*.sar=38;5;9:*.rar=38;5;9:*.alz=38;5;9:*.ace=38;5;9:*.zoo=38;5;9:*.cpio=38;5;9:*.7z=38;5;9:*.rz=38;5;9:*.cab=38;5;9:*.jpg=38;5;13:*.jpeg=38;5;13:*.gif=38;5;13:*.bmp=38;5;13:*.pbm=38;5;13:*.pgm=38;5;13:*.ppm=38;5;13:*.tga=38;5;13:*.xbm=38;5;13:*.xpm=38;5;13:*.tif=38;5;13:*.tiff=38;5;13:*.png=38;5;13:*.svg=38;5;13:*.svgz=38;5;13:*.mng=38;5;13:*.pcx=38;5;13:*.mov=38;5;13:*.mpg=38;5;13:*.mpeg=38;5;13:*.m2v=38;5;13:*.mkv=38;5;13:*.webm=38;5;13:*.ogm=38;5;13:*.mp4=38;5;13:*.m4v=38;5;13:*.mp4v=38;5;13:*.vob=38;5;13:*.qt=38;5;13:*.nuv=38;5;13:*.wmv=38;5;13:*.asf=38;5;13:*.rm=38;5;13:*.rmvb=38;5;13:*.flc=38;5;13:*.avi=38;5;13:*.fli=38;5;13:*.flv=38;5;13:*.gl=38;5;13:*.dl=38;5;13:*.xcf=38;5;13:*.xwd=38;5;13:*.yuv=38;5;13:*.cgm=38;5;13:*.emf=38;5;13:*.axv=38;5;13:*.anx=38;5;13:*.ogv=38;5;13:*.ogx=38;5;13:*.aac=38;5;45:*.au=38;5;45:*.flac=38;5;45:*.mid=38;5;45:*.midi=38;5;45:*.mka=38;5;45:*.mp3=38;5;45:*.mpc=38;5;45:*.ogg=38;5;45:*.ra=38;5;45:*.wav=38;5;45:*.axa=38;5;45:*.oga=38;5;45:*.spx=38;5;45:*.xspf=38;5;45:
LD_LIBRARY_PATH=/opt/R/3.5.0/lib64/R/lib::/lib:/usr/local/lib64:/usr/lib/jvm/java-1.8.0-openjdk-1.8.0.171-8.b10.el7_5.x86_64/jre/lib/amd64/server
TAR=/usr/bin/gtar
R_ZIPCMD=/usr/bin/zip
COBBLER_SERVER=redhat.psu.edu
PAGER=/usr/bin/less
RSTUDIO_DISABLE_PROJECT_SHARING=1
R_GZIPCMD=/usr/bin/gzip
PATH=/usr/local/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/pass/users/m/a/mah60/.local/bin:/pass/users/m/a/mah60/bin:bin/msys-ssh-1000-18
MAIL=/var/spool/mail/mah60
RSTUDIO_USER_IDENTITY=mah60
PWD=/pass/users/m/a/mah60
RSTUDIO_WINUTILS=bin/winutils
R_SHARE_DIR=/opt/R/3.5.0/lib64/R/share
RSTUDIO_SESSION_ID=107764e8
EDITOR=vi
LANG=en_US.UTF-8
R_LIBS_SITE=
RSTUDIO_TERM=A27FCF21
SSH_ASKPASS=rpostback-askpass
RMARKDOWN_MATHJAX_PATH=/usr/lib/rstudio-server/resources/mathjax-26
R_BROWSER=/usr/bin/xdg-open
HISTCONTROL=ignoredups
SHLVL=3
HOME=/pass/users/m/a/mah60
MAKE=make
RS_RPOSTBACK_PATH=/usr/lib/rstudio-server/bin/rpostback
R_UNZIPCMD=/usr/bin/unzip
LOGNAME=mah60
R_BZIPCMD=/usr/bin/bzip2
CLICOLOR_FORCE=1
LESSOPEN=||/usr/bin/lesspipe.sh %s
PROMPT_COMMAND=echo -ne "\033]0;${PWD/#${HOME}/~}\007"
R_HOME=/opt/R/3.5.0/lib64/R
GIT_ASKPASS=rpostback-askpass
DISPLAY=:0
XDG_RUNTIME_DIR=/run/user/285233
R_PLATFORM=x86_64-pc-linux-gnu
R_LIBS_USER=~/R/x86_64-pc-linux-gnu-library/3.5
R_DOC_DIR=/opt/R/3.5.0/lib64/R/doc
R_SESSION_TMPDIR=/tmp/RtmpN1E9jm
RSTUDIO_CONSOLE_COLOR=256
_=/usr/bin/env
bash-4.2$ which ssh-keygen
/usr/bin/ssh-keygen
bash-4.2$ ls -lha /usr/bin/ssh-keygen
-rwxr-xr-x. 1 root root 410K Nov 24 2017 /usr/bin/ssh-keygen
bash-4.2$ ssh-keygen --help
unknown option -- -
usage: ssh-keygen [-q] [-b bits] [-t dsa | ecdsa | ed25519 | rsa | rsa1]
[-N new_passphrase] [-C comment] [-f output_keyfile]
ssh-keygen -p [-P old_passphrase] [-N new_passphrase] [-f keyfile]
ssh-keygen -i [-m key_format] [-f input_keyfile]
ssh-keygen -e [-m key_format] [-f input_keyfile]
ssh-keygen -y [-f input_keyfile]
ssh-keygen -c [-P passphrase] [-C comment] [-f keyfile]
ssh-keygen -l [-v] [-E fingerprint_hash] [-f input_keyfile]
ssh-keygen -B [-f input_keyfile]
ssh-keygen -D pkcs11
ssh-keygen -F hostname [-f known_hosts_file] [-l]
ssh-keygen -H [-f known_hosts_file]
ssh-keygen -R hostname [-f known_hosts_file]
ssh-keygen -r hostname [-f input_keyfile] [-g]
ssh-keygen -G output_file [-v] [-b bits] [-M memory] [-S start_point]
ssh-keygen -T output_file -f input_file [-v] [-a rounds] [-J num_lines]
[-j start_line] [-K checkpt] [-W generator]
ssh-keygen -s ca_key -I certificate_identity [-h] [-n principals]
[-O option] [-V validity_interval] [-z serial_number] file ...
ssh-keygen -L [-f input_keyfile]
ssh-keygen -A
ssh-keygen -k -f krl_file [-u] [-s ca_public] [-z version_number]
file ...
ssh-keygen -Q -f krl_file file ...
And from a bash shell via ssh...
bash-4.2$ id
uid=285233(mah60) gid=1000(access) groups=1000(access),188(staff.up.cis), <REMOVED LONG LIST OF LDAP GROUPS> context=unconfined_u:unconfined_r:unconfined_t:s0-s0:c0.c1023
bash-4.2$ env
XDG_SESSION_ID=731
HOSTNAME=<REMOVED HOSTNAME>
SELINUX_ROLE_REQUESTED=
SHELL=/bin/bash
TERM=xterm-256color
HISTSIZE=1000
SSH_CLIENT=66.71.76.36 51489 1855
CONDA_SHLVL=0
SELINUX_USE_CURRENT_RANGE=
OLDPWD=/home/mah60admin
SSH_TTY=/dev/pts/4
USER=mah60
LS_COLORS=rs=0:di=38;5;27:ln=38;5;51:mh=44;38;5;15:pi=40;38;5;11:so=38;5;13:do=38;5;5:bd=48;5;232;38;5;11:cd=48;5;232;38;5;3:or=48;5;232;38;5;9:mi=05;48;5;232;38;5;15:su=48;5;196;38;5;15:sg=48;5;11;38;5;16:ca=48;5;196;38;5;226:tw=48;5;10;38;5;16:ow=48;5;10;38;5;21:st=48;5;21;38;5;15:ex=38;5;34:*.tar=38;5;9:*.tgz=38;5;9:*.arc=38;5;9:*.arj=38;5;9:*.taz=38;5;9:*.lha=38;5;9:*.lz4=38;5;9:*.lzh=38;5;9:*.lzma=38;5;9:*.tlz=38;5;9:*.txz=38;5;9:*.tzo=38;5;9:*.t7z=38;5;9:*.zip=38;5;9:*.z=38;5;9:*.Z=38;5;9:*.dz=38;5;9:*.gz=38;5;9:*.lrz=38;5;9:*.lz=38;5;9:*.lzo=38;5;9:*.xz=38;5;9:*.bz2=38;5;9:*.bz=38;5;9:*.tbz=38;5;9:*.tbz2=38;5;9:*.tz=38;5;9:*.deb=38;5;9:*.rpm=38;5;9:*.jar=38;5;9:*.war=38;5;9:*.ear=38;5;9:*.sar=38;5;9:*.rar=38;5;9:*.alz=38;5;9:*.ace=38;5;9:*.zoo=38;5;9:*.cpio=38;5;9:*.7z=38;5;9:*.rz=38;5;9:*.cab=38;5;9:*.jpg=38;5;13:*.jpeg=38;5;13:*.gif=38;5;13:*.bmp=38;5;13:*.pbm=38;5;13:*.pgm=38;5;13:*.ppm=38;5;13:*.tga=38;5;13:*.xbm=38;5;13:*.xpm=38;5;13:*.tif=38;5;13:*.tiff=38;5;13:*.png=38;5;13:*.svg=38;5;13:*.svgz=38;5;13:*.mng=38;5;13:*.pcx=38;5;13:*.mov=38;5;13:*.mpg=38;5;13:*.mpeg=38;5;13:*.m2v=38;5;13:*.mkv=38;5;13:*.webm=38;5;13:*.ogm=38;5;13:*.mp4=38;5;13:*.m4v=38;5;13:*.mp4v=38;5;13:*.vob=38;5;13:*.qt=38;5;13:*.nuv=38;5;13:*.wmv=38;5;13:*.asf=38;5;13:*.rm=38;5;13:*.rmvb=38;5;13:*.flc=38;5;13:*.avi=38;5;13:*.fli=38;5;13:*.flv=38;5;13:*.gl=38;5;13:*.dl=38;5;13:*.xcf=38;5;13:*.xwd=38;5;13:*.yuv=38;5;13:*.cgm=38;5;13:*.emf=38;5;13:*.axv=38;5;13:*.anx=38;5;13:*.ogv=38;5;13:*.ogx=38;5;13:*.aac=38;5;45:*.au=38;5;45:*.flac=38;5;45:*.mid=38;5;45:*.midi=38;5;45:*.mka=38;5;45:*.mp3=38;5;45:*.mpc=38;5;45:*.ogg=38;5;45:*.ra=38;5;45:*.wav=38;5;45:*.axa=38;5;45:*.oga=38;5;45:*.spx=38;5;45:*.xspf=38;5;45:
COBBLER_SERVER=redhat.psu.edu
PATH=/usr/local/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/home/mah60admin/.local/bin:/home/mah60admin/bin
MAIL=/var/spool/mail/mah60admin
PWD=/pass/users/m/a/mah60
LANG=en_US.UTF-8
SELINUX_LEVEL_REQUESTED=
KRB5CCNAME=FILE:/tmp/krb5cc_285233_Or4A2e
HISTCONTROL=ignoredups
HOME=/pass/users/m/a/mah60
SHLVL=2
LOGNAME=mah60
SSH_CONNECTION=66.71.76.36 51489 128.118.155.47 1855
LESSOPEN=||/usr/bin/lesspipe.sh %s
XDG_RUNTIME_DIR=/run/user/285233
_=/usr/bin/env
bash-4.2$ which ssh-keygen
/usr/bin/ssh-keygen
bash-4.2$ ls -lha /usr/bin/ssh-keygen
-rwxr-xr-x. 1 root root 410K Nov 24 2017 /usr/bin/ssh-keygen
bash-4.2$ ssh-keygen --help
unknown option -- -
usage: ssh-keygen [-q] [-b bits] [-t dsa | ecdsa | ed25519 | rsa | rsa1]
[-N new_passphrase] [-C comment] [-f output_keyfile]
ssh-keygen -p [-P old_passphrase] [-N new_passphrase] [-f keyfile]
ssh-keygen -i [-m key_format] [-f input_keyfile]
ssh-keygen -e [-m key_format] [-f input_keyfile]
ssh-keygen -y [-f input_keyfile]
ssh-keygen -c [-P passphrase] [-C comment] [-f keyfile]
ssh-keygen -l [-v] [-E fingerprint_hash] [-f input_keyfile]
ssh-keygen -B [-f input_keyfile]
ssh-keygen -D pkcs11
ssh-keygen -F hostname [-f known_hosts_file] [-l]
ssh-keygen -H [-f known_hosts_file]
ssh-keygen -R hostname [-f known_hosts_file]
ssh-keygen -r hostname [-f input_keyfile] [-g]
ssh-keygen -G output_file [-v] [-b bits] [-M memory] [-S start_point]
ssh-keygen -T output_file -f input_file [-v] [-a rounds] [-J num_lines]
[-j start_line] [-K checkpt] [-W generator]
ssh-keygen -s ca_key -I certificate_identity [-h] [-n principals]
[-O option] [-V validity_interval] [-z serial_number] file ...
ssh-keygen -L [-f input_keyfile]
ssh-keygen -A
ssh-keygen -k -f krl_file [-u] [-s ca_public] [-z version_number]
file ...
ssh-keygen -Q -f krl_file file ...
Very interesting! Thanks for the command output! There are definitely some differences to PATH and probably other environment variables, which suggests some differences in startup. It's also interesting to see identical output from id, which and ssh-keygen, which means that you can execute the executable in both contexts. As a result, my suggestion would be to focus on the options you are using with ssh-keygen and what ssh-keygen is trying to do (i.e. are defaults different and is it trying to write to a different place from bash / terminal? You could be getting "Permission denied" from trying to write to a place that you do not have permission to do so.
I.e. you could try ssh-keygen without parameters and see what happens? And then slowly add parameters and see what breaks things? If you can share the full output of a failing command, that might be helpful as well (not sure if there is a way to increase verbosity or gather more SSH client logs). If it fails trying to create a certain file, do a ls -lha /path/to/file to see what permissions are on that file.
Thanks for the reply. I think I've tried most of these same steps before, but ssh-keygen seems to fail regardless of what parameters are used when run within RStudio.
bash-4.2$ ls -lha ~/.ssh/
total 16K
drwx--x--x. 2 mah60 mah60 512 Oct 1 16:58 .
drwx--x--x. 45 root mah60 8.0K Oct 15 14:06 ..
bash-4.2$ ssh-keygen
Generating public/private rsa key pair.
Enter file in which to save the key (/pass/users/m/a/mah60/.ssh/id_rsa):
Could not stat /pass/users/m/a/mah60/.ssh: Permission denied
Enter passphrase (empty for no passphrase):
Enter same passphrase again:
Saving key "/pass/users/m/a/mah60/.ssh/id_rsa" failed: Permission denied
bash-4.2$ stat /pass/users/m/a/mah60/.ssh
File: ‘/pass/users/m/a/mah60/.ssh’
Size: 512 Blocks: 0 IO Block: 32768 directory
Device: 28h/40d Inode: 1265085 Links: 2
Access: (0711/drwx--x--x) Uid: (285233/ mah60) Gid: (1252753/ mah60)
Context: system_u:object_r:nfs_t:s0
Access: 2018-10-16 10:25:19.351429642 -0400
Modify: 2018-10-01 16:58:59.727977234 -0400
Change: 2018-10-01 16:58:59.726292521 -0400
Birth: -
Wait, is this saying that root owns your home directory? The permissions would suggest that you do not have write access to your own home directory? Can you cd /pass/users/m/a/mah60/.ssh/ and touch somefile or write a text file or something?
There is definitely some weirdness going on here 
Eeverything else works as expected, touch, mkdir, and installing R packages, etc..
bash-4.2$ touch .ssh/test
bash-4.2$ ls -alh .ssh/test
-rw-------. 1 mah60 mah60 0 Oct 16 10:37 .ssh/test
bash-4.2$ rm .ssh/test
bash-4.2$ ls -alh .ssh/test
ls: cannot access .ssh/test: No such file or directory
In fact, I can run ssh-keygen to create the ssh key in /tmp or anywhere outside of my NFS home where I have permissions to write. Obviously writing to /tmp wouldn't be a very secure thing to recommend for our users. Normally I'd chalk this up to permission quirks on our NFS space except for the fact that the same commands work when run outside of the RStudio environment.
As a workaround, I created a quick Python script to generate the key pair. Sharing for posterity...
#!/usr/bin/env python
# Matt Hansen - mah60@psu.edu - (16 Oct 2018)
import os
import sys
import paramiko
# Generate
key = paramiko.RSAKey.generate(2048)
# Private Key
key.write_private_key_file(os.path.expanduser('~/.ssh/id_rsa'))
# Public Key
pubkey = "ssh-rsa %s %s@%s\n" % (key.get_base64(), os.environ["USER"], os.environ["HOSTNAME"])
print(pubkey)
with open(os.path.expanduser('~/.ssh/id_rsa.pub'), 'w') as write_public_key_file:
write_public_key_file.write(pubkey)
1 Like
We're also running JupyterHub with a very similar setup and it appears Jupyter, within a bash shell or R notebook, has the same issue running ssh-keygen. Could it be related to the way these processes are demonized and spawning user processes? I checked the SELinux logs and didn't find anything there.
Yeah, that's a great thought. I know AppArmor is another thing that is in play on Ubuntu machines, but I know little about how it works. We discuss it in a support article here: https://support.rstudio.com/hc/en-us/articles/200717193-RStudio-Server-Will-Not-Start
In my opinion, we have entered the realm of weird linux wizardry that I do not understand. One final check would be to execute R in a normal bash shell and see if you can execute ssh-keygen there. If it works, it would very nicely point to the problem being something related to daemonized processes and not necessarily a problem with R.
Sure, good idea!
Executing R from RStudio terminal
> system("ssh-keygen -f id_rsa -t rsa -N ''")
Generating public/private rsa key pair.
Saving key "id_rsa" failed: Permission denied
Executing the same R binary from SSH
> system("ssh-keygen -f id_rsa -t rsa -N ''")
Generating public/private rsa key pair.
Your identification has been saved in id_rsa.
Your public key has been saved in id_rsa.pub.
The key fingerprint is:
...
1 Like
Interesting!! So still no idea what it is, but at least it's not R. It's some type of linux daemonization security shenanigans that is denying access to whatever ssh-keygen is using / doing.
I say that's some pretty impressive detective-work! Well done!