I have lost rstudio server while updating it on ubuntu 16: URGENT HELP needed !

RStudio IDE
1

For the last several hours my rstudio server has been pulled down and the new downloaded version fails to be installed. Numerous errors are pouring in as I try to install the latest rstudio server on my debian 9 amd64 machine on AWS EC2.

I have downloaded the debian 9 version of rstudio described here:

using the following commands:

sudo apt-get install gdebi-core
wget https://download2.rstudio.org/server/debian9/x86_64/rstudio-server-2022.02.0-443-amd64.deb
sudo gdebi rstudio-server-2022.02.0-443-amd64.deb

Current status :

$ sudo rstudio-server verify-installation
/usr/lib/rstudio-server/bin/rserver: error while loading shared libraries: libssl.so.1.0.2: cannot open shared object file: No such file or directory
rstudio@ip-172-31-30-19:~$

:expressionless:

And on entering the url for the rstudio server (hosted on Ec2 instance of AWS) I get the following error on the browser

502 Bad Gateway
nginx/1.10.3 (Ubuntu)

This is expected as the origin server is down.

I see numerous posts across rstudio community support, r-bloggers.com as well as github about this libssl.so.1.1 / 1.0 / 1.2 and it is really confusing to understand as the information is not consisent.
I saw here that there is a downgrade required. This is further confusing as I am already on an old debian 9 (stretch) / ubuntu 16.04/xenial so not sure what downgrade is needed.

One observation: There is an error inside rstudo server logs which has repeated hundreds of times in last one week:

ERROR system error 40 (Too many levels of symbolic links) [path: /home/rstudio/omni/iot]; OCCURRED AT bool rstudio::core::FilePath::exists() cons

I am not sure if the above error is linked to my rstudio server failing to install, so please ignore if this information is irrelevant. I have also removed the symlink file that may be causing this.

I have read and tried suggestions in follwing two similar issues.

and

I used this advice Rstudio server installion - #5 by andresrcs to stop, purge and reinstall. So my old rstudio is purged now.

Here are the verbatim rstudio installation shell command & the responses I got.

rstudio@ip-172-31-30-19:~$ sudo gdebi rstudio-server-2022.02.0-443-amd64.deb
Reading package lists... Done
Building dependency tree        
Reading state information... Done
Reading state information... Done

RStudio Server
 RStudio is a set of integrated tools designed to help you be more productive with R. It includes a console, syntax-highlighting editor that supports direct code execution, as well as tools for plotting, history, and workspace management.
Do you want to install the software package? [y/N]:y
Selecting previously unselected package rstudio-server.
(Reading database ... 237801 files and directories currently installed.)
Preparing to unpack rstudio-server-2022.02.0-443-amd64.deb ...
Unpacking rstudio-server (2022.02.0+443) ...
Setting up rstudio-server (2022.02.0+443) ...
useradd: user 'rstudio-server' already exists
Created symlink from /etc/systemd/system/multi-user.target.wants/rstudio-server.service to /lib/systemd/system/rstudio-server.service.
/usr/lib/rstudio-server/bin/rserver: error while loading shared libraries: libssl.so.1.0.2: cannot open shared object file: No such file or directory
● rstudio-server.service - RStudio Server
   Loaded: loaded (/lib/systemd/system/rstudio-server.service; enabled; vendor preset: enabled)
   Active: inactive (dead) since Sun 2022-03-06 23:57:39 IST; 44min ago
 Main PID: 1217
   CGroup: /system.slice/rstudio-server.service
           ├─2293 python3 ../subscriber3.py dand.ini
           ├─2294 python3 ../subscriber3.py bagru1.ini
           ├─2295 python3 ../subscriber3.py bagru2.ini
           ├─2296 python3 ../subscriber3.py jobner.ini
           └─2297 python3 ../subscriber3.py sawarda.ini

Mar 06 23:57:05 ip-172-31-30-19 sudo[7622]:  rstudio : TTY=pts/0 ; PWD=/home/rstudio ; USER=root ; COMMAND=/usr/bin/gdebi rstudio-server-2022.02.0-443-amd64.deb
Mar 06 23:57:05 ip-172-31-30-19 sudo[7622]: pam_unix(sudo:session): session opened for user root by (uid=0)
Mar 06 23:57:39 ip-172-31-30-19 systemd[1]: Stopping RStudio Server...
Mar 06 23:57:39 ip-172-31-30-19 systemd[1]: Stopped RStudio Server.
Mar 06 23:57:40 ip-172-31-30-19 rsession-rstudio[31976]: ERROR r error 4 (R code execution error) [errormsg: Error in base::try(ts2, TRUE) : object 'ts2' not found
                                                         ]; OCCURRED AT rstudio::core::Error rstudio::r::exec::executeSafely(rstudio_boost::function<void()>) src/cpp/r/REx...
Mar 06 23:57:41 ip-172-31-30-19 sudo[7622]: pam_unix(sudo:session): session closed for user root
Mar 06 23:57:44 ip-172-31-30-19 rsession-rstudio[31976]: ERROR system error 9 (Bad file descriptor); OCCURRED AT virtual rstudio::core::Error rstudio::core::system::ChildP...
Mar 07 00:40:35 ip-172-31-30-19 systemd[1]: Stopped RStudio Server.
Mar 07 00:41:19 ip-172-31-30-19 systemd[1]: Stopped RStudio Server.
Mar 07 00:41:37 ip-172-31-30-19 systemd[1]: Stopped RStudio Server.
Hint: Some lines were ellipsized, use -l to show in full.

Since the Ubuntu server has production R shiny code, I am extremely nervous that this mishap has happened.

It is past 2.30 AM here as I post this question before going to bed hoping I really get some solid suggesdtions as soon as I am up.

Lucklily the R shiny app is still working and even the python scripts that collect data every minute are still running. Note : In the above messages (2293 to 2297 python3...) are referred. Those scripts are still running (they were fired using rstudio terminal and they are still alive, while rstudio is dead). Is there a chance those scripts are interfering . I cannot kill them, as they are critical for my production, as they collect data.

2

Run openssl version in your server to check what version you have installed, if it doesn't match the required version (1.0.2), then this is the downgrade you need to make.

Try with

sudo apt install libssl1.0.2

If that version is not available for you, look for an older package repository and add it to your source.list file.

Although, If you want to update your software, I think you should consider upgrading your whole setup. Also, I would strongly recommend implementing a more robust production setup, at the very list you should have some sort of automated restore procedure and depending on how critical your application is, you could consider implementing high availability with a cluster and an orchestrator like Kubernetes.

Note: I noticed you talk about both Debian9 and Ubuntu 16 as being interchangeable but that is not strictly the case, Debian 9 is a current LTS version whereas Ubuntu 16 is no longer supported so there is no guarantee the latest RStudio version is going to work on it properly.

3

Thank you for the quick suggestions @andresrcs :raised_hands:

Here are the results.

OpenSSL 1.0.2g 1 Mar 2016

So 1.0.2g is already there.
:face_with_raised_eyebrow:
I tried installing 1.0.2 but unsuccessful. Hope this is not needed now.

rstudio@ip-172-31-30-19:/home/ubuntu$ sudo apt install libssl1.0.2
Reading package lists... Done
Building dependency tree       
Reading state information... Done
Package libssl1.0.2 is not available, but is referred to by another package.
This may mean that the package is missing, has been obsoleted, or
is only available from another source

E: Package 'libssl1.0.2' has no installation candidate
rstudio@ip-172-31-30-19:/home/ubuntu$

I will soon be working to make the set up more robust as you suggested using clustering and kubernetics type container and orchestrator a little later.

I can start with Ubuntu 16.04 to 18 update now if it is absolutely essential. SInce it will cause an outage, I have two ways:

  1. Set up another EC2 server with Ubuntu 18 or 20, install R and R studio and python3 and move all applications and data files on to it - one by one. Once the new server starts working I can point the elastic IP address to the new server.
  2. Upgrade the existing server to Ubuntu 18 and then reinbstall rstudio.

I would be looking for your advice on a more reliable way. Both have risks.

  • Option 1 main risk: Moving all the environment of ETL python & R scripts (that are fired every 1 minutes) and data files ( a large number of RDS files and csv files) from old server to new may take several days. I donot use any RDBMS.

  • Option 2 main risk: Updating process to Ubuntu 18 or 20 on a live server will immediately stop the R shiny and all ETL scripts for a few hours . The outage hours (or minutes) will depend on how fast I can get the R Shiny up and ETL scripts up. I can afford an outage of a couple of hours during slack time. So if this has higher success chance than option 1 I can afford the outage.

Is there any other faster option to switch on Rstudio on the existing Ubuntu 16 and then we do a more careful upgrade of other setup. Actually without Rstudio working it will be like one hand tied to my back while upgrading a working Shiny server and moving files around.

Many thanks in advance and looking forward for next steps.

Sanjay

4

It seems to me you are actually using Ubuntu 16 and not Debian 9, as I said Ubuntu 16 is no longer a supported version so very likely RStudio has not taken into account the libssl variant it has available. I think you could just reinstall your old RStudio version to have at least a working instance while you setup your new server on a different EC2 instance, although, it's not completely necessary, RStudio Server is independent from all the other mentioned services so they are going to keep operating normally as long as you do not introduce any other software changes, and if you need RStudio in a hurry, you could simply use it from your new instance.

About your upgrade options, I don't recommend using your second option, a hot-upgrade is very likely to end up on an unstable system, they even officially recommend against it.

Installing a new system in parallel is a better plan and while general system configuration has not a time constraint, migrating your workloads and applications does. I personally use Ansible, an application-deployment tool enabling infrastructure as code, to automate my recovery process (my application requires very low latency so a local Kubernetes cluster would be too expensive), which makes the whole thing faster and reproducible, but in your case this might be more complicated since you are using file based storage (RDS files instead of a much more suitable and robust RDBMS solution) and it seems like you are not using a version control server like GitHub for your applications so migration to a new server can be a little harder and slower.

If I was you, I would invest some time streamlining my environment and making it more robust before migration. Key aspects to consider are using a proper database, using GitHub or any other version control repository for your apps and automating your recovery process.

5

I am with you. Hot upgrade is ruled out..Was a bad option.
I have already spun a new server. This time it is on Digital Ocean as it seemed much easier with existing Rstudio and Shiny R and addresses a lot of java dependencies already built in the image (as I read). Also the image is under active maintenance unlike RStudio image (AMI) for AWS which is not updated regularly. It is too early for me to conclude that Digital ocean looks easier than AWS for R but I will soon give my verdict, as many people must be struggling for this decision.

I am indeed using github so am able to clone the code base quickly. Its just the data files that would be taking time (no I donot commit data files). But that is my headache I will have to handle.

I have still to take a decision on moving to anRDBMS or to a SparkR version of big data management. At the moment the RDS based daily files is serving the purpose well.

Your suggested Ansible based dev ops was in my pipeline for learning but got shelved as it was taking some time when I last started. Automating the Recovery process is a key next step before even moving to RDBMS / SparkR.

The immediate challenge is to get up and running all the R + python ETL scripts (API & MQTT) + Shiny App on the new digital ocean server with preinstalled latest Rstudio and R Shiny on the latest Ubuntu 20.

Thankyou so much for all the advice. :blush:

Sanjay

6

I'm personally against the use of a pre-made image for a "production" environment because they usually include very opinionated configuration options and in some cases it can be more time consuming and prone to failure to reconfigure to your liking rather than setting the configuration right from the beginning. I think using "infrastructure as code" is a much better approach and more efficient in the long run too.

7

Ah yes, this has been your recommendation earlier also.
Since I have already started with moving to Digital Ocean - code is moved and data is under movement, I will see how it goes. If problems start cropping up, will move to the ansible way of deployment on server. However I see a sharp learning curve to grasp ansible as I quickly browsed your blog and also this blog specific to Digital Ocean.

Note: I guess my main problems would be handling hard coded paths inside my R shiny code and R processing scripts. Also permisions of files could be another pitfall. Third could be the data volume mounted on AWS using a cold HDD, and the path needs to be retained in DO. None of these 3 may be addressable by ansible I guess.

8

You do not have to use Ansible specifically, I'm referring to "infrastructure as code" as a general concept, there are other software tools that implement it.

You would be surprised of how flexible Ansible is, I'm not sure I fully understand your issues but they sound relatively easy to tackle with Ansible and a little Jinja2 scripting.

1 Like
9

Screenshot 2022-03-15 at 1.11.35 PM
Screenshot 2022-03-15 at 1.11.35 PM2380×1268 281 KB

So, Ubuntu 16.04 and above are supported for Rstudio.
I just wonder is there a simple way to restart my rstudio on the ubuntu16 machine, instead of upgrading everything. Note: I already have spun a new machine with Ubuntu 20 and moved all my code on it (yes Ansible would have been better but I have yet to learn it).
At the moment I would like my old Ubuntu 16 server where R and R shiny are working, to also work, so restoring rstudio on it is still required. I will use it as a backup server.

10

I am working on your advice. But I cannot find my old Rstudio server installation binary.
This is the only place it seems Rstudio allows downloads from. My Ubuntu 16.04 does not appear anywhere.

11

I think this is just a documentation error or is not updated, it would make little sense to support RStudio for Ubuntu 16 when not even Ubuntu supports Ubuntu 16 anymore.

I simply googled "download old RStudio version" and I got this

There you can find RStudio 1.4.1106 which explicitly says, it is compatible with Ubuntu 16.04

12

Almost there.
rstudio old version installed and verified. But when I access it from browser :

IP address:8787

it is inaccessible

  1. Checked port 8787, it is opened from all IP addresses
  2. Checked server status and saw it is dead since a week,
rstudio@ip-172-31-30-19:~$ systemctl status rstudio-server
● rstudio-server.service - RStudio Server
   Loaded: loaded (/lib/systemd/system/rstudio-server.service; enabled; vendor preset: enabled)
   Active: inactive (dead) since Wed 2022-03-09 01:39:32 IST; 6 days ago
rstudio@ip-172-31-30-19:~$ systemctl start rstudio-server
  1. Tried restarting with following results.
rstudio@ip-172-31-30-19:~$ systemctl start rstudio-server
==== AUTHENTICATING FOR org.freedesktop.systemd1.manage-units ===
Authentication is required to start 'rstudio-server.service'.
Authenticating as: Ubuntu (ubuntu)
Password: 
==== AUTHENTICATION COMPLETE ===
Job for rstudio-server.service failed because the control process exited with error code. See "systemctl status rstudio-server.service" and "journalctl -xe" for details.
  1. Checked Journal logs.. Here are first few lines.
rstudio@ip-172-31-30-19:~$ sudo journalctl -u rstudio-server
-- Logs begin at Thu 2022-03-10 22:44:06 IST, end at Tue 2022-03-15 20:50:49 IST. --
Mar 15 13:39:50 ip-172-31-30-19 systemd[1]: Stopped RStudio Server.
Mar 15 20:29:25 ip-172-31-30-19 systemd[1]: Starting RStudio Server...
Mar 15 20:29:25 ip-172-31-30-19 rserver[21776]: /usr/lib/rstudio-server/bin/rserver: error while loading shared libraries: libssl.so.1.0.2: cannot open shared 
Mar 15 20:29:25 ip-172-31-30-19 systemd[1]: rstudio-server.service: Control process exited, code=exited status=127
Mar 15 20:29:25 ip-172-31-30-19 systemd[1]: Failed to start RStudio Server.
Mar 15 20:29:25 ip-172-31-30-19 systemd[1]: rstudio-server.service: Unit entered failed state.
Mar 15 20:29:25 ip-172-31-30-19 systemd[1]: rstudio-server.service: Failed with result 'exit-code'.
Mar 15 20:29:25 ip-172-31-30-19 systemd[1]: rstudio-server.service: Service hold-off time over, scheduling restart.
Mar 15 20:29:25 ip-172-31-30-19 systemd[1]: Stopped RStudio Server.
Mar 15 20:29:25 ip-172-31-30-19 systemd[1]: Starting RStudio Server...
Mar 15 20:29:25 ip-172-31-30-19 rserver[21779]: /usr/lib/rstudio-server/bin/rserver: error while loading shared libraries: libssl.so.1.0.2: cannot open shared 
Mar 15 20:29:25 ip-172-31-30-19 systemd[1]: rstudio-server.service: Control process exited, code=exited status=127
Mar 15 20:29:25 ip-172-31-30-19 systemd[1]: Failed to start RStudio Server.
Mar 15 20:29:25 ip-172-31-30-19 systemd[1]: rstudio-server.service: Unit entered failed state.
Mar 15 20:29:25 ip-172-31-30-19 systemd[1]: rstudio-server.service: Failed with result 'exit-code'.
Mar 15 20:29:26 ip-172-31-30-19 systemd[1]: rstudio-server.service: Service hold-off time over, scheduling restart.
Mar 15 20:29:26 ip-172-31-30-19 systemd[1]: Stopped RStudio Server.
Mar 15 20:29:26 ip-172-31-30-19 systemd[1]: Starting RStudio Server...
Mar 15 20:29:26 ip-172-31-30-19 rserver[21784]: /usr/lib/rstudio-server/bin/rserver: error while loading shared libraries: libssl.so.1.0.2: cannot open shared 
Mar 15 20:29:26 ip-172-31-30-19 systemd[1]: rstudio-server.service: Control process exited, code=exited status=127
Mar 15 20:29:26 ip-172-31-30-19 systemd[1]: Failed to start RStudio Server.

Looks like it is leading to the same problem:

error while loading shared libraries: libssl.so.1.0.2:

Update: Just saw this post, and it seems to be similar problem. Since it is closed I cannot comment on it. But looks like the solution could be in the same thread but is not evident to me.

Clearly this libssl seems to be a big pain area in installing rstudio server on ubuntu.

13

Using unsupported legacy software is always a pain in general, not specific to Ubuntu or RStudio.

The last think I can think of is that the libssl 1.0.2g variant you have is more recent than the RStudio binary thus not taken into account for compatibility checks.

Look into finding an apt source that provides libssl 1.0.2 (without the g) for Ubuntu 16.04 so you can downgrade it.

14

Sorry for the delay. Will find the right libssl file without the g

15

This topic was automatically closed 21 days after the last reply. New replies are no longer allowed.

If you have a query related to it or one of the replies, start a new topic and refer back with a link.