I will be teaching an ML class with ~30 students at the UG level this winter. I am currently trying to set up the base project for my class' workspace in Posit cloud on an organization account so that all students have access to the required packages. I am running into issues while setting up the python-related environment/packages. I am planning to cover basic DNNs with keras, tensorflow, etc.
I thought that it could be a space issue (I believe I have 1 GB), so I tried adding pip_options = "--no-cache-dir" to the install_keras piece of code from https://forum.posit.co/t/tensorflow-installation-in-rstudio-cloud-does-not-work/111847 (I am also getting the same "Killed" and "Error installing package(s)" error messages), but the issue wasn't resolved.
I am having a hard time figuring this issue out. Any suggestions/references/fixes would be greatly appreciated.
There’s a report of a successful install a couple of years ago here. To get that version of Tensorflow required avoiding the karas install routine, which is said to install the current version.
An alternative is to host an RStudio server instance on one of your institution’s servers, which has the added advantage of using existing authentication. It provides a near identical experience to Desktop and initial setup is simple.
Is is because miniconda wasn't installed properly, or a separate issue related to the version of tensorflow? What am I missing? Any help would be much appreciated.
and these seem to work. I assume the other missing pips can be installed similarly, such as pandas.
I haven't tried it, and it may be challenging, is to create the right virtual environment and identify it to sessions in the R console, but I hope this gets you closer.
I tried running your code. The reticulate::install_miniconda("miniconda") was working fine, but at the end got 'Killed' with an error code of 137. I was able to install all the packages using py_install, except "tensorflow==2.9.3".
When I tried to run a neural network with the above installation, I received the following error. I could not solve it using the install_tensorflow() function.
I also noticed that the miniconda folder had python3.8, whereas the virtualenvs folder had python3.9. I am wondering if the discrepancy creates an issue, but I am not sure how to specify the version of python.
What seems to be working fine for me is the following from this post.
I got errors when attempting to run with this setup:
2022-12-22 14:55:29.143731: I tensorflow/core/platform/cpu_feature_guard.cc:193] This TensorFlow binary is optimized with oneAPI Deep Neural Network Library (oneDNN) to use the following CPU instructions in performance-critical operations: AVX2 AVX512F FMA
To enable them in other operations, rebuild TensorFlow with the appropriate compiler flags.
2022-12-22 14:55:36.010353: W tensorflow/compiler/xla/stream_executor/platform/default/dso_loader.cc:64] Could not load dynamic library 'libcudart.so.11.0'; dlerror: libcudart.so.11.0: cannot open shared object file: No such file or directory; LD_LIBRARY_PATH: /opt/R/4.2.2/lib/R/lib:/usr/local/lib:/usr/lib/x86_64-linux-gnu:/usr/lib/jvm/java-11-openjdk-amd64/lib/server:/opt/R/4.2.2/lib/R/lib:/lib:/usr/local/lib:/usr/lib/x86_64-linux-gnu:/usr/lib/jvm/java-11-openjdk-amd64/lib/server
2022-12-22 14:55:36.010371: I tensorflow/compiler/xla/stream_executor/cuda/cudart_stub.cc:29] Ignore above cudart dlerror if you do not have a GPU set up on your machine.
2022-12-22 14:56:00.187446: W tensorflow/compiler/xla/stream_executor/platform/default/dso_loader.cc:64] Could not load dynamic library 'libnvinfer.so.7'; dlerror: libnvinfer.so.7: cannot open shared object file: No such file or directory; LD_LIBRARY_PATH: /opt/R/4.2.2/lib/R/lib:/usr/local/lib:/usr/lib/x86_64-linux-gnu:/usr/lib/jvm/java-11-openjdk-amd64/lib/server:/opt/R/4.2.2/lib/R/lib:/lib:/usr/local/lib:/usr/lib/x86_64-linux-gnu:/usr/lib/jvm/java-11-openjdk-amd64/lib/server
2022-12-22 14:56:00.187760: W tensorflow/compiler/xla/stream_executor/platform/default/dso_loader.cc:64] Could not load dynamic library 'libnvinfer_plugin.so.7'; dlerror: libnvinfer_plugin.so.7: cannot open shared object file: No such file or directory; LD_LIBRARY_PATH: /opt/R/4.2.2/lib/R/lib:/usr/local/lib:/usr/lib/x86_64-linux-gnu:/usr/lib/jvm/java-11-openjdk-amd64/lib/server:/opt/R/4.2.2/lib/R/lib:/lib:/usr/local/lib:/usr/lib/x86_64-linux-gnu:/usr/lib/jvm/java-11-openjdk-amd64/lib/server
2022-12-22 14:56:00.187773: W tensorflow/compiler/tf2tensorrt/utils/py_utils.cc:38] TF-TRT Warning: Cannot dlopen some TensorRT libraries. If you would like to use Nvidia GPU with TensorRT, please make sure the missing libraries mentioned above are installed properly.
Thanks for pointing this out, @jkylearmstrong . I don't think that these are errors, they are probably warning (W) and information (I) messages. They are related to GPU installations, which I was not bothered about.
My neuralnet code worked despite these messages. Also, as per the instructions here, I checked whether the installation succeeded with the following chunk of code.
This will provide you with a default installation of TensorFlow suitable for use with the tensorflow R package. Read on if you want to learn about additional installation options, including installing a version of TensorFlow that takes advantage of Nvidia GPUs if you have the correct CUDA libraries installed.
The message you mentioned are probably related to this. Hope this helps.
2022-12-22 22:14:46.706060: I tensorflow/core/platform/cpu_feature_guard.cc:193] This TensorFlow binary is optimized with oneAPI Deep Neural Network Library (oneDNN) to use the following CPU instructions in performance-critical operations: AVX2 AVX512F FMA
To enable them in other operations, rebuild TensorFlow with the appropriate compiler flags.
2022-12-22 23:35:37.758246: I tensorflow/core/platform/cpu_feature_guard.cc:193] This TensorFlow binary is optimized with oneAPI Deep Neural Network Library (oneDNN) to use the following CPU instructions in performance-critical operations: AVX2 AVX512F FMA
To enable them in other operations, rebuild TensorFlow with the appropriate compiler flags.
It seems like an informational message rather than an error. And Posit Cloud crashing seems to be an issue with the cloud system/environment rather than your installation or code. Are you working on a free account?
I am using an institutional account. Even that doesn't allow me to work with such a huge dataset and sufficiently sized network structure as you have. I have also had this issue with the Cloud crashing and quitting.
I am planning to consider the first 100 or 1000 observations of this dataset, with a small-ish network to get the idea across to my students. I have also seen some texts recommend not to run neural nets on any cloud system.
If it helps you can try to access the datasets from the dslabs package and use it accordingly. I am at least able to access the data using the following.