sparklyr: JAVA_HOME does not point valid version.

Hey All,
I'm having a bit of trouble with sparklyr, I'm not the best with the path issues but I thought it was set up correctly having followed several steps on youtube to set up hadoop recently.
I am getting the following error

Error in spark_get_java(throws = TRUE) : 
  Java is required to connect to Spark. JAVA_HOME is set to 'C:\Java\jdk1.8.0_311\bin' but does not point to a valid version. Please fix JAVA_HOME or reinstall from:

when I run...


# Available for installation

# Install 
#spark_install(version = "3.1.1")

# Version installed on computer

  spark  hadoop                                                                 dir
1 3.1.1    2.7    C:\\Users\\August\\AppData\\Local/spark/spark-3.1.1-bin-hadoop2.7
2 3.1.1    3.2    C:\\Users\\August\\AppData\\Local/spark/spark-3.1.1-bin-hadoop3.2
3 3.1.2    3.2    C:\\Users\\AUgust\\AppData\\Local/spark/spark-3.1.2-bin-hadoop3.2

# Configuration Setup (Optional)
conf <- list()
conf$`sparklyr.cores.local`         <- 10
conf$`` <- "16G"
conf$spark.memory.fraction          <- 0.5

# Connects to Spark Locally
sc <- spark_connect(
    master  = "local", 
    version = "3.1.1", 
    config  = conf

My JAVA_HOME and HADOOP environment variables are set to...


Not sure if this should be an issue.
Any advice is greatly appreciated, not really sure what to do here at all.

Hi @August , are you running a Hadoop cluster virtually in your laptop? Or are you trying just to get sparklyr started with a local connection without connecting to a Hadoop cluster?

It was a pathing issue, with R unable to find the correct version of the development version of Java, and made sure it matched my operating system x 84. To resolve the pathing issues I reinstalled Java and followed these instructions for configuration. It also seems I made a mistake in one of the XML files as well.

You live and learn :man_shrugging:, at least there is always something to be learnt like how to use shell better.

This topic was automatically closed 7 days after the last reply. New replies are no longer allowed.

If you have a query related to it or one of the replies, start a new topic and refer back with a link.