Sparklyr: Cannot connect to Spark, gateway will not respond

I'm trying to get started with Sparklyr, but I'm hitting barriers with connecting to Spark in RStudio.

For reference, I'm using Spark 3.1.1 with R v.4.0.5 and Java 11. I've tried a couple of different versions of Spark to no avail.

The issue - I think - stems from the call to spark_connect() resulting in the following error:

> options(sparklyr.log.console = TRUE)
> config$sparklyr.gateway.port <- 8890
> sc <- spark_connect(master = "local")
* Using Spark: 3.1.1
Exception in thread "main" java.lang.ExceptionInInitializerError
	at org.apache.spark.unsafe.array.ByteArrayMethods.<clinit>(
	at org.apache.spark.internal.config.package$.<init>(package.scala:1095)
	at org.apache.spark.internal.config.package$.<clinit>(package.scala)
	at org.apache.spark.deploy.SparkSubmitArguments.$anonfun$loadEnvironmentArguments$3(SparkSubmitArguments.scala:157)
	at scala.Option.orElse(Option.scala:447)
	at org.apache.spark.deploy.SparkSubmitArguments.loadEnvironmentArguments(SparkSubmitArguments.scala:157)
	at org.apache.spark.deploy.SparkSubmitArguments.<init>(SparkSubmitArguments.scala:115)
	at org.apache.spark.deploy.SparkSubmit$$anon$2$$anon$3.<init>(SparkSubmit.scala:1013)
	at org.apache.spark.deploy.SparkSubmit$$anon$2.parseArguments(SparkSubmit.scala:1013)
	at org.apache.spark.deploy.SparkSubmit.doSubmit(SparkSubmit.scala:85)
	at org.apache.spark.deploy.SparkSubmit$$anon$2.doSubmit(SparkSubmit.scala:1030)
	at org.apache.spark.deploy.SparkSubmit$.main(SparkSubmit.scala:1039)
	at org.apache.spark.deploy.SparkSubmit.main(SparkSubmit.scala)
Caused by: java.lang.reflect.InaccessibleObjectException: Unable to make private java.nio.DirectByteBuffer(long,int) accessible: module java.base does not "opens java.nio" to unnamed module @40f9161a
	at java.base/java.lang.reflect.AccessibleObject.checkCanSetAccessible(
	at java.base/java.lang.reflect.AccessibleObject.checkCanSetAccessible(
	at java.base/java.lang.reflect.Constructor.checkCanSetAccessible(
	at java.base/java.lang.reflect.Constructor.setAccessible(
	at org.apache.spark.unsafe.Platform.<clinit>(
	... 13 more
Error in spark_connect_gateway(gatewayAddress, gatewayPort, sessionId,  : 
  Gateway in localhost:8880 did not respond.

Any assistance with this issue would be hugely appreciated.

Not a spark user but I notice in the log that you set the port to 8890 but the error says it couldn't connect to 8880. Could that have some bearing on your issue?

This topic was automatically closed 21 days after the last reply. New replies are no longer allowed.

If you have a query related to it or one of the replies, start a new topic and refer back with a link.