I tried to connect sparklyr in Cloudera environment with following command:
library(sparklyr)
options(sparklyr.log.console = TRUE)
sc <- spark_connect(master = "local", config = conf, spark_version = "2.4.7", spark_home = Sys.getenv("SPARK_HOME"))
However, it does not work and gives me following error:
Error in spark_connect_gateway(gatewayAddress, gatewayPort, sessionId, :
Gateway in localhost:8880 did not respond.
Try running `options(sparklyr.log.console = TRUE)` followed by `sc <- spark_connect(...)` for more debugging info.
I included the sparklyr.log.console options, but there is no additional information given in the error message.
I tried to also install the spark again from the package instead of using my SPARK_HOME. Unfortunately, this does not help as well.
library(sparklyr)
spark_version <- "2.4.7"
spark_install(version = spark_version )
options(sparklyr.log.console = TRUE)
sc <- spark_connect(master = "local", spark_version = "2.4.7", spark_home = spark_install_find(version=spark_v)$sparkVersionDir)
Is there something that I miss here? How can I get it to run? Ultimately, I need to run this in a yarn cluster. However, both local and yarn mode does not work right now.
Java version: 1.8.0_332
> sessionInfo()
R version 4.1.1 (--)
Platform: x86_64-pc-linux-gnu (64-bit)
Running under: Ubuntu 20.04.6 LTS
Matrix products: default
BLAS/LAPACK: /usr/lib/x86_64-linux-gnu/openblas-pthread/libopenblasp-r0.3.8.so
locale:
[1] LC_CTYPE=en_US.UTF-8 LC_NUMERIC=C
[3] LC_TIME=en_US.UTF-8 LC_COLLATE=en_US.UTF-8
[5] LC_MONETARY=en_US.UTF-8 LC_MESSAGES=en_US.UTF-8
[7] LC_PAPER=en_US.UTF-8 LC_NAME=C
[9] LC_ADDRESS=C LC_TELEPHONE=C
[11] LC_MEASUREMENT=en_US.UTF-8 LC_IDENTIFICATION=C
attached base packages:
[1] stats graphics grDevices utils datasets methods base
other attached packages:
[1] sparklyr_1.8.1
loaded via a namespace (and not attached):
[1] rstudioapi_0.14 magrittr_2.0.1 rappdirs_0.3.3 tidyselect_1.2.0
[5] R6_2.5.1 rlang_1.1.0 fastmap_1.1.1 fansi_0.5.0
[9] httr_1.4.2 dplyr_1.1.1 tools_4.1.1 parallel_4.1.1
[13] config_0.3.1 utf8_1.2.2 cli_3.6.1 DBI_1.1.3
[17] withr_2.5.0 dbplyr_2.3.2 askpass_1.1 ellipsis_0.3.2
[21] htmltools_0.5.5 openssl_1.4.4 yaml_2.3.7 assertthat_0.2.1
[25] digest_0.6.27 rprojroot_2.0.3 tibble_3.2.1 lifecycle_1.0.3
[29] forge_0.2.0 tidyr_1.3.0 purrr_1.0.1 base64enc_0.1-3
[33] htmlwidgets_1.6.2 vctrs_0.6.1 glue_1.6.2 compiler_4.1.1
[37] pillar_1.9.0 r2d3_0.2.6 generics_0.1.3 jsonlite_1.8.4
[41] pkgconfig_2.0.3
Hi! Are you trying to access the Cloudera Spark or a separate Spark session only on that server?
It should be the Cloudera spark. I added the spark_home param to be sure that it takes the right one. However, not using the spark_home parameter does not change anything unfortunately.
It seems like there is something that I miss that cause the gateway to not respond.
Ultimately, I need to run the script in a Haddop Yarn cluster (master = "yarn"). I tried to use master = "yarn" instead of local, but this does not help as well.
Edit: change mode to master
Ok, I think what needs to be set to "yarn" is master, not mode. Here is a link to connecting to YARN with sparklyr: sparklyr - Configuring Spark Connections
Sorry. I meant master and not mode. I already had the correct parameter names.
Ok, can you share the code and error you get when you attempt to connect to the Cloudera cluster?
The code looks as follow:
library(sparklyr)
conf <- spark_config()
conf$sparklyr.jars.default <- "<path_to_hwc_jar>"
conf$spark.sql.hive.hiveserver2.jdbc.url <- "jdbc2:hive2://<hive_url>/<hive_port>/default"
conf$spark.hadoop.hive.metastore.uris <- "thrift://<hive_url>:9083"
conf$spark.sql.extensions <- "com.qubole.spark.hiveacid.HiveAcidAutoConvertExtension"
conf$spark.kryo.registrator <- "com.qubole.spark.hiveacid.util.HiveAcidKyroRegistrator"
conf$spark.datasource.hive.warehouse.read.via.llap <- TRUE
conf$spark.datasource.hive.warehouse.read.jdbc.mode <- "cluster"
conf$spark.deploy.mode <- "cluster"
conf$spark.driver.cores <- 2
conf$spark.driver.memory <- "2g"
conf$spark.executor.cores <- 2
conf$spark.executor.memory <- "2g"
conf$spark.executor.instances <- 2
conf$spark.hadoop.hive.exec.dynamic.partition.mode <- "nonstrict"
options(sparklyr.log.console = TRUE)
sc <- spark_connect(master = "yarn", config = conf)
The code above gives me also the gateway error code. Unfortunately, I cannot get more error message than this. The sparklyr.log.console options does not work properly. I also tried to used the option as conf$sparklyr.log.console <- TRUE but nothing happened.
Error in spark_connect_gateway(gatewayAddress, gatewayPort, sessionId, :
Gateway in localhost:8880 did not respond.
Try running `options(sparklyr.log.console = TRUE)` followed by `sc <- spark_connect(...)` for more debugging info.
The R session runs in a CDSW session if this matters.
Additional info, I managed to get a bit more error output from sparklyr.log.console.
library(sparklyr)
conf <- spark_config()
conf$sparklyr.jars.default <- "<path_to_hwc_jar>"
conf$spark.sql.hive.hiveserver2.jdbc.url <- "jdbc2:hive2://<hive_url>/<hive_port>/default"
conf$spark.hadoop.hive.metastore.uris <- "thrift://<hive_url>:9083"
conf$spark.sql.extensions <- "com.qubole.spark.hiveacid.HiveAcidAutoConvertExtension"
conf$spark.kryo.registrator <- "com.qubole.spark.hiveacid.util.HiveAcidKyroRegistrator"
conf$spark.datasource.hive.warehouse.read.via.llap <- TRUE
conf$spark.datasource.hive.warehouse.read.jdbc.mode <- "cluster"
conf$spark.deploy.mode <- "cluster"
conf$spark.driver.cores <- 2
conf$spark.driver.memory <- "2g"
conf$spark.executor.cores <- 2
conf$spark.executor.memory <- "2g"
conf$spark.executor.instances <- 2
conf$spark.hadoop.hive.exec.dynamic.partition.mode <- "nonstrict"
conf$sparklyr.log.console <- TRUE
sc <- spark_connect(master = "yarn", config = conf)
The error now also mentions that the executor memory must be a positive number. Which is weird as I inserted 2g there
Exception in thread "main" org.apache.spark.SparkException: Executor memory must be a positive number
at org.apache.spark.deploy.SparkSubmitArguments.error(SparkSubmitArguments.scala:657)
at org.apache.spark.deploy.SparkSubmitArguments.validateSubmitArguments(SparkSubmitArguments.scala:274)
at org.apache.spark.deploy.SparkSubmitArguments.validateArguments(SparkSubmitArguments.scala:251)
at org.apache.spark.deploy.SparkSubmitArguments.<init>(SparkSubmitArguments.scala:120)
at org.apache.spark.deploy.SparkSubmit$$anon$2$$anon$1.<init>(SparkSubmit.scala:909)
at org.apache.spark.deploy.SparkSubmit$$anon$2.parseArguments(SparkSubmit.scala:909)
at org.apache.spark.deploy.SparkSubmit.doSubmit(SparkSubmit.scala:81)
at org.apache.spark.deploy.SparkSubmit$$anon$2.doSubmit(SparkSubmit.scala:922)
at org.apache.spark.deploy.SparkSubmit$.main(SparkSubmit.scala:931)
at org.apache.spark.deploy.SparkSubmit.main(SparkSubmit.scala)
Error in spark_connect_gateway(gatewayAddress, gatewayPort, sessionId, :
Gateway in localhost:8880 did not respond.
2
Try running `options(sparklyr.log.console = TRUE)` followed by `sc <- spark_connect(...)` for more debugging info.
master = "local" still does not work at all unfortunately.