Sparklyr - change log level during spark-submit


I wonder if you can help me with the following problem.

What I need to achieve is to silence all the information logged to the console during ${SPARK_HOME}/bin/spark-submit execution.

I have already changed log level for spark itself by changing the appropriate line in ${SPARK_HOME}/conf/ but still when I submit my R-sparklyr-script by providing sparklyr.jar I get the following info:

[user@host] ${SPARK_HOME}/bin/spark-submit --class sparklyr.Shell 'sparklyr.jar' script.R
21/07/01 16:12:25 INFO sparklyr: Session (999991) is starting under port 4440
21/07/01 16:12:25 INFO sparklyr: Session (999991) found port 4440 is available
21/07/01 16:12:26 INFO sparklyr: Gateway (999991) is starting batch rscript

I deducted that sparklyr itself uses log4j as a logger but I don't know where to find the appropriate log4j.properites file.

Do you know any method to cope with it?

With regards,

Sparklyr actually does not use log4j (or other similar framework), and there is no way to change the log level at the moment unfortunately. I have created an issue for this at Use log4j (or similar) for logging · Issue #3145 · sparklyr/sparklyr · GitHub

Thanks a lot for the response. It seems that my deduction was false then - it was based on the observation that log produced by sparklyr is similar to the one produced by log4j. Anyway I believe that this change would be appreciated by many :slight_smile:

Once again thanks and looking forward to the changes :slight_smile:

This topic was automatically closed 21 days after the last reply. New replies are no longer allowed.

If you have a query related to it or one of the replies, start a new topic and refer back with a link.