Hi!
I wonder if you can help me with the following problem.
What I need to achieve is to silence all the information logged to the console during ${SPARK_HOME}/bin/spark-submit execution.
I have already changed log level for spark itself by changing the appropriate line in ${SPARK_HOME}/conf/log4j.properties but still when I submit my R-sparklyr-script by providing sparklyr.jar I get the following info:
[user@host] ${SPARK_HOME}/bin/spark-submit --class sparklyr.Shell 'sparklyr.jar' script.R
21/07/01 16:12:25 INFO sparklyr: Session (999991) is starting under 127.0.0.1 port 4440
21/07/01 16:12:25 INFO sparklyr: Session (999991) found port 4440 is available
21/07/01 16:12:26 INFO sparklyr: Gateway (999991) is starting batch rscript
....
I deducted that sparklyr itself uses log4j as a logger but I don't know where to find the appropriate log4j.properites file.
Do you know any method to cope with it?
With regards,
Adam.