Note to self: Changing loglevel in apache Spark

Very quick note for future reference. Please ignore.

Change loglevel in spark

Easy peasy, you can do it programatically in the application like:

spark.sparkContext.setLogLevel("WARN")

Change loglevel in yarn

This one took a while to find, you can just run spark-submit while previously exporting this envvar:

export YARN_ROOT_LOGGER=${YARN_ROOT_LOGGER:-WARN,RFA}

That´s all.