如何在pyspark中使用log4j配置特定日志记录器的日志级别？

本文是小编为大家收集整理的关于如何在pyspark中使用log4j配置特定日志记录器的日志级别？的处理/解决方法，可以参考本文帮助大家快速定位并解决问题，中文翻译不准确的可切换到English标签页查看源文。

中文
English

问题描述

来自 this stackoverflow thread，我知道如何在pyspark中获取和使用log4j logger，例如:

from pyspark import SparkContext
sc = SparkContext()
log4jLogger = sc._jvm.org.apache.log4j
LOGGER = log4jLogger.LogManager.getLogger('MYLOGGER')
LOGGER.info("pyspark script logger initialized")

与 spark-submit 脚本一起工作.

我的问题是如何修改 log4j.properties 文件以配置此特定 logger的日志级别或如何动态配置它?

推荐答案

关于如何通过log4j.properties文件配置log4j的其他答案，但是我没有看到有人动态地提及如何进行操作，因此:

from pyspark import SparkContext
sc = SparkContext()
log4jLogger = sc._jvm.org.apache.log4j
LOGGER = log4jLogger.LogManager.getLogger('MYLOGGER')

# same call as you'd make in java, just using the py4j methods to do so
LOGGER.setLevel(log4jLogger.Level.WARN)

# will no longer print
LOGGER.info("pyspark script logger initialized")

本文地址：https://www.itbaoku.cn/post/1575080.html

百度翻译此文有道翻译此文

问题描述

From this StackOverflow thread, I know how to obtain and use the log4j logger in pyspark like so:

from pyspark import SparkContext
sc = SparkContext()
log4jLogger = sc._jvm.org.apache.log4j
LOGGER = log4jLogger.LogManager.getLogger('MYLOGGER')
LOGGER.info("pyspark script logger initialized")

Which works fine with the spark-submit script.

My question is how to modify the log4j.properties file to configure the log level for this particular logger or how to configure it dynamically?

推荐答案

There are other answers on how to configure log4j via the log4j.properties file, but I haven't seen anyone mention how to do it dynamically, so:

from pyspark import SparkContext
sc = SparkContext()
log4jLogger = sc._jvm.org.apache.log4j
LOGGER = log4jLogger.LogManager.getLogger('MYLOGGER')

# same call as you'd make in java, just using the py4j methods to do so
LOGGER.setLevel(log4jLogger.Level.WARN)

# will no longer print
LOGGER.info("pyspark script logger initialized")

上一篇：如何使用log4j抑制来自外部库的日志信息？

下一篇：春天。"SimpleLogger似乎没有位置意识 "异常