Outage caused by Log4j's Async Appender
In AsyncAppender.append
, if failed to move the message to in-memory buffer
if (blocking) {
if (AbstractLogger.getRecursionDepth() > 1) { // LOG4J2-1518, LOG4J2-2031
// If queue is full AND we are in a recursive call, call appender directly to prevent deadlock
AsyncQueueFullMessageUtil.logWarningToStatusLogger();
logMessageInCurrentThread(logEvent);
} else {
// delegate to the event router (which may discard, enqueue and block, or log in current thread)
final EventRoute route = asyncQueueFullPolicy.getRoute(thread.getId(), memento.getLevel());
route.logMessage(this, memento);
}
} else {
error("Appender " + getName() + " is unable to write primary appenders. queue is full");
logToErrorAppenderIfNecessary(false, memento);
}
On default settings
- The buffer is an
ArrayBlockingQueue
blocking
is set to true- use DefaultAsyncQueueFullPolicy, which most likely uses
EventRoute.ENQUEUE
, which in turns usesBlockingQueue.put(logEvent);
This means that under default settings, if the buffer is remains full for a long time, e.g., consumer is unable to consume, your main logging thread will be blocked on that put
call, and in turn the Tomcat work thread pool will be exhausted!