Log4Shell Vulnerability Detection, Data Correlation, and Recommendations

Contributors: Dvir Sayag, Yuval Zacharia, Shahar Vaknin, Netanel Golani, Alon Slotky, Sarah Granger, Guy Yasoor, Guy Yagev, Evin Hernandez, Yonatan Khen.

Q&A Recording - Log4Shell Vulnerability: Detection, Data Correlation, and Recommendations

Executive Summary

• A new critical RCE vulnerability in the Apache Log4j package was reported last week AKA Log4Shell, tagged as CVE-2021-44228, and classified critical by Apache.
• This vulnerability was seen exploited in the wild, and the affected Log4j versions are between 2.0 to 2.14.1.
• After thorough research by Hunters’ research and threat hunting teams, we immediately implemented detection and mitigation tactics that minimize the risk.
• Hunters offer cross data-sources detection, supporting more than 40 combinations of security products (EDRs, Web application logs, and Network logs), that can enhance the ability to detect the threat of the Log4j vulnerability.
• The Hunters platform was not affected by the Log4j vulnerability and has taken measures to ensure the risk is mitigated.

Background

On December 9th, 2021, a critical RCE vulnerability in the Apache Log4j package was published and classified as CVE-2021-44228, also called Log4Shell. In the last few days, massive exploitation of the vulnerability was observed in the wild.

Hunters researched the vulnerability and relevant exploitations in order to surface servers at potential risk and hunt for exploitation attempts, as well as to develop detection methods that can detect the exploitation of Log4Shell. Given that Log4j is used in a wide range of consumer and enterprise websites, services, and applications, we first assessed our own systems and then took measures to identify potential vulnerabilities or attacks for our customers, creating a visibility dashboard within the Hunters platform. 

While endpoint-based detection can be relatively effective in many cases, correlating between cross-platform data sources can be a powerful tool for detecting a vulnerability like this that could be spread across multiple systems and devices. Firewall logs, EDR logs, and WAF logs can be overlooked if only considered on their own. The Hunters platform was built for cases like this in order to cross correlate varied data sources into one coherent threat intelligence picture.

The vulnerability in Log4j can be easily taken advantage of, allowing attackers to access company data, implant backdoors, takeover privileged users, execute remote code, and more. While some vendors are already patching their applications, hackers have already created tools to exploit the vulnerability. 

Follow our recommendations in this blog post to learn more about the exploitation process. After that, to implement cross data-sources detection methods to have the ability to detect future attempts of exploitation in your environment.

Attack Exploitation Process

An external attacker can exploit Log4Shell to run arbitrary code on an external-facing machine running a Java-based server, which uses a vulnerable Log4j package, for example, Apache webserver. The attack abuses Java Naming and Directory Interface (JNDI) API which provides lookup and access services to Java objects, locally or remotely, for Java applications.

Therefore, by exploiting Log4Shell, the attacker gains initial access into the network which can lead to a wider attack on the internal network.

Now let’s explain the attack flow and list data sources relevant for detection:

1. The attacker sends a JNDI lookup string to the Java application in a header that is most likely logged. For example: 
   

• • Logs - WAF logs, HTTP logs, RPROXY logs, HTTP logs

       2. String is logged by vulnerable Log4j logger                                                                 

• • Logs - Java application logs (usually under /var/log)

       3. JNDI API queries the malicious server for the malicious Java object in LDAP protocol over port 1337:

• • Logs - EDR network logs, Firewall logs, Netflow logs

         4. Attacker server serves the malicious Java object directory information related to the malicious Java class.

• • Logs - IPS logs, Java application logs

• • Logs - EDR logs

Hunters' Recommendations for Mitigation

The first most crucial step is to identify which systems might be vulnerable. Here is how we recommend creating a self-updated visibility report:

An anonymized example of a visibility dashboard in Hunters’ portal

The visibility dashboard details all devices and the corresponding Java binaries execution that load a component related to the Log4j package. In addition, we tagged external-facing devices and exported the version of the Log4j package in case it exists. 

Here is the SQL query that we used to create this dashboard:

First, the query tries to find all Java program executions with Log4j string in the command line (which indicates loading of Log4j). Then, it takes the fetched agent IDs and checks if there was an inbound connection from an external IP address in the past 7 days - allowing it to assess the risk of such execution. Finally, it correlates between the findings and tries to resolve the agent ID to their hostnames and external IPs.

After understanding what assets are at risk, we highly recommended applying one of the following options to prevent exploitation:

• Update Log4j packages to version 2.16.0 or above.
• For versions >2.10 this behavior can be mitigated by following the next steps -

Setting system property "log4j2.formatMsgNoLookups" to "true" or by removing the JndiLookup class from the classpath (example: zip -q -d log4j-core-*.jar org/apache/logging/log4j/core/lookup/JndiLookup.class).

Java 8u121 (see https://www.oracle.com/java/technologies/javase/8u121-relnotes.html) protects against remote code execution by defaulting "com.sun.jndi.rmi.object.trustURLCodebase" and "com.sun.jndi.cosnaming.object.trustURLCodebase" to "false".

Hunters' Log4Shell Detection Methods

When you are finished with the mitigation steps above, it’s time to implement detection methods that will minimize the risk of being exploited in the future, by detecting and alerting on Log4Shell exploitation attempts.

Rare LDAP or RMI Connection to External Server - 

The 'Rare LDAP or RMI Connection to External Server' detector detects access from an internal IP address to an external IP address in one of the ports associated with the vulnerable protocols - LDAP, LDAPS, or RMI. If the external IP address is an address that has not been accessed before, a threat signal will be generated.

For further investigation, we recommend the external IP address that will be used by the attacker's C2 server. Our recommendation is to check if it is a known IP address, using the geo IP field -- for Hunters’ customers, this is done automatically --, to understand if it is an external service of the organization that is legitimately communicated in the mentioned vulnerable protocols.

While LDAP is in some cases used for legitimate communication between internal and external servers, for example in the case of using ״AWS Managed Microsoft AD״ communication outside the organization in the RMI protocol is rarer and should be taken as suspicious.

Suspected Log4Shell Exploitation - Two Parts -

We have created two detectors that make use of multiple different data sources. When assessing customer attack surfaces, we have observed attempts to attack servers using the Log4j vulnerability. In order to filter out the noise, we decided to examine the potentially malicious attempts and see if they correlate with other data sources in two different ways.

First, we search for attack attempts in the different parts of the web app logs, which may be logged using the vulnerable application. For example: we search for these attempts in the user-agent used and in the requested URL. This works generically for many different Web application logs such as AWF WAF logs, Cloudflare logs, or IIS logs. Once an attack attempt is detected, we extract the Domain / IP which will be contacted by the victim server if the attack is successful.

Option 1 - EDR: Correlation between HTTP logs and EDR logs in order to detect exploitation attempts and successful outgoing connections to attacker IP.

In this detection, we first identify a malicious payload in the HTTP/Proxy/WAF. Then the domain and IP are correlated with EDR network logs. This means that we look at all of the EDR network connections (and DNS requests) and search for ones matching the IP or Domain we have found in the potential attack. This works generically for many different EDRs such as - MDATP, CrowdStrike, and Carbon Black.

Here is a query example for looking for log4j vulnerability execution correlated with EDR:

Option 2 - Network Traffic: Correlation between HTTP logs and network logs in order to detect exploitation attempts and successful outgoing connections to attacker IP.

In this case, the IP is correlated with Network logs (such as Firewall logs). This means that we look at all of the outgoing network traffic and search for connections to the malicious IP. This works generically for many different Network logs such as - PAN Firewall logs, AWS VPC logs, Check Point traffic logs.

Here is a query example for looking for log4j vulnerability execution correlated with web flow logs:

** Please note that Hunters does not use any specific table for this detection. We use a unified schema over many tables. Also, the queries are examples to help the reader understand how to correlate between the data sources. Threat actors may use more complex obfuscation which is not detected by this example query.

Detecting Log4Shell in the Hunters platform

Summary

Hunters has implemented our unified data, multi-platform detections, and correlation for the Log4Shell vulnerability, developing custom detections, outlined above. We recommend all security operations teams use all resources available to mitigate potential risks. We strive to cross-correlate as much data as possible, as quickly as we can, to prevent severe attacks.