Log4j critical vulnerability (CVE-2021-44228): Practical tips to protect your organisation
The Log4j critical vulnerability has exploded across industry blogs and mainstream media and is a major concern for organisations worldwide. CyberCX is helping hundreds of customers across Australia and New Zealand respond and recover from this incident. This post sets out what we’ve learned so far and how you can protect your organisation. We will update this page as we discover more.
What’s going on?
Log4j is commonly used in Java applications of all shapes and sizes. Versions up to 2.15.0 have a critical vulnerability which is being actively exploited and attackers are using it to:
- collect credentials from running servers
- install ransomware and crypto miners
- exfiltrate data from organisations.
This can impact any system, not just applications that are directly on the internet. For example, if a back-end system logs requests that are sent by a front-end application, attackers may be able to use the Log4j vulnerability to attack them.
What should you do?
- Identify vulnerable systems
- Identify systems running Java
The first step is to identify what applications you have that use a vulnerable version of Log4j.
Start with a list of software you use in your organisation. In most cases, the easiest way to tell if a product is vulnerable will be to check the vendor website. However, vendors are still working through their own product lists. Systems which are not currently on vendor lists may still be proven vulnerable.
One way to do this is to search for the Log4j JAR file on your servers or look for file hashes of vulnerable versions. However, as Log4j is included as a library in third party software, the results may not be conclusive.
Review logs
Even after doing the above, there may be applications in your environment that you aren’t aware run Java or that are vulnerable. Another way you can detect vulnerable systems is to look for signs that servers are being scanned or exploited.
- Review network logs for outbound connections on ports tcp/389, tcp/636, tcp/1099, tcp/1389and tcp/3893. These ports are commonly used in scans to detect vulnerable applications. Any systems that have started creating outbound connections since 9 December are likely vulnerable and/or exploited.
- Review DNS logs for signs of compromise or data exfiltration. If you use AWS in your environment, search for DNS queries containing “AKIA” or “ASIA”, as these may indicate compromise of AWS access tokens. You should also check for DNS queries with a long length and/or high randomness (Shannon entropy).
Patch
If possible, apply updates. Vendors are releasing updates to products and this is the best way to prevent future exploitation of your systems.
This will not always be possible. While vendors are working quickly to release patches, it takes time to identify what products include Log4j and to make sure updates don’t break functionality. In the meantime, a combination of mitigations could be used to reduce the likelihood of exploitation.
Update to 2.17.0
On Monday 14 December at 9:28 (AEDT), the Log4j developers released version 2.16.0, which completely disables JNDI by default. If you have not already updated to 2.15.0 and instead are using the environment variable mitigations, your systems may still be vulnerable in certain cases.
On Sunday 19 December, the Log4j developers released version 2.17.0, which mitigates a Denial-of-Service vulnerability present in all previous versions. This vulnerability is not helped by the environment variable mitigations.
Vendors will need to re-assess and possibly re-release patches for their applications. We recommend continuing to monitor vendor advisories and updating to 2.17.0 where possible to further reduce the risk.
Mitigate
Test your mitigations
This incident is continuing to evolve, and attackers are changing their tactics. Some early mitigations—in particular, early WAF rules—may no longer be effective.
We recommend that every organisation who is depending on these mitigations to test their systems, either with professional penetration testing companies, and/or automated vulnerability scanning tools.
Disable vulnerable functionality
You can disable the vulnerable functionality by adding ‐DLog4j2.formatMsgNoLookups=true to the JVM command for starting your application, or LOG4J_FORMAT_MSG_NO_LOOKUPS=”true” to the environment variables.
There are some cases where these mitigations will not be completely effective, and should not be solely relied upon.
These mitigations will only work on versions 2.10 and later of the vulnerable Log4j and it may not be obvious where to configure this in all applications. Your application vendors will have specific documentation about how to apply this mitigation to their products.
If you are unable to patch log4j, one mitigation option that remains effective is to manually remove the vulnerable JNDI functionality from Log4j JAR files. Instructions are available in the Apache security notification (see Resources below), however care should be taken as different applications may include JAR files in different ways, such as WAR or EAR packages.
Outbound filtering
Preventing your server from initiating outbound connections may prevent an attacker from successfully carrying out their attack. This is due to the way the exploit works–causing your system to download an exploit from attacker-controlled infrastructure.
You can implement this mitigation using a local firewall such as Windows firewall or iptables or a network firewall as appropriate for your environment. This mitigation requires caution to avoid breaking system functionality and may not be completely effective, because information can leak through DNS.
Web Application Firewalls (WAF)
WAF vendors are pushing updates to their products to detect and block attempted exploit activity. However, as attackers are changing their techniques, this may not be completely effective.
Respond: Consider your people over the holidays
This is likely to continue over the holiday period, and it’s important that you start considering how your people will be able to take time off while still being able to respond.
We have provided a standalone blog addressing this at: Log4j Critical Vulnerability (CVE-2021-44228): Planning for the holidays
Respond: Look for indicators of compromise
There are signs that the attack has been exploited since the start of December, although most of the activity started on Friday 9 December (NZT) after the official announcement was released.
It’s important to note that lots of security companies have been scanning the internet since Friday, so not all log entries indicate malicious activity. However, all alerts should be investigated to determine whether the system has been exploited.
Endpoint Detection and Response (EDR) vendors have created detection logic to look for signs of compromise. Once the system has been patched or mitigated, we recommend using these tools to look for signs if your systems have been compromised.
If you confirm compromise of your system, you should investigate for any post-compromise activity such as information stealing, executing malware or movement to other systems on your network.
Recover: Prepare to change system credentials
There is evidence of remote attackers pulling the credentials out of vulnerable servers, including environment variables which contain access keys such as AWS, Azure or Service Account credentials.
Once you have identified vulnerable systems and patched them, we recommend changing the credentials that are used by, or stored on, that server.
How can CyberCX help?
CyberCX is currently engaged with many customers across Australia and New Zealand and to detect, respond and recover from this incident. If you need assistance, we can help:
- triage your environment
- identify and apply mitigations
- explain the technical issues to non-technical management
- look for indicators of compromise
- respond to compromised systems.
For urgent assistance, contact us
Resources
We will maintain a list of useful resources here, containing links to vendor responses as well as information about attacker activity and how to detect it.
Identify
- A tool to assist you in testing systems to determine vulnerability, from Huntress Labs: https://log4shell.huntress.com/
- File hashes of known vulnerable Log4j versions: https://github.com/mubix/CVE-2021-44228-Log4Shell-Hashes
Patch and mitigate
- NCSC-NL is maintaining a list of vendor responses on GitHub: https://github.com/NCSC-NL/log4shell/tree/main/software
- Another list of vendor responses: https://gist.github.com/SwitHak/b66db3a06c2955a9cb71a8718970c592
- Apache project notification, including initial mitigations: https://logging.apache.org/log4j/2.x/security.html
Indicators of compromise (IOCs)
- Curated intelligence list of IOCs: https://github.com/curated-intel/Log4Shell-IOCs
- Remote code execution detection commands: https://gist.github.com/Neo23x0/e4c8b03ff8cdf1fa63b7d15db6e3860b
Technical security vendor advisories
-
- CrowdStrike: https://www.crowdstrike.com/blog/Log4j2-vulnerability-analysis-and-mitigation-recommendations/
- F5 (includes information about F5 products and simple iRules to mitigate vulnerable applications): https://support.f5.com/csp/article/K19026212
- Microsoft: https://www.microsoft.com/security/blog/2021/12/11/guidance-for-preventing-detecting-and-hunting-for-cve-2021-44228-Log4j-2-exploitation/
- Splunk guidance for hunting: https://www.splunk.com/en_us/blog/security/log-jammin-Log4j-2-rce.html
- More Splunk hunting advice: https://www.splunk.com/en_us/blog/security/log4shell-detecting-log4j-vulnerability-cve-2021-44228-continued.html
- Splunk article on searching for DNS exfiltration: https://www.splunk.com/en_us/blog/security/random-words-on-entropy-and-dns.html
