Alert Message Please update your browser.

We don't support this browser version anymore. Using an updated version will help protect your accounts and provide a better experience. 

Update your browser

Please update your browser.

We don't support this browser version anymore. Using an updated version will help protect your accounts and provide a better experience.

Update your browser

Close

The Log4Shell chronicles, Part 2: Lessons learned

by Haiying Guo and Panna Pavangadkar

January 1 — 9 min read

In Part 1, we saw how the Log4Shell vulnerability exploited a simple logging feature in the open-source library Apache Log4j2, threatening large industries during the 2021 holiday season. In Part 2, we’ll explore what we learned from the experience and how we plan to prevent future vulnerabilities.

Now that the Log4Shell firefighting is over, we can assess the landscape and figure out the keys to preventing a similar issue in the future. Broadly speaking, we would, given a similar issue, adopt a much more draconian remediation strategy. These are the additional improvements that can help us reduce the manual labor required.

Stay current all the time

"Always stay current" should become part of our culture. We need to update our software with the latest available libraries and languages all the time, not just when reacting to vulnerabilities. This should be further enforced by following New Banking Architecture (NBA) standards and building with Photon.

You may be saying to yourself, "Well, Log4j 1.x versions did not have issues. Why should I stay current?" Staying current lets us take advantage of the latest features and fixes to reduce risk and development costs. During remediation of Log4Shell, the hardest applications to remediate were those still running older versions of Java, where patch libraries take more time to produce.

Automate upgrades (with platforms like Proactive)

Modern programs rely on many open-source and/or vendor libraries for common functionality to accelerate development. These libraries also pull in other libraries, resulting in hundreds of direct or indirect dependencies. Any of those dependencies may be a source of vulnerability. Proactive leverages the same method we use to create bills of materials (BOMs) to detect libraries that require updates among the web of dependencies and automatically proposes a change in code to upgrade them. This takes human manual coding out of the equation. Proactive also regularly scans applications to ensure that whenever a new library version is published and marked ready to use, an upgrade will be proposed. This enforces the culture of "Stay current all the time" automatically.

By expanding the use of Proactive to more applications and to more use cases, such as JDK, an OS image, we could significantly reduce the cost of remediating future vulnerabilities.

Refine reporting

Real-time reporting coupled with a Google-like text search for impacted assets can make us even more efficient for the next incident.

Our success was built on top of accurate inventory, but inventory reporting is not yet real time. In a fast-turnaround remediation campaign, this introduced noise and confusion. We need to invest more in near-real-time reporting as our delivery speed improves.

Current inventory data is not accessible unless specially permitted by the application. We need to revisit information visibility controls in general to prevent a scramble to obtain access during a crisis like this.

It would also be an immensely powerful user experience to provide natural language search capabilities against the inventory system. The ease of use and improved precision filtering can help us get to the impacted applications/assets more quickly.

Speed and even more speed

There is an opportunity to further refine our patterned build tools and improve build time and reduce turnaround cost. The quicker we can make changes, the quicker we can react to the risk introduced by vulnerabilities like Log4Shell.

Features like logging could have a generic test harness to ensure every application using a specific logging framework act as expected. Testing against such a framework’s upgrade is no longer an application-specific test. We should be able to run the same generic test against all upgraded applications. Once the test passes, we can automatically accept the upgrade. Such tests should be created as internal open-source software and can potentially become JPMC contributions to the industry.

Stay tuned

These are just some initial thoughts on how we can improve. An upcoming Log4Shell retrospective will produce a comprehensive list of next steps. Stay tuned for Part 3!