XZ Near Miss Sheds Light on Vulnerability, Patching Issues
Vulnerability management is one of the biggest challenges security defenders face. It becomes especially difficult when a vulnerability impacts a critical device or service. When a backdoor was discovered in XZ Utils, an open-source compression library available for Linux, defenders were on high alert.
Using the XZ backdoor, a remote unauthenticated attacker could achieve remote code execution (RCE) in SSH servers that were using the compression utility in an unrecommended fashion. Thankfully, we got lucky. A researcher discovered the backdoor before malicious versions of XZ Utils were added to production versions of Debian, Ubuntu, and Fedora Linux, which would have impacted a significant portion of the internet.
The calm before the (inevitable) next vulnerability in a critical library should motivate defenders and businesses to address common stumbling blocks in the vulnerability management process.
Linux, libraries, and open source, oh my!
XZ Utils is a set of open-source tools and libraries for lossless data compression that comes preinstalled on most modern Linux distributions. Linux is the backbone of the internet — over 96% of the top 1 million web servers use Linux. So, to say Linux is ubiquitous greatly underscores the potential severity of the XZ backdoor.
Operated by the Linux foundation — which is not a software development company — and a small cohort of code maintainers check, test, and deploy code updates into the Linux ecosystem. These code maintainers are burning out, creating a path for threat actors to exploit the human element.
An April 15, 2024, statement from the Open Source Security (OpenSSF) and OpenJS Foundations acknowledged that the XZ backdoor was likely “not an isolated incident,” citing a smaller takeover attempt in two popular Javascript projects. The foundation urged code maintainers to look for potential signs of social engineering but noted that code maintainers often lack security expertise.
The calm before the (inevitable) next vulnerability in a critical library should motivate defenders and businesses to address common stumbling blocks in the vulnerability management process.
What makes a given vulnerability 'special’?
It’s impossible for every critical vulnerability to receive a “stop everything and patch now” response — security teams are often spread too thin and under-resourced. Of course, there are “special” exceptions that warrant an immediate response, such as Heartbleed or Log4Shell, a vulnerability impacting the Log4j software library used in many software applications.
The challenge is accurately predicting when a new vulnerability will evolve into a Log4j Situation. In addition to the 500% increase in the number of common vulnerabilities and exposures (CVEs) discovered since 2016, defenders must weigh the risk of any given CVE against multiple vulnerability prioritization systems.
Even when the community agrees that a given vulnerability is worthy of elevation to “special” status and all defenders are called to patch it, sometimes they just don’t.
What is patch hesitancy, and how does it apply to XZ and Log4j?
Three years later, despite a massive community response to patch and remediate, Log4j remains one of the most commonly exploited vulnerabilities. While zero-day vulnerabilities present a significant risk to businesses, threat actors also target known, unpatched vulnerabilities. A recent report by Cato Cyber found that Log4j “represented 30% of the outbound vulnerability exploitations and 18% of the inbound vulnerability exploitations detected in the first quarter of 2024.”
We often think of vulnerability management as a race between defenders, who are patching vulnerable software, and threat actors, who are seeking to gain unauthorized access. Unfortunately, the rush to patch can often be met with resistance from businesses that prioritize operational efficiency over security. Patch hesitancy is understandable to a degree, as patching can require taking a critical device (a Linux web server or a payroll system with a Log4j library) offline to implement the patch.
In the early days of cybersecurity, before ransomware became headline news, it was commonplace for businesses to wait before implementing a patch. There may be concerns that the patch would break existing workflows, disrupt network connectivity, or introduce other bugs. Unfortunately, these habits persist today, in an era when it is no longer safe to wait and stress test a patch.
"[Log4j] represented 30% of the outbound vulnerability exploitations and 18% of the inbound vulnerability exploitations detected in the first quarter of 2024.” — Cato Cyber
So, how do we fix vulnerability management?
Some security experts point to a software bill of materials (SBOM) as a fix for vulnerability management. While SBOM can have positive outcomes for security, it is not a panacea for vulnerability management.
SBOM is an inventory of a software piece's components. Security teams could then check against the inventory for components, such as a Log4j library, that have known vulnerabilities. Unfortunately, SBOM cannot enforce patching, and even if a business knows it uses a software product vulnerable to Log4Shell, it can still decide not to patch if the downtime required for the patch can disrupt business operations.
Vulnerability management is plagued by information overload and a few core issues. The main issue is identifying which vulnerabilities meet the appropriate level of criticality for a drop everything and patch response — a category XZ easily would have fallen into had it not been for early detection. Patch hesitancy is another significant issue.
SBOM will not fix vulnerability management because it cannot prioritize vulnerabilities or change deeply ingrained bad behaviors. Instead, security defenders and businesses need to build the muscle of patching early and often. Inevitably, this will mean operational downtime, but the consequences of not patching are far more severe.
How does Coalition prepare for risks like XZ?
Time is of the essence when it comes to vulnerability management. To ensure we stay ahead of the latest emerging threats, Coalition constantly collects security data by scanning the entire IPv4 space and parts of the IPv6 space, tracking new vulnerabilities, monitoring threat actor behavior with honeypots, and gathering intelligence from data leaks.
We only send alerts for zero-day vulnerabilities that present a real and immediate threat to businesses. This way, businesses can make good security decisions like patching early and often.
Businesses that want to take an active role in managing their cyber risk exposure can sign up for Coalition Control®, our integrated cyber risk management platform. Control allows businesses to better manage their digital risks through customized alerting and remediation recommendations.
