Andres Freund, a Microsoft engineer, was conducting routine tests on a Linux system when he noticed something strange. His investigation into this unusual behavior led him to discover one of the most significant backdoors in the history of open-source software. This is the true story of that discovery.
This article simplifies the technical aspects of the incident, but if you’re interested in the technical details, feel free to explore further.
What is Open Source Software?
Open-source software is software whose source code is publicly available. This allows any developer to review the code, identify vulnerabilities, suggest improvements, or contribute to the project. However, not everyone can directly update the code. Most open-source projects are hosted on platforms like GitHub, where trusted individuals called “maintainers” review contributions and decide which changes are safe to merge into the main source code.
The Project: XZ-Utils
This story revolves around an open-source project called “XZ-Utils”, a widely used compression tool in Linux systems. While Windows users typically rely on ZIP files for compression, Linux systems often use XZ. The maintainer of XZ-Utils, Lasse Collin, managed the project in his free time, as it wasn’t his full-time job. Unfortunately, his limited availability created an opportunity for attackers.
Background of the Attack
Two contributors, Kumar and Eng, were frustrated with the slow pace of changes in the XZ-Utils project and Collin’s delayed responses. They expressed their dissatisfaction and even suggested that a new maintainer might be needed. Aware of his struggles, Collin apologized, admitting he was overwhelmed by the workload since he maintained the project as an unpaid hobby.
In need of help, Collin identified an active contributor named Jia Tan, who seemed like a good fit for the role of maintainer. Unbeknownst to Collin, this was part of a larger plan orchestrated by Kumar, Eng, and Tan. Their goal was to gain control of the project.
In June 2022, Jia was promoted to core maintainer of XZ-Utils. Instead of launching an attack right away, Jia patiently built trust over time by fixing bugs and making legitimate contributions. Eventually, he managed to configure the project so that security alerts were routed to him rather than Collin.
The Attack
In February 2024, Jia launched his attack. Rather than inserting malicious code directly into the source code, he took a more covert approach. Since XZ-Utils is an open-source project, anyone could inspect the code, and if malicious changes had been introduced to the main codebase, they would have been detected quickly. Instead, Jia hid his backdoor in a test file which an area that typically receives less scrutiny.
Soon after, various Linux distributions, such as Debian, Kali, and MicroOS, began updating their systems with the latest version of XZ-Utils. However, these updates were initially rolled out in the unstable branches, which are used for testing purposes. After thorough testing, updates usually get promoted to the stable versions used in production environments.
The Discovery
This is where our hero, Andres Freund, enters the story. While running a micro-benchmark on a Debian Linux system with the new XZ version, he noticed something odd—SSHD processes (which handle secure shell connections for remote access) were consuming an unusually high amount of CPU resources. Even more concerning, SSH connections were taking about half a second longer than they should. Though a half-second delay might not seem like much, in the world of computing, it was a significant red flag.
Andres investigated further and discovered that a backdoor had been cleverly inserted (malware that allows unauthorized access to a system by bypassing normal authentication processes) Realizing the gravity of the situation, he immediately reported the vulnerability to the Debian team. If left undetected, this backdoor could have compromised a vast portion of the Linux ecosystem, leaving countless systems vulnerable to attack.
The Aftermath
After Andres published his findings, the cybersecurity world was stunned. It was a nightmare scenario that if the backdoor gone unnoticed, untold numbers of Linux systems could have been compromised. The affected Linux distributions quickly rolled back to previous safe versions of XZ-Utils. Investigators soon traced the malicious code back to Jia Tan, but it became clear that Jia was not working alone.
Interestingly, Kumar, Eng, and Jia Tan were just GitHub usernames, and it remains unclear whether they were individual attackers or part of a larger hacker group. To this day, the true identities behind the attack remain a mystery.
