Tens of thousands of websites, some operated by The Los Angeles Times, Seagate, and other reputable companies, have recently come under the spell of “Darkleech,” a mysterious exploitation toolkit that exposes visitors to potent malware attacks.
The ongoing attacks, estimated to have infected 20,000 websites in the past few weeks alone, are significant because of their success in targeting Apache, by far the Internet’s most popular Web server software. Once it takes hold, Darkleech injects invisible code into webpages, which in turn surreptitiously opens a connection that exposes visitors to malicious third-party websites, researchers said. Although the attacks have been active since at least August, no one has been able to positively identify the weakness attackers are using to commandeer the Apache-based machines. Vulnerabilities in Plesk, Cpanel, or other software used to administer websites is one possibility, but researchers aren’t ruling out the possibility of password cracking, social engineering, or attacks that exploit unknown bugs in frequently used applications and OSes.
Researchers also don’t know precisely how many sites have been infected by Darkleech. The server malware employs a sophisticated array of conditions to determine when to inject malicious links into the webpages shown to end users. Visitors using IP addresses belonging to security and hosting firms are passed over, as are people who have recently been attacked or who don’t access the pages from specific search queries. The ability of Darkleech to inject unique links on the fly is also hindering research into the elusive infection toolkit.
“Given that these are dynamically generated, there would be no viable means to do a search to ferret them out on Google, etc.,” Mary Landesman a senior security researcher for Cisco Systems’ TRAC team, told Ars. “Unfortunately, the nature of the compromise coupled with the sophisticated conditional criteria presents several challenges.”
The injected HTML iframe tag is usually constructed as IP address/hex/q.php. Sites that deliver such iframes that aren’t visible within the HTML source are likely compromised by Darkleech. Special “regular expression” searches such as this one helped Landesman ferret out reported iframes used in these attacks. Note that while the iframe reference is formed as IP/hex/q.php, the malware delivery is formed as IP/hex/hex/q.php.
In active development
With the help of Cisco Security Engineer Gregg Conklin, Landesman observed Darkleech infections on almost 2,000 Web host servers during the month of February and the first two weeks of March. The servers were located in 48 countries, with the highest concentrations in the US, UK, and Germany. Assuming the typical webserver involved hosted an average of 10 sites, that leaves the possibility that 20,000 sites were infected over that period. The attacks were documented as early as August on researcher Denis Sinegubko’s Unmask Parasites blog. They were observed infecting the LA Times website in February and the blog of hard drive manufacturer Seagate last month, an indication the attacks are ongoing. Landesman said the Seagate infection affected media.seagate.com, which was hosted by Media Temple, began no later than February 12, and was active through March 18. Representatives for both Seagate and the LA Times said the sites were disinfected once the compromises came to light.
“I regularly receive e-mails and comments to my blog posts about new cases,” Sinegubko told Ars last week. “Sometimes it’s a shared server with hundreds or thousands of sites on it. Sometimes it’s a dedicated server with some heavy-traffic site.”
Referring to the rogue Apache modules that are injected into infected sites, he added, “Since late 2012 people have sent me new versions of the malicious modules, so this malware is in active development, which means that it pays off well and the number of infected servers can be high (especially given the selectivity of the malware that prefers to stay under the radar rather than infecting every single visitor).”
Landesman picked a random sample of 1,239 compromised websites and found all were running Apache version 2.2.22 or higher, mostly on a variety of Linux distributions. According to recent blog posts published here and here by researchers from security firm Securi, Darkleech uses rogue Apache modules to inject malicious payloads into the webpages of the sites it infects and to maintain control of compromised systems. Disinfecting Web servers can prove extremely difficult since the malware takes control of the secure shell (SSH) mechanism that legitimate administrators use to make technical changes and update content to a site.
“We have noticed that they are modifying all SSH binaries and inserting a version that gives them full access back to the server,” Securi CTO Daniel Cid wrote in January. “The modifications not only allow them to remote into the server bypassing existing authentication controls, but also allow them to steal all SSH authentications and push it to their remote servers.”
Researchers from a variety of other organizations, including antivirus provider Sophos and the Malware Must Die blog, have also stumbled on servers infected by Darkleech. They note the third-party attack sites host malicious code from the Blackhole exploit kit, a suite of tools that targets vulnerabilities in Oracle’s Java, Adobe’s Flash and Reader, and a variety of other popular client software.
“It looks like the attackers were beforehand well-prepared with some penetration method to gain web exploitation which were used to gain shell access and did the privilege escalation unto root,” the writer of the latter blog post wrote last week, adding that he wasn’t at liberty to discuss the precise method. “Since the root [was] gained in all infected servers, there is no way we can trust the host or its credentials anymore.”
The writer went on to recommend that admins take infected servers offline and use backup data to reinstall the software. He also suggested that users take care to change all server credentials, since there’s a strong chance all previous administrator logins have been compromised.
Déjà vu
The Apache server compromise in many ways resembles a mass infection from 2008 that also used tens of thousands of sites to silently expose visitors to malware attacks. The challenge white hats often face in fighting these hacks is that each researcher sees only a small part of the overall damage. Because the server malware is designed to conceal itself and because so many individual systems are affected, it can be next to impossible for any one person to gain a true appreciation for the scope of attack.
Since there’s not yet consensus among researchers about exactly how Darkleech takes hold of infected systems, it’s still unclear exactly how to protect them. And as already noted, disinfecting systems can also prove challenging since backdoor and possibly even rootkit functionality may allow attackers to maintain control of servers even after the malicious modules are uninstalled. Landesman has published her own blog post about the infection here.
“This is a latent infection,” Sinegubko wrote. “It hides from server and site admins using blacklists and IPs and low-level server APIs (something that normal site scripts don’t have access to). “It hides from returning visitors. It constantly changes domains so you can’t reduce it to the facts were some particular domain was involved. I’m still waiting for someone to share any reliable information about the attack vector.”
Story updated to add details about Blackhole, add link to Landesman post.
