SaaS data loss: The problem you didn’t know you had
A US Department of Commerce agency has been chastised for spunking $2.7m chasing down a supposed major malware infection that was actually limited to a handful of PCs.
The Economic Development Administration adopted a scorched earth policy – isolating itself from the internet before destroying more than $170,000 worth of equipment including printers, TVs, and even computer mice – in a comically inept attempt to resolve the phantom outbreak.
The physical destruction of equipment only ceased after the department’s disposal budget was exhausted. “The destruction of IT components was clearly unnecessary,” the Office of the Inspector General’s (OIG) auditor said in an official report released last month.
EDA’s CIO concluded that the risk, or potential risk, of extremely persistent malware and nation-state activity (which did not exist) was great enough to necessitate the physical destruction of all of EDA’s IT components.
EDA’s management agreed with this risk assessment and EDA initially destroyed more than $170,000 worth of its IT components, including desktops, printers, TVs, cameras, computer mice, and keyboards. By August 1, 2012, EDA had exhausted funds for this effort and therefore halted the destruction of its remaining IT components, valued at over $3 million.
EDA intended to resume this activity once funds were available. However, the destruction of IT components was clearly unnecessary because only common malware was present on EDA’s IT systems.
The EDA, which promotes economic development in underperforming US regions, went into panic mode after receiving notification of a malware outbreak from the US Department of Homeland Security’s CERT (Computer Emergency Response Team) in December 2011.
The agency hired an outside security contractor, at an eventual cost of $823,000, in late January 2012. After some initial false positives, the contractor decided EDA’s systems were mostly clean. Common-or-garden malware was found on six systems, a problem that could have been repaired by reimaging the affected machines. The unnamed “common malware [was] contained in archived e-mail attachments and temporary Internet browser files”, according to OIG’s report.
Confusion and miscommunication meant that an outbreak reckoned by the Department of Commerce’s computer security response team to be limited to two components was treated as something that had spread to more than half of the EDA’s 250 computers. It was treated as an advanced persistent attack internally by the EDA, despite little evidence to substantiate this belief.
EDA’s CIO, fearing that the agency was under attack from foreign cyber-intelligence, isolated its systems from the net and initiated the policy of physical destruction. Unnecessary destruction of IT equipment alone cost $175,000.
Staff were given temporary laptops requisitioned from the Census Bureau, along with internet access and interim e-mail capability in a programme that eventually cost $1.06 million. Entire mail servers were shut down, quite unnecessarily, because of some relatively innocuous malware on a small number of client PCs.
The EDA also spent $688,000 on contractors to come up with a long-term response to the incident. The whole mess was eventually sorted out within five weeks, but even this process didn’t begin until February 2013 – over a year after the initial incident. In total the agency spunked $2.7 million – more than half its 2012 annual IT budget – in grappling with the problem.
The National Oceanic and Atmospheric Administration received the same notice from the CERT but came up with a far more measured response, isolating and cleaning up the problem by January 2012. In contrast, the EDA was still ineptly grappling with the problem until outside agencies stepped in during February the following year.
Despite the wholesale trashing of perfectly good gear, the EDA is not wholly to blame for the mess. An inexperienced staffer in the CERT initially told the EDA that it was dealing with 146 infected components. This assessment was quickly revised downwards to just two infected items in a follow-up notice issued a day later, but, crucially, the EDA were not told that the initial notice was inaccurate and misleading.
“EDA thought there were 146 infected components [which] influenced everyone’s perception of the incident and contributed to EDA’s unnecessary recovery and remediation activities,” OIG explains. EDA took drastic actions through misplaced fears that the malware infection would spread to other government bureaux.
“Deficiencies in the Department [of Commerce]’s Incident Response Program… significantly contributed to EDA’s inaccurate belief that it experienced a widespread malware infection,” OIG explains. “Consequently, the Department of Commerce Computer Incident Response Team (DOC CIRT) and EDA propagated inaccurate information that went unidentified for months after EDA’s incident.”
The OIG added:
We found that DOC CIRT’s incident handlers did not follow the Department’s incident response procedures, that its handler for EDA’s incident did not have the requisite experience or qualifications, and that DOC CIRT did not adequately coordinate incident response activities.
OIG is clear, however, that the main blame for the whole sorry episode rests with EDA, whose recovery efforts were misdirected and inept. “EDA focused its recovery efforts on replacing its IT infrastructure and redesigning its business applications. EDA should have concentrated its resources on quickly and fully recovering its IT systems (e.g. critical business applications) to ensure its operational capabilities.”
The auditor’s report, Malware Infections on EDA’s Systems Were Overstated and the Disruption of IT Operations Was Unwarranted, can be found here (PDF, 33 pages, fairly large). The report notes that the EDA’s IT infrastructure has been systematically mismanaged and insecure for years, with poor handling of patch management and other routine tasks. Many of these problems date back as far as early 2006. ®
What you need to know about cloud backup
Article source: http://go.theregister.com/feed/www.theregister.co.uk/2013/07/10/us_gov_agency_smashed_mice_tvs_malware_oig_eda/
