Electrical Explosions Cause Construction Delays at NSA’s $1.2 Billion Spy Compound

Rick Bowmer/AP

AA Font size + Print

Persistent electrical surges apparently sparked explosions and a year-long delay that NSA officials did not disclose. By Aliya Sternstein

While the government was nearing a shutdown and bracing for a default, a $1.2 billion National Security Agency supercomputer homestead in Utah, built to ingest surveillance data, reportedly was wasting hundreds of thousands of dollars on fried equipment.

NSA officials masked the woes out west, at a time when the agency already was facing accusations of lying about domestic spying practices.

As recently as June 19, agency officials told Nextgov the spy center — 15 times the size of MetLife stadium — would be “operational” by Oct. 1. That month, NSA spokeswoman Vanee Vines also said in an email the “exterior is done.  . .  It’s now time for IT guts to be installed,” referring to information technology systems.

But, behind the security perimeter, persistent electrical surges apparently sparked explosions and a year-long delay that NSA officials did not disclose. On Monday night, the Wall Street Journal reported there have been 10 meltdowns during the past 13 months that have prevented NSA from running computers.

The revelations come after ex-NSA contractor Edward Snowden this spring leaked files to the media detailing the agency’s mammoth collections of Internet and phone data, revelations later acknowledged by U.S. officials. The 200-acre Utah computer fortress was designed to warehouse this data.

In June, Harvey Davis, a top agency official, who handles military installations, as well as power, space and cooling for all NSA data centers, said in an interview that the construction – undertaken by the Corps of Engineers – has been an open project. 

The military construction process by design is a very, very transparent process,” he said. “It’s a public discourse. When we give out our request for proposal, that’s through FedBizOpps.gov,” a website that advertises jobs for prospective vendors, so “by its nature, military construction is a very, very open process.”

Project contract papers called for a 65-megawatt center with a chiller plant, fire suppression systems, electrical generators and an uninterruptible power supply backup capacity.

On Tuesday, NSA and Army Corps of Engineers officials declined to answer questions about the failure to disclose that there would be some sort of delay.

Norbert Suter, the Corps chief of construction operations, said in an email that power connectivity remains incomplete. “The final commissioning of the entire electrical system is still ongoing,” he said, declining to address cost overruns. “The Corps is ensuring it functions as required and that it will be completely reliable prior to releasing the project to the customer,” meaning NSA. Suter did not say when the site would turn on. 

He acknowledged there have been a series of glitches. ”During the testing and commissioning of the Utah data center, problems were discovered with certain parts of the electrical system. Issues such as these can arise in any project, and are the reason the Corps tests and reviews every aspect of any project prior to releasing it to the customer,” he said.

More Corps professionals were summoned and the source of the problems was discovered, Suter said, without elaborating further. 

Because of the scope of this project, a specialized team of experts from the Corps’ headquarters was sent to review and assist with addressing these issues. The cause of the electrical issues was identified by the team and is currently being corrected by the contractor,” he said.

The team also uncovered ventilation troubles. In order to function, data centers need an air management system able to cool IT equipment and remove waste heat. Suter said, “An airflow issue was discovered with the generators and the contractor is correcting the problem.”

On Monday night, Vines said, “On the whole, this is why acceptance testing is done - so that issues can be found and resolved before the government takes possession.”

While final assessments continue, Suter said the Corps is ”making preparations to release it to the customer.”

The final tab for the ultimate customer, the taxpayer, is a question mark. Davis this summer said, “I came in under cost,” referring to $100 million in savings gained partly by refusing to let contractors adjust the plan. 

Close [ x ] More from DefenseOne
 
 

Thank you for subscribing to newsletters from DefenseOne.com.
We think these reports might interest you:

  • Ongoing Efforts in Veterans Health Care Modernization

    This report discusses the current state of veterans health care

    Download
  • Modernizing IT for Mission Success

    Surveying Federal and Defense Leaders on Priorities and Challenges at the Tactical Edge

    Download
  • Top 5 Findings: Security of Internet of Things To Be Mission-Critical

    As federal agencies increasingly leverage these capabilities, government security stakeholders now must manage and secure a growing number of devices, including those being used remotely at the “edge” of networks in a variety of locations. With such security concerns in mind, Government Business Council undertook an indepth research study of federal government leaders in January 2017. Here are five of the key takeaways below which, taken together, paint a portrait of a government that is increasingly cognizant and concerned for the future security of IoT.

    Download
  • Coordinating Incident Response on Posts, Camps and Stations

    Effective incident response on posts, camps, and stations is an increasingly complex challenge. An effective response calls for seamless conversations between multiple stakeholders on the base and beyond its borders with civilian law enforcement and emergency services personnel. This whitepaper discusses what a modern dispatch solution looks like -- one that brings together diverse channels and media, simplifies the dispatch environment and addresses technical integration challenges to ensure next generation safety and response on Department of Defense posts, camps and stations.

    Download
  • Forecasting Cloud's Future

    Conversations with Federal, State, and Local Technology Leaders on Cloud-Driven Digital Transformation

    Download

When you download a report, your information may be shared with the underwriters of that document.