The NSA’s Terrorist-Hunting Computer for Pakistan May Have Targeted Innocents

Pakistani men, who were displaced with their families from Pakistan's tribal areas due to fighting between the Taliban and the army, sit on a roadside on the outskirts of Islamabad, Pakistan, Saturday, Nov. 2, 2013.

Muhammed Muheisen/AP

AA Font size + Print

Pakistani men, who were displaced with their families from Pakistan's tribal areas due to fighting between the Taliban and the army, sit on a roadside on the outskirts of Islamabad, Pakistan, Saturday, Nov. 2, 2013.

A new report suggests that the agency has been using a machine-learning program to identify potential terrorists, but thousands of Pakistanis may have been mislabeled.

Artificially intelligent programs can learn to walk like we do, beat game shows and board games, and even drive cars. But are they ready to find and fight enemies for us?

A new report from Ars Technica suggests that the US National Security Agency (NSA) has been using a machine learning program to identify potential terrorists in Pakistan—but its methodology may have led to thousands of innocent Pakistanis being mislabeled.

The NSA’s program—inexplicably named SKYNET, like the homicidal AI program of the Terminator film franchise—was first unveiled by documents leaked by Edward Snowden to The Intercept in 2015. According to a leaked 2012 government Powerpoint presentation, SKYNET uses “analytic triage” to calculate the probability that individuals are terrorists, using an 80-point analytical test, that evaluates factors like a person’s phone calls, location, social media activity, and travel patterns.

The system apparently flagged Al-Jazeera’s Islamabad bureau chief Ahmad Zaidan as a potential target, the Intercept’s data showed, as he often travels to conflict areas to report.

In the leaked slides, NSA claimed that SKYNET has a false-positive rate of only 0.008%, in certain instances. But Pakistan has a population of about 182 million, and the NSA was using phone records from about 55 million people for SKYNET. As Ars points out, even a that minute rate, many innocent people are likely to end up mislabeled. Some of the NSA’s tests in the leaked slides saw error rates of 0.18%, which could mean mislabeling about 99,000 people out of the 55 million.

SKYNET can be compared to the machine learning systems that businesses use to find sales leads—both methods learn a person’s traits, and compares them to model profiles based on those traits. SKYNET was trained by feeding the system with the data from 100,000 random people, and seven known terrorists. It was then tested with the task of identifying one of those seven terrorists. What’s troubling is that SKYNET does not appear to have been tested with new data, which would have shown whether the system could work in new situations, according to an expert who examined the leaked slides for Ars.

“There are very few ‘known terrorists’ to use to train and test the model,” Patrick Ball, a data scientist and director of the Human Rights Data Analysis Group, explained to Ars Technica. “If they are using the same records to train the model as they are using to test the model, their assessment of the fit is completely bullshit.”

It’s not clear yet what purpose SKYNET serves. Although it could be part of non-violent surveillance activities, such as monitoring suspected terrorists’ travel patterns, Ars suggests the technology could potentially be used to target drone strikes. Since 2004, the US government has carried out hundreds of drone strikes in Pakistan against alleged terrorists, according to the Bureau of Investigative Journalism.

Last year, the UN warned against nations developing autonomous weapons, due to concerns about what they might do without a human’s moral judgement. The NSA was not immediately available to comment on how SKYNET was used, or how it was trained.

Close [ x ] More from DefenseOne
 
 

Thank you for subscribing to newsletters from DefenseOne.com.
We think these reports might interest you:

  • Ongoing Efforts in Veterans Health Care Modernization

    This report discusses the current state of veterans health care

    Download
  • Modernizing IT for Mission Success

    Surveying Federal and Defense Leaders on Priorities and Challenges at the Tactical Edge

    Download
  • Top 5 Findings: Security of Internet of Things To Be Mission-Critical

    As federal agencies increasingly leverage these capabilities, government security stakeholders now must manage and secure a growing number of devices, including those being used remotely at the “edge” of networks in a variety of locations. With such security concerns in mind, Government Business Council undertook an indepth research study of federal government leaders in January 2017. Here are five of the key takeaways below which, taken together, paint a portrait of a government that is increasingly cognizant and concerned for the future security of IoT.

    Download
  • Coordinating Incident Response on Posts, Camps and Stations

    Effective incident response on posts, camps, and stations is an increasingly complex challenge. An effective response calls for seamless conversations between multiple stakeholders on the base and beyond its borders with civilian law enforcement and emergency services personnel. This whitepaper discusses what a modern dispatch solution looks like -- one that brings together diverse channels and media, simplifies the dispatch environment and addresses technical integration challenges to ensure next generation safety and response on Department of Defense posts, camps and stations.

    Download
  • Forecasting Cloud's Future

    Conversations with Federal, State, and Local Technology Leaders on Cloud-Driven Digital Transformation

    Download

When you download a report, your information may be shared with the underwriters of that document.