Twitter CEO Jack Dorsey, accompanied by Facebook COO Sheryl Sandberg, testify before the Senate Intelligence Committee hearing on 'Foreign Influence Operations and Their Use of Social Media Platforms' on Capitol Hill, Wednesday, Sept. 5, 2018, in DC.

Twitter CEO Jack Dorsey, accompanied by Facebook COO Sheryl Sandberg, testify before the Senate Intelligence Committee hearing on 'Foreign Influence Operations and Their Use of Social Media Platforms' on Capitol Hill, Wednesday, Sept. 5, 2018, in DC. AP Photo/Jose Luis Magana

Social-Media Companies Are Scanning for Potential Terrorists — Islamic Ones, Anyway

Big platforms like Facebook others have come a long way in detecting and preventing the spread of Islamic extremist content and tracking potential Muslim terrorists. Why aren’t they doing more about other kinds?

Following the politically motivated shooting in Pittsburgh and the mailing of pipe bombs to political officials and journalists across the country, public outcry has risen against social-media companies. Suspected pipe-bomber Cesar Sayoc and Pittsburgh shooting suspect Robert Bowers used various platforms to post content indicating their potential for ideological violence. Some have asked why social-media companies didn’t do more, sooner, to stop the threat.

It’s a question that Facebook, Twitter, YouTube, and others have faced before, going back to 2014 when the problem was content from extremists of a different sort: violent jihadist groups such as ISIS.

Since then, many social-media giants have developed technological and policy-based ways to help prevent extremist content from proliferating across their sites and even to help law enforcement better track potential violent actors. But those efforts were aimed at foreign Islamic extremists, not domestic threats.

Monika Bickert, Facebook’s head of product policy and counterterrorism, has helped her company move farther along in this regard than some others. What role did Facebook play in the events that unfolded last week? A limited one: Robert Bowers, the charged Pittsburgh shooter, kept his threats and violent posts to a relatively obscure right-wing platform called Gab. Soon after he joined Gab in January, he began to post and spread images and content threatening to Jews.

Cesar Sayoc had a Facebook profile that he used to advance conspiracy theories. He also threatened people on Twitter, such as political analyst Rochelle Ritchie. Ritchie reported the threats to the platform. Last weekend, the company apologized for failing to act sooner.

Although Sayoc had a small presence on Facebook, the company might still have had a lot of information about him. Facebook monitors extremist rhetoric and content—of the Islamist variety—on sites that aren’t Facebook. It employs contractors to watch extremist chat rooms and other places so they can be ready to identify and tag threatening language, images, and content on Facebook.

Facebook’s Erin Marie Saltman, a policy manager at Facebook who oversees counterterrorism efforts in Europe, Africa, and the Middle East, disclosed this at the GLOBSEC security summit in May.

“There are a lot of people in other parts of the world that are not Facebook and not government,” Saltman said. “They are intel providers that sit and squat on a lot of these other sites and they tell us, in as close to real time as possible, when bad content is being released and so we know about it as soon as possible. So when the [Abū Bakr al-Baghdadi] speech was released a little while ago, and it wasn’t in video form, just audio, we were able to hash[tag] it before it started hitting our site.”

Predicting potentially violent behavior requires as much digitally collected data as possible, precisely the sort of data that intel vendors watching sites like Gab might notice. But when Defense One asked Facebook representatives whether they monitor sites like Gab for such content—or potential indicators of violence—they declined to say.

“As Erin mentioned, we work with intel and research firms who monitor many platforms, but we prefer not to disclose further details as bad actors actively work to circumvent our detection techniques,” a Facebook spokesperson said. “Since the bombing attempts, and the shooting in Pittsburgh, teams across our company have been monitoring developments in real time to understand both situations and how they relate to content on our site," they added.

In 2011, J. Reid Meloy, a forensic psychologist and consultant to FBI’s Behavioral Analysis Units at Quantico, identified eight behaviors that can predict lone-wolf attacks based on ideological extremism. Sayoc and Bowers exhibited several of them across multiple social media sites. If social-media companies could search for these subtle indicators of a potentially dangerous person, behaviors, such as fixation or obsession, in the context of overtly troubling posts and comments such as direct threats, patterns could emerge to predict an individual’s behavior.

Cross-platform analysis of individuals' data residue is what contemporary microtargeting for advertisements is based on. It works to predict whether a person might be open to a specific product pitch but it also works to predict potentially harmful behavior. Facebook is already using AI to spot suicidal tendencies signaled by text patterns. The same algorithms could be applied to spot violent extremism, as could network analysis and even semantic text analysis. That information, coupled with the identification of violent messages or threats spread on other sites, could go a long way toward predicting and preventing violent behavior and the posting of extremist content. And it is, but it’s mostly violent Islamic behavior and content.

Consider the case of Demetrius Nathaniel Pitts, a Cleveland man recently charged with plotting a jihadist-inspired terrorist attack. Authorities monitored Pitts's Facebook posts carefully after he commented on a photo of an al-Qaida training camp. His posts extorted Muslims to learn how to operate firearms, posts that law enforcement officials described as "disturbing" to USA Today. But pages urging non-Muslims (or people who are not explicitly Muslim) to own and practice with firearms are common on Facebook.

"We continually enforce our Community Standards through a combination of technology, reports from our community, and human review. This includes our hate speech policy that prohibits content that attacks people based on their race, ethnicity, national origin, religious affiliations, sexual orientation, caste, sex, gender, gender identity, and serious disease or disability," said the spokesperson.

Following the public outcry against the proliferation of jihadist extremist messaging, Facebook and other sites tried a technique called hashing: essentially, to mark Islamic extremist content as individuals tried to spread it from one site to another. In 2016, Facebook executives led an effort to share data on hashed images across platforms.

“It creates the equivalent of a digital fingerprint so you can know when these things are coming up. We encourage that type of sharing, the hash sharing. Anybody using types of video, photo matching, would be able to use the hashes we are trying to share,” said Saltman.

Could hashed images and data from accounts like Bowers’s and Sayoc’s be relevant to law enforcement? Potentially, but the practice of hash sharing doesn’t involve the government, said Saltman. Instead, she said, the goal was to make a “safe tech space” for technology platforms to use whatever tools they saw fit.

“This is a by-industry, for-industry effort; it doesn’t include government or NGOs. It’s really so we can create a safe space so that some of these smaller platforms that are really scared about talking outside of industry—and admitting you have a problem is step one—can come together in a safe tech space and start operationalizing around some of this.”

In a conversation with New York Times reporters on Sunday, Gab founder Andrew Torba denied that he or any Gab employer should monitor content on the site. “Twitter and other platforms police ‘hate speech’ as long as it isn’t against President Trump, white people, Christians, or minorities who have walked away from the Democratic Party,” he wrote. “This double standard does not exist on Gab.”

X
This website uses cookies to enhance user experience and to analyze performance and traffic on our website. We also share information about your use of our site with our social media, advertising and analytics partners. Learn More / Do Not Sell My Personal Information
Accept Cookies
X
Cookie Preferences Cookie List

Do Not Sell My Personal Information

When you visit our website, we store cookies on your browser to collect information. The information collected might relate to you, your preferences or your device, and is mostly used to make the site work as you expect it to and to provide a more personalized web experience. However, you can choose not to allow certain types of cookies, which may impact your experience of the site and the services we are able to offer. Click on the different category headings to find out more and change our default settings according to your preference. You cannot opt-out of our First Party Strictly Necessary Cookies as they are deployed in order to ensure the proper functioning of our website (such as prompting the cookie banner and remembering your settings, to log into your account, to redirect you when you log out, etc.). For more information about the First and Third Party Cookies used please follow this link.

Allow All Cookies

Manage Consent Preferences

Strictly Necessary Cookies - Always Active

We do not allow you to opt-out of our certain cookies, as they are necessary to ensure the proper functioning of our website (such as prompting our cookie banner and remembering your privacy choices) and/or to monitor site performance. These cookies are not used in a way that constitutes a “sale” of your data under the CCPA. You can set your browser to block or alert you about these cookies, but some parts of the site will not work as intended if you do so. You can usually find these settings in the Options or Preferences menu of your browser. Visit www.allaboutcookies.org to learn more.

Sale of Personal Data, Targeting & Social Media Cookies

Under the California Consumer Privacy Act, you have the right to opt-out of the sale of your personal information to third parties. These cookies collect information for analytics and to personalize your experience with targeted ads. You may exercise your right to opt out of the sale of personal information by using this toggle switch. If you opt out we will not be able to offer you personalised ads and will not hand over your personal information to any third parties. Additionally, you may contact our legal department for further clarification about your rights as a California consumer by using this Exercise My Rights link

If you have enabled privacy controls on your browser (such as a plugin), we have to take that as a valid request to opt-out. Therefore we would not be able to track your activity through the web. This may affect our ability to personalize ads according to your preferences.

Targeting cookies may be set through our site by our advertising partners. They may be used by those companies to build a profile of your interests and show you relevant adverts on other sites. They do not store directly personal information, but are based on uniquely identifying your browser and internet device. If you do not allow these cookies, you will experience less targeted advertising.

Social media cookies are set by a range of social media services that we have added to the site to enable you to share our content with your friends and networks. They are capable of tracking your browser across other sites and building up a profile of your interests. This may impact the content and messages you see on other websites you visit. If you do not allow these cookies you may not be able to use or see these sharing tools.

If you want to opt out of all of our lead reports and lists, please submit a privacy request at our Do Not Sell page.

Save Settings
Cookie Preferences Cookie List

Cookie List

A cookie is a small piece of data (text file) that a website – when visited by a user – asks your browser to store on your device in order to remember information about you, such as your language preference or login information. Those cookies are set by us and called first-party cookies. We also use third-party cookies – which are cookies from a domain different than the domain of the website you are visiting – for our advertising and marketing efforts. More specifically, we use cookies and other tracking technologies for the following purposes:

Strictly Necessary Cookies

We do not allow you to opt-out of our certain cookies, as they are necessary to ensure the proper functioning of our website (such as prompting our cookie banner and remembering your privacy choices) and/or to monitor site performance. These cookies are not used in a way that constitutes a “sale” of your data under the CCPA. You can set your browser to block or alert you about these cookies, but some parts of the site will not work as intended if you do so. You can usually find these settings in the Options or Preferences menu of your browser. Visit www.allaboutcookies.org to learn more.

Functional Cookies

We do not allow you to opt-out of our certain cookies, as they are necessary to ensure the proper functioning of our website (such as prompting our cookie banner and remembering your privacy choices) and/or to monitor site performance. These cookies are not used in a way that constitutes a “sale” of your data under the CCPA. You can set your browser to block or alert you about these cookies, but some parts of the site will not work as intended if you do so. You can usually find these settings in the Options or Preferences menu of your browser. Visit www.allaboutcookies.org to learn more.

Performance Cookies

We do not allow you to opt-out of our certain cookies, as they are necessary to ensure the proper functioning of our website (such as prompting our cookie banner and remembering your privacy choices) and/or to monitor site performance. These cookies are not used in a way that constitutes a “sale” of your data under the CCPA. You can set your browser to block or alert you about these cookies, but some parts of the site will not work as intended if you do so. You can usually find these settings in the Options or Preferences menu of your browser. Visit www.allaboutcookies.org to learn more.

Sale of Personal Data

We also use cookies to personalize your experience on our websites, including by determining the most relevant content and advertisements to show you, and to monitor site traffic and performance, so that we may improve our websites and your experience. You may opt out of our use of such cookies (and the associated “sale” of your Personal Information) by using this toggle switch. You will still see some advertising, regardless of your selection. Because we do not track you across different devices, browsers and GEMG properties, your selection will take effect only on this browser, this device and this website.

Social Media Cookies

We also use cookies to personalize your experience on our websites, including by determining the most relevant content and advertisements to show you, and to monitor site traffic and performance, so that we may improve our websites and your experience. You may opt out of our use of such cookies (and the associated “sale” of your Personal Information) by using this toggle switch. You will still see some advertising, regardless of your selection. Because we do not track you across different devices, browsers and GEMG properties, your selection will take effect only on this browser, this device and this website.

Targeting Cookies

We also use cookies to personalize your experience on our websites, including by determining the most relevant content and advertisements to show you, and to monitor site traffic and performance, so that we may improve our websites and your experience. You may opt out of our use of such cookies (and the associated “sale” of your Personal Information) by using this toggle switch. You will still see some advertising, regardless of your selection. Because we do not track you across different devices, browsers and GEMG properties, your selection will take effect only on this browser, this device and this website.