Orbon Alija

How the Intelligence Community Can Get Better at Open Source Intel

Several factors make it harder to use publicly available information in all-source assessment than classified information.

During his House Appropriations Committee testimony on May 27, Chairman of the Joint Chiefs of Staff Gen. Mark Milley presciently stated that a country must master ubiquitous intelligence-surveillance-reconnaissance to win the next major war. Indeed, in an era in which the United States can no longer rely on economic or military overmatch to guarantee its security, we need a more intimate understanding of an adversary’s goals, intentions, capabilities, and actions to safeguard our national interests. The chairman’s challenge begs important questions. How do we ensure that our civilian and military leaders enjoy decision advantage in an increasingly contested environment? Channeling the mindset of Sun Tzu, how can we win before the first shot is fired—or prevail without any shots at all? 

Senior leaders consistently assert that the key to decision advantage in an ISR construct is timely acquisition and analysis of the best information. Historically, the analysis produced to meet intelligence requirements was based predominantly on data acquired by government collectors and government technologies. Unlike open data sources, intelligence officers can task sensitive intelligence sources and methods to target the specific people, places and events that drive the intelligence needs of our policymakers and commanders. 

But the world has changed. In today’s digital age, people and organizations carry out activities using technology that projects data about their background, actions, and preferences onto public platforms. Simultaneously, private companies have built business models around co-opting the two pursuits that animate most intelligence work—identifying and predicting personal behaviors. The combined result of these phenomena is a data ecosystem that couples extraordinary quantities of information with sophisticated processing tools to produce stunningly diverse insights—including insights relevant to the hardest intelligence problems. 

The ubiquity and accessibility of this public data disrupts the assumed superiority of the government’s proprietary intelligence sources of methods—a reality the national security community has been slow to recognize and accept. The open-source intelligence, or OSINT, derived from the vast pools of publicly/commercially available information, or PAI/CAI, in the public domain will not replace traditional intelligence—but at the very least it can enrich and enhance this data, and for many intelligence requirements it may be a better, safer and cheaper option of first resort. OSINT can also provide overwatch for areas and topics that fall outside the zone of active IC coverage. 

Unfortunately, we are poorly postured to realize these benefits because technology and industry have far outpaced the law, policy, and governance required to optimize the use of OSINT in the national security community. Current national security frameworks did not anticipate the explosion of internet-based PAI, the increased demand for public-private sector collaboration, or the wide availability of certain data aggregated and sold by private companies that the government could not legally collect on its own. We need an expedited policy response that generates the laws, policies, resources, and tradecraft required for the IC to confidently bring the full force of OSINT to bear on the serious threats we face, while ensuring that the acquisition and exploitation of PAI/CAI protects the privacy of U.S. persons and adheres to the ethics and practices of the intelligence profession. 

One of the most formidable obstacles to this end-state is cultural. Within the IC the production of OSINT is not regarded as a unique intelligence discipline but as research incident to all-source analysis or as a media production service. This approach was reasonable when open data sources consisted primarily of media and academic output. Now, however, given the volume and diversity of internet-derived PAI/CAI, OSINT more closely resembles SIGINT than library research. In the IC, the SIGINT discipline is a strong center of gravity—managing a diverse array of unique data types, each with its own unique collection, discovery, processing, exploitation, tradecraft, sharing and oversight processes—governed and championed by a dedicated agency. 

OSINT, on the other hand, remains a distributed activity that functions more like a collection of cottage industries. While OSINT has pockets of excellence, intelligence community OSINT production is largely initiative based, minimally integrated, and has little in the way of common guidance, standards, and tradecraft. The conspicuous absence of common standards, community coordination, and enterprise governance make PAI/CAI more difficult to use in all-source assessment than classified information produced by the IC’s more established disciplines. SIGINT, GEOINT and HUMINT reporting, for example, is well understood by analysts, has established tradecraft, and is supported by a deep bench of specialists who help interpret and evaluate the reporting. In the world of open source, the lack of common tradecraft and reach-back support to discipline specialists makes evaluating the accuracy and validity of PAI/CAI challenging, and all-source analysts may be understandably reluctant to use that reporting—substantially diminishing the value of a significant resource. The intelligence community must make OSINT a true intelligence discipline on par with the traditional functional disciplines, replete with leadership and authority that enables the OSINT enterprise to govern itself and establish a brand that instills faith and trust in open source information. Only then will OSINT have the advocacy, commitment, and structure to move from a cottage industry to the core discipline it must become.

A second significant obstacle is resources. OSINT activity is not funded at a level nearly commensurate with its value, though congressional intelligence oversight committees have pressed repeatedly for greater use of OSINT. Foreseeably, the absence of a dedicated OSINT advocate and champion results in severe underinvestment in open source information at a time that our national security posture demands a Manhattan Project mentality to harvesting the immense intelligence value of open source data. Instead, the nascent OSINT initiatives in each intelligence community element compete within their organization’s budgetary Thunderdomes, leading to predictably anemic outcomes. 

Finally, we need to update the antiquated data handling rules and policies designed to regulate older technologies and mitigate different risks. The astronomical volume of internet derived information, innovative machine analytics, and the proliferation of private commercial data providers present a unique opportunity for the intelligence community to achieve a scope of geographic and topical coverage unimaginable ten years ago. However, the lack of enterprise-wide legal and policy guidance impedes the widespread use of PAI/CAI, and increases the risk of inappropriate activity when the scant existing guidance is interpreted inconsistently by agencies across the community. The question of what constitutes an inherently governmental collection function, for example, directly influences the formation of partnerships between the intelligence community and commercial data providers. The private sector is witnessing a proliferation of commercial companies which not only provide unique data, but also bundle that data with analytic tools that enable new and striking insights. Maximizing the utility of OSINT requires the open source community to partner with industry in much the same way that the GEOINT community currently partners with commercial imagery providers. We need anticipatory policy that enables the IC to safely, effectively and ethically team with the private sector to evolve critical open source collection and analytic capabilities.

Operationalizing OSINT is crucial to the success of the intelligence community and to the security of our nation. Creating a foreign-intelligence OSINT capability for an era of peer adversaries and heightened global uncertainty will be challenging. Adjusting structure, policy, and resourcing will require a concerted effort across Congress, the Office of the Director of National Intelligence, and the Department of Defense. The intelligence community must act boldly to shift OSINT into a true intelligence discipline. Policy and oversight must keep pace with efforts to nurture the public-private partnerships that enable government to leverage the rapid technical advances occurring in the commercial and NGO sector. Resource programming must support these efforts and align budget expenditures to realize the tremendous and growing value of OSINT to the intelligence mission. 

The adversaries our nation faces are very real, very capable, and very dangerous. We can no longer afford complacency. Now is the time for audacious action to realize the Chairman’s vision of ubiquitous ISR, and aggressive exploitation of PAI/CAI will be the linchpin. The competition for decision advantage starts well before the first shot is fired and spans all the national instruments of power: diplomatic, information, military, and economic. Technology and data available in the public domain could close critical knowledge gaps now. Government must match and embrace these capabilities with commitment, policy, and resources. Failure to do so risks disastrous consequences. Our adversaries will capitalize on these opportunities and so must we – we are already behind. 

The views expressed are the authors’ and do not imply endorsement by the Office of the Director of National Intelligence or any other U.S. government agency. 

Bob Ashley is a former director of the Defense Intelligence Agency and now an adjunct senior fellow at the Center for a New American Security. 

Neil Wiley is former Principal Executive in the Office of the Director of National Intelligence and a former Chair of the National Intelligence Council.

X
This website uses cookies to enhance user experience and to analyze performance and traffic on our website. We also share information about your use of our site with our social media, advertising and analytics partners. Learn More / Do Not Sell My Personal Information
Accept Cookies
X
Cookie Preferences Cookie List

Do Not Sell My Personal Information

When you visit our website, we store cookies on your browser to collect information. The information collected might relate to you, your preferences or your device, and is mostly used to make the site work as you expect it to and to provide a more personalized web experience. However, you can choose not to allow certain types of cookies, which may impact your experience of the site and the services we are able to offer. Click on the different category headings to find out more and change our default settings according to your preference. You cannot opt-out of our First Party Strictly Necessary Cookies as they are deployed in order to ensure the proper functioning of our website (such as prompting the cookie banner and remembering your settings, to log into your account, to redirect you when you log out, etc.). For more information about the First and Third Party Cookies used please follow this link.

Allow All Cookies

Manage Consent Preferences

Strictly Necessary Cookies - Always Active

We do not allow you to opt-out of our certain cookies, as they are necessary to ensure the proper functioning of our website (such as prompting our cookie banner and remembering your privacy choices) and/or to monitor site performance. These cookies are not used in a way that constitutes a “sale” of your data under the CCPA. You can set your browser to block or alert you about these cookies, but some parts of the site will not work as intended if you do so. You can usually find these settings in the Options or Preferences menu of your browser. Visit www.allaboutcookies.org to learn more.

Sale of Personal Data, Targeting & Social Media Cookies

Under the California Consumer Privacy Act, you have the right to opt-out of the sale of your personal information to third parties. These cookies collect information for analytics and to personalize your experience with targeted ads. You may exercise your right to opt out of the sale of personal information by using this toggle switch. If you opt out we will not be able to offer you personalised ads and will not hand over your personal information to any third parties. Additionally, you may contact our legal department for further clarification about your rights as a California consumer by using this Exercise My Rights link

If you have enabled privacy controls on your browser (such as a plugin), we have to take that as a valid request to opt-out. Therefore we would not be able to track your activity through the web. This may affect our ability to personalize ads according to your preferences.

Targeting cookies may be set through our site by our advertising partners. They may be used by those companies to build a profile of your interests and show you relevant adverts on other sites. They do not store directly personal information, but are based on uniquely identifying your browser and internet device. If you do not allow these cookies, you will experience less targeted advertising.

Social media cookies are set by a range of social media services that we have added to the site to enable you to share our content with your friends and networks. They are capable of tracking your browser across other sites and building up a profile of your interests. This may impact the content and messages you see on other websites you visit. If you do not allow these cookies you may not be able to use or see these sharing tools.

If you want to opt out of all of our lead reports and lists, please submit a privacy request at our Do Not Sell page.

Save Settings
Cookie Preferences Cookie List

Cookie List

A cookie is a small piece of data (text file) that a website – when visited by a user – asks your browser to store on your device in order to remember information about you, such as your language preference or login information. Those cookies are set by us and called first-party cookies. We also use third-party cookies – which are cookies from a domain different than the domain of the website you are visiting – for our advertising and marketing efforts. More specifically, we use cookies and other tracking technologies for the following purposes:

Strictly Necessary Cookies

We do not allow you to opt-out of our certain cookies, as they are necessary to ensure the proper functioning of our website (such as prompting our cookie banner and remembering your privacy choices) and/or to monitor site performance. These cookies are not used in a way that constitutes a “sale” of your data under the CCPA. You can set your browser to block or alert you about these cookies, but some parts of the site will not work as intended if you do so. You can usually find these settings in the Options or Preferences menu of your browser. Visit www.allaboutcookies.org to learn more.

Functional Cookies

We do not allow you to opt-out of our certain cookies, as they are necessary to ensure the proper functioning of our website (such as prompting our cookie banner and remembering your privacy choices) and/or to monitor site performance. These cookies are not used in a way that constitutes a “sale” of your data under the CCPA. You can set your browser to block or alert you about these cookies, but some parts of the site will not work as intended if you do so. You can usually find these settings in the Options or Preferences menu of your browser. Visit www.allaboutcookies.org to learn more.

Performance Cookies

We do not allow you to opt-out of our certain cookies, as they are necessary to ensure the proper functioning of our website (such as prompting our cookie banner and remembering your privacy choices) and/or to monitor site performance. These cookies are not used in a way that constitutes a “sale” of your data under the CCPA. You can set your browser to block or alert you about these cookies, but some parts of the site will not work as intended if you do so. You can usually find these settings in the Options or Preferences menu of your browser. Visit www.allaboutcookies.org to learn more.

Sale of Personal Data

We also use cookies to personalize your experience on our websites, including by determining the most relevant content and advertisements to show you, and to monitor site traffic and performance, so that we may improve our websites and your experience. You may opt out of our use of such cookies (and the associated “sale” of your Personal Information) by using this toggle switch. You will still see some advertising, regardless of your selection. Because we do not track you across different devices, browsers and GEMG properties, your selection will take effect only on this browser, this device and this website.

Social Media Cookies

We also use cookies to personalize your experience on our websites, including by determining the most relevant content and advertisements to show you, and to monitor site traffic and performance, so that we may improve our websites and your experience. You may opt out of our use of such cookies (and the associated “sale” of your Personal Information) by using this toggle switch. You will still see some advertising, regardless of your selection. Because we do not track you across different devices, browsers and GEMG properties, your selection will take effect only on this browser, this device and this website.

Targeting Cookies

We also use cookies to personalize your experience on our websites, including by determining the most relevant content and advertisements to show you, and to monitor site traffic and performance, so that we may improve our websites and your experience. You may opt out of our use of such cookies (and the associated “sale” of your Personal Information) by using this toggle switch. You will still see some advertising, regardless of your selection. Because we do not track you across different devices, browsers and GEMG properties, your selection will take effect only on this browser, this device and this website.