Arin Burgess / Military Review

Will Commanders Trust Their New AI Weapons and Tools?

A study of DOD’s artificial-intelligence efforts reveals a key gap.

As the Department of Defense races to develop AI-enabled tools and systems, there are outstanding questions about exactly where their investments are going, and what benefits and risks might result. One key unknown: will commanders and troops trust their new tools enough to make them worth the effort?

Drawing on publicly available budgetary data about the DOD science and technology program, the Center for Security and Emerging Technology, or CSET, examined the range of autonomy and AI-related research and development efforts advanced by the U.S. Army, Navy, Marines, Air Force, and DARPA. Among the main research lines are programs dedicated to increasing automation, autonomy, and AI capabilities in air, ground, surface, and undersea unmanned vehicles and systems; increasing the speed and accuracy of information processing and decision making; and increasing precision and lethality throughout the targeting processes. We recently released a two-part analysis of the scope and implications of these efforts. One of the most consistent themes is an emphasis on human-machine collaboration and teaming.

Indeed, while in the public imagination the integration of AI into military systems foretells a future where machines replace humans on the battlefield and wartime decisions are made without human input or control, in our assessment of U.S. military research on emerging technologies, humans remain very much in the loop. 

The U.S. Army’s flagship Next Generation Combat Vehicle research program is a good example of human-machine teaming cutting across different use cases and applications of autonomy and AI. One of its research activities develops AI and ML software and algorithms to “team with soldiers in support fully autonomous maneuver of NGCV and other autonomous systems, both physical and non-embodied.” This effort is meant to develop capabilities for NGCV that will “increase autonomy, unburdening the soldier operator, with a high degree of survivability and lethality in a highly contested environment.” Another applied NGCV research initiative looks to new “ML and reinforcement learning methods” to enable “joint human-intelligent agent decision making, optimizing the strengths of each in the decision process and creating an adaptive, agile team.” In other words, this research envisions intelligent technologies that interact, collaborate, and integrate with soldiers, optimizing performance for both humans and human-machine teams in different environments and missions.

DARPA’s applied research on “Artificial Intelligence and Human Machine Symbiosis,” however, takes the human-machine teaming concept a step further, aiming to develop technologies that “enable machines to function not only as tools that facilitate human action but as trusted partners to human operators.” 

Trust is central to human-machine teaming because it affects the willingness of humans to use autonomous and AI-enabled systems and accept their recommendations. CSET research, however, identifies important gaps in U.S. military research on the role of trust in human-machine teams and gaps in investment toward building trustworthy AI systems. In fact, the ‘Budgetary Assessment’ report which analyzes the 2020 U.S. military science and technology programs finds that only 18 out of the 789 research components related to autonomy and 11 out of the 287 research components related to AI mention the word “trust.” 

As the Department of Defense AI ethics principles dictate, humans are ultimately responsible for the development, use, and outcomes of AI systems in both combat and non-combat situations. If left unaddressed, gaps in research on the role of trust in human-machine teams could hinder the development and fielding of AI systems in operational environments. 

For instance, AI-enabled decision support technologies in particular are meant to help commanders process and assess more options for action and make better and faster decisions in complex and time-sensitive situations. Advances in real-time analytics and recommender systems technologies could help warfighters adapt and gain the initiative in dynamic operational settings. But if human operators don’t trust the system, they would be reluctant to follow its recommendations. Thus, without trust in human-machine teams, the U.S. military may not be able to capitalize on the advantages in speed and precision AI promises to deliver.

Understanding the role of trust in human-machine teams is also important for effective collaboration with allies. Some research suggests that trust in AI varies by country of origin. If commanders from some allied countries are less willing to trust and use AI-enabled systems during multinational operations, such divergence could undermine coordination, interoperability, and overall effectiveness.

Finally, in addition to research into human attitudes toward technology, progress toward advanced human-machine teaming will also depend on making the AI systems themselves trustworthy, in part by making them more transparent, explainable, auditable, reliable and resilient. 

The U.S. military has several S&T programs focused explicitly on increasing AI system robustness and resilience, strengthening security against deceptive and adversarial attacks, and developing systems that behave reliably in operational settings. DARPA’s “AI Next Campaign” is a prominent effort. Yet as previously noted, in our assessment, most of the research programs related to autonomy and AI don’t mention those safety and security attributes that are so pertinent to trustworthiness. Now, beyond hampering progress toward advanced human-machine teaming, our “Strategic Assessment” report also finds that failing to consistently invest in reliable, trustworthy, and resilient AI systems could increase the risk from miscalculations, misuse, and unintended escalation, especially in contested environments such as the South China Sea. 

Today’s research and development investments will set the course for the future of AI in national security. By directing more attention to understanding the role of trust in human-machine teams, the U.S. military could accelerate the development and fielding of intelligent agents and systems as trusted partners to human operators across different missions and environments. Furthermore, by prioritizing reliable and secure AI systems, the United States will be able to ensure long-term military, technological, and strategic advantages. 

For a deeper dive into our findings and recommendations, read: “U.S. Military Investments in Autonomy and AI: A Budgetary Assessment” and “U.S. Military Investments in Autonomy and AI: A Strategic Assessment.” 

X
This website uses cookies to enhance user experience and to analyze performance and traffic on our website. We also share information about your use of our site with our social media, advertising and analytics partners. Learn More / Do Not Sell My Personal Information
Accept Cookies
X
Cookie Preferences Cookie List

Do Not Sell My Personal Information

When you visit our website, we store cookies on your browser to collect information. The information collected might relate to you, your preferences or your device, and is mostly used to make the site work as you expect it to and to provide a more personalized web experience. However, you can choose not to allow certain types of cookies, which may impact your experience of the site and the services we are able to offer. Click on the different category headings to find out more and change our default settings according to your preference. You cannot opt-out of our First Party Strictly Necessary Cookies as they are deployed in order to ensure the proper functioning of our website (such as prompting the cookie banner and remembering your settings, to log into your account, to redirect you when you log out, etc.). For more information about the First and Third Party Cookies used please follow this link.

Allow All Cookies

Manage Consent Preferences

Strictly Necessary Cookies - Always Active

We do not allow you to opt-out of our certain cookies, as they are necessary to ensure the proper functioning of our website (such as prompting our cookie banner and remembering your privacy choices) and/or to monitor site performance. These cookies are not used in a way that constitutes a “sale” of your data under the CCPA. You can set your browser to block or alert you about these cookies, but some parts of the site will not work as intended if you do so. You can usually find these settings in the Options or Preferences menu of your browser. Visit www.allaboutcookies.org to learn more.

Sale of Personal Data, Targeting & Social Media Cookies

Under the California Consumer Privacy Act, you have the right to opt-out of the sale of your personal information to third parties. These cookies collect information for analytics and to personalize your experience with targeted ads. You may exercise your right to opt out of the sale of personal information by using this toggle switch. If you opt out we will not be able to offer you personalised ads and will not hand over your personal information to any third parties. Additionally, you may contact our legal department for further clarification about your rights as a California consumer by using this Exercise My Rights link

If you have enabled privacy controls on your browser (such as a plugin), we have to take that as a valid request to opt-out. Therefore we would not be able to track your activity through the web. This may affect our ability to personalize ads according to your preferences.

Targeting cookies may be set through our site by our advertising partners. They may be used by those companies to build a profile of your interests and show you relevant adverts on other sites. They do not store directly personal information, but are based on uniquely identifying your browser and internet device. If you do not allow these cookies, you will experience less targeted advertising.

Social media cookies are set by a range of social media services that we have added to the site to enable you to share our content with your friends and networks. They are capable of tracking your browser across other sites and building up a profile of your interests. This may impact the content and messages you see on other websites you visit. If you do not allow these cookies you may not be able to use or see these sharing tools.

If you want to opt out of all of our lead reports and lists, please submit a privacy request at our Do Not Sell page.

Save Settings
Cookie Preferences Cookie List

Cookie List

A cookie is a small piece of data (text file) that a website – when visited by a user – asks your browser to store on your device in order to remember information about you, such as your language preference or login information. Those cookies are set by us and called first-party cookies. We also use third-party cookies – which are cookies from a domain different than the domain of the website you are visiting – for our advertising and marketing efforts. More specifically, we use cookies and other tracking technologies for the following purposes:

Strictly Necessary Cookies

We do not allow you to opt-out of our certain cookies, as they are necessary to ensure the proper functioning of our website (such as prompting our cookie banner and remembering your privacy choices) and/or to monitor site performance. These cookies are not used in a way that constitutes a “sale” of your data under the CCPA. You can set your browser to block or alert you about these cookies, but some parts of the site will not work as intended if you do so. You can usually find these settings in the Options or Preferences menu of your browser. Visit www.allaboutcookies.org to learn more.

Functional Cookies

We do not allow you to opt-out of our certain cookies, as they are necessary to ensure the proper functioning of our website (such as prompting our cookie banner and remembering your privacy choices) and/or to monitor site performance. These cookies are not used in a way that constitutes a “sale” of your data under the CCPA. You can set your browser to block or alert you about these cookies, but some parts of the site will not work as intended if you do so. You can usually find these settings in the Options or Preferences menu of your browser. Visit www.allaboutcookies.org to learn more.

Performance Cookies

We do not allow you to opt-out of our certain cookies, as they are necessary to ensure the proper functioning of our website (such as prompting our cookie banner and remembering your privacy choices) and/or to monitor site performance. These cookies are not used in a way that constitutes a “sale” of your data under the CCPA. You can set your browser to block or alert you about these cookies, but some parts of the site will not work as intended if you do so. You can usually find these settings in the Options or Preferences menu of your browser. Visit www.allaboutcookies.org to learn more.

Sale of Personal Data

We also use cookies to personalize your experience on our websites, including by determining the most relevant content and advertisements to show you, and to monitor site traffic and performance, so that we may improve our websites and your experience. You may opt out of our use of such cookies (and the associated “sale” of your Personal Information) by using this toggle switch. You will still see some advertising, regardless of your selection. Because we do not track you across different devices, browsers and GEMG properties, your selection will take effect only on this browser, this device and this website.

Social Media Cookies

We also use cookies to personalize your experience on our websites, including by determining the most relevant content and advertisements to show you, and to monitor site traffic and performance, so that we may improve our websites and your experience. You may opt out of our use of such cookies (and the associated “sale” of your Personal Information) by using this toggle switch. You will still see some advertising, regardless of your selection. Because we do not track you across different devices, browsers and GEMG properties, your selection will take effect only on this browser, this device and this website.

Targeting Cookies

We also use cookies to personalize your experience on our websites, including by determining the most relevant content and advertisements to show you, and to monitor site traffic and performance, so that we may improve our websites and your experience. You may opt out of our use of such cookies (and the associated “sale” of your Personal Information) by using this toggle switch. You will still see some advertising, regardless of your selection. Because we do not track you across different devices, browsers and GEMG properties, your selection will take effect only on this browser, this device and this website.