U.S. Air Force
Project Maven Shows Need for Better Acquisition Policies, DOD Watchdog Finds
The Air Force's AI project moved fast. One of its lessons is the need for better acquisition policies and documentation.
Before the Defense Department leans into using artificial intelligence more broadly, it may have to update policies and standardize how it buys other emerging technologies.
The DOD's inspector general found that the lack of a formal documentation process for acquisition metrics and procedures could prevent future growth of Project Maven, which launched in 2017 with the aim of helping warfighters decipher information from images autonomously using machine learning and deep learning.
In a report publicly released Jan. 10, the inspector general wrote while the program, also referred to as the Algorithmic Warfare Cross-Functional Team, properly managed and monitored its contracts with inspections and meetings, the lack of formalized documentation could increase the risk of "lapses" in oversight particularly as project personnel change.
"Without formalized and documented processes, there is an increased risk of lapses occurring in the monitoring and management of the Project Maven contracts as the program grows and as project personnel change," the inspector general wrote. "In addition, as AI and machine learning programs expand within the DOD, future DOD acquisitions related to this complex, rapidly moving technology may not benefit from the monitoring and management lessons learned" by Project Maven's team.
The program had four contracts: three with ECS Federal LLC and one cooperative agreement with Carnegie Mellon University, according to the report. The cross functional team, "did not document its approach to contract oversight by formalizing, in writing, the reporting metrics and procedures for monitoring and managing" those contracts.
Instead, the team used provisional processes and procedures, as it "was learning how to work at the speed of commercial development within the DOD enterprise." That required "close monitoring and management techniques that are not captured in current procedures and best practices that are used by the DOD acquisitions community," including daily interactions with the contractor, in-depth weekly meetings and detailed research reviews.
The inspector general recommended the group submit formal documentation on strategy for managing, surveilling and monitoring the projects, contract monitoring procedures that go beyond typical acquisition requirements; financial reviews; interacting with contractors, reporting requirements, and necessarily surveillance needs, including training and personnel.
The inspector general's report also hinted at the need for acquisition officials to get ahead of setting policies and norms to manage programs like Project Maven as the Defense Department broadens its buying of AI and other emerging technologies, recommending the assistant secretary of defense for acquisition review Project Maven's acquisition processes and procedures to determine whether they "should be further formalized in acquisition policies, implemented by other programs and cross-functional teams, or incorporated into relevant curriculum and training."
In reply comments, Christopher O'Donnell, currently acting as assistant secretary of defense for acquisition, agreed with the recommendations and noted that DOD's existing Software Acquisition Pathway could provide a model for how to build documentation and accountability into highly iterative processes.