To confirm technologies and procedures during the Engineering and Manufacturing Development phase

Defense Acquisition: Employing Best Practices Can Shape Better Weapon
System Decisions (Testimony, 04/26/2000, GAO/T-NSIAD-00-137).

Pursuant to a congressional request, GAO discussed issues the Department
of Defense (DOD) faces in its acquisition of weapon systems.

GAO noted that: (1) the pressures of competing for the funds to launch
and sustain a weapon system program create incentives for starting
programs too early; overpromising performance capabilities; and
understating expected costs, schedules, and risks associated with
developing and producing the weapon; (2) leading commercial firms have
adopted a knowledge-based approach to developing new products that
embodies incentives that encourage realism, candor, and meeting product
expectations; (3) making sure that new technology is mature--that is,
demonstrated that it works--is the foundation for this approach; (4)
DOD's variances from best commercial practices result in higher costs,
compromised performance, and questionable cost benefit approaches; (5) a
knowledge-based approach can be used to reshape DOD's acquisition
process; (6) by itself, such a process will not produce better program
outcomes unless it influences the decisions made on individual weapon
systems; (7) if a knowledge-based acquisition process is put in place
and used, there are potential benefits that transcend individual program
outcomes; and (8) specifically, the ability to execute a program more
predictably within cost and schedule estimates would lessen the need to
offset cost increases by disrupting other modernization programs.

--------------------------- Indexing Terms -----------------------------

 REPORTNUM:  T-NSIAD-00-137
     TITLE:  Defense Acquisition: Employing Best Practices Can Shape
	     Better Weapon System Decisions
      DATE:  04/26/2000
   SUBJECT:  Weapons systems
	     Defense procurement
	     Defense cost control
	     Private sector practices
	     Procurement planning

******************************************************************
** This file contains an ASCII representation of the text of a  **
** GAO Testimony.                                               **
**                                                              **
** No attempt has been made to display graphic images, although **
** figure captions are reproduced.  Tables are included, but    **
** may not resemble those in the printed version.               **
**                                                              **
** Please see the PDF (Portable Document Format) file, when     **
** available, for a complete electronic file of the printed     **
** document's contents.                                         **
**                                                              **
******************************************************************

GAO/T-NSIAD-00-137

   * For Release on Delivery
     Expected at 10:00 a.m. EST

Wednesday,

April 26, 2000

GAO/T-NSIAD-00-137

DEFENSE ACQUISITION

Employing Best Practices Can Shape Better Weapon System Decisions

        Statement of David M. Walker, Comptroller General of the United
        States

Testimony

Before the Subcommittee on Readiness and Management Support, Committee on
Armed Services, U.S. Senate

United States General Accounting Office

GAO

Mr. Chairman and Members of the Subcommittee:

I am pleased to be here today to discuss issues the Department of Defense
(DOD) faces in its acquisition of weapon systems and the application of best
practices to help address those issues. With DOD's annual research,
development, and production spending for weapon systems at about $85
billion-coupled with suggestions from within DOD and the Congress that it
should be substantially higher-the Subcommittee's oversight of acquisition
policy can have a major impact on the value the taxpayer gets for that
expenditure.

After having done hundreds of reviews of major weapon systems over the last
20 years, we have seen many of the same problems recur-cost increases,
schedule delays, and performance problems. At your request, we have
undertaken a body of work that examines weapon acquisition issues from a
different, more cross-cutting perspective-one that draws lessons learned
from the best commercial product development efforts to see if they apply to
weapon system development. In the past few years, leading commercial firms
have developed increasingly sophisticated products in significantly less
time and at lower cost. These firms include the Boeing Commercial Airplane
Group, Bombardier Aerospace, Caterpillar, Chrysler Corporation, Ford Motor
Company, Hughes Space and Communications, Motorola Corporation, and 3M. Our
work shows that valuable lessons can be learned from the commercial sector
and can be applied to the development of weapon systems.

My testimony focuses on how best commercial practices help shape
improvements in the way DOD operates and makes decisions through the weapons
acquisition process. Specifically, I will discuss why weapon system
acquisition problems persist and how a process based on best practices can
lead to better weapon system outcomes. These issues have significant
implications for decisions made on individual programs and for DOD's larger
modernization goals, which I will also discuss.

Results in Brief

Persistent Problems Are Reinforced By the Current Process

The differences in the practices employed by the leading commercial firms
and DOD reflect the different demands imposed on programs by the
environments in which they are managed. Specific practices take root and are
sustained because they help a program succeed in its environment. The way
success and failure are defined for commercial and defense product
developments differs considerably, which creates a different set of
incentives and evokes different behaviors from the people managing the
programs. Attempts at reforming weapon acquisitions have not succeeded
because they did not change these incentives. All of the participants in the
acquisition process play a part in creating incentives. In this context, the
Office of the Secretary of Defense, the Congress, and the services'
organizations each play a critical role in getting the better outcomes
sought on major weapon system programs.

The basic management goal for a weapon system program in DOD is similar to
that of a major commercial undertaking: to develop and deliver a product
that meets the customer's needs. The product development efforts of leading
commercial firms are guided by achieving a sensible balance among customer
needs, product capabilities, and investment resources. However, the
pressures of successfully competing for the funds to start and sustain a DOD
acquisition program make for a much different business case. Compared with
commercial programs, the DOD environment encourages launching product
developments that embody more technical unknowns and less knowledge about
the performance and production risks they entail. A new weapon system is
encouraged to possess performance features that significantly distinguish it
from other systems. Consequently, DOD program managers have incentives to
promote performance features and design characteristics that rely on
immature technologies. These managers rely much more on maturing technology
during product development-when attention should be focused on design and
manufacturing-than do commercial programs.

Even though less information about a new product development is available at
the time of launch, the competition for funding requires detailed
projections to be made based on this information. Product development cannot
be launched unless a program's development and production cost, as well as
timing, coincides with available funding. Because DOD relies largely on
forecasts of cost, schedule, and performance that are comparatively soft,
competing for funds encourages the estimates to be squeezed into profiles of
available funding. Additional requirements, such as high reliability and
maintainability, serve to make the fit even tighter. As competition for
funding will continue throughout the program's development, success becomes
identified with the ability to secure the next installment. The risks
associated with developing new technologies and the product together-within
tight estimates-have come to be accepted as standard. Production realities
are usually too distant to have a curbing effect on early technology
decisions.

Rewards for discovering and recognizing potential problems early in a weapon
system development are few. Less available knowledge makes it harder for
program managers to say "no" to unreasonable expectations. Other factors,
such as short tenures and career pressures, discourage program managers from
saying no. In contrast with leading commercial firms, not having attained
knowledge-such as the demonstrated maturity of a technology or a key
manufacturing process-can be perceived as better than knowing that problems
exist. For these reasons, the practices associated with successful
commercial ventures are not readily adopted in DOD.

How a Knowledge-Based Process Can Lead to Better Weapons Acquisitions

On the other hand, product development in commercial ventures is a clearly
defined undertaking for which firms insist on having in hand the technology
that meets customers' needs before starting. The process of developing a
product culminates in delivery, and therefore, gives great weight to design
and production. The firms demand-and receive-specific knowledge about a new
product before production begins. A program does not go forward unless a
strong business case on which the program was originally justified continues
to hold true. This process is depicted in figure 1.

Figure 1: Knowledge-based Process for Applying Best Practices to the
Development of New Products

   * when a match is made between the customer's needs and the available
     technology;
   * when the product's design meets performance requirements,
   * and when the product can be produced within cost, schedule, and quality
     targets.

There is a synergy in this process, as the attainment of each successive
knowledge point builds on the preceding one. Metrics gauge when the
requisite level of knowledge has been attained. The result is a product
development process that delivers excellent results in a predictable manner.
On the other hand, in DOD's current acquisition process, technology, design,
and manufacturing knowledge is attained concurrently-in the higher cost
environment of product development-throughout the weapon system phases. This
process and the knowledge gathering that takes place is shown in figure 2.

We have reported our findings on best commercial practices to DOD, and have
made recommendations on how they can be applied to weapon systems. A listing
of these reports is included in the appendix. DOD has agreed with these
recommendations and they have been reflected in DOD's draft revision of its
regulations that guide the acquisition process. We have incorporated some of
the terms from the draft regulations in the acquisition process outlined in
the pages that follow.

Technology and Requirements Must Match Before a Program Can Be Launched

Other practices contribute to successfully developing technology. These are:

   * flexibility in both the resources provided and the product's
     performance requirements to allow for the uncertainties of technical
     progress;
   * disciplined paths for technology to take toward inclusion in products,
     with strong gatekeepers to decide when to allow it into a product
     development program;
   * high standards for judging the maturity and readiness of technology;
   * the imposition of strict product development cycle times; and
   * rules concerning how much innovation can be accepted on a product
     before the next generation must be launched.

Collectively, these factors create a healthy environment for developing
technology and making good decisions on what to include in a product.

These practices can be adapted to the front end of DOD's weapon acquisition
process. They can relieve the pressure to launch programs with immature
technologies and underestimated resources that are overmatched by
requirements. In fact, in some cases because of unique circumstances, DOD
has applied these practices with successful outcomes. For example, the
Defense Advanced Research Projects Agency matured a revolutionary periscope
technology to a very high maturity level before it was included on the
Virginia class attack submarine development program. To date, this new
technology has not caused cost and schedule problems for the submarine's
development program. More typically, however, DOD launches programs with
immature technologies and does not get a match between mature technology and
the weapon's requirements until late in product development. For example, we
found key technologies for the Army's Brilliant Antiarmor Submunition were
at very low maturity levels when the acquisition program was launched. This
was a significant gap between technology maturity and weapon system
requirements that was not closed until well into the program. Problems with
the technologies were a main contributor to the program's 88-percent cost
growth and 62-percent slip in schedule.

DOD's current acquisition process begins with a determination that a
military need must be met with a new weapon system. Different concepts are
explored to determine the best type of weapon system to meet the need. Once
the concept has been selected, DOD typically launches the development
program and assigns a program manager responsibility for formulating the
weapon system design and maturing essential technologies. DOD refers to this
phase as program definition and risk reduction. In applying best practices,
we have recommended that DOD not launch a program until the technologies
needed to meet a new weapon's requirements are mature. To separate this
technology development from the program, best practices suggest that a
technology and concept maturation phase follow concept exploration and
precede program launch. This and the concept exploration phase are
illustrated in figure 3.

Figure 3: Weapon Acquisition Phases That Should Precede Launching A New
Program

Once the concept phase has concluded with selecting a weapon system concept,
such as a satellite, to meet a need, the technology and concept maturation
phase would start. Practices included in this phase are:

   * assigning science and technology organizations responsibility for
     identifying key technologies and maturing them to acceptable levels;
   * developing a preliminary system design;
   * identifying enabling technologies and applying standards for managing
     and assessing their maturity using technology readiness levels;
   * using a hybrid organization or some other means to smoothly transition
     the responsibility for technologies from science and technology
     organizations to the program manager, and
   * establishing a baseline set of weapon system requirements, within
     reasonable cost and schedule expectations, that can be met with mature
     technologies.

These practices would separate technology development from an acquisition
program. They would also result in an established baseline requirement, a
best technical approach, and technologies matured to a high standard. This
would make it possible to achieve a match between performance requirements
and mature technologies-a key knowledge point-and would enable
decisionmakers to not launch a program if the match was not demonstrated.
Given a baseline design, decisionmakers could also defer unmet requirements
and the associated technologies to future upgrades, allowing technology
development to take place in a more flexible, failure-tolerant environment.
Collectively, these practices would curb incentives to overpromise the
capabilities of a new weapon system and to rely on immature technologies.
This would put the program manager in a better position to succeed once the
program is launched.

Product Development Should Deliver a Mature Design and Key Processes

   * a clear market (customer requirements) exists;
   * the firm possesses the technology and engineering capability to design
     the product and bring it to market on time;
   * the firm can afford the financial investment to develop the product;
     and
   * the product can be manufactured on time, at high quality, and at a cost
     that enables an acceptable return on investment.

If the firm does not have confidence in one or more of these elements, it
does not launch the product. On the other hand, if all the elements are
there, the manager of the product development effort is in a good position
to succeed. The technology needed for the product is proven, the firm has
evidence that it can develop and produce the product as scheduled, and the
venture can be afforded. Time constraints are an important element. If the
product development cannot be completed within a set schedule, competitors
might beat the firm to market and the cost of schedule overruns would hurt
profits. Time constraints are also used to limit the initial product's
requirements. Another, perhaps more basic, consideration limits the product
development schedule: the amount of time one can reasonably expect people to
drive to a goal like delivering a product. One commercial executive observed
that it is unreasonable to expect people to focus on a goal (such as
production start-up) that is 4 or more years away. To live within these time
constraints and keep innovation alive, commercial firms have adopted the
practice of saving requirements that cannot be met with proven technologies
for the next generation product.

In DOD's current acquisition process, the decision point that most closely
resembles commercial firms' product development phase is the engineering and
manufacturing development decision. Although the decision to launch the
weapon system acquisition program has already been made years earlier, the
engineering and manufacturing development phase represents DOD's effort to
design, test, and begin production of the new weapon. Its broad scope places
an enormous burden on the program manager. Technologies still must be
matured and integrated into a system, testing conducted, and production
processes readied. Although this range of activities can take 8 to 10 years,
the program manager is expected to develop and be accountable for a precise
cost and schedule estimate made at the decision to go forward with
development. Because of their short tenures, it normally takes several
program managers to complete this phase alone. Consequently, the program
manager that commits to the cost and schedule estimate during the program
definition and risk reduction phase is not the same person responsible for
achieving it.

Best practices can be applied to this scope of activities, starting with the
premise that key technologies are matured and matched to a basic set of user
requirements. Best practices suggest a product development phase consisting
of two elements-system integration and system demonstration-to be completed
in 5 years. Leading commercial firms commit to product development as a
single phase that combines both elements. About midway through product
development, they become certain of the product design's ability to meet
customer requirements, indicated by having 90 percent of engineering
drawings completed. In addition to this knowledge point, they also have all
key production processes under statistical control by the end of product
development-the third knowledge point. In applying best practices to DOD,
there may be more discovery-that is, more potential to find the
unexpected-when the individual components and technologies are brought
together as a whole. Thus, a distinct system integration effort makes sense.
Similarly, DOD programs may also have more discovery on production
processes, given the long time spans between programs. Therefore, key
production processes should be demonstrated by the end of system
demonstration, with full process control being achieved during low rate
initial production. Figure 4 illustrates a product development phase
incorporating best practices.

Figure 4: Product Development Phase to Deliver a Mature Design and Key
Processes

The system integration component of this phase is the first point in the
process that vests full responsibility in the program manager, but
recognizes that a fair amount of system-level discovery can still occur when
components and technologies are integrated into a complete weapon system.
Because of this, the program manager should not be held to precise cost and
schedule estimates to complete development and production because requisite
knowledge is absent. Therefore, system integration activity should be
accomplished in a flexible cost and contract environment. System integration
should be used by the program manager to gather the knowledge necessary to
build firm cost and schedule targets and to firm the weapon's design.
Activities included in this phase are:

   * program launch;
   * integration of all technologies into a complete system;
   * maturation of the system design;
   * completion of at least 90 percent of the system's engineering drawings;
   * development of knowledge-based cost and schedule estimates; and
   * finalization of the system's requirements.

System integration should conclude with a critical design review of
engineering drawings and confirmation that the system's design will meet
requirements-a key knowledge point. It should also result in firm cost and
schedule targets and a final set of requirements for the initial version of
the weapon. Decision-makers should insist on a mature design-supported with
completed engineering drawings-before proceeding to system demonstration.
Having such knowledge at this point in time contributes to product success.

For example, Boeing had released over 90 percent of the engineering drawings
for its 777-200 airplane half-way through its product development program.
This allowed Boeing to have near certainty that the design for the 777-200
airplane would meet requirements. On the other hand, DOD's C-17 program had
released only 56 percent of its engineering drawings at approximately the
same point in development. We estimate that the C-17 program did not release
over 90 percent of its initial drawings until more than three years later
and after 7 production aircraft had been delivered. The C-17 encountered
numerous technical problems in testing that resulted in redesigns, cost
increases, and schedule delays.

I have mentioned the value of setting time constraints for product
development. The leading commercial firms we have visited consciously
limited their development phases, ranging from 18 months to just over 4
years. We have identified a 5-year limit for weapon developments that
reflects both commercial practices and guidance provided by the Under
Secretary of Defense for Acquisition, Technology, and Logistics that weapons
should complete development and initial production within 7 years. This
would make it possible for DOD to extend the tenure for a single program
manager to the entire product development phase-providing the manager a more
realistic responsibility but with more accountability. Accordingly, the
system integration phase should last no longer than half-2.5 years-of the
product development phase.

System demonstration is the point in product development when the program
manager can legitimately commit to firm cost, schedule, and performance
targets within which to deliver the weapon system. In turn, the program
manager should secure a full funding commitment from DOD fund managers.
Included in system demonstration are:

   * building of weapon system prototypes;
   * test and evaluation of those prototypes; and
   * test and evaluation of the key processes used to build them.

Demonstrating the system should conclude with demonstrations of the product
in a realistic environment and of the key manufacturing processes used to
build the product. It should be completed no later than 5 years after the
start of system integration. We believe this activity would make the program
manager and contractor accountable for a low-risk product development phase
with precise cost and schedule targets. It would enable the use of a
tighter, less flexible contract that contains a pricing structure for which
the contractor can be held accountable. At its conclusion, decisionmakers
could make an accurate, informed decision to move from product development
into production.

Successful Product Development Allows a True Production Focus

In the current weapon acquisition process, DOD breaks production into two
phases-low-rate initial production and full-rate production. Ostensibly, the
purpose of low-rate production is to produce the minimum quantity of a new
weapon for operational tests and establishment of a production base that
allows for an orderly increase to full production rates. Typically, however,
development work is not concluded before low-rate production is started and
thus both are done concurrently. Consequently, significant cost increases
that cause schedule and other problems are not uncommon in early production,
and production processes are often not able to be statistically controlled
even when full-rate production begins.

Applying best commercial practices to the production of weapon systems could
relieve the pressures to complete product development concurrently with
low-rate production. If the product design and key manufacturing processes
are fully demonstrated in product development, the scope of low-rate
production can be reduced to building operational test articles and maturing
the production processes. Low-rate production could safely conclude when a
product has met user requirements in operational conditions and
manufacturing processes are under statistical process control-the third key
knowledge point.

Putting all of the pieces together, DOD could adopt a weapon acquisition
process that is knowledge-based and reflects best commercial practices, as
shown in figure 5.

This process centers on the separation of technology development from
product development and the accumulation of technology, design, and process
knowledge measured at key junctures. As I noted earlier, DOD is currently
rewriting its acquisition policy to reflect these concepts among many other
changes. It should be recognized that the acquisition offices preparing this
new policy cannot succeed without the agreement and cooperation of other key
communities that influence the acquisition process. These include the
organizations responsible for writing and approving weapon system
requirements, conducting science and technology, and programming
modernization funds.

Implications of a Knowledge-Based Acquisition Process for the Future

The principles embodied in policy have proven difficult to apply on
individual programs. Pressing circumstances are invoked as reasons for
making exceptions for the program at hand. Thus, on a case-by case basis,
decisions to approve programs despite knowledge shortfalls or their failure
to live up to advertised expectations can be rationalized in a number of
ways: an urgent threat needs to be met; a production capability needs to be
preserved; despite shortfalls, the new system is more capable than the one
it is replacing; the new system's problems will be fixed in the future. The
challenge for acquisition participants is to treat individual program
decisions as more than the case at hand. They must weigh and be accountable
for the broader implications of what is acceptable or "what will work" and
be willing to say no to programs that run counter to best practices. We have
made recommendations along these lines to the Secretary of Defense and the
Congress. Acting on them will be key to the success of an acquisition policy
anchored in best practices.

If DOD succeeds in applying knowledge-based practices to the weapons
acquisition process, the benefits transcend the management of individual
programs. Programs managed using best practices are more likely to be
executed within cost and schedule estimates. This will minimize surprises in
the form of cost and schedule increases that are often accommodated by
disrupting the funding of other programs. Without such disruptions, DOD can
better stabilize its overall modernization effort. Decisionmakers may be
more able to focus on a balanced investment strategy rather than continue
the practice of making decisions about how much each program's funding and
quantities should be reduced so that all programs can continue to be funded.

We also believe that the idea of a phased investment strategy for a
program-with contracts tailored for more clearly defined phases of work,
such as technology maturation, system integration, and system
demonstration-can leave the potential for competition open longer, both
across the defense and nondefense industries. Finally, limiting product
development cycle times to 5 years or less would allow for more frequent
assimilation of new technologies into weapon systems, thereby reducing
obsolescence and allowing better management of the industrial base. Reduced
cycle times, coupled with planned upgrades, could make for more frequent and
predictable work in production, where contractors and the industrial base
can profit.

Contact and Acknowledgments

Appendix I

Related GAO Products

Best Practices: Better Management of Technology Development Can Improve
Weapon System Outcomes (GAO/NSIAD-99-162, July 30, 1999).

Defense Acquisitions: Best Commercial Practices Can Improve Program Outcomes
(GAO/T-NSIAD-99-116, Mar. 17, 1999).

Defense Acquisition: Improved Program Outcomes Are Possible
(GAO/T-NSIAD-98-123, Mar. 17, 1998).

Best Practices: Successful Application to Weapon Acquisition Requires
Changes in DOD's Environment (GAO/NSIAD-98-56, Feb. 24, 1998).

Best Practices: Commercial Quality Assurance Practices Offer Improvements
for DOD (GAO/NSIAD-96-162, Aug. 26, 1996).

(707497)

        Orders by Internet

For information on how to access GAO reports on the Internet, send an e-mail
message with "info" in the body to:

[email protected]

or visit GAO's World Wide Web home page at:

http://www.gao.gov

        Web site: http://www.gao.gov/fraudnet/fraudnet.htm

E-mail: [email protected]

1-800-424-5454 (automated answering system)
  
*** End of document. ***

What is the engineering and manufacturing development phase?

What authorizes entry into the EMD phase?

Which defense acquisition system phase includes the development of the product support strategy and refinement of requirements and metrics?