Which document supports the defense acquisition system?

Defense Acquisitions: DOD's Revised Policy Emphasizes Best Practices, but More Controls Are Needed (10-NOV-03, GAO-04-53). The Department of Defense's (DOD) investment in new weapon systems is expected to exceed $1 trillion from fiscal years 2003 to 2009. To reduce the risk of cost and schedule overruns, DOD revamped its acquisition policy in May 2003. The policy provides detailed guidance on how weapon systems acquisitions should be managed. The Senate report accompanying the National Defense Authorization Act for Fiscal Year 2004 required GAO to determine whether DOD's policy supports knowledge-based, evolutionary acquisitions and whether the policy provides the necessary controls for DOD to ensure successful outcomes, such as meeting cost and schedule goals. The report also required GAO to assess whether the policy is responsive to certain requirements in the Bob Stump National Defense Authorization Act for Fiscal Year 2003 concerning DOD's management of the acquisition process. -------------------------Indexing Terms------------------------- REPORTNUM: GAO-04-53 ACCNO: A08843 TITLE: Defense Acquisitions: DOD's Revised Policy Emphasizes Best Practices, but More Controls Are Needed DATE: 11/10/2003 SUBJECT: Best practices Decision making Defense procurement Internal controls Procurement policy Weapons research and development Policy evaluation Reporting requirements ****************************************************************** ** This file contains an ASCII representation of the text of a ** ** GAO Product. ** ** ** ** No attempt has been made to display graphic images, although ** ** figure captions are reproduced. Tables are included, but ** ** may not resemble those in the printed version. ** ** ** ** Please see the PDF (Portable Document Format) file, when ** ** available, for a complete electronic file of the printed ** ** document's contents. ** ** ** ****************************************************************** GAO-04-53 United States General Accounting Office GAO Report to the Senate and House Committees on Armed Services November 2003 DEFENSE ACQUISITIONS DOD's Revised Policy Emphasizes Best Practices, but More Controls Are Needed GAO-04-53 November 2003 DEFENSE ACQUISITIONS DOD's Revised Policy Emphasizes Best Practices, but More Controls Are Needed DOD's new policy supports knowledge-based, evolutionary acquisitions by adopting lessons learned from successful commercial companies. One of those lessons is a knowledge-based approach, which requires program managers to attain the right knowledge at critical junctures-also known as knowledge points-so they can make informed investment decisions throughout the acquisition process. The policy also embraces an evolutionary or phased development approach, which sets up a more manageable environment for attaining knowledge. The customer may not get the ultimate capability right away, but the product is available sooner and at a lower cost. Leading firms have used these approaches-which form the backbone of what GAO calls the best practices model-to determine whether a project can be accomplished with the time and money available and to reduce risks before moving a product to the next stage of development. By adopting best practices in the acquisition policy, DOD's leadership has taken a significant step forward. The next step is to provide the necessary controls to ensure a knowledge-based, evolutionary approach. Implementing the necessary controls at all three knowledge points along the acquisition process helps decision makers ensure a knowledge-based approach is followed. Without controls in the form of measurable criteria that decision makers must consider, DOD runs the risk of making decisions based on overly optimistic assumptions. Each successive knowledge point builds on the preceding one, and having clearly established controls helps decision makers gauge progress in meeting goals and ensuring successful outcomes. DOD Policy Incorporates Best Practices but Does Not Have Sufficient Controls Best practices' intent in Sufficient controls in Knowledge point DOD policy? DOD policy? Highlights of GAO-04-53, a report to the Senate and House Committees on Armed Services The Department of Defense's (DOD) investment in new weapon systems is expected to exceed $1 trillion from fiscal years 2003 to 2009. To reduce the risk of cost and schedule overruns, DOD revamped its acquisition policy in May 2003. The policy provides detailed guidance on how weapon systems acquisitions should be managed. The Senate report accompanying the National Defense Authorization Act for Fiscal Year 2004 required GAO to determine whether DOD's policy supports knowledge-based, evolutionary acquisitions and whether the policy provides the necessary controls for DOD to ensure successful outcomes, such as meeting cost and schedule goals. The report also required GAO to assess whether the policy is responsive to certain requirements in the Bob Stump National Defense Authorization Act for Fiscal Year 2003 concerning DOD's management of the acquisition process. GAO recommends that the Secretary of Defense strengthen DOD's acquisition policy by requiring additional controls to ensure decision makers will follow a knowledge-based, evolutionary approach. DOD partially concurred with GAO's recommendations. Knowledge point 1 (at program Yes No launch): Technologies, time, funding, and other resources match customer needs Knowledge point 2 (between integration and Yes No demonstration): Design performs as expected Knowledge point 3 Yes No (at production commitment): Production meets cost, schedule, and quality targets Sources: DOD (data), GAO (analysis). DOD was responsive to the requirements in the Bob Stump National Defense Authorization Act for Fiscal Year 2003. DOD's responses reflected the www.gao.gov/cgi-bin/getrpt?GAO-04-53. committee's specific concerns about the application of certain statutory and To view the full product, including the scope regulatory requirements to the new evolutionary acquisition process, for and methodology, click on the link above. more guidance for implementing spiral development, and about technologyFor more information, contact Katherine V. readiness at program initiations. Schinasi at (202) 512-4841 or [email protected] Contents Letter Results in Brief Background DOD's Revised Policy Provides a Framework for Knowledge-Based, Evolutionary Acquisitions Controls Are Needed to Ensure Knowledge-Based Approach Is Followed DOD Is Responsive to Congressional Requirements about New Acquisition Process Conclusions Recommendations for Executive Action Agency Comments and Our Evaluation Scope and Methodology 1 2 3 5 8 10 14 14 15 16 Appendix I Section 802, Evolutionary Acquisition Appendix II Section 803, Spiral Development Appendix III Section 804, Technology Maturity, and Section 822, Independent Technology Readiness Assessments Appendix IV Comments from the Department of Defense Related GAO Products Tables Table 1: Excerpts of Best Practices Contained in DOD's New Policy 6 Table 2: Comparison of Controls Used in Best Practices Model and DOD Policy 8 Table 3: How DOD Policy Addresses Section 802's Statutory and Regulatory Requirements 19 Table 4: How DOD Responded to Section 802's Requirements Regarding Specific Matters for Each Increment of an Evolutionary Acquisition Process 21 Table 5: How DOD Policy Addresses Section 803's Requirements 24 Table 6: How DOD Responded to Section 804 and Section 822 Requirements 26 Figures Figure 1: Distribution of Development Funds for Fiscal Year 2003 5 Figure 2: Comparison of DOD's Framework and Commercial Best Practices Model 7 Abbreviations DOD Department of Defense DODD DOD Directive DODI DOD Instruction MDA Milestone Decision Authority TEMP Test and Evaluation Master Plan This is a work of the U.S. government and is not subject to copyright protection in the United States. It may be reproduced and distributed in its entirety without further permission from GAO. However, because this work may contain copyrighted images or other material, permission from the copyright holder may be necessary if you wish to reproduce this material separately. United States General Accounting Office Washington, DC 20548 November 10, 2003 The Honorable John W. Warner Chairman The Honorable Carl Levin Ranking Minority Member Committee on Armed Services United States Senate The Honorable Duncan Hunter Chairman The Honorable Ike Skelton Ranking Minority Member Committee on Armed Services House of Representatives The Department of Defense (DOD) plans to spend more than $1 trillion from fiscal years 2003 to 2009 for developing and procuring weapon systems. To get the most out of its investment, DOD has set goals to develop and procure weapons faster and at less cost. Guiding this effort is DOD's newly revised acquisition policy. Issued in May 2003, the policy seeks to foster greater efficiency while building flexibility in the acquisition process. The policy embraces a knowledge-based, evolutionary framework that emphasizes shorter development times. DOD intends to use its policy to improve its record for meeting cost and schedule estimates and for delivering capabilities as promised. Congress has expressed support for DOD's efforts to revamp its acquisition process and has established requirements that DOD must take to ensure a disciplined approach. Recognizing that a consistent and disciplined application of policies will be key to achieving desired outcomes, the Senate Armed Services Committee directed us to assess DOD's current acquisition policies to determine whether they (1) support knowledge-based, evolutionary acquisitions, (2) provide the necessary controls to ensure DOD's policy intent is followed, and (3) respond to specific requirements in the Bob Stump National Defense Authorization Results in Brief Act for Fiscal Year 2003 (Public Law 107-314) regarding DOD plans to manage its new acquisition process.1 DOD has made major improvements to its acquisition policy by adopting knowledge-based, evolutionary practices used by successful commercial companies. If properly applied, these best practices can put DOD's decision makers in a better position to deliver high-quality products on time and within budget. The policy requires decision makers to have the knowledge they need before moving to the next phase of development. To ensure that the acquisition environment is conducive to a knowledge-based approach, the policy embraces evolutionary development, which allows managers to develop a product in increments rather than trying to incorporate all the desired capabilities in the first version that comes off the production line. Leading companies who use the knowledge-based evolutionary approach have shown that these best practices help reduce risks at critical junctures during a product's development and help ensure that decision makers get the most out of their investments. DOD's recent policy changes are a significant step forward. The next step is for DOD to provide the necessary controls to ensure a knowledge-based, evolutionary approach is followed. For example, the policy does not establish measures to gauge design and manufacturing knowledge at critical junctures in the product development process. Without specific requirements to demonstrate knowledge at key points, the policy allows significant unknowns to be judged as acceptable risks, leaving an opening for decision makers to make uninformed decisions about continuing product development. DOD was responsive to the requirements in the Defense Authorization Act for Fiscal Year 2003. DOD's responses reflected the committee's specific concerns about the application of certain statutory and regulatory requirements2 to the new evolutionary acquisition process, for more guidance for implementing spiral development, and about technology readiness at program initiations. 1 S. Rept. No. 108-46, Report of the Senate Armed Services Committee accompanying S. 1050, National Defense Authorization Act for Fiscal Year 2004, at page 346 (May 13, 2003). 2 As used here, the term "regulatory requirements" refers to policies governing DOD's acquisition system. Background This report makes recommendations that the Secretary of Defense strengthen DOD's acquisition policy by requiring additional controls to ensure decision makers will follow a knowledge-based, evolutionary approach. DOD partially concurred with our recommendations. DOD believes the current acquisition framework includes the controls necessary to achieve effective results, but department officials will continue to monitor the process to determine whether other controls are needed to achieve the best possible outcomes. DOD agreed it should record and justify program decisions for moving from one stage of development to next but did not agree with the need to issue a report outside of the department. Traditionally, DOD's strategy for acquiring major weapon systems has been to plan programs that would achieve a big leap forward in capability. However, because the needed technologies often are not yet mature, programs stay in development for years until the technologies are demonstrated. As a result, weapon systems have frequently been characterized by poor cost, schedule, and performance outcomes. This has slowed modernization efforts, reduced the buying power of the defense dollar, delayed capabilities for the warfighter, and forced unplanned-and possibly unnecessary-trade-offs among programs. Our extensive body of work shows that leading companies use a product development model that helps reduce risks and increase knowledge when developing new products. This best practices model enables decision makers to be reasonably certain about their products at critical junctures during development and helps them make informed investment decisions. This knowledge-based process can be broken down into three cumulative knowledge points. o Knowledge point 1: A match must be made between the customer's needs and the developer's available resources-technology, engineering knowledge, time, and funding-before a program starts. o Knowledge point 2: The product's design must be stable and must meet performance requirements before initial manufacturing begins. o Knowledge point 3: The product must be producible within cost, schedule, and quality targets and demonstrated to be reliable before production begins. To bolster the knowledge-based process, leading companies use evolutionary product development, an incremental approach that enables developers to rely more on available resources rather than making promises about unproven technologies. While the user may not initially receive the ultimate capability under this approach, the initial product is available sooner and at a lower, more predictable cost. Also, leading companies know that invention cannot be scheduled and its cost is difficult to estimate. They do not bring technology into new product development unless that technology has been demonstrated to meet the user's requirements. Allowing technology development to spill over into product development puts an extra burden on decision makers and provides a weak foundation for making product development estimates. DOD understands that it must improve acquisition process outcomes if it is to modernize its forces within currently projected resources. To help achieve this goal, DOD has revised its acquisition policy, called the 5000 series, to reflect best practices from successful commercial and DOD programs.3 The policy covers most-but not all-major acquisitions. The Secretary of Defense has delegated authority to the Missile Defense Agency and to the National Security Space Team to develop separate guidance for missile defense and space systems, respectively.4 Approximately 35 percent of DOD's development funds in 2003 went to these systems. (Figure 1 shows how $43.1 billion in development funds were distributed across space, missile defense, and systems covered by the 5000 series.) This report addresses policy for the defense programs covered exclusively under the 5000 series. 3 DOD Directive 5000.1, The Defense Acquisition System, describes the management principles for DOD's acquisition programs. DOD Instruction 5000.2, The Operation of the Defense Acquisition System, outlines a framework for managing acquisition programs. Collectively, these are known as the 5000 series. 4 Program elements of the ballistic missile defense program enter the formal DOD acquisition cycle at milestone C (production commitment) and are subject to the 5000 series from that point on. To use the streamlined process of the National Security Space Acquisition Policy, a Space System Program Director/Program Manager must request that the DOD Space Milestone Decision Authority (the Under Secretary of the Air Force) grant a waiver and an exemption to the processes and procedures described in DOD Instruction 5000.2 (the waiver authority does not include DOD Directive 5000.1). Notwithstanding these policy exceptions, statutory requirements for major defense acquisition programs continue to apply to missile defense and space programs. Figure 1: Distribution of Development Funds for Fiscal Year 2003 Note: Includes research, development, test, and evaluation budget activities for advanced component development and prototypes; system development and demonstration; and operational systems development. DOD's leaders have made significant improvements to DOD's acquisition policy by adopting the knowledge-based, evolutionary approach used by leading commercial companies. The revised policy has the potential to transform DOD's acquisition process by reducing risks and increasing the chances for successful outcomes. The policy provides a framework for developers to ask themselves at key decision points whether they have the knowledge they need to move to the next phase of acquisition. If rigorously applied, this knowledge-based framework can help managers gain the confidence they need to make significant and sound investment decisions for major weapon systems. In placing greater emphasis on evolutionary product development, the policy sets up a more manageable environment for achieving knowledge. Another best practice reflected in the policy's framework is separating technology development from product development, which reduces technological risk at the start of a program. DOD's Revised Policy Provides a Framework for Knowledge-Based, Evolutionary Acquisitions As shown in table 1, DOD's policy emphasizes best practices used by leading companies. Table 1: Excerpts of Best Practices Contained in DOD's New Policy Best practices DOD policy Knowledge-based acquisition DOD Directive 5000.1 (enclosure 1, p. 5): Program managers "shall provide knowledge about key aspects of a system at key points in the acquisition process. ... shall reduce technology risk, demonstrate technologies in a relevant environment ... prior to program initiation. They shall reduce integration risk and demonstrate product design prior to the design readiness review. They shall reduce manufacturing risk and demonstrate producibility prior to full-rate production." Evolutionary acquisition DOD Instruction 5000.2 (p. 3): "Evolutionary acquisition is the preferred DOD strategy for rapid acquisition of mature technology for the user. ... The objective is to balance needs and available capability with resources, and to put capability into the hands of the user quickly. The success of the strategy depends on consistent and continuous definition of requirements, and the maturation of technologies ... that provide increasing capability towards a materiel concept." Separating technology development from DOD Instruction 5000.2 (p. 6): "The project shall exit Technology Development when an product development affordable increment of militarily-useful capability has been identified, the technology for that increment has been demonstrated in a relevant environment, and a system can be developed for production within a short timeframe (normally less than five years)." Source: DOD. Similar to the best practices model, DOD's policy divides its acquisition process into phases, as shown in figure 2. Key decisions are aligned with the three critical junctures of a product's development, or knowledge points. Figure 2: Comparison of DOD's Framework and Commercial Best Practices Model In other similarities, DOD's framework pinpoints program start at milestone B, about the same point as program start on the best practices model. At the midway point on both approaches, a stable product design should be demonstrated. With DOD's framework, managers are required to know-by the time full-rate production decision review occurs-whether the product can be produced within cost, schedule, and quality targets. This requirement occurs earlier in the best practices model, before production begins, or at knowledge point 3. Leading companies have used this approach to reduce risks and to make costs and delivery dates more predictable. Controls Are Needed to Ensure Knowledge-Based Approach Is Followed While DOD has strengthened its acquisition policy with a knowledge-based, evolutionary framework, the policy does not include many of the same controls that leading companies rely on to attain a high level of knowledge before making additional significant investments. Controls are considered effective if they are backed by measurable criteria and if decision makers are required to consider them before deciding to advance a program to the next level. Controls used by leading companies help decision makers gauge progress in meeting cost, schedule, and performance goals and ensure that managers will (1) conduct activities to capture relevant product development knowledge, (2) provide evidence that knowledge was captured, and (3) hold decision reviews to determine that appropriate knowledge was captured to move to the next phase. To determine if DOD has the necessary controls, we compared controls in DOD's policy with those used in the best practices model at three critical junctures. Table 2 shows the presence or absence of controls for various versions of DOD policy since 1996, including the May 2003 revision. Table 2: Comparison of Controls Used in Best Practices Model and DOD Policy Controls used in DOD policy March 1996 October 2000 October 2002 May 2003 Controls used in commercial best practices model policy policy interim policy policy Knowledge point 1: Occurs at program launch. Match exists between requirements and resources. Technologies needed to meet essential product requirements have been demonstrated to work in their intended environment and the producer has completed a preliminary design of the product. Demonstrate technologies to high readiness levels X X X Ensure that requirements for the product are informed by the systems engineering process X Establish cost and schedule estimates for product based on knowledge from preliminary design using systems engineering tools Conduct decision review for program launch X X X X Knowledge point 2: Occurs between integration and demonstration. Design is stable and has been demonstrated through prototype testing. Ninety percent of engineering drawings are releasable to manufacturing organizations. Complete 90 percent of design drawings Complete subsystem and system design reviews Demonstrate with prototype that design meets requirements X X X Obtain stakeholders concurrence that drawings are complete and producible Complete failure modes and effects analysis Identify key system characteristics Controls used in DOD policy March 1996 October 2000 October 2002 May 2003 policy policy interim policy policy Controls used in commercial best practices model Identify critical manufacturing processes Establish reliability targets and growth plan based on demonstrated reliability rates of components and subsystems Conduct decision review to enter system demonstration X X Knowledge point 3: Occurs at production commitment. Product is ready to be manufactured within cost, schedule, and quality targets. All key manufacturing processes have come under statistical control and product reliability has been demonstrated. Demonstrate manufacturing processes Build production representative prototypes Test production representative prototypes to achieve reliability goal Test production representative prototypes to demonstrate product in operational environment Collect statistical process control data Demonstrate critical processes capable and in statistical control Conduct decision review to begin production X X X X Sources: GAO and DOD. At all three knowledge points, DOD's policy does not provide all the necessary controls used by commercial companies. For example, at program launch (milestone B) or when knowledge point 1 should be reached, the policy requires decision makers to identify and validate a weapon system's key performance requirements and to have a technical solution for the system before program start. This information is then used to form cost and schedule estimates for the product's development. However, the policy does not emphasize the use of a disciplined systems engineering process for balancing a customer's needs with resources to deliver a preliminary design. The lack of effective controls at knowledge point 1 could result in gaps between requirements and resources being discovered later in development. At the design readiness review or when knowledge point 2 should be reached, DOD's policy does not require specific controls to document that a product is ready for initial manufacturing and demonstration. DOD's policy suggests appropriate criteria, such as number of subsystem and system design reviews completed, percentage of drawings completed, planned corrective actions to hardware and software deficiencies, adequate development testing, completed failure modes and effects analysis, identification of key system characteristics and critical manufacturing processes, and availability of reliability targets and growth plans. However, these criteria are not required. For example, we found that a key indicator of a product's design stability is the completion of 90 percent of the engineering drawings supported by design reviews. DOD's policy does not require that a certain percentage of drawings or design reviews be completed to ensure the design is mature enough to enter the system demonstration phase. As a result, a decision maker has no benchmark to consider when deciding to advance a program to the next level of development. Finally, at production commitment or when knowledge point 3 should be reached, DOD's policy does not require specific controls to document that a product can be manufactured to meet cost, schedule, and quality targets before moving into production. For example, the policy states there should be "no significant manufacturing risks" at the start of low-rate production but does not define what this means or how it is to be measured. DOD's policy does not require the demonstrated control of manufacturing processes and the collection of statistical process control data until full-rate production begins but even then fails to specify a measurable control. Given that low-rate production can last several years, a significant number of products can be manufactured before processes are brought under control, creating a higher probability of poor cost and schedule outcomes. While supporting efforts to build more flexibility into the DOD acquisition process and to develop weapon systems using an evolutionary approach, Congress asked DOD to be more disciplined in its approach. The Defense Authorization Act for Fiscal Year 2003 required DOD to address (1) the way it plans to meet certain statutory and regulatory requirements for managing its major acquisition programs, (2) needed guidance for implementing spiral developments, and (3) technology readiness (at acquisition program initiation). DOD was responsive to all three requirements. With regard to the second requirement, a description of the process that would be used to independently validate that measurable exit DOD Is Responsive to Congressional Requirements about New Acquisition Process criteria for applying a spiral development process5 have been met was unclear. DOD stated that the milestone decision authority6 provides that independent validation as part of DOD's milestone approval process. DOD's responses to the relevant sections of the act are summarized below. More detailed comparisons are provided in appendixes I, II, and III. Section 802: Evolutionary Acquisition Requirements: This section directed DOD to report on its plan to meet certain statutory and regulatory requirements for managing its major acquisition programs applying an evolutionary acquisition process. These include establishing and approving operational requirements and cost and schedule goals for each increment, meeting requirements for operational and live fire testing for each increment, and optimizing total system performance and minimizing total ownership costs. DOD response: In April 2003, DOD submitted its report reflecting how these requirements are addressed in its acquisition policy. According to the report, the policy addresses the statutory and other requirements applicable to all major defense acquisition programs, including each increment of evolutionary acquisition programs. For example, the policy requires that each program or increment of an evolutionary acquisition have a milestone B decision to approve program initiation and to permit entry into system development and demonstration. The policy specifies the statutory and regulatory information necessary to support the decision. Section 803: Spiral Development Requirements: This section authorizes DOD to conduct a research and development program for a major defense acquisition program using spiral development only if approved by the Secretary of Defense or authorized 5 A spiral development program is defined in section 803 of the Defense Authorization Act for Fiscal Year 2003 as a research and development program conducted in discrete phases or blocks, each of which will result in the development of fieldable prototypes and will not proceed into acquisition until specific performance parameters, including measurable exit criteria, have been met. 6 The milestone decision authority is typically the DOD component acquisition executive or designee, or for certain large programs, the head of the component or the Under Secretary of Defense for Acquisition, Technology, and Logistics. high-level designee.7 A program cannot be conducted as a spiral development unless the Secretary of Defense or designee approves a plan that describes such things as the program strategy, test plans, performance parameters, and measurable exit criteria. The section also requires the Secretary of Defense to issue guidance addressing the appropriate processes for an independent validation that exit criteria have been met, the operational assessment of fieldable prototypes, and the management of these types of programs. It further requires the Secretary to report to Congress on the status of each program applying spiral development by September 30 of each year from 2003 to 2008. DOD response: DOD established a technology development strategy in the new policy to address this requirement. The strategy must be completed before a program can enter the technology development phase. The strategy also documents the cost and schedule goals, the test plans, the number of prototypes, and a program strategy for the total research and development program. The strategy requires a test plan to ensure the goals and exit criteria for the first technology spiral demonstration are met, and the policy requires an independent operational assessment for the release of each product increment to the user. What is unclear in DOD's guidance is the process that will be used for independently validating whether measurable cost, schedule, and performance exit criteria have been met. However, DOD stated that the milestone decision authority provides independent validation that exit criteria have been met as part of DOD's milestone approval process. As of October 23, 2003, DOD's report on the status of each program applying spiral development was still in draft and not yet submitted. DOD's current draft report states that there are no research and development programs that have been approved as spiral development programs as of September 30, 2003. Section 803 requirements were implemented in DOD Instruction 5000.2, which was effective in May 2003. DOD anticipates that there will be approved spiral development programs to report in 2004. 7 Section 803 authorized the Secretary of Defense to delegate authority to approve a spiral development plan to the Under Secretary of Defense for Acquisition, Technology, and Logistics or to the senior acquisition executive of the military department or defense agency concerned. The authority may not be delegated further. Section 822: Independent Technology Readiness Assessments Requirements: This section added a requirement to section 804 of the National Defense Authorization Act for Fiscal Year 2002 (Public Law 107-107) that directed DOD to report by March of each year between 2003 and 2006 on the maturity of technology at the initiation of major defense acquisition programs. Each report is required to (1) identify any major acquisition program that entered system development and demonstration during the preceding calendar year with immature key technology that was not demonstrated in, at minimum, a relevant environment, as required by the new policy; (2) justify the incorporation of any key technology on an acquisition program that does not meet that requirement; (3) identify any instances that the Deputy Under Secretary of Defense for Science and Technology did not concur with the technology assessment and explain how the issue has been or will be resolved; (4) identify each case in which a decision was made not to conduct an independent technology readiness assessment for a critical technology on a major defense acquisition; and (5) explain the reasons for the decision each year through 2006. DOD response: In March 2003, DOD reported that two programs entered system development and demonstration in 2002 with critical technologies that did not meet demonstration requirements and provided justification for them.8 DOD did not identify or report any cases where an independent technology readiness assessment was not conducted or where the Under Secretary disagreed with assessment findings. 8 The two programs were the Composite Health Care System II and the Joint Tactical Radio System, Cluster 1. Conclusions Recommendations for Executive Action DOD can maximize its $1 trillion investment in new weapons over the next 6 years by ensuring effective implementation of the new acquisition policy. DOD's leaders have taken noteworthy steps by incorporating into the policy a framework that supports a knowledge-based, evolutionary acquisition process, similar to one used by leading commercial companies to get successful outcomes. A framework is an important and significant step. DOD must now turn its attention to establishing controls. As leading companies have found, having clearly established controls to capture and use appropriate knowledge to make decisions at critical junctures is crucial for delivering affordable products as planned. DOD's policy addresses specific congressional requirements and includes some controls that leading companies use to capture knowledge at the start of a program. However, additional controls are needed to ensure that decisions made throughout product development are informed by demonstrated knowledge. DOD must design and implement necessary controls to ensure that appropriate knowledge is captured and used at critical junctures to make decisions about moving a program forward and investing more money. We recommend that the Secretary of Defense require additional controls for capturing knowledge at three key points-program launch, design readiness review for transitioning from system integration to system demonstration, and production commitment. The additional controls for program launch (milestone B) should ensure the capture of knowledge about the following: o Cost and schedule estimates based on knowledge from a preliminary design using systems engineering tools. The additional controls for transitioning from system integration to system demonstration (design readiness review) should ensure the capture of knowledge about the following: o Completion of 90 percent of engineering drawings. o Completion of subsystem and system design reviews. o Agreement from all stakeholders that drawings are complete and the design is producible. o Completion of failure modes and effects analysis. o Identification of key system characteristics. o Identification of critical manufacturing processes. o Reliability targets and a reliability growth plan based on demonstrated reliability rates of components and subsystems. The additional controls for the production commitment (milestone C) should ensure the capture of knowledge about the following: o Completion of production representative prototypes. o Availability of production representative prototypes to achieve reliability goal and demonstrate the product in an operational environment. o Collection of statistical process control data. o Demonstration that critical manufacturing processes are capable and in statistical control. Because knowledge about technology, design, and manufacturing at critical junctures can lower DOD's investment risk, decisions that do not satisfy knowledge-based criteria should be visible and justified. Therefore, we also recommend that the Secretary of Defense document the rationale for any decision to move a program to the next stage of development without meeting the knowledge-based criteria, including those listed in the first recommendation. The responsible milestone decision authority should justify the decision in the program's acquisition decision memorandum and in a report to Congress. Agency Comments and Our Evaluation DOD provided us with written comments on a draft of this report. The comments appear in appendix IV. DOD partially concurred with our recommendation that the Secretary require additional controls for capturing knowledge at three key points: program launch, design readiness review for transitioning from system integration to system demonstration, and production. DOD stated that it agrees in principle with the advantages of using knowledge-based controls at key points in the acquisition process to assess risk and ensure readiness to proceed into the next phase of the acquisition process. DOD believes the current acquisition framework includes the controls necessary to achieve effective results, but it will continue to monitor the process to determine whether others are necessary to achieve the best possible outcomes. While we believe DOD's effort to establish a solid framework for evolutionary acquisitions is a giant step forward, our work has shown that a disciplined application of controls in the process is needed to implement the framework if better acquisition outcomes are to be achieved. DOD's policy does not include all the necessary controls to ensure a high level of product knowledge is attained and used for making decisions to move a program forward in the product development process. Leading product developers use additional controls, as listed in our first recommendation, to achieve the knowledge necessary to reduce risk to reasonable levels at critical junctures before making additional significant investments in product development. Simply monitoring the process may not be enough for DOD to achieve the best outcomes. Therefore, we are retaining our recommendation that the Secretary require additional controls at three critical points in the acquisition process. DOD also partially concurred with our recommendation that the Secretary document in each program's acquisition decision memorandum and in a report to Congress the rationale for any decision to move a program to the next stage of development without meeting the knowledge-based criteria, including those described in the first recommendation. DOD agreed that it should record and be accountable for program decisions. Decision makers will continue to use the acquisition decision memorandum to document program decisions and the rationale for them. DOD did not concur with the need for a report outside the department. Because we believe strongly that knowledge-based criteria used to gauge a product's development progress at critical junctures can lower DOD's investment risks, we think it is important that decisions made without satisfying knowledge-based criteria be justified in a visible and transparent way to hold managers accountable for moving a program forward absent this knowledge. Therefore, we are retaining our recommendation for reporting the basis for decisions to move forward in a report to Congress. We reviewed DOD's revised and past acquisition policies, DOD Directive 5000.1, DOD Instruction 5000.2, and DOD 5000.2-R,9 which provide management principles and mandatory policies and procedures for managing acquisitions programs. We contacted an official in the Office of the Under Secretary of Defense for Acquisition, Technology, and Logistics who is responsible for the development of the policy to better understand its content. We also reviewed information from the Defense Acquisition University that provided educational material on the policies. We reviewed the relevant sections of the Bob Stump National Defense Authorization Act for Fiscal Year 2003 and the accompanying Senate Scope and Methodology 9 DOD 5000.2-R was canceled and replaced with the Interim Defense Acquisition Guidebook. Armed Services Committee report to identify the requirements applicable to DOD's acquisition policy. We compared these requirements with DOD's responses to determine whether they have been addressed. Finally, we used information from more than 10 GAO products that examine how commercial best practices can improve outcomes for various DOD programs. During the past 6 years, we have gathered information based on discussions and visits with the following companies: o 3M o Boeing Commercial Airplane Group o Chrysler o Bombardier Aerospace o Ford Motor o Hughes Space and Communication o Motorola o Xerox o Hewlett-Packard o Caterpillar o Cummins o General Electric Aircraft Engines o Toyota o Harris Semiconductor o Honda o Texas Instruments o John Deere o Varian Oncology Systems o Ethicon-Endo Surgery (division of Johnson & Johnson) Although the approaches varied, these companies consistently applied the basic processes and standards in use. We compared this information with the acquisition framework and controls established by DOD's policy. We concentrated on whether the policy provides a framework for a knowledge-based, evolutionary process and the controls necessary to carry out this intent. We conducted our review from April 2003 to September 2003 in accordance with generally accepted government auditing standards. We are sending copies of this report to the Secretary of Defense; the Secretaries of the Air Force, Army, and Navy; and the Director of the Office of Management and Budget. We will also provide copies to others on request. In addition, the report will be available at no charge on the GAO Web site at http://www.gao.gov. Please contact me at (202) 512-4841 if you have any questions concerning this report. Other key contributors to this report were Lily Chin, Chris DePerro, Matt Lea, Mike Sullivan, and Adam Vodraska. Katherine V. Schinasi Director, Acquisition and Sourcing Management Appendix I: Section 802, Evolutionary Acquisition Section 802 of the Defense Authorization Act for Fiscal Year 2003 required the Secretary of Defense to submit a report to Congress explaining how the Department of Defense (DOD) plans to meet certain statutory and regulatory requirements for acquisition programs following an evolutionary approach. In April 2003, the Secretary reported how these requirements were addressed in DOD's policy (such as in tables of statutory and regulatory information requirements contained in enclosure 3 of Instruction 5000.2). According to the report, DOD's policy requires that each program-including an increment of an evolutionary acquisition-have a milestone B decision to approve program initiation and to permit entry into systems development and demonstration. DOD's policy specifies the statutory and regulatory information necessary to support the decision. We examined the policy to ensure the statutes and regulations identified in section 802 were addressed. Table 3 provides a list of the statutory and regulatory requirements identified in section 802, a corresponding document and page number where the requirement appears in DOD's policy, and a description of the requirement from the policy. Table 3: How DOD Policy Addresses Section 802's Statutory and Regulatory Requirements Section 802 requirements DOD policy reference examples Brief description from policy Requirements of chapter 144 of title 10, United States Code Sec. 2430, major defense acquisition Department of Defense Instruction Table E.2.T1 provides a description (criteria) and the program defined (DODI) 5000.2, enclosure 2, p. 16 decision authority by acquisition program category. Sec. 2431, weapons development Not referenceda and procurement schedules Sec. 2432, Selected DODI 5000.2, Required at milestone B or Acquisition enclosure 3, p.19 program initiation and Reports annually thereafter; end of quarter following milestone C decision; full-rate production decision. Also required when there is a baseline breach. Sec. 2433, unit cost DODI 5000.2, reports enclosure 3, p. 19 Required on quarterly basis. Sec. 2434, independent cost estimates; operational manpower requirements DODI 5000.2, enclosure 6, p. 30 DODI 5000.2, enclosure 3, p. 19 Required at program initiation for ships (cost assessment only). Required at milestones B and C and full-rate production decision. Sec. 2435, baseline description DODI 5000.2, enclosure 3, p. 19 Required at program initiation for ships. Required at DODI 5000.2, p. 8 milestones B and C and full-rate production decision. Program deviation report required immediately upon program deviation. Sec 2440, technology and industrial DODI 5000.2, enclosure 3, p. 19 Required at milestones B and C (part of acquisition base plans strategy). Appendix I: Section 802, Evolutionary Acquisition Section 802 requirements DOD policy reference examples Brief description from policy Sec. 139, Director of Operational Department of Defense Directive Cites Director of Operational Test and Evaluation as Test and Evaluation (DODD) 5000.1, p. 3 DODI 5000.2, enclosure 3 DODI 5000.2, enclosure 5, pp. 27-29 key official of defense acquisition system. Director responsible for assessing adequacy of operational tests and live fire tests and evaluating operational effectiveness, suitability, and survivability of systems. Sec. 181, Joint Requirements DODD 5000.1, p. 3 Chairman of the Joint Chiefs of Staff, with assistance Oversight Council DODI 5000.2, p. 4 of Joint Requirements Oversight Council, responsible for assessing and providing advice regarding capability needs for defense acquisition programs. Chairman also responsible for validating and approving capabilities documents. Sec. 2366, major systems and DODI 5000.2, enclosure 3, p. 19 Live fire testing and reporting required for all covered munitions programs: survivability DODI 5000.2, enclosure 5, p. 29 systems. Strategy or live fire waiver and alternate testing and lethality testing required DODI 5000.2, p. 10 plan required at milestone B. Live fire test and before full-scale production evaluation report required for full-rate production decision. Sec. 2399, operational test and DODI 5000.2, enclosure 3, p. 18 Director of Operational Test and Evaluation shall evaluation of defense acquisition DODI 5000.2, pp. 9-10 determine operational effectiveness and suitability of programs DODI 5000.2, enclosure 5, pp. 27-28 system under realistic conditions. Beyond Low Rate Initial Production Report required at full-rate production decision. DOD may not conduct operational testing until Director of Operational Test and Evaluation approves test plan. Sec. 2400, low-rate initial production DODI 5000.2, enclosure 3, p. 19 Low-rate initial production quantities will be of new systems DODI 5000.2, pp. 9-10 determined by milestone B. DODD 5000.1 DODD 5000.1, pp. Policies in directive apply to 1, 2 all acquisition programs. Evolutionary acquisition strategies are preferred approach to satisfying operational needs. DODI 5000.2 Instruction applies to all DODI 5000.2, p. 1 defense technology projects and acquisition projects. Chairman of the Joint Chairman of Joint Chiefs of Staff Chiefs of Staff DODI 5000.2, p. 4 provides advice Instruction 3170.01Bb through Instruction 3170.01. Capability documents required at concept decision and at milestones A, B, and C. Other provisions of law and regulations DODI 5000.2, enclosure 3, pp. 18-22 Several other statutory, regulatory, and contract (including successor documents) that requirements are addressed in the policy. are applicable to such programs Sources: Defense Authorization Act for Fiscal Year 2003, DOD, and GAO. aAlthough the statutory requirement of 10 U.S.C., section 2431 is not specifically cited in the policy, the same summarized information is submitted to Congress in the Selected Acquisition Reports for the first quarter of a fiscal year as required by 10 U.S.C., section 2432. The requirement for Selected Acquisition Reports is addressed by DOD's acquisition policy. The statute concerning weapons development and procurement schedules requires the Secretary of Defense to submit budget justification documents regarding this information for each weapon system for which fund authorization is required (and for which procurement funds are requested in the budget) not later than 45 days after the President submits the budget to Congress. bChairman of the Joint Chiefs of Staff Instruction 3170.01B was revised and reissued June 24, 2003, as Joint Chiefs of Staff Instruction 3170.01C with an accompanying manual. Appendix I: Section 802, Evolutionary Acquisition Section 802 also required DOD to report on its plans for addressing certain acquisition process issues regarding each increment of an evolutionary process. DOD reported on how it plans to establish and approve operational requirements and cost and schedule goals; meet requirements for operational and live fire testing; monitor cost and schedule performance; achieve interoperability; and consider total system performance and total ownership costs. We compared DOD's response with section 802's reporting requirements. As shown in table 4, DOD was responsive to the 802 requirements. Table 4: How DOD Responded to Section 802's Requirements Regarding Specific Matters for Each Increment of an Evolutionary Acquisition Process Section 802 requirements DOD response Policy reference "The manner in which the Secretary plans to establish and approve, for each increment of an evolutionary acquisition process-" "operational requirements; and" "Operational Requirements: Each program is required to have documented, approved operational requirements in accordance with authorized Joint Staff procedures. For evolutionary acquisition programs, the requirements documents are typically time-phased and specify the capability expected of each increment." "cost and schedule goals." "Cost and Schedule Goals: At program initiation, each program and program increment is required to have an Acquisition Program Baseline approved by the Milestone Decision Authority (MDA). The Acquisition Program Baseline includes cost and schedule goals." Capability development document, including key performance parameters, required at milestone B or program initiation (DODI 5000.2, p. 7, and enclosure 3, p. 20). Acquisition program baseline required at milestone B or program initiation (DODI 5000.2, p. 8, and enclosure 3, p. 19). Appendix I: Section 802, Evolutionary Acquisition Section 802 requirements DOD response Policy reference "The manner in which the Secretary plans, for each increment of an evolutionary acquisition process-" "to meet requirements for operational testing and live fire testing;" "to monitor cost and schedule performance; and" "to comply with laws requiring reports to Congress on results testing and on cost and schedule performance." "At program initiation, each program or program increment must have a Test and Evaluation Master Plan (TEMP) approved by the Director, Operational Test and Evaluation. The TEMP includes requirements for operational and live fire testing." "Progress against cost and schedule goals [for each increment] is addressed via automated reporting systems at both the Office of Secretary of Defense staff level and at the Service staff level, and by reviews conducted in the context of the acquisition oversight model." "DoD acquisition policy requires independent assessment by the operational test authority prior to release of each successive increment to the user. All reports by statute will be submitted." Test and Evaluation Master Plan required at milestone B or program initiation (DODI 5000.2, enclosure 3, p. 21, and enclosure 5, p. 25). DOD's policy contains an enclosure on resource estimation (DODI 5000.2, enclosure 6, pp. 30-31). Also, several reporting requirements address monitoring cost and schedule performance such as acquisition program baselines and selected acquisition reports (DODI 5000.2, enclosure 3, pp. 18-21). The service shall perform an independent operational assessment prior to release of each successive increment to the user (DODI 5000.2, enclosure 5, p. 29). Also, several statutory reports are listed in the policy addressing testing and cost and schedule performance (DODI 5000.2, enclosure 3, pp. 18-21). Appendix I: Section 802, Evolutionary Acquisition Section 802 requirements DOD response Policy reference "The manner in which the Secretary plans to ensure that each increment of an evolutionary acquisition process is designed-" "to achieve interoperability within and among United States forces and United States coalition partners; and" "to optimize total system performance and minimize total ownership costs by giving appropriate consideration to-" o "logistics planning;" o "manpower, personnel, and training;" o "human, environmental, safety, occupational health, accessibility, survivability, operational continuity and security factors;" o "protection of critical program information; and" o "spectrum management" "Each increment of an evolutionary acquisition program is required to have an Acquisition Program Baseline, approved by the MDA, which includes an Interoperability key performance parameter." "Each increment of an evolutionary acquisition program is required to have an acquisition strategy, approved by the MDA, that addresses [where applicable]: logistics planning; manpower, personnel and training; human, environmental, safety, occupational health; accessibility (human), survivability, operational continuity (as required by requirements document), security factors; critical program information; and spectrum management. These factors are taken into account as the Department of Defense considers total system performance and total ownership costs." Acquisition program baseline required at milestone B or program initiation (DODI 5000.2, p. 8, and enclosure 3, p. 19). Acquisition strategy required at milestone B or program initiation (DODI 5000.2, p. 7, and enclosure 3, p. 20). Sources: Defense Authorization Act for Fiscal Year 2003, DOD, and GAO. Appendix II: Section 803, Spiral Development Section 803 of the Defense Authorization Act for Fiscal Year 2003 authorized the Secretary of Defense to conduct major defense acquisition programs as spiral development programs. However, the section placed a limitation on these programs. It stated that a research and development program for a major acquisition may not be conducted as a spiral development program unless the Secretary of Defense or authorized high-level designee gives approval. The section requires the Secretary of Defense to issue guidance for the implementation of such programs to address appropriate processes for ensuring the independent validation of exit criteria being met, the operational assessment of fieldable prototypes, and the management of these types of programs. DOD responded to these requirements principally by incorporating into the acquisition policy the requirement for a technology development strategy. This strategy is a prerequisite for a project to enter the technology development phase of the acquisition process, or milestone A. Table 5 compares the spiral development plan requirements in the act with the technology development strategy requirements in DOD's May 2003 acquisition policy. Table 5: How DOD Policy Addresses Section 803's Requirements Section 803 requirements DOD policy "A spiral development plan for research and development Section 3.5.4 of DOD Instruction 5000.2 states that the technology program for a major defense acquisition program shall, at a minimum, include the following matters:" development strategy document the following: "A rationale for dividing the research and development program The rationale for adopting an evolutionary strategy. For an into separate spirals, together with a preliminary identification of evolutionary acquisition, either spiral or incremental, the the spirals to be included." strategy shall include a preliminary description of how the program will be divided into technology spirals and development increments. "A program strategy, including overall cost, schedule and A program strategy, including overall cost, schedule, and performance goals for the total research and development." performance goals for the total research and development program. "Specific cost, schedule, and performance parameters, Specific cost, schedule, and performance goals, including exit including measurable exit criteria, for the first spiral to be criteria, for the first technology spiral demonstration. conducted." "A testing plan to ensure that performance goals, parameters, A test plan to ensure that the goals and exit criteria for the first and exit criteria are met." technology spiral demonstration are met. "An appropriate limitation on the number of prototype units that An appropriate limitation on the number of prototypes units that may be produced under the research and development may be produced and deployed during technology program." development. Appendix II: Section 803, Spiral Development Section 803 requirements DOD policy "Specific performance parameters, including measurable exit Specific performance goals and exit criteria that must be met criteria, that must be met before the major defense acquisition program proceeds into production units in excess of the limitation on the number of prototype units." before exceeding the number of prototypes that may be produced under the research and development program. Sources: Defense Authorization Act for Fiscal Year 2003, DOD, and GAO. As shown in the table, DOD's policy generally responded to the requirements in the act concerning guidance for implementation of spiral development programs. While the policy includes a technology development strategy that requires a test plan to ensure the goals and exit criteria for the first technology spiral demonstration are met and an independent operational assessment for the release of each product increment to the user, it is unclear what the process is for independently validating that cost, schedule, and performance exit criteria have been. However, DOD stated that the milestone decision authority provides independent validation that exit criteria have been met as part of DOD's milestone approval process. Section 803 also requires that a spiral development plan include "[s]pecific cost, schedule, and performance parameters, including measurable exit criteria, for the first spiral to be conducted." DOD's policy substituted "parameters" for "goals" and did not use the term "measurable" in describing the required exit criteria. Finally, section 803 requires the Secretary of Defense to submit to Congress by September 30 yearly from 2003 through 2008 a status report on each spiral development program. The report is to include information on unit costs for the projected prototypes. As of October 23, 2003, DOD's report on the status of each program applying spiral development was still in draft and not yet submitted. DOD's current draft report states that there are no research and development programs that have been approved as spiral development programs as of September 30, 2003. Section 803 requirements were implemented in DOD Instruction 5000.2, which was effective in May 2003. DOD anticipates that there will be approved spiral development programs to report in 2004. Appendix III: Section 804, Technology Maturity, and Section 822, Independent Technology Readiness Assessments Section 804 of the Defense Authorization Act for Fiscal Year 2002 required DOD to report on the maturity of technology at the initiation of major defense acquisition programs. The act directed DOD to report by March 1 of each year between 2003 and 2006 on a requirement in DOD's policy that technology must have been demonstrated in a relevant environment (or, preferably, in an operational environment) to be considered mature enough to use for product development in systems integration. Each report is required to (1) identify any major acquisition program that entered system development and demonstration during the preceding calendar year with immature key technology that was not demonstrated in, at minimum, a relevant environment, as required by the new policy; (2) justify the incorporation of any key technology on an acquisition program that does not meet that requirement; (3) and identify any instances that the Deputy Under Secretary of Defense for Science and Technology did not concur and explain how the issue has been or will be resolved, including information on the use of independent readiness assessments. Section 822 of the Defense Authorization Act for Fiscal Year 2003 amended section 804 by adding a requirement that the Secretary of Defense identify each case in which an authoritative decision has been made within DOD not to conduct an independent technology readiness assessment for a critical technology on a major defense acquisition program and explain the reasons for the decision. On March 18, 2003, DOD submitted its first report. Table 6 shows the specific requirements for the report and DOD's response. Table 6: How DOD Responded to Section 804 and Section 822 Requirements Reporting requirements DOD's response Section 804: "identify each case in which a major defense DOD reported two programs (Joint Tactical Radio System Cluster acquisition program entered system development and 1 and Composite Health Care System II) that entered system demonstration during the preceding calendar year and into which development and demonstration with key technologies that did not key technology has been incorporated that does not meet the meet the technological maturity requirement. While DOD did not technological maturity requirement ... and provide a justification specifically identify all the technologies for these programs that did for why such key technology was incorporated."a not meet the requirements, it did provide its justification for why the technologies were incorporated. Section 804: "identify any determination of technological maturity DOD reported that, in all cases, the Deputy Under Secretary of with which the Deputy Under Secretary of Defense for Science Defense for Science and Technology concurred with the and Technology did not concur and explain how the issue has technology readiness assessment of the program manager and been or will be resolved." the milestone decision authority. Appendix III: Section 804, Technology Maturity, and Section 822, Independent Technology Readiness Assessments Reporting requirements DOD's response Section 822: "identify each case in which an authoritative decision DOD did not report on or identify any cases. has been made within the Department of Defense not to conduct an independent technology readiness assessment for a critical technology on a major defense acquisition program and explain the reasons for the decision." Sources: National Defense Authorization Acts for Fiscal Years 2002 and 2003, DOD, and GAO. aTechnology must have been demonstrated in a relevant environment (or, preferably in an operational environment) to be considered mature enough to use for product development in systems integration. Appendix IV: Comments from the Department of Defense Appendix IV: Comments from the Department of Defense Now on pp. 14 and 15. Appendix IV: Comments from the Department of Defense Now on p. 15. Related GAO Products Best Practices: Setting Requirements Differently Could Reduce Weapon Systems' Total Ownership Costs. GAO-03-57. Washington, D.C.: February 11, 2003. Best Practices: Capturing Design and Manufacturing Knowledge Early Improves Acquisition Outcomes. GAO-02-701. Washington, D.C.: July 15, 2002. Defense Acquisitions: DOD Faces Challenges in Implementing Best Practices. GAO-02-469T. Washington, D.C.: February 27, 2002. Best Practices: Better Matching of Needs and Resources Will Lead to Better Weapon System Outcomes. GAO-01-288. Washington, D.C.: March 8, 2001. Best Practices: A More Constructive Test Approach Is Key to Better Weapon System Outcomes. GAO/NSIAD-00-199. Washington, D.C.: July 31, 2000. Defense Acquisition: Employing Best Practices Can Shape Better Weapon System Decisions. GAO/T-NSIAD-00-137. Washington, D.C.: April 26, 2000. Best Practices: DOD Training Can Do More to Help Weapon System Program Implement Best Practices. GAO/NSIAD-99-206. Washington, D.C.: August 16, 1999. Best Practices: Better Management of Technology Development Can Improve Weapon System Outcomes. GAO/NSIAD-99-162. Washington, D.C.: July 30, 1999. Defense Acquisitions: Best Commercial Practices Can Improve Program Outcomes. GAO/T-NSIAD-99-116. Washington, D.C.: March 17, 1999. Defense Acquisition: Improved Program Outcomes Are Possible. GAO/T-NSIAD-98-123. Washington, D.C.: March 17, 1998. Best Practices: DOD Can Help Suppliers Contribute More to Weapon System Programs. GAO/NSIAD-98-87. Washington, D.C.: March 17, 1998. Related GAO Products Best Practices: Successful Application to Weapon Acquisition Requires Changes in DOD's Environment. GAO/NSIAD-98-56. Washington, D.C.: February 24, 1998. Best Practices: Commercial Quality Assurance Practices Offer Improvements for DOD. GAO/NSIAD-96-162. Washington, D.C.: August 26, 1996. (120214) GAO's Mission Obtaining Copies of GAO Reports and Testimony The General Accounting Office, the audit, evaluation and investigative arm of Congress, exists to support Congress in meeting its constitutional responsibilities and to help improve the performance and accountability of the federal government for the American people. GAO examines the use of public funds; evaluates federal programs and policies; and provides analyses, recommendations, and other assistance to help Congress make informed oversight, policy, and funding decisions. GAO's commitment to good government is reflected in its core values of accountability, integrity, and reliability. The fastest and easiest way to obtain copies of GAO documents at no cost is through the Internet. GAO's Web site (www.gao.gov) contains abstracts and full-text files of current reports and testimony and an expanding archive of older products. The Web site features a search engine to help you locate documents using key words and phrases. You can print these documents in their entirety, including charts and other graphics. Each day, GAO issues a list of newly released reports, testimony, and correspondence. GAO posts this list, known as "Today's Reports," on its Web site daily. The list contains links to the full-text document files. To have GAO e-mail this list to you every afternoon, go to www.gao.gov and select "Subscribe to e-mail alerts" under the "Order GAO Products" heading. Order by Mail or Phone The first copy of each printed report is free. Additional copies are $2 each. A check or money order should be made out to the Superintendent of Documents. GAO also accepts VISA and Mastercard. Orders for 100 or more copies mailed to a single address are discounted 25 percent. Orders should be sent to: U.S. General Accounting Office 441 G Street NW, Room LM Washington, D.C. 20548 To order by Phone: Voice: (202) 512-6000 TDD: (202) 512-2537 Fax: (202) 512-6061 Contact: To Report Fraud, Web site: www.gao.gov/fraudnet/fraudnet.htm Waste, and Abuse in E-mail: [email protected] Federal Programs Automated answering system: (800) 424-5454 or (202) 512-7470 Jeff Nelligan, Managing Director, [email protected] (202) 512-4800 Public Affairs U.S. General Accounting Office, 441 G Street NW, Room 7149 Washington, D.C. 20548 *** End of document. ***

What is the defense acquisition system?

What is the primary objective of defense acquisition?

What are the three phases of the acquisition process?

What federal government regulations govern DoD acquisitions activities?