Note: Where available, the PDF/Word icon below is provided to view the complete and fully formatted document
Public Accounts and Audit—Joint Committee—Report 456—Defence Major Equipment Procurement and Evaluation, and Great Barrier Reef Regulation: Review of Auditor-General Reports Nos 51-52 (2014-15) and Nos 1-10 (2015-16)—May 2016


Download PDF Download PDF

The Parliament of the Commonwealth of Australia

Report 456

Defence Major Equipment Procurement and Evaluation, and Great Barrier Reef Regulation Review of Auditor-General Reports Nos 51-52 (2014-15) and Nos 1-10 (2015-16)

Joint Committee of Public Accounts and Audit

May 2016 Canberra

© Commonwealth of Australia 2016

978-1-74366-525-1 Printed version

978-1-74366-526-8 HTML version

This work is licensed under the Creative Commons Attribution-NonCommercial-NoDerivs 3.0 Australia License.

The details of this licence are available on the Creative Commons website: http://creativecommons.org/licenses/by-nc-nd/3.0/au/.

Contents

Foreword ............................................................................................................................................ vii

Membership of the Committee ............................................................................................................ ix

Terms of reference .............................................................................................................................. xi

List of recommendations .................................................................................................................... xii

1 Summary .............................................................................................................. 1

Background ............................................................................................................................... 3

Relevant public sector frameworks ......................................................................................... 3

2 Performance Audit Report No. 52 (2014-15) ...................................................... 7

ADF’s Medium and Heavy Vehicle Fleet Replacement (Land 121 Phase 3B) ........ 7 Inquiry context and background .............................................................................................. 7

ANAO report overview .............................................................................................................. 9

Audit objective and scope ........................................................................................................... 9

Audit conclusion .......................................................................................................................... 9

Audit recommendations ............................................................................................................ 12

Committee review of evidence ............................................................................................... 13

Implementation of ANAO recommendations ............................................................................. 13

Initial tender: poor test and evaluation ...................................................................................... 14

Initial tender: flawed value-for-money assessment ................................................................... 16

Initial tender: advice to Ministers ............................................................................................... 17

Ultimate extra cost of failed process ......................................................................................... 17

Accountability: who was responsible and what lessons have been learned? ............................ 19

New risk assessment procedures through First Principles Review ........................................... 20

iv

Committee comment ............................................................................................................... 21

3 Performance Audit Report No. 3 (2015-16) ...................................................... 25

Regulation of Great Barrier Reef Marine Park Permits and Approvals ................ 25

Inquiry context and background ............................................................................................ 25

ANAO report overview ............................................................................................................ 27

Audit objective, scope and criteria ............................................................................................. 27

ANAO overall conclusion .......................................................................................................... 27

ANAO recommendations and agency response ....................................................................... 28

Committee review of evidence ............................................................................................... 29

Key audit themes and relevant public sector frameworks ......................................................... 29

Relevant reviews, reports and audits ........................................................................................ 29

Key issues ................................................................................................................................. 30

Implementation of ANAO recommendations and stakeholder consultation ............................... 30

Permit application processing, assessment and approval ......................................................... 34

Managing compliance ............................................................................................................... 40

Responding to non-compliance ................................................................................................. 40

Committee comment ............................................................................................................... 42

4 Performance Audit Report No. 9 (2015-16) ...................................................... 47

Test and Evaluation of Major Defence Equipment Acquisitions........................... 47

Inquiry context and background ............................................................................................ 47

ANAO report overview ............................................................................................................ 49

Audit objective, scope and criteria ............................................................................................. 49

ANAO overall conclusion .......................................................................................................... 49

ANAO recommendations and agency response ....................................................................... 50

Committee review of evidence ............................................................................................... 51

Key audit themes and relevant public sector frameworks ......................................................... 51

Relevant reviews, reports and audits ........................................................................................ 51

Key issues ................................................................................................................................. 52

Implementation of ANAO recommendations and other review findings..................................... 52

T&E enterprise-level governance .............................................................................................. 55

Managing acquisition risks, including HMAS Canberra case study ........................................... 59

v

Committee comment ............................................................................................................... 65

Appendix A - Submissions ..................................................................................... 71

Appendix B - Public Hearings ................................................................................. 73

vi

Foreword

In this report the Committee chose to inquire into the following three audit reports in detail:

• Defence Medium and Heavy Vehicle Fleet Replacement: Land 121 Phase 3B (ANAO Report No. 52, 2014-15) • Regulation of Great Barrier Reef Marine Park Permits and Approvals (Audit Report No. 3, 2015-16) • Test and Evaluation of Major Defence Equipment Acquisitions (Audit Report No. 9,

2015-16)

The key theme emerging from the Committee’s inquiries was the importance of effective risk management.

The Committee was strongly concerned that shortcomings in regulating the permit system for the Great Barrier Reef Marine Park may be undermining the system as a means of managing risks to the Park. The Committee’s key recommendations were that the Authority accelerate its projected timeframe for implementation of the ANAO audit recommendations and report back on progress to the Committee.

Effective risk management in the context of procurement was emphasised in the Committee’s inquiry into Defence’s medium and heavy vehicle fleet replacement program (Land 121 Phase 3B). The Committee notes the failed first tender process

for this matter resulted in an estimated seven-year delay to delivering new vehicles and a need to sustain the existing fleet far beyond initial specifications. The Committee requested evidence that Defence has adequately learnt and embedded the lessons from this acquisition.

Effective risk management in the context of procurement was again emphasised in the Committee’s inquiry into Defence’s test and evaluation (T&E) processes for major equipment acquisitions. Several past ANAO audits and external reviews have identified deficiencies in aspects of Defence’s T&E program. The Committee found that good progress has recently been made by Defence, but the Committee

viii

has asked Defence to provide an update report on future improvements to the governance of T&E under the First Principles Review, amongst other matters.

Over the 44th Parliament, the Committee has conducted 25 separate inquiries on matters in ANAO reports, brought together in eleven reports tabled in the Parliament. Through its inquiries, the Committee has investigated a diverse range of matters involving significant public expenditure, including: major grants programs; Centrelink services; Commonwealth cyber security; regulation of the Great Barrier Reef; Medicare services; compliance with the Environment Protection and Biodiversity Conservation Act 1999; and Defence major projects.

The Committee examines a broad range of evidence when determining its report comments and recommendations. In addition to the ANAO reports, the Committee receives submissions from audited agencies and the public, and conducts public hearings. To provide broader context, the Committee also considers the relevant public sector frameworks—including the Public Governance, Performance and Accountability Act 2013 (PGPA Act), rules and guidance—and findings of other relevant reviews, reports and audits.

I thank Committee members, past and present, for their deliberation on these important matters. I also thank agency representatives who appeared at public hearings for assisting the JCPAA in its important role of holding Commonwealth agencies to account for the efficiency and effectiveness with which they use public monies.

Hon Ian Macfarlane MP Chair

Membership of the Committee

Chair Hon Ian Macfarlane MP

Deputy Chair Mr Pat Conroy MP

Members Hon Anthony Albanese MP Senator Cory Bernardi

Ms Gai Brodtmann MP Senator Katy Gallagher (until 12 Nov 15)

Mr Andrew Giles MP Senator Chris Ketter

Dr David Gillespie (from 19 Oct 2015) Senator Jenny McAllister (from 12 Nov 15)

Dr Peter Hendy MP (to 12 Oct 2015) Senator Bridget McKenzie

Mr Craig Laundy MP Senator Dean Smith

Mrs Jane Prentice MP

Mr Angus Taylor MP

Mr Ken Wyatt MP (to 12 Oct 2015)

x

Committee Secretariat

Secretary Mr David Brunoro

Inquiry Secretary Dr Kate Sullivan

Research Officer Dr Andrew Gaczol

Administrative Officer Ms Tamara Palmer

Terms of reference

Having considered Auditor-General Reports Nos 51-52 (2014-15) and Nos 1-10 (2015-16), on Thursday 3 December 2015, the Joint Committee of Public Accounts and Audit (JCPAA) resolved to review the following audit reports in detail:

 Audit Report No. 52 (2014-15) Australian Defence Force’s Medium and

Heavy Vehicle Fleet Replacement (Land 121 Phase 3B)

 Audit Report No. 3 (2015-16) Regulation of Great Barrier Reef Marine Park

Permits and Approvals

 Audit Report No. 9 (2015-16) Test and Evaluation of Major Defence

Equipment Acquisitions

Under section 8 of the legislation establishing the JCPAA, the Public Accounts and Audit Committee Act 1951, one of the duties of the Committee is to ‘examine all reports of the Auditor-General (including reports of the results of performance audits) that are tabled in each House of the Parliament’ and ‘report to both Houses of the Parliament, with any comment it thinks fit, on any items or matters in those … reports, or any circumstances connected with them, that the Committee thinks should be drawn to the attention of the Parliament’.

List of recommendations

2 ADF’s Medium and Heavy Vehicle Fleet Replacement (Land 121 Phase 3B) Recommendation 1

The Committee recommends that the Department of Defence provide evidence to the Committee that the lessons learnt from the failed first tender process for LAND 121 Phase 3B have been incorporated into the department’s standard operating procedures for acquisitions. This evidence can take the form of:

 revised manuals;

 specific internal case studies and training briefings; and

 any other relevant material.

3 Regulation of Great Barrier Reef Marine Park Permits and Approvals Recommendation 2

To improve the effectiveness of the permit system as a means of managing risks to the Great Barrier Reef Marine Park, the Committee recommends that the Great Barrier Reef Marine Park Authority (GBRMPA):

 appropriately accelerate its projected timeframe, currently planned over two tranches in 2017 and 2020, for implementation of the audit recommendations in ANAO Report No. 3 and other improvements identified by GBRMPA as part of its project to strengthen the permit system

 report back to the Committee within six months with details of new implementation dates and milestones, and how the accelerated timeframe will be achieved

xiii

Recommendation 3

To improve the effectiveness of the permit system as a means of managing risks to the Great Barrier Reef Marine Park, the Committee recommends that the Great Barrier Reef Marine Park Authority implement more effective performance information, including targets, for permit application processing, assessment and approval timeframes, and continue to monitor and publicly report on performance outcomes in this area.

Recommendation 4

To improve the effectiveness of the permit system as a means of managing risks to the Great Barrier Reef Marine Park, the Committee recommends that the Great Barrier Reef Marine Park Authority report back to the Committee at 18 months from the tabling of the Committee’s report, on:

 whether it has met the new implementation dates and milestones as previously advised to the Committee in response to recommendation 1

 specific implementation details and dates achieved on the following:

⇒ for permit application processing, assessment and approval— finalisation of standard operating procedures, guidance materials, and improvements in: documentation by officials, templates, assessment reports and standard permit conditions

⇒ for compliance management—finalisation of standard operating procedures and implementation of a risk-based program

⇒ for response to non-compliance—finalisation of guidance materials and improvements in: identification of non-compliance, records management, documentation by officials and verification of enforcement action having been undertaken prior to closure of investigation

4 Test and Evaluation of Major Defence Equipment Acquisitions Recommendation 5

To promote continued improvement in test and evaluation (T&E) of major equipment acquisitions by the Department of Defence, the Committee recommends that the department report back to the Committee at 12 months from the tabling of the Committee’s report, on:

xiv

 how T&E is being coordinated across the department and how this coordination is consistent with the recommended outcomes of the First Principles Review: Creating One Defence (2015)

 key improvements to T&E performance monitoring and reporting

 each element of the Australian National Audit Office recommendation on T&E competency and training, including the key findings of and actions from the competency and training needs analysis

1

Summary

1.1 The importance of effective risk management was the key theme that emerged from this inquiry. The Committee emphasises that risk management is vital to reduce risks to the Great Barrier Reef from permitted activities and to reduce financial risk for Australian taxpayers in terms of major Defence spending.

1.2 Effective risk management in the context of better practice regulation was emphasised in the Committee’s inquiry into the audit of Great Barrier Reef Marine Park permits and approvals. The Committee notes the seriousness of the Auditor-General’s overall conclusion that shortcomings in the Great Barrier Reef Marine Park Authority’s (GBRMPA) regulatory processes and, more particularly, its regulatory practices have undermined the effectiveness of the permit system as a means of managing risks to the Marine Park. The Committee was strongly concerned about this finding, given the environmental, social and economic significance of the Great Barrier Reef Marine Park and its World Heritage listing. The Committee therefore recommended that GBRMPA:

 appropriately accelerate its projected timeframe for implementation of

the audit recommendations in the Australian National Audit Office’s (ANAO) Report No. 3 and report back to the Committee within six months on how it has achieved this, including details of new implementation dates and milestones

 implement more effective performance information, including targets,

for permit application processing, assessment and approval timeframes, and continue to monitor and publicly report on performance outcomes in this area

 report back to the Committee at 18 months from the tabling of the

Committee’s report on whether it has met the new implementation dates and milestones in response to recommendation 1, and on implementation details and dates achieved for a range of initiatives

2 REPORT 456: DEFENCE LAND 121, GBRMPA AND DEFENCE T&E

related to permit application assessment, compliance management and response to non-compliance

1.3 Effective risk management in the context of procurement was emphasised in the Committee’s inquiry into the audit of Australian Defence Force’s medium and heavy vehicle fleet replacement program (Land 121 Phase 3B). The Committee notes the failed first tender process resulted in an estimated seven-year delay to delivering new vehicles and a need to sustain the existing fleet far beyond initial specifications. The Committee expressed strong disappointment in this flawed process and costly outcome. It is difficult to understand how such fundamental mistakes as poor test and evaluation (T&E) of the proposals, significantly flawed value-for-money assessments, and failure to inform the Minister of the significant capability and technical risks could be made. Defence presented a strong argument that the lessons of the failed first tender process have been learned, and the Committee acknowledged that the reforms underway as part of the First Principles Review could minimise the risks of similar issues reoccurring. However, given the significance of the audit findings, the Committee recommended that:

 Defence provide additional evidence that the lessons learned from this

acquisition are being firmly embedded and broadly disseminated through incorporation into standard operating procedures

1.4 Effective risk management in the context of procurement was again emphasised in the Committee’s inquiry into the audit of Defence’s T&E processes for major equipment acquisitions. The Committee notes that, over a number of years, several ANAO audits and external reviews have identified deficiencies in aspects of Defence’s T&E program. Defence have made progress in a number of areas towards implementing recommendations from these reports, but this has taken considerable time. Overall, Defence still needs to improve key aspects of its T&E administration, and to do so in a timely manner. Reforms to T&E competency and training arrangements need to be completed. Further, as the conduct of T&E was distributed across 12 Defence organisations, Defence needs to introduce arrangements to provide a more coordinated, whole-of-Defence T&E governance through enterprise-level advice to senior leaders, integrated T&E policy and procedures, and improved performance measures. To promote continued improvement in T&E of major equipment acquisitions by Defence, the Committee recommended that the department report back to the Committee at 12 months from the tabling of the Committee’s report, on:

SUMMARY 3

 how T&E is being coordinated across the department and how this

coordination is consistent with the recommended outcomes of the First Principles Review: Creating One Defence (2015)

 key improvements to T&E performance monitoring and reporting

 each element of the ANAO recommendation on T&E competency and

training, including the key findings of and actions from the competency and training needs analysis

Background

1.5 One of the duties of the Joint Committee of Public Accounts and Audit (JCPAA) is to ‘examine all reports of the Auditor-General … that are tabled in each House of the Parliament’ and ‘report to both Houses of the Parliament, with any comment it thinks fit, on any items or matters in those … reports, or any circumstances connected with them …’1

1.6 In selecting audit reports for inquiry, the Committee considers:

 the significance of the program or issues raised in audit reports

 the audited agencies response to the audit

 the potential public interest benefits in conducting an inquiry

1.7 The Committee examines a broad range of evidence when determining its comments and recommendations. In addition to the ANAO reports, the Committee receives submissions from audited agencies and the public, and conducts public hearings. To provide broader context, the Committee also considers the relevant public sector frameworks and findings of other relevant reviews, reports and audits.

1.8 In this way, the Committee seeks to identify key issues, and recommend actions needed to both remedy any immediate concerns and promote key learnings for future activities.

Relevant public sector frameworks

1.9 Key public sector frameworks referenced by the Committee are the relevant sections of the Public Governance, Performance and Accountability Act 2013 (PGPA Act) and PGPA Rule 2014, and associated resource management guidance.

1 Section 8 of the Public Accounts and Audit Committee Act 1951.

4 REPORT 456: DEFENCE LAND 121, GBRMPA AND DEFENCE T&E

1.10 As the primary piece of Commonwealth resource management legislation, the PGPA Act establishes a single system of governance and accountability for public resources. In particular, s16 of the PGPA Act, supported by the Commonwealth Risk Management Policy (2014), requires agencies to establish appropriate systems for the management of risk. The goal of the Commonwealth Risk Management Policy is to embed risk management as part of the culture of Commonwealth agencies where the shared understanding of risk leads to well informed decision making.

1.11 Additional key references for the Committee’s inquiry into Audit Report No. 3 (2015-16), Regulation of Great Barrier Reef Marine Park Permits and Approvals, include the ANAO Better Practice Guide on Administering Regulation: Achieving the Right Balance (2014), which states that risk management is an ‘integral component of good regulatory administration’ and can be used to ‘support regulatory administration, the monitoring and management of regulatory compliance, and the efficient allocation of available resources’.2

1.12 Additional key references for the Committee’s inquiry into Audit Report No. 52 (2014-15), Australian Defence Force’s Medium and Heavy Vehicle Fleet Replacement (Land 121 Phase 3B), include the Commonwealth Procurement Framework through the Commonwealth Procurement Rules: Achieving Value for Money (CPRs), Department of Finance (July 2014), issued under the PGPA Act.3 The CPRs focus on achieving value for money through the efficient, effective, economical and ethical use of public resources, and ensuring accountability and transparency in government procurement activities. In terms of management of procurement risk, section 8.2 of the CPRs state that ‘relevant entities must establish processes for the identification, analysis, allocation and treatment of risk when conducting a procurement … Relevant entities should consider risks and their potential impact when making decisions relating to value for money assessments, approvals of proposals to spend relevant money and the terms of the contract’.

1.13 Additional key references for the Committee’s inquiry into Audit Report No. 9 (2015-16), Test and Evaluation of Major Defence Equipment Acquisitions, include Defence’s First Principles Review—Creating One Defence (April 2015) and the Defence Capability Development Manual (December 2014), which

2 ANAO Better Practice Guide, Administering Regulation: Achieving the Right Balance (2014), p. 14. See also Department of Prime Minister and Cabinet, The Australian Government Guide to Regulation (2014). 3 The procurement framework for the period covered by the ANAO audit was through the July

2012 Commonwealth Procurement Rules and, before that, the Commonwealth Procurement Guidelines—each of these documents require procurements to represent value for money for the Commonwealth and encourage competition in procurement.

SUMMARY 5

states that Defence T&E arrangements provide decision-makers with factual information to ‘help assess risks to achieving the desired capability. T&E in Defence is a deliberate and evidentiary process applied … to ensure that a system is fit-for-purpose, safe to use and that Defence personnel have been trained and provisioned with the enduring operating procedures and tactics to be an effective military force’.4

4 Defence, ‘Introduction to Test and Evaluation’, Defence Capability Development Manual (December 2014), p. 1.

6 REPORT 456: DEFENCE LAND 121, GBRMPA AND DEFENCE T&E

2

Performance Audit Report No. 52 (2014-15)

ADF’s Medium and Heavy Vehicle Fleet Replacement (Land 121 Phase 3B)

2.1 Chapter 2 focuses on the Joint Committee of Public Accounts and Audit (JCPAA) review of Australian National Audit Office (ANAO) Audit Report No. 52 (2014-15) Australian Defence Force’s Medium and Heavy Vehicle Fleet Replacement (Land 121 Phase 3B). The chapter comprises:

 inquiry context and background

 ANAO report overview

 Committee review

 Committee comment

Inquiry context and background

2.2 Project Overlander Land 121 is a multi-phased project to provide the Australian Defence Force (ADF) with new field vehicles and trailers to enhance ground mobility. Phase 3B1 of the project is to acquire medium and heavy trucks, modules and trailers, at a budgeted cost of $3.386 billion. The vehicles are a core element of ADF capability, and essential for the conduct of operations. They will be used for the movement of Army troops, assets and supplies in combat theatres,

1 Phase 3B forms the medium and heavy field vehicle, module and trailer component of Project Overlander LAND 121. Other current phases of Land 121 include Phase 3A light/lightweight vehicles; and Phase 4 Protected Mobility Vehicle. The ANAO report generally refers to Project Overlander LAND 121 Phase 3B as the medium and heavy vehicle fleet acquisition.

8

humanitarian operations, natural disaster relief, general peacetime operations and training.2

2.3 Land 121 Phase 3 received government first-pass approval in June 2004. At the time, the Department of Defence (Defence) considered that the medium and heavy vehicle acquisition was a relatively low risk military off-the-shelf (MOTS) procurement. Defence originally released a Request for Tender (RFT) for the medium and heavy vehicle segment in December 2005, but decided to retender in December 2008, due to concerns over the selected vehicles. Key milestones for the acquisition included:

 in August 2007, Defence received government second-pass approval to

enter negotiations with Stewart and Stevenson as the supplier for the Phase 3B vehicles and modules, and with Haulmark Trailers for the Phase 3B trailers;

 in August 2008, Defence withdrew from negotiations with Stewart and

Stevenson, citing technical and probity issues, and a tender resubmission process was initiated;

 in April 2011, Defence endorsed Rheinmetall MAN Military Vehicles-

Australia (RMMV-A) as the preferred supplier for the vehicles and modules, and Haulmark Trailers was confirmed as the preferred supplier for the provision of trailers; and

 in July 2013, Land 121 Phase 3B received a revised government

second-pass approval and Defence entered into contracts with RMMV-A and Haulmark Trailers.3

2.4 Defence is acquiring 2,536 medium and heavy trucks, and 2,999 modules, from RMMV-A; and 1,582 trailers from Haulmark Trailers. The total number of vehicles being acquired under Phase 3B is, however, 2707 following a decision to also acquire 122 Mercedes-Benz G-Wagon vehicles and 49 Thales Bushmaster vehicles. The capability will comprise a variety of vehicles including semi-trailers, recovery trucks, hook lift trucks and flatbeds in both protected and unprotected configurations.4

2 ANAO Audit Report No.52 (2014-15), p. 11. 3 ANAO Audit Report No.52 (2014-15), pp. 11-12. 4 ANAO Audit Report No.52 (2014-15), p. 12.

ADF’S MEDIUM AND HEAVY VEHICLE FLEET REPLACEMENT (LAND 121 PHASE 3B) 9

ANAO report overview

Audit objective and scope 2.5 The ANAO’s audit objective was to assess the effectiveness of Defence's management of the acquisition of medium and heavy vehicles, associated modules and trailers for the ADF. The audit focused on the acquisition of

the medium and heavy vehicle fleet from first-pass approval in 2004 through to early 2015.5

2.6 The high-level criteria developed by ANAO to assist in evaluating Defence's performance were:

 requirements definition, acquisition strategies and plans, and capability

development processes met Defence policy and procedures;

 procurement processes complied with the Financial Management and

Accountability Act 1997 (FMA Act) and Regulations, and other relevant Commonwealth and Defence procurement requirements; and

 the acquisition has progressed to the expectations of the

Commonwealth in terms of cost, schedule and delivery of required capability.6

Audit conclusion

Delayed capability 2.7 ANAO’s assessment of Defence’s initial tender process to acquire a replacement medium and heavy vehicle fleet was that this was flawed, resulting in a failed tender and a second approach to market, which

contributed to long delays in the acquisition of a modern medium and heavy vehicle capability.7

2.8 ANAO reported that Defence conducted a more effective tender resubmission process from 2008, but the process was protracted and Defence did not enter into contracts to supply the replacement fleet until July 2013. The aborted initial tender process and the time taken to finalise the tender resubmission process have delayed the scheduled achievement of Final Operational Capability by seven years to 2023. In the intervening period, Defence will continue to rely on an aged fleet of medium and heavy vehicles that is increasingly costly to operate, maintain and repair.8

5 ANAO Audit Report No.52 (2014-15), p. 13. 6 ANAO Audit Report No.52 (2014-15), p. 13. 7 ANAO Audit Report No.52 (2014-15), p. 14. 8 ANAO Audit Report No.52 (2014-15), p. 14.

10

Initial tender 2.9 Defence originally considered that the medium and heavy vehicle acquisition was a relatively low risk military off-the-shelf procurement. ANAO assessed that the difficulties subsequently experienced by Defence

in acquiring a new medium and heavy vehicle fleet can mostly be attributed to shortcomings in its initial tender process between 2005 and 2007. Defence did not conduct any preliminary test and evaluation of vehicles before recommending a single supplier to the then Government. In selecting a preferred supplier, the ANAO assessed that Defence also did not have sufficient regard to all relevant costs and benefits identified in its tender evaluation process, so as to adhere to the Government’s core principle of value for money. Defence’s 2007 Source Evaluation Report initially ranked a proposal from Stewart and Stevenson last of five tenders on the basis of value-for-money, but elevated the proposal to the position of preferred tender because it was the most affordable—notwithstanding Defence’s assessment of significant vehicle deficiencies against its specific requirements, and the identification of many acquisition risks in the course of the tender process.9

2.10 Further, ANAO reported that Defence did not advise Ministers of the significant capability and technical risks it had identified, before recommending a single supplier. Defence confirmed the previously identified shortcomings through test and evaluation after the acquisition entered an Offer Definition and Refinement Process, and the preferred supplier’s vehicles were tested. Defence subsequently cancelled contract negotiations with the preferred supplier.10

Tender re-submission 2.11 In December 2008, Defence again approached the market and implemented a more robust tender process, drawing on key lessons learned from the initial tender process. Defence conducted preliminary

test and evaluation of vehicles supplied by five companies, before shortlisting three suppliers and asking them to submit tenders. In April 2011, Defence selected RMMV-A as its preferred vehicle supplier on the basis of value-for-money. However, the protracted Offer Definition and Refinement Process with RMMV-A required escalation to senior leaders and, as a consequence, Defence was not in a position to approach the then Government for second-pass approval until July 2013.11

9 ANAO Audit Report No.52 (2014-15), pp. 14-15. 10 ANAO Audit Report No.52 (2014-15), p. 15. 11 ANAO Audit Report No.52 (2014-15), p. 15.

ADF’S MEDIUM AND HEAVY VEHICLE FLEET REPLACEMENT (LAND 121 PHASE 3B) 11

Capability 2.12 ANAO reported that Defence has not applied a rigorous approach to capability definition throughout the acquisition of the medium and heavy vehicle fleet. Defence did not complete or update its mandated Capability

Definition Documents for the initial and revised government second-pass approvals in 2007 and 2013, or when negotiating and entering into contractual arrangements. Defence also developed a variety of non-standard documents to compensate for the absence of updated Capability Definition Documents; an approach which unnecessarily added to procurement risk. In addition, Defence applied different methodologies over time to determine the acquisition’s Basis of Provisioning,12 a process intended to measure the number of each vehicle type required by Army to meet its capability objectives. Further, Defence’s Basis of Provisioning for the medium and heavy vehicle fleet has been amended on many occasions during the acquisition process to reflect the number of vehicles Defence could afford, rather than the number of vehicles it required to deliver the defined capability—a pragmatic approach which did not align with the key purpose of the Basis of Provisioning process. In the light of this experience, ANAO advised that Defence should review its 1999 Instruction to provide contemporary guidance on the Basis of Provisioning for the acquisition of specialist military equipment for the ADF.13

2.13 Defence advised that, as at March 2015, total expenditure on the medium and heavy vehicle fleet acquisition was $112 million, with most expenditure to be incurred from mid-2016 when truck production commences. Defence further advised that there was sufficient budget remaining for the project to complete against its agreed scope, and the project had not applied any contingency funding to date. Under applicable budgeting arrangements, Defence is able to use approved funding later in the project, if it is not spent at the time initially anticipated due to project delays.14

Systems integration 2.14 Defence remains confident that it will meet the acquisition’s current critical milestones, the first being the commencement of Introduction Into Service Training in September 2016. Key issues that have affected the

12 The Basis of Provisioning is a process for determining and recording the quantity of an asset that Army is required to hold in order to support preparedness and mobilisation objectives. Defence, Defence Instruction (Army) 64-1 Basis of Provisioning, December 1999, p. AL1 cited in ANAO Audit Report No.52 (2014-15), p. 19. 13 ANAO Audit Report No.52 (2014-15), pp. 15-16. 14 ANAO Audit Report No.52 (2014-15), p. 16.

12

project since contract signature include: delays experienced by RMMV-A in engaging sub-contractors to develop modules; and a range of systems integration issues. The ANAO has previously observed that cost and schedule risks tend to rise when acquisition programs approach the complex stage of systems integration, and Defence will need to maintain a focus on managing the remaining integration issues. Defence has worked with RMMV-A to manage the vehicle production schedule and production of the initial test vehicles commenced in April 2015. 15

Budgeted expenditure and value-for-money 2.15 The overall project delay of seven years has obliged Defence to continue to operate its in-service fleet of vehicles, delivered between 1982 and 2003. The current fleet is becoming increasingly unreliable and costly to

maintain, and Defence has sought to achieve savings by disposing of uneconomical vehicles. While Defence currently expects to deliver the project within budget, the audit illustrates the impact of protracted procurement and approval processes on both Defence and industry suppliers.16

2.16 Against a background of other major Land Systems acquisitions approaching key milestones, this audit underlines the benefits of early test and evaluation of prospective vehicles, which strengthen Defence’s ability to identify and mitigate risks, and provide informed advice for decision-making on a preferred supplier. Further, having commenced a tender process, Defence needs to keep in view the Government’s core rule of achieving value-for-money, which continues to require consideration of relevant financial and non-financial costs and benefits of each proposal.17

Audit recommendations Table 2.1 ANAO recommendations, Audit Report No.52 (2014-15)

1 To provide greater certainty in the development of relevant

assessments and advice, the ANAO recommends that Defence develop contemporary guidance on the Basis of Provisioning for the acquisition of specialist military equipment for the Australian Defence Force.

Defence response: Agreed.

15 ANAO Audit Report No.52 (2014-15), pp. 16-17. 16 ANAO Audit Report No.52 (2014-15), p. 17. 17 ANAO Audit Report No.52 (2014-15), p. 17.

ADF’S MEDIUM AND HEAVY VEHICLE FLEET REPLACEMENT (LAND 121 PHASE 3B) 13

Committee review of evidence

2.17 The Committee’s key sources of evidence for this inquiry were the ANAO’s audit report, evidence given by representatives from the ANAO and Defence at the Committee’s public hearing on 3 March 2016, and agency submissions (see details of public hearings and submissions at Appendixes A and B). The Committee also investigated relevant public sector frameworks.

2.18 The Committee focused on the following matters:

 implementation of ANAO recommendations

 initial tender: poor test and evaluation

 initial tender: flawed value-for-money assessment

 initial tender: advice to Ministers

 ultimate extra cost of failed process

 accountability: who was responsible and what happened to them?

 new risk assessment procedures through First Principles Review

Implementation of ANAO recommendations 2.19 At the public hearing, Defence provided an update on implementing the ANAO recommendation:

Defence Instruction (Army) Administration 64-1 forms the basis of provisioning guidance and is the extant policy on [Basis of Provisioning], with the extant document released on 10 December 1999. A significant rewrite has been undertaken as a result of the audit, to bring it up to contemporary demands. This contemporary approach has been effectively used recently on a trial basis in the approval of Land 121 Phase 4. The draft rewrite of the DIA admin 64-1 has been completed and is now out for consultation and discussion with subject matter experts, both internal and external to Army, for review. Army expects to have the review comments finalised by 1 April 2016, with the updated

policy released no later than 1 July 2016.18

2.20 Defence also provided an example of where it had put its improved processes into practice:

The audit report, quite appropriately, told us to revise that process… Under the Hawkei provision, for example, we have a

18 Major General Fergus McLachlan, Head, Modernisation and Strategic Planning, Army, Department of Defence, Committee Hansard, 3 March 2016, p. 1.

14

provision inside the Defence Capability Plan, as it was then—it would now be the Integrated Investment Program—that reflects what we have to achieve our acquisition under. We do the best we can in partnership with our acquisition agency as they go into a contract negotiation with the supplier to get the price down to the extent that is possible. We make cost/capability trade-offs through that process... We made sensible trade-offs about the number of those that we would need and we came up with a number for the vehicles that was short of our optimum but was better than we had at the start of the process. The simple reality is that it is a process that we have to responsibly take within the provision that we have been allocated by government.19

2.21 Finally, Defence also explained what procedure have been put in place to meet the LAND 121 acquisition’s current critical milestones:

As far as schedule is concerned, we are now on schedule and on budget. There were some initial delays with Rheinmetall MAN Military Vehicles in providing a level of technical support required, and Rheinmetall has markedly improved its performance in this area. The issue was raised to board level. There is a weekly videoconference between Victoria Barracks, Melbourne, and Germany at a senior level to ensure that we remain on track.

The first vehicles have arrived in Australia. They arrived last month and the first 12 vehicles will be delivered by April 2016. As far as trailers are concerned, we are slightly ahead in trailers of where we need to be and the modules are on track.20

Initial tender: poor test and evaluation 2.22 Defence was asked to explain the failure of process where they did not conduct any preliminary test and evaluation (T&E) of vehicles before recommending a single supplier to the then Government. Defence

responded:

The initial issue with this as a test and evaluation was that the initial acquisition predated the formal establishment of the Australian Defence Test and Evaluation Office. There were some elements of policy and process that we had not fully resolved in terms of how we would plan for and conduct preview T&E [test

19 Major General Fergus McLachlan, Defence, Committee Hansard, 3 March 2016, p. 7. 20 Major General David Coghlan, Head, Land Systems Division, Department of Defence Committee Hansard, 3 March 2016, p. 3.

ADF’S MEDIUM AND HEAVY VEHICLE FLEET REPLACEMENT (LAND 121 PHASE 3B) 15

and evaluation]. In the initial acquisition, based on the assessment that this was an off-the-shelf acquisition with elements of commercial off-the-shelf and elements of military off-the-shelf, the decisions that were taken at that time suggested that we did not need to conduct detailed preview test and evaluation.21

2.23 Importantly, Defence itself acknowledged the ANAO’s criticisms:

I fully accept the criticisms of the audit office here that, with hindsight, we should have done a much better job in the way we conducted this test and evaluation and the risks we took. At the time, the people who were there obviously made an assessment and made that decision. With hindsight we accept the recommendations of the audit office, but we also have implemented, through the recommendations of the First Principles Report, a significant change to the T&E regime and the risk assessments that we do when there is an environment where there is a modification or there is a requirement to take a commercial off-the-shelf piece of equipment. I think at that point in time we were probably in an environment where we were just taking COTS vehicles at face value without actually understanding the true risk, and we have learnt from that. We are looking at a number of different acquisitions at the moment, and we are doing things very differently these days.22

2.24 Defence outlined its risk management approach to this area at the time:

The dilemma between first and second pass in the early stages of LAND 121 Phase 3B certainly illustrated very clearly the risk of: no matter how mature we think a design might be, the risk was in relying on what ended up being a paper based down-selection based on advice from the suppliers or the tenderers that were involved at the time and involving only the tendered documents of would-be suppliers to make those assessments. It results in a situation where Defence engineers have misunderstood or not fully understood the technical and performance risks in those tenders without the accompanying opportunity to do the preview test and evaluation. In effect, that is where we are now going...

[T]he assessment at the time, based on the data that we had, in accordance with the procurement rules, was that the risks were tolerable and that the understanding of the detail and data that

21 Air Vice Marshal Mel Hupfeld, Acting Chief, Capability Development Group, Department of Defence, Committee Hansard, 3 March 2016, p. 2. 22 Mr Kim Gilles, Deputy Secretary, Capability Acquisition and Sustainment Group, Department of Defence, Committee Hansard, 3 March 2016, p. 4.

16

was provided by the tenderers was of sufficient quality and technical coherency that we would be able to proceed without doing any test and evaluation. And, indeed, as I said, our test and evaluation policy and process then was not as robust as it is now.23

2.25 Defence further explained what measures they were putting in place to address the mistakes made:

We are now putting policy governance and process in place to address how we would do preview test and evaluation in the future to learn lessons from what has occurred under Land 121 Phase 3B. The key point is that it is still a risk based approach to take to look at what is necessary given that there are significant costs involved in doing any sort of test and evaluation, including preview test and evaluation. That was omitted in the first part of the acquisition phase for Land 121 Phase 3B.24

Initial tender: flawed value-for-money assessment 2.26 There was also interest in further understanding why, in the initial tender, Defence had elevated the most affordable tender above all other tenders despite it initially being ranked last of five on a value-for-money basis and

since Defence had already assessed the proposal as having significant vehicle deficiencies against its specific requirements and many acquisition risks.25

2.27 Importantly, the ANAO noted that Commonwealth procurement guidelines at the time:

…had an expectation that in assessing value for money in a procurement context there would be a comparative analysis of all relevant costs and benefits throughout the procurement cycle. The point we were making in the audit … was that it seemed inconsistent with that general principle in the procurement guidelines of the day … that somehow in this process—which had competitive elements; it was not a sole source; I think there were five tenderers in the process—for some reason the least competitive on an important set of dimensions was elevated to the key position.26

23 Air Vice Marshal Hupfeld, Defence, Committee Hansard, 3 March 2016, pp. 3-4. 24 Air Vice Marshal Hupfeld, Defence, Committee Hansard, 3 March 2016, p. 2. 25 ANAO Audit Report No.52 (2014-15), pp. 14-15. 26 Dr Tom Ioannou, Group Executive Director, Performance Audit Services Group, Australian

National Audit Office, Committee Hansard, 3 March 2016, p. 4.

ADF’S MEDIUM AND HEAVY VEHICLE FLEET REPLACEMENT (LAND 121 PHASE 3B) 17

Initial tender: advice to Ministers 2.28 Defence acknowledged that the then Defence Minister had not been advised of the significant capability and technical risks it had identified, before recommending a single supplier. 27 When asked why the Minister

hadn’t been informed, Defence responded:

The preferred tenderer was advised to Government in August 2007 as part of Second Pass approval where the technical risk was assessed as medium. This did not involve the selection of a particular vehicle. The preferred approach was to proceed to an Offer Definition and Refinement Period (ODRP) with a single preferred tenderer. Government was advised that prior to contract signature Defence would define, refine and confirm the tendered offers based on the key issues, activities and deliverables identified during evaluation.

A comparative ODRP to mitigate risk leading to final source selection occurred in the second quarter of 2008.

Introductory negotiations during the ODRP, along with a vehicle demonstration and compliance test, identified an increased level of technical risk and gave rise to significant concerns with the preferred tenderer’s capacity to deliver against its tendered offer.

Due to increased levels of technical, cost and schedule risk identified, Government was advised in June 2008 that Defence could exercise its discretion under the conditions of tender and invite all initial tenderers back into the evaluation process, and initiate a tender resubmission process.28

Ultimate extra cost of failed process 2.29 Questions at the public hearing investigated whether revisions to the acquisition over time may have resulted in less protected and more unprotected vehicles being purchased, but for higher overall cost. Defence

was asked to provide more definitive numbers of protected and unprotected vehicles and the reason for the significant increase in cost. 29 In response, Defence provided the following tables30 that give exact number of vehicles:

27 ANAO Audit Report No.52 (2014-15), pp. 64-65. 28 Department of Defence, Submission 4.1, p. 7. 29 Acting Chair, Mr Pat Conroy MP, Committee Hansard, 3 March 2016, p. 7. 30 Tables are included in Department of Defence, Submission 4.1, p. 10.

18

2.30 Defence also provided the following comparison of the Second Pass approval costs for the original proposal of 2007, and the second in 2013:

The total cost of the Medium-Heavy Capability element of LAND 121 Phase 3 at the 2007 Second Pass approval was $2,572.5 million (2012-13 Portfolio Budget Statement Outturned).

The total cost of LAND 121 Phase 3B (Medium-Heavy Capability) at the 2013 Second Pass approval was $3,298.2 million (2013-14 Portfolio Budget Statement Outturned).

The difference between the 2007 and 2013 figures is $725.7 million. This included $712.2 million of LAND 121 supplementation approved by Government as part of the second pass approval to deliver the required capability. Key factors included an increase in vehicle numbers under a revised basis of provisioning; changes to operational requirements (including updated requirements for digitisation); and changes to legislative requirements (including environmental and work health and safety).

A further $13.5 million was received to provide for exchange rate variation between June 2012 and Second Pass approval in 2013.33

2.31 In addition, Defence was asked what the overall total cost of the delay in providing the new vehicles has been. Defence responded:

The cost of the LAND 121 Phase 3B acquisition delay is approximately $25 million (excluding GST).34

31 Basis of Provisioning 32 Basis of Provisioning 33 Department of Defence, Submission 4.1, p. 11. 34 Department of Defence, Submission 4.1, p. 1

Basis of Provisioning at 2007 2nd Pass

Total BOP 31 Protected Unprotected

2471 1433 1038

Basis of Provisioning at 2013 - 2nd Pass

Total BOP 32 Protected Unprotected

2707 1172 1535

ADF’S MEDIUM AND HEAVY VEHICLE FLEET REPLACEMENT (LAND 121 PHASE 3B) 19

Accountability: who was responsible and what lessons have been learned? 2.32 The Committee has, in the past, asked questions about whether there are ramifications for the senior executives of organisations involved in making

poor procurement and management decisions.

2.33 The Committee again asked Defence whether there were ramifications for the individuals involved when mistakes have been acknowledged, noting that the Committee did not seek any information that would identify the individuals. The committee merely sought information as to whether there were any disciplinary or other consequences for the individuals that made the poor procurement and management decisions. However, at the time of finalising this report Defence had not responded to this question. The Committee looks forward to receiving a response and remains interested in this matter.

2.34 Defence were also asked how corporate knowledge is maintained regarding T&E - i.e. how will the lessons learnt from projects such as LAND 121 Phase 3B be kept and transferred to new staff. Defence responded:

Corporate knowledge is maintained through the standardisation of Test and Evaluation (T&E) documents such as the Test Concept Document (TCD) at First Pass, the Early Test Plan at Second Pass and the Test & Evaluation Master Plan (TEMP) post Second Pass. These documents are endorsed and approved by senior stakeholders at a One Star/SES Band 1 level and ensure continuity of the test planning phase. Trial Reports are also approved by the senior T&E manager in an agreed policy and quality accredited format, which are formal Defence records.

Knowledge transfer can be a challenge particularly relating to staff turn-over. Military posting cycles allow for a three year period and, in the main, members are able to conduct a handover to ensure corporate knowledge is maintained, with postings promulgated 3-6 months in advance. Australian Public Service employees tend to stay longer and provide consistency in regard to corporate knowledge.

Knowledge transfer in Defence is encouraged through a number of initiatives such as mentoring, networking, and transition to retirement programs. In addition, some areas of Defence plan a staggered rotation of staff, and some flexibility exists to extend military staff for an additional year to assist with this planning.

The individual T&E competencies for Defence are now being reviewed through a formal Skills Census that commenced 7 March

20

2016 and is due to be completed by 27 May 2016. The results will be used to improve T&E training and management, as recommended by the ANAO audit Test and Evaluation of Major Defence Equipment Acquisitions, Report No 9, 2015-16, and to update the current policy in the Defence Capability Development Manual. 35

New risk assessment procedures through First Principles Review 2.35 The First Principles Review, announced in April 2015, is a fundamental re-structure of Defence, particularly with regard to equipment procurement and sustainment given the abolition of the Defence Materiel Organisation

and the re-absorption of its functions into the Department proper.

2.36 Defence explained the relationship between the First Principles Review and the improvement of procurement and T&E policies:

The First Principles Review is certainly one of the strong elements that would look at addressing some of these concerns. It certainly drives to accountabilities within the department and details those accountabilities at every phase of the capability life cycle. The Department of Defence is committed to that and we are obliged to return to government based on their commissioning of the report. We have to report back and demonstrate that we are implementing under the First Principles Review….

Inevitably, the business that we are in is about trying to balance affordability against the capability needs that we have and making the appropriate trade-offs to provide government with the best options to deliver the capability that is required to meet the strategic requirements that are given for the Department of Defence. Those trade-offs entail assessment of risk. What we have been discussing this morning is being able to better understand, articulate and then make appropriate recommendations to government for approvals based on those risks to get the best possible outcomes that we can so that we get efficiency in both the acquisition and then ongoing sustainment. The First Principles Review goes to the heart of those issues. We are right in the middle of planning those elements. We will shortly start to implement, once we gain approvals through the appropriate senior leadership of the department up to government.36

35 Department of Defence, Submission 4.1, pp. 4-5. 36 Air Vice Marshal Hupfeld, Defence, Committee Hansard, 3 March 2016, pp. 5-6.

ADF’S MEDIUM AND HEAVY VEHICLE FLEET REPLACEMENT (LAND 121 PHASE 3B) 21

2.37 Defence explained how its new smart buyer model was consistent with what the First Principles Review was trying to achieve. When asked about ‘undue processes’ in the long LAND 121 acquisition process in comparison with the much shorter acquisition of similar vehicles conducted by New Zealand,37 Defence responded:

I think you have just synthesised what the new capability development life cycle is, the smart buyer model, the First Principles and what Defence is implementing currently, which is that, for these types of processes, there are smarter acquisition processes… In essence, the change that has happened—and it is currently happening as we speak—the move towards the new capability development life cycle, which is a risk based assessment, against what is the best acquisition strategy for less complex, complex and significantly complex acquisitions, and the smart buyer model which is: how would a very efficient and effective organisation best utilise its resources to buy equipment for the Defence Force? That is the undertaking that I have on behalf of the secretary in CDF.38

Committee comment

2.38 The results of the LAND 121 Phase 3B acquisition process emphasise the need for better risk management practices for Defence acquisition projects.

2.39 The failed first tender process resulted in an estimated seven-year delay to delivering the vehicles and a need to sustain the existing fleet far beyond initial specifications. The total cost to taxpayers has been quantified by Defence at $25 million. However, Defence only provided an annual cost,39 and therefore the total cost of sustaining the existing fleet remains unknown. The Committee is disappointed that a comprehensive answer

37 Relating specifically to LAND 121 Phase 3, the Asia Pacific Defence Reporter made an unfavourable comparison between Australia and New Zealand which had also ordered Rheinmetall MAN Military Vehicles Australia (RMMV-A) trucks. They said: Compare this to the situation in New Zealand, where their Department of Defence announced on

May 15 the acquisition of 200 Rheinmetall-MAN vehicles for NZ $135 million... Despite ordering vehicles of identical complexity to Australia’s, the NZDF will take delivery of all of their trucks by the end of 2014 - at least two years earlier than the ADF. Every New Zealand truck will be in service by the end of 2015, while the last of Australia’s will still be arriving in 2020. Bergmann, K., “Truck contract - is this finally the end?” Asia Pacific Defence Reporter, 25 August 2013, accessed 24 February 2016. 38 Mr Kim Gilles, Defence, Committee Hansard, 3 March 2016, p. 5. 39 ANAO Audit Report No.52 (2014-15), p. 97.

22

was not provided by Defence given that the total sustainment costs can be assumed to be in the hundreds of millions.

2.40 Regarding the additional cost of providing the new capability, the Committee notes that the second pass approval budget increased by $725.7 million between 2007 and 2013 as a result of a revised Basis of Provisioning and other factors. This cost increase is in the context of acquiring 261 fewer protected vehicles, which are the most expensive class of vehicles in this project.

2.41 The Committee expresses strong disappointment in this flawed process and costly outcome.

2.42 Given the frequency with which Defence procures new equipment, it is difficult to understand how such fundamental mistakes as poor test and evaluation of the proposals, significantly flawed value-for-money assessments, and failure to inform the Minister of the significant capability and technical risks could be made. These are all public sector procurement basics, and the size and complexity of the procurement should have only served to increase attention on such fundamentals.

2.43 Despite these mistakes, Defence did not provide information on whether there were any ramifications for the individuals involved. This exposes Defence to criticism that there are no ramifications.

2.44 The Committee also notes the ANAO observations and recommendation on Defence’s Basis of Provisioning for the medium and heavy vehicle fleet. This process was amended on many occasions during the acquisition process to reflect the number of vehicles Defence could afford, rather than the number of vehicles it required to deliver the defined capability—an approach which did not align with the key purpose of the Basis of Provisioning process. The Committee was therefore pleased to note Defence’s progress in implementing the ANAO recommendation, in particular that they are nearing finalisation of a updated administrative instruction on this matter and have started to put improved processes into practice.

2.45 When asked why ANAO had made only one recommendation in its report despite the series of failures uncovered, ANAO responded:

… we do not recommend that which already exists or is required to be done... We believe that Defence has for a long time understood the importance of test and evaluation. We believe that the essentials for running a good process and an effective process were understood in Defence, but they were not necessarily followed in the first process. There was evidence of improvement in the second process, and we have reported that. On balance we

ADF’S MEDIUM AND HEAVY VEHICLE FLEET REPLACEMENT (LAND 121 PHASE 3B) 23

did not feel it was necessary to remind Defence of what it already knows.40

2.46 Defence presented a strong argument that the lessons of the failed first tender process have been learned and the ANAO acknowledged that many of the issues they exposed have been recognised by Defence and reforms are being implemented.41 In addition, the Committee acknowledges that the reforms underway as part of the FPR process could minimise the risks of similar issues. A good example of this is the strengthening of the T&E office and its increased senior officer accountability.

2.47 However, given the significance of the audit findings and that the FPR reforms are still ongoing, the Committee seeks additional assurance that the lessons learned through this acquisition are being firmly embedded and broadly disseminated across Defence.

Recommendation 1

2.48 The Committee recommends that the Department of Defence provide evidence to the Committee that the lessons learnt from the failed first tender process for LAND 121 Phase 3B have been incorporated into the department’s standard operating procedures for acquisitions. This evidence can take the form of:

 revised manuals;

 specific internal case studies and training briefings; and

 any other relevant material.

40 Dr Tom Ioannou, Australian National Audit Office, Committee Hansard, 3 March 2016, p. 6. 41 See to comment by Dr Tom Ioannou, Australian National Audit Office, Committee Hansard, 3 March 2016, p. 4: “This audit itself shows that when Defence conducted its second process it was more soundly based.”

24

3

Performance Audit Report No. 3 (2015-16)

Regulation of Great Barrier Reef Marine Park Permits and Approvals

3.1 Chapter 3 focuses on the Joint Committee of Public Accounts and Audit (JCPAA) inquiry into Australian National Audit Office (ANAO) Report No. 3 (2015-16), Regulation of Great Barrier Reef Marine Park Permits and Approvals, Great Barrier Reef Marine Park Authority (GBRMPA). The chapter comprises:

 inquiry context and background

 ANAO report overview

 Committee review of evidence

 Committee comment

Inquiry context and background

3.2 In recognition of the environmental significance of the reef, the Australian Government established the Great Barrier Reef Marine Park (Marine Park) under the Great Barrier Reef Marine Park Act 1975 (GBRMP Act).1 The Marine Park was established to provide for the long term protection and conservation of the environment, biodiversity and heritage values of approximately 344,000 square kilometres of the Great Barrier Reef region.

1 The regulatory framework for the Marine Park includes the GBRMP Act and a number of subsidiary legislative instruments: the GBRMP Zoning Plan 2003; Plans of Management; and the GBRMP Regulations 1983.

26

Subsequently, in 1981, the Great Barrier Reef was declared a World Heritage Area.

3.3 GBRMPA was established in 1975 under the GBRMP Act. The GBRMP Act prohibits the conduct of particular activities in the Marine Park without a permit granted by GBRMPA, including most commercial activities; operation of jetties, marinas, pontoons, and moorings; significant works, such as dredging and spoil dumping; and educational and research programs.2 GBRMPA and the Queensland Parks and Wildlife Service operate a joint application and assessment process for permit requests covering the Marine Park and Queensland’s Great Barrier Reef Coast Marine Park. Under current arrangements, GBRMPA assumes the lead role in the assessment of joint permits, although delegates from each jurisdiction are required to approve the permits.3

3.4 GBRMPA assesses approximately 400 permit applications each year. Over 10 years from 2004-05 to 2013-14, 4296 permits (excluding permit transfers) were issued, containing 6337 individual permissions.4 As at August 2014, 1334 permits containing 2408 individual permissions were current—85.8 per cent related to tourism operations (1488), operating a facility or mooring (311) and research activities (267).5 Monitoring of permit holders’ compliance with permit conditions is undertaken through a combination of desk-based compliance monitoring and targeted site inspections by GBRMPA’s Environmental Assessment and Protection (EAP) Section, and vessel, aerial and land-based patrols/surveillance by GBRMPA and its partner agencies6 under the Joint Field Management Program. At any one time, GBRMPA and its partner agencies are responsible for monitoring the compliance of approximately 1300 permit holders. In the period from July 2012 to June 2014, there were 76 reported breaches of permit conditions, which resulted in 59 investigations.7 (See

2 ANAO, Regulation of Great Barrier Reef Marine Park Permits and Approvals, Audit Report No. 3 (2015-16), p. 13. 3 Most visitors to the Marine Park do not require a permit for recreational activities, and most recreational and commercial fishing activities in the Marine Park are subject to state permits

and licences issued by Fisheries Queensland outside of the joint permitting system, ANAO, Audit Report No. 3, p. 13, p. 42. 4 Each permit may contain one or more individual permissions. While permit approvals for new applicants are generally issued for one year, multi-year permits are available for those seeking

replacement of an expiring permit, ANAO, Audit Report No. 3, p. 13. 5 ANAO, Audit Report No. 3, pp. 13-14. 6 Including Border Protection Command, Queensland Boating and Fishing Patrol, and the Queensland Police Service. The Field Management Compliance Unit, funded under the

program, comprises officers from GBRMPA and QPWS. 7 ANAO, Audit Report No. 3, p. 15. (See Table 7.1, p. 113, for investigation results.)

REGULATION OF GREAT BARRIER REEF MARINE PARK PERMITS AND APPROVALS 27

the ANAO report for a detailed description of the Marine Park permit system.)

ANAO report overview

Audit objective, scope and criteria 3.5 The ANAO’s audit objective was to assess the effectiveness of GBRMPA’s regulation of permits and approvals within the Great Barrier Reef Marine Park.8 To form a conclusion against this objective, the ANAO adopted the

following high-level criteria:

 an effective process to assess permit applications and attach

enforceable conditions has been established;  a structured risk management framework to assess and manage

compliance risks has been implemented;  an effective risk-based compliance program to communicate

regulatory requirements and to monitor compliance with permit conditions and regulatory objectives has been implemented; and  arrangements to manage non-compliance are effective.9

3.6 The ANAO reviewed GBRMPA’s files and records,10 accompanied departmental staff on compliance monitoring activities and assessed the controls for two relevant IT systems. Staff from GBRMPA, and staff of QPWS assigned to the Joint Field Management Program, were also interviewed, and the views of relevant stakeholders sought.11

ANAO overall conclusion 3.7 Overall, the ANAO report concluded that:

identified shortcomings in GBRMPA’s regulatory processes and, more particularly, its regulatory practices have undermined the effectiveness of the permitting system as a means of managing risks to the Marine Park. These shortcomings were identified across a broad range of GBRMPA’s regulatory activities, including

8 ANAO, Audit Report No. 3, p. 39. 9 ANAO, Audit Report No. 3, p. 39. The ANAO examined GBRMPA’s assessment of permit applications against Commonwealth requirements only. 10 The ANAO examined samples of permit application assessments and monitoring of current

permits over the period July 2012 to June 2014, and all ‘breach of permit’ enforcement actions over the period July 2012 to June 2014, ANAO, Audit Report No. 3, p. 39. 11 This included eight responses from permit holders (from 152 requests) and 17 responses from general stakeholders (from 67 requests), as well as two unsolicited responses, ANAO, Audit

Report No. 3, p. 40.

28

its assessment of permit applications, monitoring of permit holder compliance and response to non-compliance.12

ANAO recommendations and agency response 3.8 Table 3.3 sets out the recommendations from ANAO Report No. 3— GBRMPA agreed to all five recommendations without qualification.13

Table 3.3 ANAO recommendations, Report No. 3 (2015-16)

1

2

3

4

5

To improve the processing of Marine Park permit applications, the ANAO recommends that GBRMPA: (a) review and finalise standard operating procedures and administrative guidance for the permit application and

assessment process (b) reinforce to staff the need to document whether permit application assessment requirements have been addressed To improve the rigour of permit application assessment and decision- making processes, the ANAO recommends that GBRMPA: (a) prepare and revise permit application and risk assessment

templates to better address assessment considerations and risks relevant to the various permit types (b) reinforce to staff the importance of preparing assessment reports for delegates that adequately address regulatory assessment

requirements To improve the effectiveness of permit conditions used to manage risks to the Marine Park from permitted activities, the ANAO recommends that GBRMPA periodically review the adequacy of standard permit conditions To improve the effectiveness of permit compliance monitoring, the ANAO recommends that GBRMPA: (a) develop and enhance standard operating procedures for

undertaking compliance monitoring activities (including in relation to post-approval reporting requirements) (b) implement a coordinated, risk-based program of compliance monitoring activities To improve processes for responding to instances of permit non-compliance, the ANAO recommends that GBRMPA: (a) update and finalise guidance documentation for managing non-

compliance (b) reinforce to staff the need for all instances of non-compliance by permit holders to be reported and recorded in the Compliance Management Information System (c) document the reasons for key decisions taken during permit

investigations, including whether to investigate incidents and enforcement decisions (d) verify that enforcement action has been undertaken prior to the closure of investigations

12 ANAO, Audit Report No. 3, p. 15. 13 For details of GBRMPA’s response to the ANAO’s recommendations, see ANAO, Audit Report No. 3, pp. 23-24, pp. 125-126.

REGULATION OF GREAT BARRIER REEF MARINE PARK PERMITS AND APPROVALS 29

Committee review of evidence

3.9 The Committee’s key sources of evidence for this inquiry were the ANAO’s audit report, evidence given by representatives from the ANAO and GBRMPA at the Committee’s public hearing on 11 February 2016, and submissions from agencies and industry (see details of public hearings and submissions at Appendixes A and B).

3.10 The Committee also investigated relevant public sector frameworks, and past reviews, reports and audits as detailed below.

Key audit themes and relevant public sector frameworks 3.11 A key audit theme emerging from the Committee’s inquiry into ANAO Report No. 3 is effective risk management in the context of better practice administration of regulation. Key frameworks here are the relevant

sections of the Public Governance, Performance and Accountability Act 2013 (PGPA Act) and PGPA Rule 2014, and associated guidance. As the primary piece of Commonwealth resource management legislation, the PGPA Act establishes a single system of governance and accountability for public resources. In particular, s16 of the PGPA Act, supported by the Commonwealth Risk Management Policy (2014), requires agencies to establish appropriate systems for the management of risk. In terms of GBRMPA’s regulation of Marine Park permits, this includes implementation of a structured risk management framework to assess and manage compliance risks, and an effective risk-based compliance program to communicate regulatory requirements and monitor compliance with permit conditions. Another key reference is the ANAO Better Practice Guide on Administering Regulation: Achieving the Right Balance (2014).14

Relevant reviews, reports and audits 3.12 There have been a range of reviews, reports and audits concerning GBRMPA and the Marine Park, including:

 ANAO Report No. 33, Commonwealth Management of the Great Barrier

Reef (1998)

 ANAO Report No. 8, Commonwealth Management of the Great Barrier Reef

Follow-up Audit (2003)

14 See also Department of Prime Minister and Cabinet, Australian Government Guide to Regulation (2014). Chapter 1 further discusses risk management in the Commonwealth public sector as a key audit theme across the three ANAO reports covered by the Committee’s report.

30

 GBRMPA, Great Barrier Reef Outlook Report 2014 (2014), Great Barrier Reef

Region Strategic Assessment (2014) and Great Barrier Reef Region Strategic Assessment: Program Report (2014)

 Senate Environment and Communications References Committee,

Management of the Great Barrier Reef (September 2014)

 Australian and Queensland Governments, Reef 2050 Long-Term

Sustainability Plan (March 2015)

Key issues 3.13 The Committee focused on four matters regarding the ANAO report findings and evidence provided at the public hearing and in submissions to the inquiry:

 implementation of ANAO recommendations and stakeholder

consultation

 permit application processing, assessment and approval

 managing compliance

 responding to non-compliance

Implementation of ANAO recommendations and stakeholder consultation 3.14 Two previous audits of GBRMPA have been undertaken.15 During the Committee’s inquiry, the ANAO raised no issues regarding

implementation of recommendations from these reports.

3.15 GBRMPA agreed to all five recommendations from the latest ANAO audit. At the public hearing and in its submission to the inquiry, GBRMPA provided an update on implementation of these recommendations—each of which is discussed in the relevant sections below.16 Importantly, the Auditor-General noted that GBRMPA had ‘acknowledged weaknesses in its permit assessment and compliance management … and commenced work on a number of initiatives to strengthen existing practices’.17

3.16 GBRMPA emphasised that a project to strengthen the permissions system had commenced prior to the ANAO audit (the ANAO report was

15 ANAO Report No. 33, Commonwealth Management of the Great Barrier Reef, 1998, and ANAO Report No. 8, Commonwealth Management of the Great Barrier Reef Follow-up Audit, 2003. 16 Implementation progress is tracked through GBRMPA’s Audit Committee, as well as its board and Executive Management Group, GBRMPA, Submission 1, p. 5. GBRMPA’s Corporate Plan

2015-2020 also reflects the ANAO recommendations, ANAO Report No. 3, p. 24. 17 Mr Grant Hehir, Auditor-General, ANAO, ‘Opening statement by Auditor-General’, Submission 3, p. 2.

REGULATION OF GREAT BARRIER REEF MARINE PARK PERMITS AND APPROVALS 31

completed in August 2015), in response to commitments in its 2014 Great Barrier Reef Region Strategic Assessment: Program Report:

The 2014 Great Barrier Reef strategic assessment and program report had already highlighted improvements were needed to reduce duplication and to improve rigor, consistency and transparency in our permission system. Prior to the audit, we had initiated a project to do this work and with the results of the audit and the advice from the ANAO, we were readily able to incorporate those into the project and implement the recommendations of the audit report to better address risks and regulatory requirements.18

3.17 GBRMPA noted that it had therefore ‘already identified the need to strengthen its permissions system through commitments in the … Program Report and commenced “strengthening permissions system” activities as part of its 2014-15 work program’, with the ANAO audit recommendations then being rolled into this project.19 Actions undertaken during 2014-15 included reviewing policies, drafting guidelines and updating risk assessments, implementation of changes to ensure all alleged non-compliance is recorded and managed through the Compliance Management Information System, and development of a training program on the permissions system.20

3.18 GBRMPA also outlined its two-tranche timeline for implementation of improvements to the permit system over four years.21 The first tranche is due for implementation on 1 July 2017 and the second tranche in 2020.22 An important component of this work is GBRMPA’s Strengthening Permissions Compliance Action Plan 2015-2020 (September 2015) and Annual Permissions Compliance Plan 2015-16 (October 2015), which are intended to deliver outcomes including an ‘enhanced process to identify, develop and apply enforceable permission conditions in order to manage risks to the Marine Park’ and an ‘enhanced risk-based program for the assessment of

18 Dr Russell Reichelt, Chairman, GBRMPA, Committee Hansard, 11 February 2016, p. 1. GBRMPA’s Great Barrier Reef Region Strategic Assessment: Program Report (2014) included an assessment of its permit system and made commitments to progressively strengthen relevant policies, guidance material and support tools, ANAO, Audit Report No. 3, p. 37. 19 ANAO, Audit Report No. 3, pp. 23-24. 20 ANAO, Audit Report No. 3, p. 24. 21 Mr Bruce Elliot, General Manager, Biodiversity Conservation and Sustainable Use, GBRMPA,

Committee Hansard, 11 February 2016, p. 1. 22 GBRMPA, Submission 1, pp. 4-5.

32

regulatory risks so enforcement resources and consequential actions can be efficiently, effectively and proportionately targeted’.23

3.19 There was interest at the public hearing in further exploring GBRMPA’s implementation timeframe for its project to strengthen the permit system, as well as progress to date. Of particular interest was why this process appeared to be taking such a lengthy period of time, with the project due to be completed in 2020, some six years after the 2014 Great Barrier Reef Region Strategic Assessment: Program Report had originally identified the need for improvements in the permit system and action in response to this report had commenced. GBRMPA responded that the ‘work has begun’, but ‘essentially, there were no new people to do it’:

Our dilemma is that we had to take people off processing permits to improve the permit system, because we are also handling a general tightening of staff numbers and things that has occurred right across the APS. Essentially, there were no new people to do it, but to actually improve the system you have to take some people away to do that.24

3.20 GBRMPA had reduced overall staffing levels by 17 full-time equivalents (FTE) in the 2014-15 financial year in response to the completion of non-ongoing programs and ongoing efficiencies.25 Three out of around 21 FTE in GBRMPA’s permit section—some 14 per cent—had been shifted from permit approvals to improving the permit framework and GBRMPA ‘added additional funding to that section for some contract work to be done’.26 GBRMPA also explained that its two-tranche approach to implementation would enable any changes made to the permit system, ‘be they regulatory changes, policies, guidelines’, to be introduced in ‘lump sum so that it does not have a continuous change for the permittees … But the work to prepare for each of those tranches takes a little bit of time’.27 GBRMPA confirmed that some projects were due to be completed by the end of this financial year and progress was ‘on target against the project plan’.28

23 GBRMPA, Strengthening Permissions Compliance Action Plan 2015-2020 (September 2015), p. 4— see Senate Standing Committee on Environment and Communications Legislation, GBRMPA: Answer to Question on Notice No. 154, Supplementary Budget Estimates 2015-16, 19 October 2015. 24 Dr Reichelt, GBRMPA, Committee Hansard, 11 February 2016, p. 2. 25 GBRMPA, Submission 1.2, p. 1. The following efficiency dividends have been applied to

GBRMPA: 2014-15 financial year: $289 000; and 2015-16 financial year: $540 000, GBRMPA, Submission 1.2, p. 1. 26 Mr Elliot, GBRMPA, Committee Hansard, 11 February 2016, p. 2. 27 Mr Elliot, GBRMPA, Committee Hansard, 11 February 2016, p. 1. 28 Mr Elliot, GBRMPA, Committee Hansard, 11 February 2016, pp. 1-2.

REGULATION OF GREAT BARRIER REEF MARINE PARK PERMITS AND APPROVALS 33

3.21 The ANAO noted that GBRMPA’s response to the audit recommendations was ‘encouraging’—‘the response to the report itself outlined a number of initiatives that the authority was implementing. They were encouraging given some of the areas that we had identified’.29 However, the ANAO pointed to the need for GBRMPA to more rapidly progress some of the ‘easy win type activities’ and ‘quicker initiatives’, to generate efficiencies that could then provide added momentum in terms of the overall process to improve the permit system:

The weaknesses we have identified were pretty much across the continuum of regulatory activity from the initial receipt of applications through until investigations at the other end. There is a large body of work in front of the authority … they have put a plan in place and they are looking to do that in tranches. From our perspective, we would be looking for some of those easy win type activities but also some of the quicker initiatives such as guidance and procedures. What we did notice from the authority is that they tried to establish as an efficient a process as possible, so they put in place templates, template risk assessments and template assessments. Some of those were not quite tailored well enough to enable people to use them as they currently are, so some more work in that space should generate some efficiencies, which … will then be able to be built into the process.30

3.22 There was also interest in how GBRMPA was consulting with stakeholders on changes to the permit system. Telstra’s submission to the Committee’s inquiry emphasised the importance of stakeholder consultation on this matter, noting that it is in the ‘interest of all stakeholders for … guidance and procedures to be made publicly available and for stakeholders to have an opportunity to comment on relevant provisions’.31 GBRMPA confirmed that its consultation with stakeholders was ongoing,32 and that it had recently completed a two-month public consultation period that had also targeted primary stakeholders—‘as part of the project we consulted quite heavily with our stakeholders on their views in terms of some of these issues which were raised in the audit and the efficiencies we are trying to gain’.33 GBRMPA explained that it had regularly consulted on this matter through a range of

29 Mr Mark Simpson, Executive Director, Performance Audit Services Group, ANAO, Committee Hansard, 11 February 2016, p. 4. 30 Mr Simpson, ANAO, Committee Hansard, 11 February 2016, p. 4. 31 Telstra, Submission 2, p. 2. 32 Dr Reichelt, GBRMPA, Committee Hansard, 11 February 2016, p. 6. 33 Mr Elliot, GBRMPA, Committee Hansard, 11 February 2016, pp. 5-6.

34

formal mechanisms as well, including reef advisory committees on tourism and Indigenous management, and 12 local marine advisory committees with representatives from tourism, fishing, recreation and local government.34 It had also met regularly with the tourism industry

and the Queensland Ports Association.35

Permit application processing, assessment and approval 3.23 The Auditor-General concluded that, ‘while GBRMPA has well-established arrangements for processing and assessing permit applications’, there were ‘weaknesses in the quality and completeness of

assessments caused by fragmented and incomplete guidance for staff, incomplete records, insufficient consideration of assessment requirements and limited assurance from quality control processes’.36

3.24 The ANAO findings regarding permit application processing focused on improved guidance materials and documentation. A summary of the relevant key points from the ANAO report is set out below:

 Guidance materials: While GBRMPA has produced a range of

guidance materials to underpin its processing of permit applications, the materials are fragmented and unclear in parts and do not clearly address all relevant requirements37  Documentation: over half of the assessment checksheets (56 per

cent) examined by the ANAO were incomplete … which increases the risk that relevant information was not obtained to inform the delegate’s decision on whether to grant or refuse a permit38

3.25 The ANAO findings regarding permit application assessment focused on improved templates and assessment reports. A summary of the relevant key points from the ANAO report is set out below:

 Assessment and risk templates: While the template-based

assessments [for routine permit applications] took into account many requirements, they were, in general, not sufficiently tailored to address all applicable requirements—particularly discretionary considerations. Similarly, the customised assessments prepared for non-routine applications also took into account most, but not all, mandatory and discretionary regulatory requirements39

34 Mr Elliot, GBRMPA, Committee Hansard, 11 February 2016, p. 6. 35 Mr Elliot, GBRMPA, Committee Hansard, 11 February 2016, p. 6. 36 Mr Hehir, ANAO, ‘Opening statement by Auditor-General’, Submission 3, p. 1. 37 ANAO, Audit Report No. 3, p. 17. 38 ANAO, Audit Report No. 3, p. 18, p. 51. 39 ANAO, Audit Report No. 3, pp. 18-19.

REGULATION OF GREAT BARRIER REEF MARINE PARK PERMITS AND APPROVALS 35

 Assessment reports: assessment reports prepared for delegates

did not incorporate all relevant information to inform the delegate’s decision to issue or refuse a permit40

3.26 The ANAO findings regarding permit approval focused on improved standard permit conditions and documentation. A summary of the relevant key points from the ANAO report is set out below:

 Standard permit conditions: While most permit conditions

have been satisfactorily designed to address many of the identified high and medium-rated risks to the Marine Park environment, some conditions do not sufficiently address identified risks41  Documentation: Overall … the basis on which delegates have

decided to grant or refuse a permit have been appropriately documented … [but] the basis on which the delegate decided to grant a permit for an activity posing high risks to the Marine Park [to dump dredge spoil off the coast of Abbot Point] was not fully documented until one and a half months after the initial decision had been made42

3.27 The ANAO made three recommendations regarding GBRMPA’s permit application processing, assessment and approval.

3.28 On the ANAO’s recommendation that GBRMPA improve permit application processing by finalising standard operating procedures and administrative guidance, and documenting whether permit application assessment requirements have been addressed (Recommendation 1), GBRMPA confirmed that a project is ‘well underway to develop guidelines for permit applications and assessments. The draft guidelines are planned for public consultation in mid-2016’.43 Further, internal training on the new guidelines is scheduled for early 2017, and training in more detailed operating procedures will be rolled out progressively during 2017.44 GBRMPA also noted that, as the agency moves towards a more fully automated system of managing the permissions application assessment process, it will become ‘easier to ensure all requirements have been addressed during the assessment process’.45

3.29 On the ANAO’s recommendation that GBRMPA improve permit application assessment by revising permit application and risk assessment templates to better address risks relevant to the various permit types, and

40 ANAO, Audit Report No. 3, p. 19. 41 ANAO, Audit Report No. 3, p. 20. 42 ANAO, Audit Report No. 3, p. 79. 43 GBRMPA, Submission 1, p. 1. 44 GBRMPA, Submission 1, p. 2. 45 GBRMPA, Submission 1, p. 2.

36

prepare assessment reports for delegates that adequately address regulatory assessment requirements (Recommendation 2), GBRMPA noted that it is in the ‘initial development stages of producing an online application system, which will link directly to the Reef Management System database to allow more efficient processing of applications’ and is ‘updating the risk assessment report template to explain why each criterion was deemed relevant or irrelevant’.46 Further, GBRMPA is ‘proposing making consideration of all assessment criteria mandatory, as recommended by the audit report’, and the ‘risk assessment framework is currently under review to bring it into alignment with the agency’s broader risk management framework’, with this work anticipated to be completed by June 2016 so that public consultation can occur.47

3.30 On the ANAO’s recommendation that GBRMPA improve permit approval processes by periodically reviewing the adequacy of standard permit conditions used to manage risks from permitted activities, (Recommendation 3), GBRMPA noted that work in this area is ‘ongoing’, including updating the assessment report template and procedures to draw delegates’ attention to any special permit conditions or changes to standard permit conditions, and ensure delegates clearly document reasons for making decisions at the time the decision is made.48 Additional permit conditions are also being considered to better manage certain risks, and letter templates are being progressively updated to ensure the decision notice highlights any permissions that were not granted, or any changes to conditions where a permit allows an existing use to continue.49

3.31 The public hearing further explored the timeliness of GBRMPA’s permit application assessment. Comments provided to the ANAO by permit holders and general stakeholders identified permit assessment timeliness as an area of concern.50 Similarly, Telstra’s submission to the inquiry observed that the permit process would ‘benefit from the introduction of mandated timeframes surrounding the permit decision making process’.51 The ANAO report noted that ‘assessment of permit applications by GBRMPA has not been timely’ and that ‘over the period from July 2012 to June 2014, GBRMPA achieved its 60-day target timeframe for assessing routine applications in 57 per cent of cases (413 of the 720 routine

46 GBRMPA, Submission 1, p. 2. 47 GBRMPA, Submission 1, p. 2. 48 GBRMPA, Submission 1, p. 3. 49 GBRMPA, Submission 1, p. 3. 50 ANAO, Audit Report No. 3, p. 19. See also Mr Simpson, ANAO, Committee Hansard,

11 February 2016, p. 5. 51 Telstra, Submission 2, p. 2.

REGULATION OF GREAT BARRIER REEF MARINE PARK PERMITS AND APPROVALS 37

applications), with a further 81 applications taking between 120 days and around two years to complete’.52

3.32 GBRMPA confirmed it no longer had a key performance indicator (KPI) for 60 days (12 weeks) and was currently advising permittees that its average assessment time for a Level 1 permit is 16 weeks: ‘it has actually been around 16 weeks for the last couple of years, and it will probably remain there until such time as we start to see the efficiencies of the work that we are doing right now’.53 GBRMPA explained that the average permit assessment time had therefore not effectively changed—‘it is still what it was, so that 57 per cent is probably because it was already 16 weeks in reality. We have checked the statistics back several years. It has been 14 to 16 weeks probably for the last four or so years’.54 Reasons for the average 16-week processing time included that ‘we had to take people off processing permits to improve the permit system … Another reason is that the numbers of … level 2 permits have gone up’.55

3.33 As to current outcome reporting against the performance target of a 16-week average permit assessment time, since 1 January 2015 GBRMPA had received 239 Level 1 permit applications, with 190 (79 per cent) being decided within the 16 week timeframe.56 GBRMPA noted that it expects the 16-week KPI to ‘improve as the efficiencies being implemented as part of our program to strengthen our permissions system are realised’.57 As GBRMPA further commented, ‘we have already started to implement an online permit system which will allow for permittees to more simply put their applications in … so our whole permit decision process will become quicker … the efficiencies that we are looking at doing at the moment … will allow us to bring that down to more reasonable time frames’.58

3.34 There was interest in what a revised permit assessment KPI might be as result of these efficiencies. However, Mr Bruce Elliot, General Manager, GBRMPA, responded:

52 ANAO, Audit Report No. 3, p. 19. 53 Mr Elliot, GBRMPA, Committee Hansard, 11 February 2016, p. 2. 54 Mr Elliot, GBRMPA, Committee Hansard, 11 February 2016, p. 2. 55 Dr Reichelt, GBRMPA, Committee Hansard, 11 February 2016, p. 2. 56 GBRMPA, Submission 1.1, p. 1. (It was noted that this figure does not include Level 1

applications received in December 2015, which have yet to be finalised because the 16-week timeframe has not yet been reached, p. 1.) Permit applications are graded by GBRMPA on a four-point scale based on the risk that the proposed activity poses to the Marine Park. Permit assessment templates are used for Level 1 (routine) permits, representing over 90 per cent of all permit applications, and permit application assessments are customised for higher level (non-standard) permits, ANAO, Audit Report No. 3, p. 43. 57 GBRMPA, Submission 1.1, p. 1. 58 Mr Elliot, GBRMPA, Committee Hansard, 11 February 2016, pp. 1-2.

38

It is not a question I can answer at the moment because we have not really got a feel for how these efficiencies we are looking at at the moment will tighten things up. For example, the permits online system: it will not accept an application until all of the fields have been filled in and, therefore, we have sufficient information to do the processing. That will mean that it will obviously be at least 30 days because of the native title notification, and we also have to do an assessment against it and make sure that that assessment is double-checked by a manager or supervisor.59

3.35 GBRMPA concluded that, ‘in theory, we could get down to that KPI of 60 days, but the other thing it will depend on is the volume we have coming in at any one time’.60 On this point, it was clarified that the permit application workflow ‘tends to be seasonal’—‘we tend to have less at the end of the calendar year and we tend to have more in the middle of the calendar year, so it does go through a cycle each year’.61 On whether GBRMPA’s staffing was correspondingly seasonal, Mr Elliot explained that, ‘no, staffing is consistently flat. We can surge if we need to, because there are other people in the agency who understand our permit system and who have worked there before’.62

3.36 As to why a 60-day (12-week) KPI had originally been established, given an actual average permit assessment time of 16 weeks, Mr Elliot, responded: ‘I cannot answer that question, because the KPI was probably set at a time, and was probably sitting there for some years without people reviewing it, but I would say that that KPI of 60 days was probably unachievable and probably will be unachievable for quite some time’.63 GBRMPA Chairman, Dr Russell Reichelt, further observed:

this is the type of thing that our strategic assessment prior to the audit highlighted maybe three years ago. It really called us to radically overhaul this system, and that is what we had already begun, and I think the ANAO have correctly pointed out, firstly, that we are on the right track but that we need to do more. So we acknowledge that we need to improve it, but the KPI was established prior to the strategic assessment.64

3.37 There was interest in further understanding the critical path running through the permit approval process that had resulted in a 16-week

59 Mr Elliot, GBRMPA, Committee Hansard, 11 February 2016, p. 3. 60 Mr Elliot, GBRMPA, Committee Hansard, 11 February 2016, p. 3. 61 Mr Elliot, GBRMPA, Committee Hansard, 11 February 2016, p. 3. 62 Mr Elliot, GBRMPA, Committee Hansard, 11 February 2016, p. 3. 63 Mr Elliot, GBRMPA, Committee Hansard, 11 February 2016, p. 2. 64 Dr Reichelt, GBRMPA, Committee Hansard, 11 February 2016, p. 2.

REGULATION OF GREAT BARRIER REEF MARINE PARK PERMITS AND APPROVALS 39

timeframe for permit assessment as opposed to a 60-day (12-week) timeframe. GBRMPA explained that there is a minimum processing timeframe of 30 days, and the processing time starts from the moment the application is lodged, even if the application is incomplete:

the minimum that can occur is 30 days because there is the mandatory native title notification, and that is a process that takes 30 days. The other thing that tends to create most of the time—and this problem will be solved by our online application system that we are developing at the moment—is when we receive an application, it often does not have sufficient information for us to do an assessment on it. We have to have enough information to do an assessment against the mandatory and discretionary criteria in our regulations. So we spend a lot of time going backwards and forwards to the ‘permittee’ to source the additional information we need. That is one of the things that does slow it down …

If we started the clock when we had all information necessary to do an application, it would be a significantly shorter period of time.65

3.38 Another area of interest was the potential economic impact of permit approval delays. GBRMPA clarified that, ‘if it is a continuation of a permit, there is no impact because our legislation allows that as long as they put the application in before their permit expires, their permit can continue to be used on their existing permit until the continuation is processed’.66 GBRMPA further emphasised that new applications ‘get priority’,67 and that it also received ‘other short notice applications’—‘with the potential for coral bleaching on the Great Barrier Reef this year, we have had a flood of research permit applications. They all have to be processed reasonably quickly because they relate to this summer. Recently we have had one for a desalination plant at Palm Island because of water shortages. Obviously we are going to do that one as quickly as we can’.68 GBRMPA also confirmed its policy of encouraging early consultation, prior to an organisation making a permit application.69

65 Mr Elliot, GBRMPA, Committee Hansard, 11 February 2016, p. 3. 66 Mr Elliot, GBRMPA, Committee Hansard, 11 February 2016, p. 4. 67 Dr Reichelt, GBRMPA, Committee Hansard, 11 February 2016, p. 4. 68 Mr Elliot, GBRMPA, Committee Hansard, 11 February 2016, p. 4. 69 Dr Reichelt, GBRMPA, Committee Hansard, 11 February 2016, p. 5.

40

Managing compliance 3.39 The Auditor-General concluded that, ‘in general, permit monitoring undertaken collectively by GBRMPA and its partner agencies has been insufficient to determine permit holders’ compliance with conditions’.70

3.40 The ANAO findings regarding compliance intelligence, risk assessment and managing compliance focused on developing standard operating procedures for compliance monitoring and implementing a risk-based approach. A summary of the relevant key points from the ANAO report is set out below:

 Standard operating procedures and risk-based approach:

Improved monitoring of permit holders’ compliance with post-approval reporting requirements and the establishment of a risk-based program of supplementary monitoring would better position GBRMPA to manage the risks posed to the Marine Park by permitted activities … GBRMPA … did not initiate or conduct monitoring activities, on a risk basis, for 104 permits (89.7 per cent) examined by the ANAO over the period from July 2012 to July 201471

3.41 On the ANAO’s recommendation that GBRMPA improve permit compliance monitoring by enhancing standard operating procedures and implementing a risk-based program (Recommendation 4), GBRMPA pointed to the development of its Strengthening Permissions Compliance Action Plan 2015-2020 (September 2015) and Annual Permissions Compliance Plan 2015-16 (October 2015), noting that these plans will deliver an enhanced permission compliance program, including ongoing development of guidelines and prioritisation of risks.72

Responding to non-compliance 3.42 The Auditor-General concluded that, ‘until recently, many instances of permit holder non-compliance were not identified by GBRMPA staff and not recorded centrally for assessment and possible enforcement action’. 73

Further, ‘limited guidance for investigators when determining appropriate enforcement responses to non-compliance, when coupled with poorly documented reasons for enforcement actions, makes it difficult for GBRMPA to demonstrate the basis for its enforcement decision-making’.74

70 Mr Hehir, ANAO, ‘Opening statement by Auditor-General’, Submission 3, p. 1. 71 ANAO, Audit Report No. 3, pp. 21-22. 72 GBRMPA, Submission 1, p. 3.

73 Mr Hehir, ANAO, ‘Opening statement by Auditor-General’, Submission 3, p. 1. 74 Mr Hehir, ANAO, ‘Opening statement by Auditor-General’, Submission 3, pp. 1-2.

REGULATION OF GREAT BARRIER REEF MARINE PARK PERMITS AND APPROVALS 41

3.43 The ANAO findings regarding responding to non-compliance focused on improved guidance materials, record-keeping, documentation and timeliness. A summary of the relevant key points from the ANAO report is set out below:

 Guidance materials: While GBRMPA is working to develop

revised compliance policy, strategy and guidance documentation, the material developed to date is generally in draft form and does not address all fundamental regulatory requirements75  Record-keeping: many instances of non-compliance that were

evident from permit monitoring activities undertaken by the EAP Section were not reported to the FMCU and, as a result, were not recorded in the Compliance Management Information System (CMIS) to enable analysis and assessment for potential enforcement action76  Documentation and timeliness: extended timeframes were

required to finalise a significant proportion of investigations … In general, the documentation of enforcement decision-making in relation to permit-rated non-compliance has been poor … there were a small number of cases (related to the education of permit holders) where the investigations were closed despite enforcement action not having been undertaken77

3.44 On the ANAO’s recommendation that GBRMPA improve its response to permit non-compliance by finalising guidance documentation, recording non-compliance in the Compliance Management Information System, documenting reasons for key decisions and verifying enforcement action has been undertaken prior to closure of investigations (Recommendation 5), GBRMPA noted that implementation of the Strengthening Permissions Compliance Action Plan 2015-2020 (September 2015) includes milestones tied to the delivery of improvements in enforcement.78 Development of permissions compliance guidelines was ‘ongoing’, with individual guidelines progressed on a risk basis.79 GBRMPA is also finalising its whole-of-agency compliance management policy, due for release in 2016.80 Other actions included delivery of a training program to improve identification of non-compliance; updating of electronic systems used to monitor and report compliance, to include specific permissions compliance fields; and updating of the Compliance

75 ANAO, Audit Report No. 3, p. 22. 76 ANAO, Audit Report No. 3, p. 22. 77 ANAO, Audit Report No. 3, p. 23. 78 GBRMPA, Submission 1, p. 4. 79 GBRMPA, Submission 1, p. 4. 80 GBRMPA, Submission 1, p. 4.

42

Management Information System to support capture and management of permissions compliance incidents.81 Compliance Management and Investigation Procedures have also been updated to include requirements for documenting the decision making process, and a procedure for recording investigation outcomes, differentiating between those matters that do and do not proceed to prosecution.82

Committee comment

3.45 The Committee notes the seriousness of the Auditor-General’s overall conclusion, that shortcomings in GBRMPA’s regulatory processes and, more particularly, its regulatory practices have undermined the effectiveness of the permit system as a means of managing risks to the Marine Park.83 Further, these shortcomings were identified across a broad range of GBRMPA’s regulatory activities, including its assessment of permit applications, monitoring of permit holder compliance and response to non-compliance.84

3.46 The Committee is strongly concerned about this finding, given the environmental, social and economic significance of the Great Barrier Reef Marine Park and its World Heritage listing.

3.47 The Committee noted that GBRMPA had acknowledged weaknesses in its permit assessment, compliance management and response to non-compliance—and has made some progress in implementing the ANAO audit recommendations. The Committee was pleased to note GBRMPA’s current stakeholder consultation process on this matter, including through formal mechanisms.85

3.48 Overall, however, the Committee is of the view that GBRMPA needs to accelerate its current timeline for implementation of improvements to its regulatory processes and practices—where this is appropriate and will not negatively impact on assessment and compliance processes and outcomes. GBRMPA’s two-tranche project to strengthen the permit system is due to be completed in 2020, some six years after GBRMPA’s 2014 Great Barrier Reef Region Strategic Assessment: Program Report had originally identified the need for improvements. Accordingly, future milestones are distant

81 GBRMPA, Submission 1, p. 4. 82 GBRMPA, Submission 1, p. 4. 83 Mr Hehir, ANAO, ‘Opening statement by Auditor-General’, Submission 3, p. 1. 84 Mr Hehir, ANAO, ‘Opening statement by Auditor-General’, Submission 3, p. 1. 85 Dr Reichelt, GBRMPA, Committee Hansard, 11 February 2016, p. 6; and Mr Elliot, GBRMPA,

Committee Hansard, 11 February 2016, p. 6.

REGULATION OF GREAT BARRIER REEF MARINE PARK PERMITS AND APPROVALS 43

and progress, while having been made, might be more rapidly advanced. As the ANAO noted, while GBRMPA’s response to the audit recommendations has been ‘encouraging’, it needs to more rapidly progress some of the ‘easy win type activities’ and ‘quicker initiatives’, such as guidance and procedures.86

3.49 GBRMPA’s resource constraints are acknowledged.87 The Committee appreciates that GBRMPA is under resource pressures and has already reprioritised staff from an assessment to an improvement role. The Committee understands that implementation by GBRMPA of its Strengthening Permissions Compliance Action Plan 2015-20 should better place the agency to target its limited resources to those risks posing the greatest threat to the Marine Park.88 The action plan is intended to deliver outcomes including an ‘enhanced risk-based program for the assessment of regulatory risks so enforcement resources … can be efficiently … targeted’.89 As the ANAO Better Practice Guide on Administering Regulation: Achieving the Right Balance states, risk management is an ‘integral component of good regulatory administration’ and can be used to support the ‘efficient allocation of available resources’.90 Further, GBRMPA and the ANAO both noted that as system improvements are realised this will generate efficiencies that can be built into the overall process.91 Accordingly, with the Committee’s recommendation to GBRMPA to appropriately accelerate its project to strengthen the permit system, comes the Committee’s acknowledgment that there will be a requirement here for either further Commonwealth funding or further targeted risk management.

Recommendation 2

3.50 To improve the effectiveness of the permit system as a means of managing risks to the Great Barrier Reef Marine Park, the Committee recommends that the Great Barrier Reef Marine Park Authority (GBRMPA):

86 Mr Simpson, ANAO, Committee Hansard, 11 February 2016, p. 4. 87 Dr Reichelt, GBRMPA, Committee Hansard, 11 February 2016, p. 2. See also GBRMPA, Submission 1.1, p. 1 and Submission 1.2, p. 1. 88 ANAO, Audit Report No. 3 (2015-16), p. 91. 89 GBRMPA, Strengthening Permissions Compliance Action Plan 2015-2020 (September 2015), p. 4—

see Senate Standing Committee on Environment and Communications Legislation, GBRMPA: Answer to Question on Notice No. 154, Supplementary Budget Estimates 2015-16, 19 October 2015. 90 ANAO Better Practice Guide, Administering Regulation: Achieving the Right Balance (2014), p. 14. 91 Mr Elliot, GBRMPA, Committee Hansard, 11 February 2016, pp. 1-2; Mr Simpson, ANAO,

Committee Hansard, 11 February 2016, p. 4; and GBRMPA, Submission 1.1, p. 1.

44

 appropriately accelerate its projected timeframe, currently

planned over two tranches in 2017 and 2020, for implementation of the audit recommendations in ANAO Report No. 3 and other improvements identified by GBRMPA as part of its project to strengthen the permit system

 report back to the Committee within six months with details of

new implementation dates and milestones, and how the accelerated timeframe will be achieved

3.51 A particular issue that emerged during the Committee’s inquiry process was concern regarding the timeliness of GBRMPA’s permit assessments (covering processing, assessment and approvals). The ANAO found that GBRMPA has only met its 60-day (12-week) timeframe for assessment of routine permits in 57 per cent of cases.92 At the public hearing, GBRMPA confirmed that its current average assessment time for routine permits is 16 weeks, which is what it had actually been for the last four years.93

3.52 Although it is far from ideal that assessments have been taking longer than expected, the Committee is pleased to note GBRMPA is now communicating a more realistic timeframe to stakeholders and has started implementing an online permit application system which will help speed up the assessment process.94

3.53 However, the Committee points to the need for more meaningful and well defined performance information and targets for permit assessment timeframes.95 Such performance information could consist of individual targets for each stage of processing, assessment and approval, as well as a target that clearly communicates when stakeholders should expect to be informed of the outcome of their application. The Committee suggests that any targets set should be adjusted over time to reflect improvements in permit assessment timeliness, such as after implementation of the new online application system.

Recommendation 3

3.54 To improve the effectiveness of the permit system as a means of managing risks to the Great Barrier Reef Marine Park, the Committee recommends that the Great Barrier Reef Marine Park Authority implement more effective performance information, including targets,

92 ANAO, Audit Report No. 3 (2015-16), p. 19. 93 Mr Elliot, GBRMPA, Committee Hansard, 11 February 2016, p. 2. 94 Mr Elliot, GBRMPA, Committee Hansard, 11 February 2016, pp. 1-2. 95 The new Commonwealth performance framework, established under the PGPA Act, and

associated PGPA Rule and resource management guidance are important references here.

REGULATION OF GREAT BARRIER REEF MARINE PARK PERMITS AND APPROVALS 45

for permit application processing, assessment and approval timeframes, and continue to monitor and publicly report on performance outcomes in this area.

3.55 The Committee notes the Auditor-General’s observation of weaknesses in the quality and completeness of assessments caused by fragmented and incomplete guidance for staff, incomplete records, insufficient consideration of assessment requirements and limited assurance from quality control processes.96

3.56 Of particular concern to the Committee was the Auditor-General’s conclusion that, in general, permit monitoring undertaken collectively by GBRMPA and its partner agencies has been insufficient to determine permit holders’ compliance with conditions and that, until recently, many instances of permit holder non-compliance were not identified by GBRMPA staff and not recorded centrally for assessment and possible enforcement action.97

3.57 Where permits are assessed and issued but compliance is inadequately monitored and non-compliance inadequately enforced, this undermines the regulatory system and raises concerns about risks to the reef. Future regulatory activity needs to be focused on both process and outcomes— the Committee therefore welcomes GBRMPA’s update on how it is implementing the ANAO’s recommendations on compliance matters.98

3.58 However, given the issues identified across the continuum of GBRMPA’s regulatory activities—including permit application assessment, compliance management and response to non-compliance—the Committee is of the view that GBRMPA should report back to the JCPAA on implementation of the ANAO recommendations across each of these specific areas. The Committee does not take reporting back lightly—the administrative burden of this has been considered—but at stake here is improved management of risks to the Great Barrier Reef Marine Park.

Recommendation 4

3.59 To improve the effectiveness of the permit system as a means of managing risks to the Great Barrier Reef Marine Park, the Committee recommends that the Great Barrier Reef Marine Park Authority report back to the Committee at 18 months from the tabling of the Committee’s

96 Mr Hehir, ANAO, ‘Opening statement by Auditor-General’, Submission 3, p. 1. 97 Mr Hehir, ANAO, ‘Opening statement by Auditor-General’, Submission 3, p. 1. 98 See GBRMPA, Submission 1, pp. 1-5, and evidence from GBRMPA officials at the public hearing, Committee Hansard, 11 February 2016.

46

report, on:

 whether it has met the new implementation dates and

milestones as previously advised to the Committee in response to recommendation 1

 specific implementation details and dates achieved on the

following:

⇒ for permit application processing, assessment and

approval—finalisation of standard operating procedures, guidance materials, and improvements in: documentation by officials, templates, assessment reports and standard permit conditions

⇒ for compliance management—finalisation of standard

operating procedures and implementation of a risk-based program

⇒ for response to non-compliance—finalisation of guidance

materials and improvements in: identification of non-compliance, records management, documentation by officials and verification of enforcement action having been undertaken prior to closure of investigation

4

Performance Audit Report No. 9 (2015-16)

Test and Evaluation of Major Defence Equipment Acquisitions

4.1 Chapter 4 focuses on the Joint Committee of Public Accounts and Audit (JCPAA) inquiry into Australian National Audit Office (ANAO) Audit Report No. 9 (2015-16), Test and Evaluation of Major Defence Equipment Acquisitions, Department of Defence (Defence). The chapter comprises:

 inquiry context and background

 ANAO report overview

 Committee review of evidence

 Committee comment

Inquiry context and background

4.2 Defence’s capital equipment acquisition program includes aircraft, maritime vessels and land-based equipment in various stages of engineering development and delivery. In 2013-14 the program comprised some 180 approved projects, with a total value of $79 billion.1 The 2012 Defence Capability Plan contains an additional 111 projects, or project phases, planned for either First or Second Pass government approval over the four-year forward estimates period, worth some $153 billion in capital costs.2

1 ANAO Report No. 9 (2015-16), Test and Evaluation of Major Defence Equipment Acquisition, p. 7. 2 ANAO Report No. 9, p. 7.

48

4.3 Each of these projects relies on test and evaluation (T&E) processes to identify areas of cost, schedule and capability risk to be reduced or eliminated. T&E is a key component of systems engineering and its primary function is to provide feedback to engineers, program managers and capability managers on whether a product or system is achieving its design goals in terms of cost, schedule, function, performance and sustainment. It also enables capability acquisition and sustainment organisations to account for their financial expenditure in terms of the delivery of products or systems that are safe to use, fit for purpose and meet the requirements approved by government.3 Figure 4.1 provides an overview of the T&E phases and Defence acquisition milestones.

Figure 4.1 Overview of T&E phases and Defence acquisition milestones

Source ANAO Report No. 9, p. 16.

4.4 Key developments in Defence T&E include the establishment of the Australian Defence Test and Evaluation Office (ADTEO) in 2007; the finalisation of an overarching T&E policy, the Defence Capability Development Manual (DCDM) in 2015; and the 2015 First Principles Review: Creating One Defence, which recommends significant organisational design change affecting T&E governance and seeks to encourage:

 A stronger and more strategic centre able to provide clear

direction, contestability of decision-making, along with enhanced organisational control of resources and monitoring of organisational performance;  An end-to-end approach for capability development with

Capability Managers having clear authority and accountability as sponsors for the delivery of capability outcomes to time and budget, supported by an integrated capability delivery function and subject to stronger direction setting and contestability from the centre;  Enablers that are integrated and customer-centric with greater

use of cross-functional processes, particularly in regional locations; and

3 ANAO Report No. 9, p. 7.

TEST AND EVALUATION OF MAJOR DEFENCE EQUIPMENT ACQUISITIONS 49

 A planned and professional workforce with a strong

performance management culture at its core.4

ANAO report overview

Audit objective, scope and criteria 4.5 The ANAO’s audit objectives were to examine the effectiveness of Defence’s management of the T&E aspects of its major capital equipment acquisition program, and to report on Defence’s progress in implementing

T&E recommendations made in the Senate Foreign Affairs, Defence and Trade References Committee’s August 2012 report, Procurement Procedures for Defence Capital Projects.5

4.6 To form a conclusion against the objectives, the ANAO adopted the following high-level criteria:

 Defence’s organisational structures, roles and responsibilities

enable the coordinated application of adequate T&E at each stage of the capital equipment project life cycle  Defence’s T&E policy and procedures are suitably designed and

applied as intended  Defence invests in a broad range of training and skills

development for T&E personnel to enable the application of necessary T&E expertise throughout the capital equipment project life cycle; and  the T&E aspects of capital equipment acquisition are

transparently reported to inform decision making and management of technical risks that may impact the development and maintenance of the major systems component of the Fundamental Inputs to Capability.6

ANAO overall conclusion 4.7 Overall, the ANAO report concluded that:

the conduct of T&E remains distributed across 12 Defence organisations, placing a premium on the effectiveness of Defence’s T&E governance as a means of mitigating the risk of inconsistent conduct of T&E. Defence’s administration of T&E would be further strengthened by introducing arrangements to provide enterprise-level advice to senior responsible leaders on key issues,

4 First Principles Review: Creating One Defence, 2015, pp. 5-6. See the ANAO report for a detailed description of Defence T&E. 5 ANAO Report No. 9, p. 7. 6 ANAO Report No. 9, pp. 7-8. (See p. 18 for audit method and scope.)

50

introducing performance measures and compliance assurance for T&E, and completing reforms to T&E personnel competency and training arrangements …

The case studies examined in this audit highlight the important role played by T&E in managing acquisition risks for major capital equipment … In the case of the first Landing Helicopter Dock (LHD), HMAS Canberra, key management decisions were usefully informed by Defence’s T&E, which identified numerous defects and deficiencies for resolution. Defence decided, on balance, to accept HMAS Canberra on the understanding that the deficiencies would be addressed during the ship’s operational phase. In doing so, the Chief of Navy accepted greater risks than would have been the case had System Acceptance been based on more complete objective quality evidence of compliance with contracted specifications, and had Initial Materiel Release been based on less qualified findings by Defence’s regulators concerning compliance.7

ANAO recommendations and agency response 4.8 Table 4.1 sets out the recommendations from ANAO Report No. 9— Defence agreed to all three recommendations without qualification.8

Table 4.1 ANAO recommendations, Audit Report No.9 (2015-16)

1 To strengthen the enterprise-level management of the T&E workforce, the ANAO recommends that Defence:

(a) identifies the training and competencies of the existing Defence T&E workforce;

(b) conducts a T&E personnel competency and training needs analysis for the whole entity; and

(c) monitors the availability of sufficient, appropriately trained T&E personnel in specific competency areas and takes steps to address any gaps identified.

2 To reduce risk and assist the transition of capability from the acquisition phase to operations, the ANAO recommends that prior to System Acceptance, Defence ensures that material deficiencies and defects are identified and documented, and plans for their remediation established.

3 In the context of its implementation of reforms arising from the First Principles Review, the ANAO recommends that Defence introduce arrangements to provide the Vice Chief of the Defence Force and Capability Managers with enterprise-level advice on the coordination, monitoring and evaluation of the adequacy and results of Defence T&E activities.

7 ANAO Report No. 9, p. 8. 8 For details of Defence’s response to the ANAO’s recommendations, see ANAO, Audit Report No. 9, pp. 11-12.

TEST AND EVALUATION OF MAJOR DEFENCE EQUIPMENT ACQUISITIONS 51

Committee review of evidence

4.9 The Committee’s key sources of evidence for this inquiry were the ANAO’s audit report, evidence given by representatives from the ANAO and Defence at the Committee’s public hearing on 3 March 2016, and agency submissions (see details of public hearings and submissions at Appendixes A and B).

4.10 The Committee also investigated relevant public sector frameworks, and past reviews, reports and audits as detailed below.

Key audit themes and relevant public sector frameworks 4.11 The key audit theme that emerged from the Committee’s inquiry into ANAO Report No. 9 is effective risk management in terms of the role of test and evaluation (T&E) in managing acquisition risks for major Defence

equipment.

4.12 Key frameworks for Defence T&E are the Defence Capability Development Manual (December 2014) and First Principles Review—Creating One Defence (April 2015). Broader frameworks here are the relevant sections of the Public Governance, Performance and Accountability Act 2013 (PGPA Act) and PGPA Rule 2014, and associated guidance. As the primary piece of Commonwealth resource management legislation, the PGPA Act establishes a single system of governance and accountability for public resources. In particular, s16 of the PGPA Act, supported by the Commonwealth Risk Management Policy (2014), requires agencies to establish appropriate systems for the management of risk. As regards Defence T&E of major equipment acquisition, this includes ensuring Defence managers have the information needed to effectively manage risk as regards assessing investment costs and benefits; resolving equipment function, performance and sustainment issues; and maximising reliable operational effectiveness. T&E is a key risk mitigation technique capable of providing managers with the information feedback needed to effectively manage risk.9

Relevant reviews, reports and audits 4.13 There have been a range of reviews, reports and audits concerning Defence and T&E:

9 Chapter 1 further discusses risk management in the Commonwealth public sector as a key audit theme across the three ANAO reports covered by the Committee’s report.

52

 Senate Foreign Affairs, Defence and Trade References Committee,

Report on the Inquiry into Materiel Acquisition and Management in Defence (2003)

 ANAO Report No. 30 (2001-02), Test and Evaluation of Major Defence

Equipment Acquisitions (2002)—aspects of T&E have also featured in a range other recent ANAO audits10

 Kinnaird Review, Defence Procurement Review (2003)

 Department of Defence, Defence Test and Evaluation Roadmap (2008)

 Senate Foreign Affairs, Defence and Trade References Committee,

Procurement Procedures for Defence Capital Projects (2012)

Key issues 4.14 The Committee focused on the following three matters:

 implementation of ANAO recommendations and other review findings

 T&E enterprise-level governance, including policy and procedures,

performance information, and competency and training

 managing acquisition risks, including HMAS Canberra case study

Implementation of ANAO recommendations and other review findings 4.15 Defence agreed to all three recommendations from the ANAO audit without qualification. At the public hearing and in its submission to the inquiry, Defence provided an update on implementation of the

recommendations (this is further discussed in the relevant sections below). In particular, Defence confirmed that, as part of the First Principles Review process, a range of activities are currently underway to strengthen T&E arrangements across the department, to ‘better achieve the outcomes and the recommendations that have been put forward by the ANAO’.11

4.16 The ANAO emphasised that a ‘well-managed T&E program consists of suitably qualified and experienced personnel undertaking T&E in accordance with sound policy and with support from appropriate institutional arrangements’.12 However, several recent ANAO audits and external reviews of Defence, as listed earlier, have identified deficiencies in these aspects of Defence’s T&E program and made recommendations for improvement—Figure 4.2 sets out common themes from these reports.

10 See list of audit reports, ANAO Report No. 9, p. 16. 11 Air Vice Marshal Mel Hupfeld, Acting Chief, Capability Development Group, Defence, Committee Hansard, 3 March 2016, p. 10. 12 ANAO Report No. 9, p. 19.

TEST AND EVALUATION OF MAJOR DEFENCE EQUIPMENT ACQUISITIONS 53

4.17 One of the ANAO’s two audit objectives was to report on Defence’s progress in implementing T&E recommendations made in the Senate Foreign Affairs, Defence and Trade References Committee’s August 2012 report, Procurement Procedures for Defence Capital Projects.13 The Senate report identified several deficiencies in the way T&E was being utilised to support Defence major capital equipment acquisitions and made five recommendations directly related to T&E.14

Figure 4.2 Common themes in previous reviews (ticks indicate whether an issue was raised)

Theme 2002 ANAO

T&E audit 2003 Senate Inquiry 2003

Kinnaird Review

2008 T&E Roadmap 2012 Senate Inquiry

Inconsistent conduct of T&E.     

Inadequate oversight of T&E training.     

Inadequate resources for T&E.     

Poor translation of T&E policy and process into practice.











Misunderstanding of T&E’s role as an assurance mechanism for the delivery of expected capability.











Source ANAO Report No. 9, p. 17

4.18 The ANAO particularly identified the need for further progress by Defence with regard to two Senate report recommendations relating to T&E competency and training requirements’ (recommendations 23 and 24), concluding that Defence had made ‘slow progress’15 (this matter is further discussed below in the section on ‘T&E governance’).

4.19 On the three remaining Senate report recommendations related to T&E, the ANAO identified progress in a number of areas:

 The Senate report recommended improvements in Defence’s Technical

Risk Assessment and Technical Risk Certification processes (recommendation 11). The ANAO confirmed that, in conducting this audit, it had ‘observed significant improvements in Defence’s Technical Risk Assessment policy and process manual’; however, the ‘DCDM

13 ANAO Report No. 9, p. 7. 14 See p. 74 of the ANAO report for details of these recommendations. 15 ANAO Report No. 9, p. 9.

54

would be improved if it included a more complete reference to this manual’.16

 The Senate report recommended that Defence capability managers

require their developmental T&E practitioners to be an equal stakeholder with the Defence Science and Technology Organisation in the pre-First Pass risk analysis and to conduct the pre-contract evaluation so they are aware of risks before committing to a project (recommendation 22). The ANAO confirmed that, in response, ‘Defence included guidance covering preview T&E in the DCDM’.17 The ANAO noted that well-conducted preview T&E policy should result in improved pre-First Pass risk analysis and pre-contractual cost and benefit evaluation, ‘resulting in Defence being more aware of acquisition risks’ and so better informing the First and Second Pass approval process’.18

 The Senate report recommended that Defence mandate a default

position of engaging specialist T&E personnel pre-First Pass (recommendation 25). The ANAO confirmed that Defence had implemented this recommendation through its recently released DCDM.19 The ANAO observed that, ‘when implemented well, Defence preview T&E has mitigated acquisition risks, particularly with respect to off-the-shelf (OTS) equipment acquisitions’.20 However, the ANAO emphasised that acquisition risks still need to be managed through the conduct of preview T&E and operational T&E.21

4.20 The ANAO’s current audit also followed up its 2002 audit on this matter, Report No. 30 (2001-02), Test and Evaluation of Major Defence Equipment Acquisitions (January 2002). The earlier audit found there was little evidence of effective corporate initiatives to support efficient and effective use of Defence’s T&E resources, and made five recommendations, including that Defence:

 review and update its T&E policy organisation and responsibilities

 establish an office responsible for common standards for, and

independent oversight of, operational T&E

16 ANAO Report No. 9, p. 26. 17 ANAO Report No. 9, p. 26. 18 ANAO Report No. 9, p. 26. 19 ANAO Report No. 9, p. 9, p. 38. 20 ANAO Report No. 9, p. 9. 21 ANAO Report No. 9, p. 9.

TEST AND EVALUATION OF MAJOR DEFENCE EQUIPMENT ACQUISITIONS 55

 T&E practitioners have training and skills adequate for their

responsibilities, through a consistent policy and program that encourage training and education in T&E22

4.21 Against the background of these previous ANAO and Senate Committee recommendations, the ANAO found that a number of recent developments had provided Defence with a ‘stronger basis for the management of T&E’.23 However, the ANAO concluded that ‘scope remains to improve key aspects of Defence’s administration’ across a number of areas, including T&E enterprise-level governance and managing acquisition risks24—as discussed in the relevant sections below.

T&E enterprise-level governance 4.22 The Auditor-General noted that Defence had ‘strengthened its enterprise-level management of test and evaluation in recent years.25 In particular, the ANAO pointed here to the establishment of the ADTEO in 2007 and

the T&E Principals’ Forum in 2008, along with the finalisation of an overarching T&E policy in 2015, the DCDM.26 Importantly, the ANAO noted that publication of Part Three of the DCDM completed a ‘longstanding commitment to the Parliament’.27 (However, the ANAO observed that this process had taken some 12 years from when the ANAO first made a recommendation on this matter.28)

4.23 Notwithstanding these positive developments, the Auditor-General found that conduct of T&E was ‘distributed across 12 Defence organisations, placing a premium on the effectiveness of Defence’s T&E governance’.29

22 ANAO Report No. 30 (2001-02), Test and Evaluation of Major Defence Equipment Acquisitions, pp. 23-24. 23 ANAO Report No. 9, p. 10. 24 ANAO Report No. 9, p. 10. 25 Mr Grant Hehir, Auditor-General, ANAO, ‘Opening statement by Auditor-General’,

Submission 3.2, p. 1. 26 ANAO Report No. 9, p. 66. 27 ANAO Report No. 9, p. 24. 28 In 2003 Defence advised the Parliament that a review of Defence T&E policies had been

initiated, following a recommendation on this matter in the 2002 ANAO report. This review had not been completed by the time of the 2012 Senate Committee inquiry, with the Committee recommending immediate policy finalisation. A completed version of Part Three of the DCDM was finally published in June 2015, ‘some 12 years after Defence had originally advised Parliament that a review and redevelopment of Defence T&E policy and procedures was underway’, ANAO Report No. 9, pp. 24-25. 29 Mr Hehir, ANAO, ‘Opening statement by Auditor-General’, Submission 3.2, p. 1. The 12

Defence T&E agencies are listed in ANAO Report No. 9, pp. 22-23. Each of these organisations has its own T&E manuals and reports T&E activities independently to its respective capability manager, and are all required to comply with one or more of defence’s three technical

56

The Auditor-General concluded that Defence’s administration of T&E would be therefore be strengthened by completing reforms to T&E personnel competency and training arrangements, and introducing arrangements to provide ‘enterprise-level advice to senior responsible leaders’ through integrated T&E policy and procedures and improved performance measures.30 These measures would provide ‘greater assurance’ as regards administration of Defence T&E and be ‘consistent with reforms underway within Defence to establish a stronger “strategic centre”’, through the First Principles Review.31 A summary of the relevant key points on these matters from the ANAO report is set out below:

 Competency and training: Defence has made slow progress in

implementing the 2012 Senate Inquiry recommendations relating to T&E personnel competency and training requirements. No whole-of-Defence T&E personnel competency and training needs analysis has been conducted and T&E personnel training and competency requirements management vary significantly between the armed Services … and the Capability Acquisition and Sustainment Group (CASG) … Defence’s approach to the training of T&E personnel has been the subject of ongoing concern. In 2001, the ANAO found that Defence’s approach to providing T&E training was decentralised and ad hoc, and not well linked in terms of coordination or information sharing32  Policy and procedures: Defence has more to do to provide a

comprehensive and integrated T&E framework to its project offices by ensuring: the DCDM aligns with Navy, Army and Aerospace regulatory management manuals; the DCDM is aligned with new organisational structures arising from the implementation of the First Principles Review; and that subsidiary T&E policy and procedural guidance manuals used by the various project offices are consistent with the DCDM33  Performance information: The ANAO has … suggested that

the T&E Principals’ Forum should make the establishment of key T&E performance indicators a matter of priority34

4.24 On the ANAO’s recommendation that, to strengthen the enterprise-level management of the T&E workforce, Defence conduct a T&E competency and training needs analysis, and monitor the availability of appropriately

regulatory management manuals. The exception to those arrangements is the Army, which has embedded a significant proportion of its operational T&E staff in ADTEO, p. 9. 30 Mr Hehir, ANAO, ‘Opening statement by Auditor-General’, Submission 3.2, p. 1. 31 Mr Hehir, ANAO, ‘Opening statement by Auditor-General’, Submission 3.2, pp. 1-2. 32 ANAO Report No. 9, p. 9, p. 30. See ANAO Report No. 30 (2001-02). 33 ANAO Report No. 9, p. 9. 34 ANAO Report No. 9, p. 65.

TEST AND EVALUATION OF MAJOR DEFENCE EQUIPMENT ACQUISITIONS 57

trained T&E personnel and address any gaps identified (Recommendation 1), Defence noted that a tender had been released to contract a training needs analysis between 7 March 2016 and 27 May 2016 and, following the outcomes of this analysis, Defence would ‘develop processes to monitor the availability of training of test and evaluation staff, leading to closing out the recommendation from the ANAO report’.35 It is also intended under the First Principles Review that ADTEO (as part of the Vice Chief of the Defence Force Group) be accountable for this matter within Defence.36

4.25 On the ANAO’s recommendation that Defence introduce arrangements to provide the Vice Chief of the Defence Force (VCDF) and capability managers with enterprise-level advice on the coordination, monitoring and evaluation of the adequacy and results of T&E activities (Recommendation 3), Defence noted that ‘this recommendation will be closed out’, by expanded T&E reporting through ADTEO to VCDF.37 ADTEO currently provides a report on Defence activities covering trials, activities and test planning in support of projects, with this reporting to be ‘expanded to include input from other test and evaluation agencies and promulgated to the Vice Chief of the Defence Force and the capability managers from April onwards’.38

4.26 There was interest at the public hearing in further exploring whether T&E had been integrated into a consistent, whole-of-Defence framework across Navy, Army and Air Force such that ‘everyone has T&E firmly on the radar’. Defence again emphasised that the ‘key element’ here is the 2007 establishment of ADTEO, in providing a ‘joint context’:

From then, we have been improving the approach … we have individual service test and evaluation organisations both in Navy and in Air Force. We have the central office under the Australian Defence Test and Evaluation Office, who also up till this point have been conducting the test and evaluation elements for Army … most importantly, the Australian Defence Test and Evaluation Office have been providing that joint context that provides those policies, the standards, the governance and the management across all of those elements so that we do pull that together across a whole-of-Defence perspective.39

35 Air Vice Marshal Hupfeld, Defence, Committee Hansard, 3 March 2016, p. 9. 36 Air Vice Marshal Hupfeld, Defence, Committee Hansard, 3 March 2016, p. 9. 37 Air Vice Marshal Hupfeld, Defence, Committee Hansard, 3 March 2016, pp. 9-10. 38 Air Vice Marshal Hupfeld, Defence, Committee Hansard, 3 March 2016, pp. 9-10. 39 Air Vice Marshal Hupfeld, Defence, Committee Hansard, 3 March 2016, p. 15.

58

4.27 Defence explained that this approach would be further strengthened under the First Principles Review—‘we will be strengthening the joint aspect … of the Australian Defence Test and Evaluation Office. That may change its name … the outcomes and structures for this are pre-decisional, but the intent is clear’.40 As Defence further highlighted, ‘under the First Principles Review we are about to undergo some significant changes to both our processes and our organisational structure’, with ADTEO proposed to be relocated to the VCDF Group to ‘provide a strong strategic centre for test and evaluation in Defence’.41 As to when the First Principles Review reform process is planned to be completed, Defence responded that the ‘key time line and transition … will depend on the approvals and agreement of that new process. We estimate that it may well be by the middle of the year … A two-year implementation time line was provided by the first principles review’.42 Asked whether there would be benefit in establishing a formal legislative framework for T&E, Defence pointed to its overarching T&E policy, as set out in the DCDM, and implementation of the recommendations of the First Principles Review and the ANAO audit as sufficiently ensuring ‘consistent application of T&E principles’.43

4.28 Under these new arrangements, capability managers will be responsible for their respective T&E agencies, ‘allowing more direct reporting to them’ and ADTEO will provide ‘standard policy across Defence and governance of test and evaluation activities’.44 Defence confirmed that the ‘recommendations and advice contained in the ANAO report informed the process that led to this direction’.45 As to whether this integrated approach, devolved to capability managers, might lead to inconsistency between individuals and inconsistency between the services, Defence reiterated that ‘policy and standards associated with both training and competency assessments will be managed from a joint perspective’.46 As Mr Kim Gillis, Deputy Secretary, Capability Acquisition and Sustainment Group, Defence, emphasised: ‘the vice-chief and I, working with the

40 Air Vice Marshal Hupfeld, Defence, Committee Hansard, 3 March 2016, p. 15. 41 Air Vice Marshal Hupfeld, Defence, Committee Hansard, 3 March 2016, p. 9—this is the ‘proposed approach. It is still preapproval through the department, let alone government’, p. 9. 42 Air Vice Marshal Hupfeld, Defence, Committee Hansard, 3 March 2016, p. 16. 43 Defence, Submission 4.1, p. 3. 44 Air Vice Marshal Hupfeld, Defence, Committee Hansard, 3 March 2016, p. 9. 45 Air Vice Marshal Hupfeld, Defence, Committee Hansard, 3 March 2016, p. 9. 46 Air Vice Marshal Hupfeld, Defence, Committee Hansard, 3 March 2016, p. 16.

TEST AND EVALUATION OF MAJOR DEFENCE EQUIPMENT ACQUISITIONS 59

capability managers, are taking a lead on the reform of this, and the one mantra that we are pushing is commonality right across Defence’:47

as the delivery agency for the vast majority of the Defence projects, I need to have a common approach to all of these things that are being delivered, in the way they are being tested and evaluated— especially when you start getting into the joint programs, where you have aspects of Navy, Army or Air Force involved in some of these components. Historically, that has sometimes caused issues because of the differences in approaches. It is about getting to that common focus and a common set of standards, qualifications, experience and training and a unified organisation while still,

however, maintaining that accountability of the capability managers. That is the nuance of this, but we have to maintain that commonality.48

4.29 In terms of how the accountability of capability managers for this area would be made clear and strengthened, Defence responded that these arrangements would be ‘clearly articulated in the design of the new process in terms of roles and responsibilities … it will be for the secretary and CDF to ensure that that is clear’.49 Mr Gillis also pointed to the ‘joint accountability of the Defence leadership’:

We have to work as a single, unified ‘One Defence’ organisation for all of these attributes. That is the cultural and behavioural change that we need to make, so, when I say, ‘I work closely with the capability managers, the chief, the vice-chief et cetera,’ who have to deliver this, we all share that accountability. None of us look at it as if it is theirs or mine; it is our responsibility to deliver this capability.50

Managing acquisition risks, including HMAS Canberra case study 4.30 The ANAO examined the management of acquisition risks in terms of reducing risk in the transition of capability from the acquisition phase to operations. The focus here was T&E for Australia’s evolving amphibious

deployment and sustainment capability—specifically:

 development and acceptance T&E for two Canberra Class

Landing Helicopter Docks (LHDs); and

47 Mr Kim Gillis, Deputy Secretary, Capability Acquisition and Sustainment Group, Defence, Committee Hansard, 3 March 2016, p. 16. 48 Mr Gillis, Defence, Committee Hansard, 3 March 2016, p. 16. 49 Air Vice Marshal Hupfeld, Defence, Committee Hansard, 3 March 2016, p. 16. 50 Mr Gillis, Defence, Committee Hansard, 3 March 2016, p. 16.

60

 operational T&E for the two LHDs, 47 MRH90 helicopters, and

12 LHD Landing Craft.51

4.31 The Auditor-General noted that the ANAO’s case study on the first LHD, HMAS Canberra, indicated key management decisions were ‘usefully informed’ by T&E, which ‘identified numerous defects and deficiencies for resolution’.52 Defence decided, ‘on balance, to accept HMAS Canberra on the understanding that the deficiencies would be addressed during the ship’s operational phase’.53 However, in doing so, the Auditor-General concluded that the ‘Chief of Navy accepted greater risks than would have been the case had System Acceptance been based on more complete objective quality evidence of compliance with contracted specifications, and had Initial Materiel Release been based on less qualified findings by Defence’s regulators concerning compliance with technical, operational and safety management system requirements’.54 The Auditor-General observed that, as operational T&E is still underway and is not due for completion until the fourth quarter of 2017, it ‘remains to be seen what impact, if any, this elevated risk has on the achievement of Final Operational Capability’.55

4.32 By way of background on the ANAO’s finding in this area, at System Acceptance (as set out in Table 4.1 of the ANAO report), only 31 per cent of Harbour Acceptance Trials Test Reports and 39 per cent of Sea Acceptance Trials Test Reports had been approved.56 Early operational T&E of HMAS Canberra also commenced against a ‘backdrop of significant work required to verify contractual compliance with 451 function and performance specifications, which had not occurred at the time of System Acceptance and Initial Materiel Release’.57

4.33 On the ANAO’s recommendation that, to reduce risk and assist the transition of capability from the acquisition phase to operations, Defence should ensure that material defects are documented and plans for remediation established prior to System Acceptance (Recommendation 2), Defence noted it had agreed to this recommendation ‘upon the understanding that extant Defence policy and procedures were in place to satisfy the recommendation’, with work in this area also being

51 ANAO Report No. 9, p. 45. The overall total approved acquisition budget for these elements of Defence’s amphibious capability is $7.072 billion (June 2015). Of that amount, $5.600 billion had been spent by June 2015, p. 45. 52 Mr Hehir, ANAO, ‘Opening statement by Auditor-General’, Submission 3.2, p. 2. 53 Mr Hehir, ANAO, ‘Opening statement by Auditor-General’, Submission 3.2, p. 2. 54 Mr Hehir, ANAO, ‘Opening statement by Auditor-General’, Submission 3.2, p. 2. 55 Mr Hehir, ANAO, ‘Opening statement by Auditor-General’, Submission 3.2, p. 2. 56 ANAO Report No. 9, p. 48. 57 ANAO Report No. 9, p. 10.

TEST AND EVALUATION OF MAJOR DEFENCE EQUIPMENT ACQUISITIONS 61

strengthened under the First Principles Review.58 Defence further commented that it was ‘gathering evidence of compliance with these policies and procedures to close off this recommendation’.59

4.34 At the public hearing, the ANAO and Defence further discussed this recommendation. The ANAO commented that, as T&E is a basis for assessing risk and for advising capability managers, in this case the Chief of Navy, of the risks in making decisions, ‘we thought it would be best if at least the material deficiencies found as a consequence of T&E work were properly documented and a process was in place for providing a level of advice and assurance … that it was being managed effectively’.60 Commodore Stephen Hughes, Director General, Navy Capability Transition and Sustainment, Defence, acknowledged that the ‘ANAO audit … hit the key issues for Navy around what we would like to see better done if we had our time again’.61 However, Commodore Hughes emphasised that T&E is ‘one part of a numerous armoury of tools we use to make a decision for the Chief of Navy’—‘I predicate that the Chief of Navy did not make this decision, I would say, easily or haphazardly; it was a very long process we followed to come to the conclusion that the vessel was at an acceptable level of understanding and risk to move forward into the follow-on test and evaluation process’.62 As Defence further noted, T&E is ‘costly in terms of both dollars and time, so we always will have to have an element of risk-based assessment on that’— ‘we have learnt the lessons of preview T&E and how we then take test and evaluation between first and second pass and, indeed, into acceptance into service … but there will always exist a tension between how much you would like to do and how much you can do, and we would try to balance that’.63

4.35 In terms of the outstanding harbour and sea acceptance trials test reports, at the time of system acceptance, as identified by the ANAO in Table 4.1 of its report, Defence explained that ‘not all those reports were about issues where we had not done a test. Those reports might have not been delivered for a number of reasons—there were spelling mistakes or they were not in the right format. There were some trials which had not been

58 Air Vice Marshal Hupfeld, Defence, Committee Hansard, 3 March 2016, p. 9. 59 Air Vice Marshal Hupfeld, Defence, Committee Hansard, 3 March 2016, p. 9. 60 Dr Tom Ioannou, Group Executive Director, Performance Audit Services Group, ANAO, Committee Hansard, 3 March 2016, p. 10. 61 Commodore Stephen Hughes, Director General, Navy Capability Transition and Sustainment,

Defence, Committee Hansard, 3 March 2016, p. 11. 62 Commodore Hughes, Defence, Committee Hansard, 3 March 2016, p. 11. 63 Air Vice Marshal Hupfeld, Defence, Committee Hansard, 3 March 2016, p. 10.

62

completed, so it was a mixed bag’.64 As Commodore Hughes further observed, ‘in a number sense I agree with the advice in Table 4.1 [of the ANAO report], but the context around that is how much understanding Navy had around what was missing at the time’:

we had eyes and ears on the ground seeing the issues as they were raised and reporting back into both the project and into the Navy … All that fed back as a body of knowledge for us to present to the CN [Chief of Navy] about the risks he was taking forward … In summary, I think that CN at the time had a very comprehensive view of the risk he was taking.65

4.36 As to why, at the time of the audit report, nearly 12 months after system acceptance, the percentage of approved harbour and sea acceptance trial test reports, as listed in Table 4.1, had risen to only 38 per cent and 60 per cent (from 31 per cent and 39 per cent respectively).66 Defence clarified that a ‘number of those were not intended to be completed until after operational availability’.67 Commodore Hughes acknowledged there were a ‘number of documents that were interim type documents, not formal, signed-off, cleared OQE [objective quality evidence] as per the contract’, and concluded: ‘I do not think I would like to see that again, because one of the issues the ANAO rightly brought up was that it made us have to rely on our own professional judgement based on what evidence we had. It is not that there was not any evidence or the trial did not occur; it was that we did not necessarily have a formal [piece] of OQE that we could take forward with us’.68 As at August 2015, 61 per cent of harbour acceptance trial test reports were outstanding, and as at February 2016, 52 per cent were still outstanding. As at August 2015, 39 per cent of sea acceptance trial test reports were outstanding, and as at February 2016, 30 per cent were still outstanding.69

64 Commodore Hughes, Defence, Committee Hansard, 3 March 2016, p. 11. 65 Commodore Hughes, Defence, Committee Hansard, 3 March 2016, p. 11. For example, Commodore Hughes noted that some of the trials were ‘hampered because a certain system on the day did not turn on or off’ and the ‘full capacity or capability of some of those systems

could not be fully tested because you needed 800 people’, Committee Hansard, 3 March 2016, p. 11. 66 ANAO Report No. 9, pp. 48-49. 67 Mr Patrick Fitzpatrick, Director-General, Future Frigate, Defence, Committee Hansard, 3 March

2016, p. 12. Mr Gillis, Defence, similarly noted: ‘there are some practical things that stand out that we have to use in operational tests to really test them … there is a whole detailed part where you need a fully trained and operating crew, and that is why we do it in that particular phase of the operation’, Committee Hansard, 3 March 2016, p. 13. 68 Commodore Hughes, Defence, Committee Hansard, 3 March 2016, p. 12. 69 Defence, Submission 4.1, p. 13.

TEST AND EVALUATION OF MAJOR DEFENCE EQUIPMENT ACQUISITIONS 63

4.37 In terms of the 451 requirements not yet certified as complying with the LHD function and performance specifications at the time of HMAS Canberra’s Initial Materiel Release on 31 October 2014, Defence explained that ‘we … had a plan with CASG [Capability Acquisition and Sustainment Group] of how to burn down that list of defects and deficiencies between November 2014 into May 2015 … What we did underestimate is the ability of the contractor to actually burn down that list’:

as time went on and the list was not burnt down and corrected, the risk we were carrying was increasing. That caused us a great deal of concern. We had a number of roundtables with CASG and the contractor. We got it all back on track. Then, by June 2015, when the ship achieved its unit readiness work-up state, we were a lot more comfortable about the material state of the ship.70

4.38 A range of contractual matters were also discussed at the public hearing. As to whether acceptance of the ship in this way by Chief of Navy might affect the contractual position of the Commonwealth, including in terms of leverage, Defence confirmed that ‘we still hold the contractor fully liable for those things outstanding. That is drafted into the contractual obligations’.71 On the contract options available to Defence through payment milestones and other mechanisms, particularly after acceptance of a product has been confirmed, Mr Gillis, Deputy Secretary, Capability Acquisition and Sustainment Group, Defence, responded: ‘my view is that we did a better job on Adelaide, but we did not lose the contractual position where we told the contractor that they are still accountable and responsible to deliver that; they still have to remediate any defects’.72 As Mr Gillis further emphasised, ‘they do not get that final payment, and they do not get the payment for that particular milestone, until they actually complete those tests. So we still hold them contractually’.73

4.39 There were interest in whether a lesson learnt here by Defence related to structuring a contract so that the testing process allows a stronger position for Defence in negotiating rectification of defects. As the person who signed that contract, Mr Gillis replied: ‘yes, I have learnt that lesson, and I will be applying that lesson … I accept that recommendation’.74 Mr Gillis confirmed his accountability for the contract on this project: ‘I signed that contract on behalf of the Commonwealth … I have read this audit report,

70 Commodore Hughes, Defence, Committee Hansard, 3 March 2016, pp. 11-12. 71 Mr Gillis, Defence, Committee Hansard, 3 March 2016, p. 13. 72 Mr Gillis, Defence, Committee Hansard, 3 March 2016, p. 14. 73 Mr Gillis, Defence, Committee Hansard, 3 March 2016, p. 14. 74 Mr Gillis, Defence, Committee Hansard, 3 March 2016, p. 13.

64

and I take note that we should have done this in ship 1 and in ship 2 … So I take that accountability, and I have taken on that lesson’.75

4.40 As to whether Defence sufficiently appreciated the contractual and potential public image risks of this area, Defence responded: ‘we did learn a lot from the experience of Canberra and we did make significant improvements in that second of class in Adelaide. The procedures were better; the processes were better. This was the first of class of the largest ship the Australian Navy has ever taken on board’.76 Mr Gillis observed that ‘one of the lessons that I have observed over the last 10 years or so is that sometimes we need to put the equipment in the hands of the war fighters to find out: what is the capability that we really have, and what is the outcome that we are achieving?’77 Mr Gillis pointed to the Wedgetail program as an example of this, noting that it was a ‘very, very difficult platform’ to do T&E against because ‘we had not actually used it in full operations’—‘what we have found is that, subsequent to the formal handover and the acceptance of that, it has achieved much better outcomes in some areas than we knew at the point of acceptance. We cannot do that until we have actually got it in the field and we have tested it. That is a difficult part of this differential between a contractual acceptance’.78

4.41 As to the ANAO’s view about whether these sorts of practices might pose a contractual risk or a risk to the Commonwealth’s financial position, the Auditor-General responded that ‘with the nature of the decision, what we point out in our report is that by doing things the way they were done increased risk was taken on’.79 As the Auditor-General further noted:

What we do not say is that that risk was taken on without the eyes of the procurer being open, so we are not making a judgment that that was a poor decision. I think they are fully aware of all of those risks, and a process that takes longer to get to conclusion changes the risk, including in how you manage the contract and hold the other party to account.80

4.42 Finally, there was interest in how HMAS Canberra was operating at the moment and whether there were any outstanding issues. Defence explained that, ‘in a complex system of this size, there will always be small issues, small faults, in the system … They will happen through the life of

75 Mr Gillis, Defence, Committee Hansard, 3 March 2016, p. 12. 76 Mr Gillis, Defence, Committee Hansard, 3 March 2016, p. 13. 77 Mr Gillis, Defence, Committee Hansard, 3 March 2016, p. 14. 78 Mr Gillis, Defence, Committee Hansard, 3 March 2016, p. 14. 79 Mr Hehir, ANAO, Committee Hansard, 3 March 2016, p. 13. 80 Mr Hehir, ANAO, Committee Hansard, 3 March 2016, p. 13.

TEST AND EVALUATION OF MAJOR DEFENCE EQUIPMENT ACQUISITIONS 65

the system’.81 Commodore Hughes concluded that, while there are ‘still outstanding issues … for this capability for the next … 40 or 50 years’, which will have to be managed through a risk approach, the ‘vessel is exceeding our expectations of where we thought we would be 18 months

into a program’:

in under a year we achieved IOC, initial operating capability, which I think is unheard of for maritime capability … One thing that I am very confident about is that we really understood the safety component of this capability. T&E did this for us in a number of ways. There has not been one issue raised in both vessels in the last 18 months which we did not foresee in our safety case. Again, I think that is a unique outcome for Navy …

to come back to your ultimate question, there is still a lot of fundamental, deep-level work being done to ensure the next 40 years sustainability of capability, and it takes a huge team out of the CASG and the industry partners.82

Committee comment

4.43 The Committee identified Defence T&E as an audit priority of Parliament in December 2012 and May 2014, in recognition of the ongoing importance of this area.83 The Committee notes that, over a number of years, several ANAO audits and external reviews have identified deficiencies in aspects of Defence’s T&E program and provided recommendations for improvement.84 Defence has made progress in a number of areas towards implementing recommendations from these reports, but this has taken considerable time. Overall, Defence still needs to improve key aspects of its T&E administration—as reflected in the ANAO recommendations—and to do so in a timely manner.

4.44 The Committee notes that Defence agreed to the ANAO recommendations without qualification and, at the public hearing and in its submission to the inquiry, provided an update on implementation progress. Importantly, Defence confirmed that, as part of the First Principles Review process, a

81 Mr Gillis, Defence, Committee Hansard, 3 March 2016, p. 14. 82 Commodore Hughes, Defence, Committee Hansard, 3 March 2016, pp. 14-15. 83 ANAO Report No. 9, p. 17. 84 In particular, ANAO Report No. 30 (2001-02), Test and Evaluation of Major Defence Equipment

Acquisitions, and the Senate Foreign Affairs, Defence and Trade References Committee report, Procurement Procedures for Defence Capital Projects (2012). Figure 4.2 sets out common themes from these reports.

66

range of activities are currently underway to strengthen T&E arrangements across the department, to ‘better achieve the outcomes and the recommendations that have been put forward by the ANAO’.85

4.45 In terms of T&E enterprise-level governance, the Committee was pleased to note the Auditor-General’s finding that Defence had strengthened its enterprise-level management of T&E conducted in support of major equipment acquisitions in recent years.86 After the 2007 establishment of the ADTEO as lead authority for T&E, and with the T&E Principals’ Forum to foster consistency of approach, Defence has now developed an overarching policy on T&E in the form of the DCDM. This completes a longstanding commitment to the Parliament—albeit after a significant 12-year delay.87

4.46 However, the Committee is mindful of the Auditor-General’s overall conclusion that conduct of T&E was distributed across 12 Defence organisations, placing a premium on the effectiveness of Defence’s T&E governance.88 This decentralised structure increased the risk of an inconsistent approach to T&E.

4.47 Defence provided a detailed response at the public hearing to the ANAO recommendation that it should introduce arrangements to provide VCDF and capability managers with enterprise-level advice through integrated T&E policy and procedures, including guidance manuals that are consistent with the DCDM, and improved performance measures. Defence described their proposal for ADTEO to be relocated to the VCDF Group— with the ANAO recommendations having ‘informed the process that led to this direction’.89 ADTEO will provide a ‘joint context’ through policies, standards, governance and management to ensure a consistent ‘whole-of-Defence perspective’.90 Defence also assured the Committee that capability managers’ accountability for consistent T&E will be strengthened through emphasis on the ‘joint accountability of the Defence leadership’.91 In particular, the Committee was pleased to note the confirmation from Mr Kim Gillis, Deputy Secretary, Capability Acquisition and Sustainment Group, Defence, that ‘the vice-chief and I, working with the capability managers, are taking a lead on the reform of this, and the one mantra that

85 Air Vice Marshal Hupfeld, Defence, Committee Hansard, 3 March 2016, p. 10. 86 Mr Hehir, ANAO, ‘Opening statement by Auditor-General’, Submission 3.2, p. 1. 87 ANAO Report No. 9, p. 25. 88 Mr Hehir, ANAO, ‘Opening statement by Auditor-General’, Submission 3.2, p. 1. 89 Air Vice Marshal Hupfeld, Defence, Committee Hansard, 3 March 2016, p. 9. 90 Air Vice Marshal Hupfeld, Defence, Committee Hansard, 3 March 2016, p. 15. 91 Mr Gillis, Defence, Committee Hansard, 3 March 2016, p. 16. See also Air Vice Marshal Hupfeld,

Defence, Committee Hansard, 3 March 2016, p. 16.

TEST AND EVALUATION OF MAJOR DEFENCE EQUIPMENT ACQUISITIONS 67

we are pushing is commonality right across Defence’.92 The Committee is also pleased to note that Defence is strengthening its performance monitoring and reporting at the enterprise-level through establishment of formal reporting arrangements to the VCDF.93

4.48 In terms of improving the T&E workforce, the Committee was disappointed to note the ANAO’s finding that Defence had made slow progress in implementing the 2012 Senate inquiry recommendations on T&E competency and training, with no whole-of-Defence competency and training needs analysis having been conducted.94 At the public hearing Defence confirmed that a tender has now been released to contract a training needs analysis and that ADTEO, within VCDF Group, would be accountable for implementation of whole-of-Defence competency and training in this area.95 Defence further commented that they would monitor the availability of staff training. The Committee is pleased that some actions are now being taken in this area but notes that Defence will also need to take steps to address any competency and training gaps identified before the Committee would consider this ANAO recommendation to be ‘closed out’.

4.49 In summary, reforms to T&E competency and training arrangements need to be completed and Defence needs to introduce arrangements to provide a more coordinated, whole-of-Defence T&E governance through enterprise-level advice to senior leaders, integrated T&E policy and procedures, and improved performance measures. The Committee notes the significant reforms underway within Defence to establish a stronger ‘strategic centre’, as part of the First Principles Review process which is scheduled to be completed by approximately April 2017. It also notes that Defence is addressing the ANAO recommendations as part of these reforms.96 Defence—understandably—therefore made references to proposed restructures and future actions as a result of the review that are ‘still preapproval through the department, let alone government’.97 Against this background and in the context of the sometimes slow progress by Defence in implementing past audit and review recommendations, the Committee is of the view that Defence should report back to the JCPAA on implementation of a range of T&E matters

92 Mr Gillis, Defence, Committee Hansard, 3 March 2016, p. 16. 93 Air Vice Marshal Hupfeld, Defence, Committee Hansard, 3 March 2016, p. 9. 94 ANAO Report No. 9, p. 9. See ANAO Report No. 30 (2001-02). 95 Air Vice Marshal Hupfeld, Defence, Committee Hansard, 3 March 2016, p. 9. 96 ANAO Report No. 9, p. 11. 97 Air Vice Marshal Hupfeld, Defence, Committee Hansard, 3 March 2016, p. 9.

68

with regard to the ANAO recommendations and First Principles Review reforms.

Recommendation 5

4.50 To promote continued improvement in test and evaluation (T&E) of major equipment acquisitions by the Department of Defence, the Committee recommends that the department report back to the Committee at 12 months from the tabling of the Committee’s report, on:

 how T&E is being coordinated across the department and how

this coordination is consistent with the recommended outcomes of the First Principles Review: Creating One Defence (2015)

 key improvements to T&E performance monitoring and

reporting

 each element of the Australian National Audit Office

recommendation on T&E competency and training, including the key findings of and actions from the competency and training needs analysis

4.51 In terms of improved management of acquisition risks, with reference to the ANAO’s case study on Australia’s new LHD, HMAS Canberra, the Committee notes the Auditor-General’s finding that key management decisions were usefully informed by T&E, which identified numerous defects and deficiencies for resolution.98 However, Defence decided, on balance, to accept HMAS Canberra on the understanding that the deficiencies would be addressed during the ship’s operational phase.99 The Auditor-General concluded that the Chief of Navy had therefore accepted greater risks than would have been the case had System Acceptance been based on more complete objective quality evidence of compliance with contracted specifications, and had Initial Materiel Release been based on less qualified findings by Defence’s regulators concerning compliance with technical, operational and safety management system requirements.100

4.52 The Committee was pleased to note Defence’s confirmation—in response to the ANAO recommendation that it ensure material defects are documented and remediation plans established prior to System Acceptance—that this area is being strengthened under the First Principles

98 Mr Hehir, ANAO, ‘Opening statement by Auditor-General’, Submission 3.2, p. 2. 99 Mr Hehir, ANAO, ‘Opening statement by Auditor-General’, Submission 3.2, p. 2. 100 Mr Hehir, ANAO, ‘Opening statement by Auditor-General’, Submission 3.2, p. 2.

TEST AND EVALUATION OF MAJOR DEFENCE EQUIPMENT ACQUISITIONS 69

Review, and that Defence is ‘gathering evidence of compliance with these policies and procedures to close off this recommendation’.101 Defence emphasised its comprehensive view of the complexity of risks involved— that T&E is ‘one part of a numerous armoury of tools’ used to make a decision for the Chief of Navy’102—but acknowledged it had ‘learnt the lessons of preview T&E’ and that the ‘ANAO audit … hit the key issues for Navy around what we would like to see better done if we had our time again’.103

4.53 Overall, the Committee appreciated the confirmation of accountability for this matter by Mr Kim Gillis, Deputy Secretary, Capability Acquisition and Sustainment Group, Defence: ‘I signed that contract on behalf of the Commonwealth, so I take the accountability and responsibility for that signature in 2007. I have read this audit report, and I take note that we should have done this in ship 1 and in ship 2 … I take that accountability, and I have taken on that lesson’.104

4.54 In summary, while acceptance of the LHD, HMAS Canberra, was not ideal,105 the Committee notes that risks appear to have been satisfactorily managed by Navy and that HMAS Canberra has been successfully deployed.106 However, this example illustrates that Defence requires improved T&E to avoid the need for later, possibly heightened, risk management. A more coordinated whole-of-Defence approach to T&E has now commenced but is still a work in progress—hence the Committee’s recommendation above.

The Hon Ian Macfarlane MP Chair Date: 3 May 2016

101 Air Vice Marshal Hupfeld, Defence, Committee Hansard, 3 March 2016, p. 9. See also on this point, Commodore Hughes, Defence, Committee Hansard, 3 March 2016, p. 11. 102 Commodore Hughes, Defence, Committee Hansard, 3 March 2016, p. 11. See also Air Vice Marshal Hupfeld, Defence, Committee Hansard, 3 March 2016, p. 10. 103 Commodore Hughes, Defence, Committee Hansard, 3 March 2016, p. 11. 104 Mr Gillis, Defence, Committee Hansard, 3 March 2016, p. 12. 105 The Auditor-General noted that, as operational T&E is still underway and is not due for

completion until the fourth quarter of 2017, ‘it remains to be seen what impact, if any, this elevated risk has on the achievement of Final Operational Capability’, Mr Hehir, ANAO, ‘Opening statement by Auditor-General’, Submission 3.2, p. 2. 106 Commodore Hughes, Defence, Committee Hansard, 3 March 2016, p. 14.

70

A

Appendix A - Submissions

1. Great Barrier Reef Marine Park Authority

1.1 Supplementary Submission

1.2 Supplementary Submission

2. Telstra

3. Australian National Audit Office

3.1 Supplementary Submission

3.2 Supplementary Submission

4. Department of Defence

72

B

Appendix B - Public Hearings

11 February 2016

Australian National Audit Office

Mr Grant Hehir, Auditor-General

Ms Rona Mellor, Deputy Auditor-General

Ms Michelle Kelly, Group Executive Director, Performance Audit Services Group

Mr Mark Simpson, Executive Director, Performance Audit Services Group

Mr Grant Caine, Senior Director, Performance Audit Services Group

Great Barrier Reef Marine Park Authority

Dr Russell Reichelt, Chairman

Mr Bruce Elliot, General Manager, Biodiversity Conservation and Sustainable Use

3 March 2016

Australian National Audit Office

Mr Grant Hehir, Auditor-General

Dr Tom Ioannou, Group Executive Director, Performance Audit Services Group

Dr Tom Clarke, Executive Director, Performance Audit Services Group

Mr Alex Wilkinson, Director, Performance Audit Services Group

74

Department of Defence

Mr Kim Gillis, Deputy Secretary, Capability Acquisition and Sustainment Group

Major General David Coghlan, Head, Land Systems Division

Major General Fergus McLachlan, Head, Modernisation and Strategic Planning

Air Vice Marshal Mel Hupfeld, Acting Chief, Capability Development Group

Commodore Stephen Hughes, Director-General, Navy Capability Transition and Sustainment

Mr Patrick Fitzpatrick, Director-General, Future Frigate