Note: Where available, the PDF/Word icon below is provided to view the complete and fully formatted document
Public Accounts and Audit - Joint Committee Reports 439 Review of Auditor-General's reports Nos 11 to 31 (2012-13) June 2013


Download PDF Download PDF

The Parliament of the Commonwealth of Australia

Report 439 Review of Auditor-General’s Reports Nos. 11 to 31 (2012-13)

Joint Committee of Public Accounts and Audit

June 2013 Canberra

© Commonwealth of Australia 2013

ISBN 978-1-74366-075-1 (Printed version)

ISBN 978-1-74366-076-8 (HTML version)

This work is licensed under the Creative Commons Attribution-NonCommercial-NoDerivs 3.0 Australia License.

The details of this licence are available on the Creative Commons website: http://creativecommons.org/licenses/by-nc-nd/3.0/au/.

Contents

Foreword ............................................................................................................................................. v

Membership of the Committee ........................................................................................................... vii

List of abbreviations ............................................................................................................................ ix

List of recommendations ..................................................................................................................... xi

1 Introduction ......................................................................................................... 1

Background to the review ........................................................................................................ 1

The Committee’s report ............................................................................................................ 2

2 Performance Audit Report No.17 (2012-13) Design and Implementation of the Energy Efficiency Information Grants Program .......................................... 3

Introduction ............................................................................................................................... 3

Audit objective and scope ........................................................................................................... 4

Audit conclusion .......................................................................................................................... 4

Audit recommendations .............................................................................................................. 6

The Committee’s review ........................................................................................................... 6

Resourcing of the program .......................................................................................................... 7

Record-keeping deficiencies ....................................................................................................... 9

Improvements to second funding round .................................................................................... 11

Committee Comment .............................................................................................................. 13

iv

3 ANAO Report No.28 (2012-13) The Australian Government Performance Measurement and Reporting Framework - Pilot Project to Audit Key Performance Indicators..................................................................................... 15

Introduction ............................................................................................................................. 15

Previous audit reports and JCPAA reviews ............................................................................... 15

The pilot project ...................................................................................................................... 17

Objective and scope .................................................................................................................. 17

Funding considerations ............................................................................................................. 18

Findings and conclusions .......................................................................................................... 19

The Committee’s review ......................................................................................................... 23

The need for greater focus on performance assessment .......................................................... 23

A more flexible framework ......................................................................................................... 25

Future directions for reform ....................................................................................................... 26

Committee Comment .............................................................................................................. 28

Appendix A - Submissions ..................................................................................... 31

Appendix B - Public Hearing ................................................................................... 33

Appendix C - Department of Finance and Deregulation response to Recommendations 1 to 3 of JCPAA Report 430..................................................... 35

Foreword

In the 43rd Parliament, the Joint Committee of Public Accounts and Audit (JCPAA) has spent quite some time on how government measures its own performance, and the process of transparent (or otherwise) grant approvals.

It is therefore appropriate that one of our final oversight reports covers a Key Performance Indicator trial undertaken by the Audit Office; as well as looking at the Energy Efficiency Information Grants program, which once again showed concerns around process.

With public sector reform happening through the Commonwealth Financial Accountability Review, having a culture of very clear and measurable Key Performance Indicators is important.

Getting the processes right on grant programs is also important. The Auditor- General has repeatedly reported his concerns on this issue, and the JCPAA shares this frustration. It is time for Parliament and the Executive to place greater priority on getting this right.

This is the 100th anniversary of the oversight and scrutiny work of the JCPAA in Australia. Oversight and audit is expanding throughout many democracies, and Australia is increasingly being looked on for leadership and guidance in this area.

I make this point for two reasons. The JCPAA is a special institution of its own to be valued into the future. Its status deserves ongoing respect.

I also make this point to acknowledge the Secretariat, and their very good work over the past three years. They play a vital role between Executive Government and Parliament, and have done so very professionally. They have done the history of oversight in Australia proud.

Robert Oakeshott MP Chair

vi

Membership of the Committee

Chair Mr Robert Oakeshott MP

Deputy Chair Ms Gai Brodtmann MP

Members Hon Dick Adams MP Senator Mark Bishop

Mr Jamie Briggs MP Senator the Hon Kim Carr

(from 14 May 2013)

Mr Darren Cheeseman MP Senator Louise Pratt

Mr Josh Frydenberg MP Senator Anne Ruston

Mr Harry Jenkins MP Senator Dean Smith

Ms Deborah O’Neill MP

Ms Laura Smyth MP

Hon Alex Somlyay MP

viii

Committee Secretariat

Secretary Mr David Brunoro

Inquiry Secretaries Mr James Nelson

Ms Vikki Darrough

Research Officer Ms Elina Gilbourd

Administrative Officers Ms Jazmine Rakic

Ms Karen Underwood

List of abbreviations

ACAG Australasian Council of Auditors-General

ANAO Australian National Audit Office

CFAR Commonwealth Financial Accountability Review

DCCEE Department of Climate Change and Energy Efficiency

EEIG Energy Efficiency Information Grants

Finance Department of Finance and Deregulation

JCPAA Joint Committee of Public Accounts and Audit

KPI Key Performance Indicator

PAC Program Advisory Committee

PM&C Department of the Prime Minister and Cabinet

RET Department of Resources, Energy and Tourism

x

List of recommendations

The Australian Government Performance Measurement and Reporting Framework - Pilot Project to Audit Key Performance Indicators

Recommendation 1

The Joint Committee of Public Accounts and Audit recommends that the Department of Finance and Deregulation, in consultation with the Australian National Audit Office, prioritise the review and update of the performance measurement and reporting framework. A goal should be to have clear policy and guidance in place for the 2014-15 financial year that can be used by agencies to produce auditable Key Performance Indicators, irrespective of the passage of the Public Governance, Performance and Accountability Bill 2013.

Recommendation 2

The Joint Committee of Public Accounts and Audit recommends that:

 the Government reinforce the requirements for agencies to incorporate specific performance monitoring, reporting and evaluation activities into the design and costing of their programs;

 agencies be appropriately funded to carry out these activities; and

 monitoring be used to provide assurance that these activities are implemented.

xii

1

Introduction

Background to the review

1.1 The Joint Committee of Public Accounts and Audit (JCPAA) has a statutory duty to examine all reports of the Auditor-General that are presented to the Australian Parliament, and report the results of its deliberations to both Houses of Parliament. In selecting audit reports for detailed review, the Committee considers factors such as:

 the significance of the program or issues raised in audit reports;

 the significance of the audit findings;

 the arguments advanced by the audited agencies; and

 the public interest in the report.

1.2 Upon consideration of twenty-one audit reports presented to the Parliament by the Auditor-General between November 2012 and May 2013, the Committee selected two reports for further scrutiny.

1.3 The reports reviewed by the JCPAA in detail were:

 Performance Audit Report No.17 2012-13 Design and Implementation of

the Energy Efficiency Information Grants Program; and

 ANAO Report No.28 2012-13 The Australian Government Performance

Measurement and Reporting Framework: Pilot Project to Audit Key Performance Indicators.

1.4 Public hearings for the two reports were held in Canberra on Wednesday, 19 June 2013.

2 REPORT 439: REVIEW OF AUDITOR-GENERAL’S REPORTS NOS. 11 TO 31 (2012-13)

The Committee’s report

1.5 This report of the Committee’s examination draws attention to the main issues raised at the public hearing. Where appropriate, the Committee has commented on unresolved or contentious issues, and has made recommendations.

1.6 The Committee’s report is structured as follows:

 Chapter 2 - Performance Audit Report No.17 2012-13 Design and

Implementation of the Energy Efficiency Information Grants Program

 Chapter 3 - ANAO Report No.28 2012-13 The Australian Government

Performance Measurement and Reporting Framework: Pilot Project to Audit Key Performance Indicators.

1.7 The following appendices provide additional information:

 Appendix A - List of submissions

 Appendix B - List of public hearings and witnesses

 Appendix C - Department of Finance and Deregulation response to

Recommendations 1 to 3 of JCPAA Report 430.

1.8 A copy of this report, transcripts of hearings and submissions are available on the Committee’s website: www.aph.gov.au/jcpaa

2

Performance Audit Report No.17 (2012-13)

Design and Implementation of the Energy Efficiency Information Grants Program

Introduction

2.1 The Energy Efficiency Information Grants (EEIG) program is a $40 million competitive merit-based grants program. It was designed to provide funding to industry associations and non-profit organisations to work with small and medium size business enterprises and community organisations to provide information about the smartest ways to use energy, empowering them to make informed decisions about energy efficiency. The program was introduced as part of the Government’s Clean Energy Future Plan.1

2.2 The Department of Climate Change and Energy Efficiency (DCCEE) was responsible for the administration of the EEIG program during the audit period.2 As part of Machinery of Government changes announced in March 2013, the DCCEE was abolished and its energy efficiency functions transferred to the Department of Resources, Energy and Tourism (RET). RET now has responsibility for the continuing administration of the program.

1 Australian National Audit Office (ANAO) Audit Report No.17 2012-13, p. 22. 2 ANAO Audit Report No.17 2012-13, p. 23.

4 REPORT 439: REVIEW OF AUDITOR-GENERAL’S REPORTS NOS. 11 TO 31 (2012-13)

2.3 The program’s first funding round was opened to applications on 13 February 2012 and closed on 16 March 2012.3

2.4 Following eligibility assessment by the DCCEE, an independent program advisory committee (PAC) was responsible for assessing each of the 188 eligible applications against three merit criteria, and allocating a merit ranking to each. The department recommended to the Minister for Climate Change and Energy Efficiency the 28 applications that were put forward by the PAC as being the most meritorious, each of which was approved for funding by the Minister in May 2012. The total value of the projects approved in the first round was approximately $20 million.4

2.5 A second EEIG funding round was opened to applications on 30 October 2012, with a closing date of 20 December 2013.5 At the time of writing, the successful applicants had not been publicly announced. According to the department’s website, applicants were due to be notified of the second round outcome in May 2013.6

Audit objective and scope

2.6 The objective of the Australian National Audit Office (ANAO)’s audit was to assess the effectiveness of the design and implementation of the EEIG program. The focus of the audit was the preparation for, and conduct of, the program’s first funding round.

2.7 The audit examined the program against relevant policy and legislative requirements for the expenditure of public money and the grants administration framework (including the Commonwealth Grant Guidelines).7

Audit conclusion

2.8 The audit report concluded that the DCCEE ‘was well resourced to design and implement the EEIG, and the design of the program was effective’, but there were ‘significant shortcomings in the conduct of the assessment process for applications’.8

3 ANAO Audit Report No.17 2012-13, p. 23. 4 ANAO Audit Report No.17 2012-13, p. 23. 5 ANAO Audit Report No.17 2012-13, pp. 23-24. 6 Department of Resources, Energy and Tourism, ‘Funding: Round Two’:

viewed 21 June 2013. 7 ANAO Audit Report No.17 2012-13, p. 12. 8 ANAO Audit Report No.17 2012-13, pp. 12-13.

DESIGN AND IMPLEMENTATION OF THE ENERGY EFFICIENCY INFORMATION GRANTS PROGRAM 5

2.9 The ANAO found that program governance arrangements and a range of internal documentation to support the administration of the program were developed, and a robust set of program guidelines were published. The department also planned a sound approach to undertaking application eligibility checks, assessing the merit of eligible applications and ranking them for the Minister’s consideration.9

2.10 However, the audit found that the merit assessment process ‘departed in important respects from that outlined in the program guidelines, and inadequate records were made and retained to demonstrate that each application was assessed in accordance with the published eligibility and merit criteria’.10 The main shortcomings outlined in the report were that:

 most eligible applications were allocated to one of four merit

categories, with each of the 28 applications allocated to the highest merit category (termed ‘outstanding’) then placed into one of six ranking bands (in case the Minister did not wish to approve all 28 ‘outstanding’ applications). Neither process was foreshadowed by the program guidelines. In addition, the allocation of the 28 ‘outstanding’ applications to a ranking band used a process that did not relate to the score each application had achieved in terms of the published merit criteria;

 DCCEE destroyed records made by each PAC member of the

assessment of each eligible application against the three published merit criteria, notwithstanding that the contractual arrangements specified that these were official records and that DCCEE had made no arrangements to otherwise record the scoring by PAC members. The minutes of the PAC meetings were also too brief to provide any insight into the merit assessment and scoring of each eligible application; and

 fortunately for DCCEE, one PAC member made and retained

his own electronic record of some of the scoring results but there is no record, official or otherwise, as to how each of the 28 recommended (and approved) applications had been assessed and scored against the three published merit criteria.11

2.11 The DCCEE advised the ANAO that it had confidence in the PAC, had followed probity advice, and had ‘no reason to believe that the successful projects were not the most competitive against the merit criteria’. However, the ANAO concluded that it was ‘not possible to be satisfied

9 ANAO Audit Report No.17 2012-13, pp. 12-13. 10 ANAO Audit Report No.17 2012-13, p. 13. 11 ANAO Audit Report No.17 2012-13, pp. 13-14.

6 REPORT 439: REVIEW OF AUDITOR-GENERAL’S REPORTS NOS. 11 TO 31 (2012-13)

that the most meritorious eligible applications … were recommended to the Minister for approval’.12

Audit recommendations

2.12 Noting that a second EEIG funding round was underway, the audit report made four recommendations, as shown in Table 2.1.

Table 2.1 ANAO recommendations, Audit Report No.17 2012-13

1. In circumstances where an advisory panel or committee is used to assess grant applicants and to provide funding recommendations, ANAO recommends that the Department of Climate Change and Energy Efficiency:

(a) develop program governance arrangements that inform panel/committee members that they are formally operating under the Commonwealth’s financial framework; and

(b) implement secretarial support arrangements that require key assessment decisions and their basis to be minuted.

DCCEE response: Agreed

2. To be able to demonstrate that grant program applicants were treated equitably and in accordance with the published program guidelines, ANAO recommends that the Department of Climate Change and Energy Efficiency maintain adequate records of its assessment of applications in terms of their eligibility and compliance with other mandatory (or gateway) criteria.

DCCEE response: Agreed

3. The ANAO recommends that the Department of Climate Change and Energy Efficiency, in planning for the implementation of competitive, applications‐based grant programs, develop strategies to effectively manage the risk of funding rounds being over‐subscribed including, as appropriate, identifying in the program guidelines any adjustments that may be made to the eligibility and/or merit assessment approach where this situation arises.

DCCEE response: Agreed

4. The ANAO recommends that, consistent with the transparency and public accountability principles of grants administration outlined in the Commonwealth Grant Guidelines, the Department of Climate Change and Energy Efficiency ensures that the assessment and merit ranking of applications and any significant changes as they proceed through the assessment process are appropriately documented.

DCCEE response: Agreed

The Committee’s review

2.13 Representatives of the following organisations gave evidence at the Committee’s public hearing on 19 June 2013:

 Australian National Audit Office

 Department of Resources, Energy and Tourism.

12 ANAO Audit Report No.17 2012-13, p. 14.

DESIGN AND IMPLEMENTATION OF THE ENERGY EFFICIENCY INFORMATION GRANTS PROGRAM 7

2.14 The Committee’s evidence covered the following issues:

 Resourcing of the program

 Record-keeping deficiencies

 Improvements in second funding round.

Resourcing of the program

2.15 The EEIG program guidelines stated that the PAC would assess the eligible applications against the three weighted merit criteria—‘project effectiveness’, ‘project design and management’ and ‘value for money’— allocating a merit ranking to each and advising the Minister, through the department, on projects suitable for funding.13

2.16 The DCCEE’s planning was based on receiving an estimated 30 to 40 applications for funding. In fact, the department received 207 first round applications, of which 188 were assessed to be eligible. The ANAO found that the ‘impact of the high number of applications received … was particularly evident at the merit assessment stage’ conducted by the PAC.14

2.17 The ANAO highlighted that the DCCEE had not developed strategies for how it would respond to receiving significantly more applications than it anticipated.15 It observed that, given the higher than expected number of applications, the time allotted for the assessment of the EEIG applications was ‘insufficient’ in the context of the resources available. It suggested that possible management strategies that could have been employed included ‘increasing the resources allocated to the merit assessment task and/or extending the planned timeframe’. The ANAO recommended that, in planning future grants programs, the DCCEE develop strategies to effectively manage the risk of funding rounds being oversubscribed.16

2.18 In its response to the ANAO’s recommendation, the DCCEE advised that it would ensure ‘sufficient resources are made available for secretariat support to the PAC’.17

2.19 At the Committee’s public hearing, RET confirmed that, of the $40 million total funding for the program, approximately $6 million (or 15 per cent)

13 ANAO Audit Report No.17 2012-13, p. 59. 14 ANAO Audit Report No.17 2012-13, pp. 46-47, 54. 15 ANAO Audit Report No.17 2012-13, p. 14. 16 ANAO Audit Report No.17 2012-13, pp. 73-74. 17 ANAO Audit Report No.17 2012-13, p. 41.

8 REPORT 439: REVIEW OF AUDITOR-GENERAL’S REPORTS NOS. 11 TO 31 (2012-13)

had been allocated to administration. When the Committee questioned why the administration costs were this high, RET explained that:

It is a program that runs for three or four years. There are some start-up costs in terms of assessing that program I think there were some comments made by the auditors in terms of benchmarking that. But it is not just a one-off. We have got to start this and we are actually managing this through. So we will manage that series of projects over the life of the projects.18

2.20 Asked about the planned outcomes of the department’s use of the part of this administration funding for ‘ongoing monitoring’, the department explained that there were two elements to this work:

The first is the contract monitoring [by] departmental staff, and most

of the contracts have multiple milestones over several years. We also

have an independent evaluation which looks at the influence of the

program—if you like, the efficacy of the information generated—and

whether that has influenced behaviour in the recipient businesses.19

2.21 Noting that the funding allocated to the department for administration seemed relatively high, the Committee asked how the deficiencies reported by the Auditor-General came about. The department identified the ‘resourcing of the secretariat function of the merit assessment and the support of the merit assessment committee’ as being one ‘specific element’ contributing to the shortcomings. It noted that the department’s experience of allocating more time and resources to those processes in the second funding round had ‘certainly led to a better product in terms of records taken’.20

2.22 The department also confirmed for the Committee that there was no involvement from the Minister’s office in the application assessment process, aside from the initial approval of program guidelines and the final consideration of the department’s recommendations for funding.21

18 Mr Greg Divall, First Assistant Secretary, RET, Committee Hansard, Canberra, 19 June 2013, p. 2. 19 Professor Gary Richards, Assistant Secretary, RET, Committee Hansard, Canberra, 19 June 2013, p. 2. 20 Prof. Richards, Committee Hansard, Canberra, 19 June 2013, p. 2. 21 Mr Divall, Committee Hansard, Canberra, 19 June 2013, p. 2.

DESIGN AND IMPLEMENTATION OF THE ENERGY EFFICIENCY INFORMATION GRANTS PROGRAM 9

Record-keeping deficiencies

2.23 The audit report documented a number of deficiencies in the records kept by the DCCEE of decisions which informed its recommendations to the Minister. These deficiencies included that:

 Minutes of PAC meetings contained insufficient detail relating to

assessments of applications and the presence or absence of members with conflict of interest declarations;22

 Decisions made by departmental officials who conducted initial

eligibility assessments were not well-documented;23

 The scoring templates used by PAC members to assess each eligible

application were destroyed by the department because they were seen as ‘rough working drafts’;24 and

 The department did not seek to capture the final agreed PAC scores for

each eligible application against the merit criteria, or the resulting aggregate scores for the 28 applications that were categorised as ‘outstanding’.25

2.24 The ANAO summarised its findings in relation to record-keeping as follows:

Some important records made by the PAC members of their initial assessment and scoring of each application were destroyed by DCCEE, and the department did not otherwise document the results of the merit assessment process. Consequently, it is not possible to be satisfied that the published merit criteria were applied in a consistent and robust manner, or for DCCEE to demonstrate that the most meritorious applications were identified and recommended to the Minister for approval.26

2.25 The audit report did not include details of how individual scoring templates came to be destroyed by the department. However, at a Senate Estimates hearing in February 2013, the ANAO stated that the destruction was carried out by ‘the more junior staff’. It explained that when the PAC members returned their copies of the applications to the department, the attached scoring sheets had been destroyed along with the applications.

22 ANAO Audit Report No.17 2012-13, pp. 39-40. 23 ANAO Audit Report No.17 2012-13, p. 50. 24 ANAO Audit Report No.17 2012-13, p. 62. 25 ANAO Audit Report No.17 2012-13, p. 63. 26 ANAO Audit Report No.17 2012-13, p. 74.

10 REPORT 439: REVIEW OF AUDITOR-GENERAL’S REPORTS NOS. 11 TO 31 (2012-13)

The ANAO advised that, by ‘asking a lot of questions to the relevant people involved’, they were satisfied that this destruction was a result of oversight rather than intent. However, the ANAO also advised that this ‘unauthorised destruction of a record’ was a breach of the Archives Act.27

2.26 In the Committee’s public hearing, RET agreed unreservedly with the ANAO’s recommendations and stated that record-keeping:

… has been recognised very clearly by the department as an area that needs to be improved, and it has been improved in the subsequent round.28

2.27 However, RET advised that it considered the destruction of the initial assessment documents of panel members to be appropriate. The department outlined that the decision had been made by a senior executive officer, based on advice from the probity adviser and a working understanding of what would constitute ‘normal administrative practice’ under the Archives Act 1983. The department added:

The advice that we had at the time from the probity adviser and from our subsequent reading of what is known as normal administrative practice under the Archives Act is that they were working notes. They were working notes of committee members acting as individuals not of the committee as a whole. In that sense, we still do not think that they were records under the definition in the Archives Act.29

2.28 The ANAO did not agree with this assessment, stating that the panel members’ contracts described the scoring template as an official record to be returned to the department. The ANAO considered the destruction to be ‘quite a significant matter’ due to the inadequacy of other documentation of the decisions underpinning the final recommendations. The ANAO explained that:

When it comes to grant assessment processes it should be not simply about recording what score is given to each application by the panel members but about the reasoning behind that score so that there can be some accountability and transparency around the basis on which panels provide recommendations to departments and departments provide recommendations to ministers so that we, auditors, parliament and other stakeholders can be confident

27 Mr Boyd, Senate Finance and Public Administration Legislation Committee Hansard, Canberra, 11 February 2013, pp. 165-167. 28 Mr Divall, Committee Hansard, Canberra, 19 June 2013, p. 1. 29 Dr Subho Banerjee, Deputy Secretary, RET, Committee Hansard, Canberra, 19 June 2013, p. 1.

DESIGN AND IMPLEMENTATION OF THE ENERGY EFFICIENCY INFORMATION GRANTS PROGRAM 11

that the most meritorious applications were identified and assessed on a basis consistent with the program guidelines.30

2.29 The ANAO added that it would expect to see this matter addressed in any future audit of the program:

We would certainly suggest that if we were auditing the second round and the department did not require something similar to these templates to be completed in some fashion so that there is a record of what each application was scored against each criteria and the reasoning for that, we would be equally critical of the second round as we were of the first round.31

2.30 Responding to these comments, RET agreed that record-keeping should fully justify the decisions made by advisory committees and that this had not adequately occurred in the first round of the program. The department affirmed that it had ‘absolutely’ changed its practices following the audit to improve its records on the deliberations of the committee.32

Improvements to second funding round

2.31 During the public hearing, RET updated the Committee on the program’s second funding round. It advised that 175 applications had been received, and 18 applications were successful. 33 In response to a question on notice, the department noted that seven of the successful second round applicants had also applied under the program’s first round.34

2.32 In a tabled opening statement, RET informed the Committee that ‘taking the lessons from the first round into the administration of the second round of the program has assisted the department to improve the program’s administration’. The department advised that it had been able to implement all of the responses to the audit findings that it had outlined, with the key actions being:

 More time for the assessment process which allowed the

Department to deal with the over-subscription, while following processes as outlined in the program guidelines.

30 Mr Boyd, Committee Hansard, Canberra, 19 June 2013, p. 5. 31 Mr Boyd, Committee Hansard, Canberra, 19 June 2013, p. 5. 32 Dr Banerjee, Committee Hansard, Canberra, 19 June 2013, p. 5. 33 Mr Divall, Committee Hansard, Canberra, 19 June 2013, p. 2. The department clarified that while

letters of offer had been sent out to successful second round applicants, it was in the ‘early stages post the assessment’ and payments had not yet been made (Prof. Richards, Committee Hansard, Canberra, 19 June 2013, p. 2). 34 RET, Submission 4.

12 REPORT 439: REVIEW OF AUDITOR-GENERAL’S REPORTS NOS. 11 TO 31 (2012-13)

 Both better procedural guidance on the secretariat functions

and further clarity in the probity briefing on the role and responsibility of the Program Advisory Committee members.

 Extra staff and clearer procedural guidance on the taking of

minutes of the deliberations of the Program Advisory Committee ensured both enhanced transparency and ability to provide feedback to applicants.35

2.33 At the public hearing, RET expanded on its implementation of the ANAO recommendations with the following comments:

 The standard operating procedures and the probity briefing to

clarify the relationship with the program advisory committee and to align with the Commonwealth financial frameworks have been put in place, so there is now clarity in terms of the role of the program advisory committee.

 We have made sure that the merit assessment process includes

assessment and commentary in relation to each evaluation criterion and ensured that flows all the way through the documentation, so the documentation is very thorough.

 A probity adviser now sits on all the governance committees, so

the program advisory committee, and is there full-time and has been there full-time in the second round.

 … we have updated our standard operating procedures to

ensure accurate and adequate record-keeping of completeness and eligibility checks.

 … we have ensured that the guidelines provide clarity in

relation to the processes around eligibility and merit assessment.

 We have implemented strategies to ensure that the volume of

applications coming in could be catered for in the process we have put in place.36

2.34 Reflecting on the broader lessons of the audit in relation to grants program processes, the Auditor-General indicated that the ANAO’s audit coverage was driven by the desire to ensure that all grant applicants are treated equitably. He noted that ‘the pleasing thing about this audit is that the department has not argued the toss at all on this matter and the matters of substance here’.37

2.35 The Auditor-General added that the department had administered ‘aspects of this program very well’, but that there were some problems particularly in the assessment area. He voiced his concern that the public

35 RET, Submission 3. 36 Mr Divall, Committee Hansard, Canberra, 19 June 2013, pp. 3-4. 37 Mr McPhee, Committee Hansard, Canberra, 19 June 2013, p. 6.

DESIGN AND IMPLEMENTATION OF THE ENERGY EFFICIENCY INFORMATION GRANTS PROGRAM 13

sector ‘do not learn the lessons of the prior experience’ as quickly as desirable. However, he expressed optimism that RET had ‘gained from this’ and flagged that the ANAO would ‘continue to emphasise the importance of sound practices in this area’.38

Committee Comment

2.36 The Committee supports the ANAO’s findings and recommendations, and agrees with its conclusion that, while aspects of the program were managed well, there were significant administrative shortcomings in the assessment phase.

2.37 It was concerning that the ANAO could not be satisfied that the most meritorious applications had been recommended for approval. However, the Committee is satisfied that the program’s shortcomings resulted from poor administrative practices, rather than any malicious intent.

2.38 From the evidence received, it was clear that RET was well-resourced for the program’s administration. However, the department did not internally allocate enough time or resources to the assessment of applications in the first funding round, significantly contributing to the program’s shortcomings.

2.39 There remains some level of disagreement between the department and the ANAO in relation to whether the individual scoring sheets completed by advisory committee members should be considered official records and formally retained under the Archives Act. However, the Committee observed agreement from both sides on the broader points that: it is essential for program records to document the basis on which recommendations for funding are made; and this had not occurred in the program’s first funding round.

2.40 The results of this audit contain valuable lessons for officials involved in grants programs about the importance of conducting fair and transparent decision-making processes that are supported by appropriate record-keeping. The findings also highlight the importance of sound risk management processes for early identification of strategies for dealing with unexpected events—in this case, a much larger than expected number of applications.

38 Mr McPhee, Committee Hansard, Canberra, 19 June 2013, p. 6.

14 REPORT 439: REVIEW OF AUDITOR-GENERAL’S REPORTS NOS. 11 TO 31 (2012-13)

2.41 Encouragingly, the key lessons from the audit appear to have been learned and applied by RET in the second funding round, with more time and resources having been allocated to the assessment of applications. The department has also pointed to a range of measures that it has put in place to improve the integrity and transparency of the program, particularly in relation to documentation.

2.42 The Committee accepts the department’s advice that it has improved its practices, and expects that any future audits of this program, or similar RET grants programs, will contain more positive findings. Any reoccurrence of issues similar to those identified in this report should be viewed as very concerning by the Auditor-General, the JCPAA, the broader Parliament and the public.

3

ANAO Report No.28 (2012-13)

The Australian Government Performance Measurement and Reporting Framework - Pilot Project to Audit Key Performance Indicators

Introduction

3.1 The Federal Government’s Outcomes and Programs Framework, introduced in 2009-10, requires entities to identify and report against the programs that contribute to government outcomes over the Budget and forward years. This requires clearly specified outcomes, program objectives and appropriate Key Performance Indicators (KPIs).1

Previous audit reports and JCPAA reviews

3.2 The topic of performance measurement and reporting has been the subject of a range of Australian National Audit Office (ANAO) audits over a number of years.2 Most recently, an audit of agency development and implementation of KPIs, released in September 2011, found that:

1 Australian National Audit Office (ANAO) Audit Report No.28 2012-13, p. 15. 2 Refer to ANAO Audit Report No.28 2012-13, pp. 34-35 for a summary of ANAO activities focused on performance measurement and reporting since 2001-02.

16 REPORT 439: REVIEW OF AUDITOR-GENERAL’S REPORTS NOS. 11 TO 31 (2012-13)

… many entities continue to find it challenging to develop and implement KPIs; in particular, effectiveness KPIs that provide quantitative and measureable information, allowing for an informed and comprehensive assessment and reporting of achievements against stated objectives.3

3.3 The audit recommended improvements to entity business planning processes around program objectives and KPIs; assessments by entities of their current use of costing information and the allocation of costs to programs; and for the Department of Finance and Deregulation (Finance) to review the development and implementation of effectiveness KPIs and to improve elements of its guidance to other entities.4

3.4 A JCPAA review of the 2011 audit, tabled in May 2012, supported the ANAO’s findings and recommended:

 that Finance include at least one recognised KPI methodology in its

guidance to other entities;

 that the methodology used by entities in the preparation of KPIs be

available for review and potentially included in annual reports; and

 that Finance report back to the Committee within six months on

progress made on improving guidance to agencies, and how the ANAO’s audit methodology was envisaged to fit within and support the overall KPI framework and support ongoing policy enhancements.5

3.5 In its February 2013 response to the Committee’s recommendations, Finance advised that it would ‘revise its policy and guidance materials for the development, monitoring and reporting of program-level KPIs’. The response indicated that the methodology for the construction of program-level KPIs would be clearer in the revised guidance, which would incorporate a practical user guide. Finance further indicated that it ‘would seek to brief the [JCPAA] on this work after the 2013-14 Budget’.

3.6 The full text of the JCPAA’s recommendations and the responses from Finance are included at Appendix C to this report.

3 ANAO Audit Report No.5 2011-12, Development and Implementation of Key Performance Indicators to Support the Outcomes and Programs Framework, p. 17. 4 ANAO Audit Report No.5 2011-12, pp. 28-29. 5 JCPAA Report 430, Review of Auditor-General’s Reports Nos. 47 (2010-11) to 9 (2011-12) and

Reports Nos. 10 to 23 (2011-12), May 2012.

THE AUSTRALIAN GOVERNMENT PERFORMANCE MEASUREMENT AND REPORTING FRAMEWORK

- PILOT PROJECT TO AUDIT KEY PERFORMANCE INDICATORS 17

The pilot project

3.7 The ANAO commenced a KPI audit pilot project in 2012-13 in the context of new powers given to the Auditor-General in 2011 with amendments to the Auditor-General Act 1997.

3.8 The amendments provided the Auditor-General with explicit authority to undertake audits of entity KPIs and reporting against KPIs, and were consistent with a JCPAA recommendation arising from its 2010 review of the Act.6 Specifically, the amendments to the Act enable the Auditor-General to ‘at any time’ conduct an audit of:

 the appropriateness of the performance indicators of a Commonwealth

entity; and

 the reporting by an entity against those indicators.7

Objective and scope

3.9 The ANAO’s pilot project was designed to:

… assess the status of the Australian Government performance measurement and reporting framework as a basis for implementation of a future program of audits of entities’ KPIs, and to develop a suitable audit methodology.8

3.10 The objective of the pilot was to:

 build an understanding of experiences from other jurisdictions

currently performing audits of KPIs as part of their financial statement audit processes, including the development of an approach and methodology;

 initiate and maintain ongoing discussions with Finance and the

Department of Prime Minister and Cabinet (PM&C) in regard to strengthening the administrative framework relating to performance measurement; and

 develop and test an audit methodology and criteria to address

the practical challenges of auditing the appropriateness of KPIs and the completeness and accuracy of reporting against them.9

6 JCPAA Report 419, Inquiry into the Auditor-General Act 1997, December 2010, Recommendation 3. 7 Auditor-General Act 1997, section 18A. 8 ANAO Audit Report No.28 2012-13, p. 17. 9 ANAO Audit Report No.28 2012-13, p. 40.

18 REPORT 439: REVIEW OF AUDITOR-GENERAL’S REPORTS NOS. 11 TO 31 (2012-13)

3.11 The implementation of the pilot included:

 assessing the Australian Government performance

measurement and reporting framework as a basis for implementation of a future program of audits of entities’ KPIs;

 reviewing the approaches taken by other relevant jurisdictions

in the implementation of KPI audit methodologies;

 working with the responsible Australian Government central

agencies; and

 testing a KPI audit approach and methodology within three

entities.10

3.12 Four entities participated collaboratively in the pilot project. Finance contributed in view of its responsibility for administering the Outcomes and Programs framework. The Australian Taxation Office, the Department of Education, Employment and Workplace Relations, and the Department of Health and Ageing, contributed as agencies with experience in applying the principles of the Outcomes and Programs framework. One outcome and program from each of these three entities was selected for assessment.11

3.13 The ANAO also worked with audit offices in other jurisdictions that have work programs with a performance information focus, in particular the Office of the Auditor-General for Western Australia and the Office of the Auditor-General of New Zealand.12

Funding considerations

3.14 The ANAO’s report noted that the pilot project had been funded to date from its existing resource base, and that ‘the future development and implementation of a broader ANAO KPI audit work program will require the ANAO to be appropriately resourced’.13

3.15 The ANAO was not successful in obtaining the budget supplementation it had requested in order to continue its KPI activities in 2013-14, but was given the opportunity to bring back the proposal for consideration in the 2014-15 Budget. Despite not receiving supplementation in 2013-14, the ANAO has informed the Committee that it aims to continue work on developing a potential KPI audit work program within its existing budget.

10 ANAO Audit Report No.28 2012-13, p. 17. 11 ANAO Audit Report No.28 2012-13, p. 40. 12 ANAO Audit Report No.28 2012-13, p. 18. 13 ANAO Audit Report No.28 2012-13, p. 17.

THE AUSTRALIAN GOVERNMENT PERFORMANCE MEASUREMENT AND REPORTING FRAMEWORK

- PILOT PROJECT TO AUDIT KEY PERFORMANCE INDICATORS 19

Findings and conclusions

3.16 The ANAO’s report concluded that ‘entities continue to experience challenges in developing and implementing meaningful KPIs, and that the administrative framework supporting the development and auditing of KPIs remains problematic’.14

The current framework and supporting guidance

3.17 While not making any formal recommendations, the ANAO’s report made a range of observations about the current performance framework and areas for potential improvement. These included:

 The need a framework that better accounts for different entity types:

The development of a framework that accommodates the diversity of public administration, and provides entities with the ability to report appropriate performance information regardless of role, is critical. … it would be desirable for the framework to recognise that the primary function of some entities is the delivery of services whereas other entities’ responsibilities include assessing the impact on the Government’s outcomes by those deliverables.15

 The use of intermediate ’milestone’ objectives for longer term outcomes:

The Outcomes and Programs framework would benefit from further consideration of intermediate objectives where an overall outcome can only be achieved over the longer‐term. 16

 The introduction of ‘efficiency’ performance indicators:

The focus of a more comprehensive model for performance measurement and reporting in the Commonwealth would include consideration of the development and implementation of ‘efficiency’ indicators to complement the ‘effectiveness’ indicator focus within the current model.17

3.18 The ANAO reported that Finance’s guidance to agencies in relation to the development of KPIs had not been significantly updated since the introduction of the Outcomes and Programs framework, and there was no

14 ANAO Audit Report No.28 2012-13, p. 19. 15 ANAO Audit Report No.28 2012-13, p. 50. 16 ANAO Audit Report No.28 2012-13, pp. 44-45. 17 ANAO Audit Report No.28 2012-13, p. 51.

20 REPORT 439: REVIEW OF AUDITOR-GENERAL’S REPORTS NOS. 11 TO 31 (2012-13)

single comprehensive source of guidance and policy.18 The ANAO also found the current framework and guidance to be unsuitable as a basis for auditing:

It is also clear from the Pilot that the current framework and accompanying guidance does not provide an effective framework against which entities’ KPIs can be reliably evaluated through an assurance audit process, as it does not specify clear standards or criteria that KPIs should satisfy. That said, it does need to be recognised that the current framework was not designed with this specific purpose in mind.19

Findings of the pilot audit

3.19 In conducting its pilot audit to assess the appropriateness of entity KPIs and the completeness and accuracy of their reporting, the ANAO developed the criteria shown in Table 3.1 and Table 3.2 below.

Table 3.1 ANAO criteria for the evaluation of the appropriateness of KPIs

Criteria Characteristics Explanation

Individual Assessment Relevant Focused The KPI should assist significantly in informing whether the program objective is being

achieved.

Understandable The KPI should be stated in plain English and signal the impacts of program activities to inform users.

Reliable Measureable The KPI should be capable of being measured to demonstrate the performance of the program.

Free from bias The KPI should allow for clear interpretation of results.

Overall Assessment Complete Balanced The set of KPIs should provide an overall picture of the impact of a program on the target group/s.

Collective The set of KPIs should demonstrate the extent of achievement against the program objective.

Table 3.2 ANAO criteria for the evaluation of the completeness and accuracy of KPI reporting

Criteria Explanation

Data Completeness and Accuracy KPIs should be reported on the basis of data and information that reflects accurately and completely all events that should have been recorded.

Disclosures Completeness and Accuracy

All disclosures relating to KPIs that should have been included in the annual report have been included (in accordance with PM&C’s Annual Reporting Requirements), and all KPIs and information relating to them in the annual reports is disclosed fairly and, where applicable, at the appropriate amounts.

Source ANAO Audit Report No.28 2012-13, pp.63- 64.

18 ANAO Audit Report No.28 2012-13, p. 53. 19 ANAO Audit Report No.28 2012-13, pp. 59-60.

THE AUSTRALIAN GOVERNMENT PERFORMANCE MEASUREMENT AND REPORTING FRAMEWORK

- PILOT PROJECT TO AUDIT KEY PERFORMANCE INDICATORS 21

3.20 The ANAO report included the following findings based on its pilot audit of a selection of KPIs from the three participating agencies:

 Five of the 31 KPIs examined ‘clearly did not meet the Finance

definition and were not assessed further against the appropriateness criteria’. Those KPIs not meeting the definition were either descriptions of activities or output indicators.20

 Only one KPI met all of the characteristics outlined in the

appropriateness criteria. Of the remaining 25 KPIs assessed, 22 met at least one of the characteristics and three partially met all but one.21

 In analysing the entities’ processes for collating KPI data, areas for

improvement included: the use of manual data entry; reliance on unverified external data sources; a lack of formally documented processes; limited quality assurance practices; and infrequency of KPI data measurement.22

 The entities’ annual reports generally met PM&C’s Annual Reporting

Requirements, although one of the entities did not identify whether its KPIs had been met and the other two entities did not provide explanations where KPIs were reported as not met.23

3.21 The report indicated that the entities involved in the pilot had been ‘receptive to the ANAO’s feedback and planned to revisit their current approach where required’.24

Future reforms

3.22 The report’s final conclusion was that:

… it is time for greater attention, investment and resourcing to be given to the quality and integrity of KPIs used by public sector entities to inform decisions about the performance of government programs. This requires a stronger and sustained focus by entities to enhance KPIs, and support provided by Finance through improved guidance. Entity leadership will be critical to success here.25

20 ANAO Audit Report No.28 2012-13, p. 70. 21 ANAO Audit Report No.28 2012-13, p. 71. 22 ANAO Audit Report No.28 2012-13, p. 77. 23 ANAO Audit Report No.28 2012-13, p. 78. 24 ANAO Audit Report No.28 2012-13, p. 79. 25 ANAO Audit Report No.28 2012-13, p. 21.

22 REPORT 439: REVIEW OF AUDITOR-GENERAL’S REPORTS NOS. 11 TO 31 (2012-13)

3.23 The report identified a key role for both the Government and the Parliament to encourage improvement in this area:

Encouragement to achieving better performance measurement can also be given by the Government and the Parliament through reviews, inquiries and the questions asked about the changes being brought about by specific programs having regard to the program’s objectives.26

3.24 The ANAO noted that the Commonwealth Financial Accountability Review (CFAR) currently being undertaken by Finance would provide an opportunity ‘to consider how to position the current performance measurement and reporting framework to respond to contemporary issues in public administration’.27

3.25 Since the ANAO’s report was released, the Government introduced into Parliament the first piece of legislation arising from CFAR. The Public Governance, Performance and Accountability Bill 2013 (PGPA Bill) seeks to replace the existing Financial Management and Accountability Act 1997 and Commonwealth Authorities and Companies Act 1997 with a single Act to govern the management of public resources and the performance of Commonwealth bodies. The Bill would be a ‘fundamental part’ of broader reforms to be introduced through CFAR.28

3.26 In relation to performance, the PGPA Bill aims to introduce a ‘framework for measuring and assessing performance, including requiring effective monitoring and evaluation’.29 The Bill, if passed, would embed performance monitoring and reporting requirements in legislation for the first time. The Bill includes clauses that require entities to measure and assess their performance in achieving their purposes, keep records of their performance, and produce annual performance statements which may be examined by the Auditor-General.30 The Bill’s Explanatory Memorandum states that the requirements for measuring performance would be outlined in rules associated with the Bill that would focus on ‘… enhancing the quality and integration of performance information required by government and the Parliament to assess actual against planned results.’31

26 ANAO Audit Report No.28 2012-13, p. 21. 27 ANAO Audit Report No.28 2012-13, p. 58. 28 Explanatory Memorandum, PGPA Bill 2013, p. 1. Refer to JCPAA Report 438, Advisory Report on the Public Governance, Performance and Accountability Bill 2013, June 2013. 29 Explanatory Memorandum, PGPA Bill 2013, p. 7. 30 Clauses 37 to 40, PGPA Bill 2013. 31 Explanatory Memorandum, PGPA Bill 2013, p. 33.

THE AUSTRALIAN GOVERNMENT PERFORMANCE MEASUREMENT AND REPORTING FRAMEWORK

- PILOT PROJECT TO AUDIT KEY PERFORMANCE INDICATORS 23

The Committee’s review

3.27 Representatives of the following organisations gave evidence at the Committee’s public hearing on 19 June 2013:

 Australian National Audit Office

 Department of Finance and Deregulation.

3.28 The Committee’s evidence covered the following issues:

 The need for greater focus on performance assessment

 A less homogenous framework

 Future directions for reform.

The need for greater focus on performance assessment

3.29 The need for a sustained and greater level of focus on performance monitoring, reporting and evaluation was a key message given to the Committee by the Auditor-General and by representatives of Finance at the public hearing.

3.30 Calling for more attention to be given to KPIs, the Auditor-General said that there was a need to set a ‘much higher level of expectation’ around agencies using KPIs to measure the effectiveness of their programs. This was a message that would need to be put out ‘collectively’ by the ANAO, Finance and the broader Government.32

3.31 The importance of government being able to assess the effectiveness of their programs was concisely summarised for the Committee by the Auditor-General:

At the end of the day, government does need to understand the impact or the effectiveness of programs because that can help government take decisions about the targeting of programs and whether they can be more effectively targeted in order to allow resources to be allocated to other areas if the lens can be tightened on some programs. So it is terribly important information, but, at the moment, there is not enough emphasis given to it within government, and we need to change that in some way.33

32 Mr Ian McPhee, Auditor-General, Committee Hansard, Canberra, 19 June 2013, p. 2. 33 Mr McPhee, Committee Hansard, Canberra, 19 June 2013, p. 2.

24 REPORT 439: REVIEW OF AUDITOR-GENERAL’S REPORTS NOS. 11 TO 31 (2012-13)

3.32 The Auditor-General stated his concern that, while agencies were ‘very clear’ about the expected benefits of their programs in submissions to Cabinet, when it came to measuring whether those objectives had been achieved ‘there is a whole range of externalities that departments then mention as to reasons why it is hard and why they cannot do it et cetera’.34

3.33 The Auditor-General particularly emphasised that KPIs needed to be part of the original design of a program, with expectations that they ‘will be carried through and delivered on’.35 He expressed optimism that the Public Service was capable of developing more effective, outcomes-focused KPIs, but that ‘we just need this particular issue to be given a higher priority in program management than it has been to date’.36 This would have implications for the allocation of resources for government programs:

Across the years, we have had different approaches to try to make sure agencies are doing this work, but the most important thing is that agencies, as part of their design and submissions to government, need to make sure they are appropriately resourced to do the full program implementation, which includes assessing the impact of programs.37

3.34 Finance agreed with the Auditor-General that the ‘capacity exists within government’ to develop rigorous, outcome-focused KPIs, adding that it was ‘very important to pay sustained attention over a period of time’.38 It suggested that a broader, evaluation-based approach might be needed, rather than an approach that focused only on KPIs:

Ultimately, we have to keep focus on assessing performance and achievement rather than focusing on the tools themselves. The tools are just a means to the end. There really is nothing better than sustained attention, both within the Public Service and from outside the Public Service to the inside, to really drive this home and ensure that adequate effort is paid in this particular area.39

3.35 Finance also cautioned that reform would require more than just changes to the architecture of the framework. It argued that ‘consistent and concerted’ interest was needed if good quality information and time series

34 Mr McPhee, Committee Hansard, Canberra, 19 June 2013, p. 2. 35 Mr McPhee, Committee Hansard, Canberra, 19 June 2013, p. 2. 36 Mr McPhee, Committee Hansard, Canberra, 19 June 2013, p. 3. 37 Mr McPhee, Committee Hansard, Canberra, 19 June 2013, p. 3. 38 Dr Stein Helgeby, Acting Secretary, Finance, Committee Hansard, Canberra, 19 June 2013, p. 3. 39 Dr Helgeby, Committee Hansard, Canberra, 19 June 2013, p. 3.

THE AUSTRALIAN GOVERNMENT PERFORMANCE MEASUREMENT AND REPORTING FRAMEWORK

- PILOT PROJECT TO AUDIT KEY PERFORMANCE INDICATORS 25

of performance data were to be achieved, and that this was what is needed to obtain a ‘solid basis on which to assess how a program is really going’:

The system needs to value performance information. You can change the architecture of the framework and you can put in place requirements and so on, but, until there is a concerted and consistent interest from government, parliament, the public and people who are responsible for administering programs, collecting performance information, using it and making it part of the public debate about the utility of public programs and so on, it is going to be hard to get progress on this front … To be honest, the Public Service has struggled on this front and I think the political system has struggled on this front.40

A more flexible framework

3.36 As noted above, the ANAO report discussed a need for the performance assessment framework to accommodate the diversity of public sector organisations, recognising the wide variety of functions carried out across different agencies.41

3.37 At the public hearing, the Committee asked Finance whether the performance assessment framework was capable of measuring agency performance in activities that went beyond the delivery of programs, such as the quality of the advice provided to government.

3.38 Finance advised that ‘the intention is and always has been to cover the totality of government activity, the bulk of which is programs’. It added that, in addition to programs, other expectations of the Public Service— such as stewardship and advice—‘need to be assessable or assessed in some kind of way’.42 Noting the diverse range of functions across the government, Finance described the difficulties associated with having a single framework:

What we have been learning and what we have experienced is that, if we take all of the different things the government does and all of the different expectations that people have of government entities such as Finance compared to, say, DHS, and put them all

40 Mr Lembit Suur, First Assistant Secretary, Finance, Committee Hansard, Canberra, 19 June 2013, pp. 3-4. 41 ANAO Audit Report No.28 2012-13, p. 50. 42 Dr Helgeby, Committee Hansard, Canberra, 19 June 2013, p. 4.

26 REPORT 439: REVIEW OF AUDITOR-GENERAL’S REPORTS NOS. 11 TO 31 (2012-13)

into the same conceptual framework or the same bucket, we will probably miss something there.43

3.39 Expanding on this topic, Finance reiterated its view that it was important to have ‘a range of tools at your disposal: not just KPIs’. It identified evaluation and review activity as other important tools, noting that the Treasury has recently undertaken an independent review of its forecasting activities, which had been publicly released.44

3.40 Finance also drew the Committee’s attention to the recent development of public service ‘capability reviews’, which apply a consistent framework to evaluating whole-of-department strengths and weaknesses. It noted that capability reviews, which were being conducted for every department and made public after 12 months, had become ‘quite a valued and a valuable part of the evaluative tool kit’, adding that:

They would have value as a benchmark or a reference point for looking at what the capability of that organisation was at a point in time, but more importantly the value would be enhanced if government was to come back in a couple of years’ time and say, ‘Let’s redo this and have a look at what has improved and what has not improved’.45

Future directions for reform

3.41 The Committee sought more information at the public hearing on how the Public Governance, Performance and Accountability Bill 2013, if passed, would impact the existing performance monitoring, reporting and evaluation framework.

3.42 Finance described the PGPA Bill as being ‘an important contribution’, but ‘certainly not a silver bullet’. It noted that the Bill, by enshrining requirements in legislation, would make performance monitoring and reporting an obligation ‘for all who work within the public sector’. This would remove the ‘level of interest’ variable that had led to inconsistent efforts in the past:

One way you can think about this area is that for the last 20 or 30 years we have relied on best endeavours and the energy and interest that individuals and organisations have had over that period. When the interest is high, then we get progress. When the

43 Dr Helgeby, Committee Hansard, Canberra, 19 June 2013, p. 4. 44 Dr Helgeby, Committee Hansard, Canberra, 19 June 2013, p. 4. 45 Dr Helgeby, Committee Hansard, Canberra, 19 June 2013, p. 5.

THE AUSTRALIAN GOVERNMENT PERFORMANCE MEASUREMENT AND REPORTING FRAMEWORK

- PILOT PROJECT TO AUDIT KEY PERFORMANCE INDICATORS 27

interest drops off, then the quality drops off. We really need to try to break that cycle … [The PGPA Bill] is certainly not a silver bullet, because it does not deal with the skills or the capability or the systems, but it does put a minimum requirement, an obligation, in place and people then have to do something. That is the real contribution that the PGPA Bill seeks to make.46

3.43 Finance further explained that one of the ‘key underpinnings’ of the CFAR process was to ‘focus more on performance’, and that the PGPA Bill was one way to provide a ‘high-level emphasis’ on this. If passed, the Bill would also have the effect of giving Finance’s work on improving requirements and skills a ‘kick along’, and would provide an opportunity for Finance to look at the work the ANAO had undertaken to try and strengthen elements of the framework.47

3.44 Asked about how the JCPAA could best support Finance in its efforts, the department indicated that ‘sustained attention’ was a fundamental issue:

It is a willingness to come back and look at terrain that you might have traversed before and to ask: ‘What has changed there’—and perhaps not just to do that in a context where it is prompted by another follow-up ANAO report but prompted by the committee itself saying: ‘We raised some issues in the past here and there around this. Let's go back and ask the people what they are doing and what they have done’. I think sustained interest would be particularly useful.48

3.45 Finance also referred to an undertaking by the Minister for Finance and Deregulation to submit the rules developed under the PGPA Act (if passed) to scrutiny by the JCPAA prior to their tabling.49 Finance suggested that engagement by the Committee on the rules for performance assessment and KPIs would be ‘particularly valued’.50 However, it noted that, while improvements to the framework were important, attention would need to shift to the ‘people who work within the frameworks’ in order to make ‘real, genuine and sustained improvements’.51

46 Dr Helgeby, Committee Hansard, Canberra, 19 June 2013, p. 3. 47 Dr Helgeby, Committee Hansard, Canberra, 19 June 2013, p. 6. 48 Dr Helgeby, Committee Hansard, Canberra, 19 June 2013, p. 6. 49 Senator the Hon Penny Wong, Minister for Finance and Deregulation, JCPAA Inquiry into

Public Governance, Performance and Accountability Bill 2013, Submission 16. 50 Dr Helgeby, Committee Hansard, Canberra, 19 June 2013, p. 6. 51 Dr Helgeby, Committee Hansard, Canberra, 19 June 2013, p. 6.

28 REPORT 439: REVIEW OF AUDITOR-GENERAL’S REPORTS NOS. 11 TO 31 (2012-13)

3.46 In regard to its role in the framework, the ANAO advised that it was working with Finance to ensure its KPI audit processes would align with any new developments. However, it noted that ‘without visibility around the content of the rules that will underpin the [PGPA] Bill, it is not clear how some provisions of the Bill … align and interact with the Auditor-General Act 1997’.52

3.47 The ANAO informed the Committee that it would ‘continue to invest resources in developing and refining the approach of the systematic audit of the appropriateness of entities’ KPIs’ in 2013-14, but that after this resources would ‘become an issue’.53 The ANAO also foreshadowed that it would be requesting ‘additional funds for 2014-15 and the forward estimates for an ongoing program of auditing performance indicators’.54

Committee Comment

3.48 The Committee welcomes the ANAO’s report for its timely and constructive analysis that will assist in the development of a more effective KPI framework, supported by a program of ongoing KPI audits.

3.49 The ANAO and Finance have both expressed their desire for a stronger focus on assessing performance in the public sector. The Committee welcomes Finance’s commitment to improving the performance monitoring and assessment framework, as was flagged in the department’s response to JCPAA Report 430. The Committee encourages Finance to continue to work closely with the ANAO in its development of a framework that better takes into account the diversity of Commonwealth agencies and provides meaningful and reliable indications of performance to the Government and the Parliament.

3.50 The Committee recognises that evaluating program and agency effectiveness is about more than just KPIs, and that there is a role for other mechanisms such as independent evaluations and capability reviews. However, KPIs remain an essential tool for informing these broader evaluations, for continuous reporting and for project management discipline.

3.51 The Committee also firmly agrees with Finance and the ANAO that it is not just the framework that is important. There is a need for strong

52 ANAO, Submission 2, p. 2;.Mr McPhee, Committee Hansard, Canberra, 19 June 2013, p. 1. 53 ANAO, Submission 2, p. 2;.Mr McPhee, Committee Hansard, Canberra, 19 June 2013, p. 1. 54 ANAO, Submission 2, p. 2.

THE AUSTRALIAN GOVERNMENT PERFORMANCE MEASUREMENT AND REPORTING FRAMEWORK

- PILOT PROJECT TO AUDIT KEY PERFORMANCE INDICATORS 29

leadership and sustained effort—both within the Government and in the Parliament, including the JCPAA—to ‘raise the level of expectation’ and promote a culture in the public service that values performance assessment. Indeed, high quality performance monitoring, reporting and evaluation should be more than aspirational: it should be demanded.

3.52 Towards this objective, the Committee supports the Auditor-General’s call for KPIs and evaluation to be built into the design of programs from their initial stages, and that appropriate funding is allocated for this purpose. It is also critical that agencies work together to ensure these activities actually take place—for which monitoring and reporting will be essential.

3.53 The PGPA Bill, by elevating performance assessment requirements into legislation, offers a positive step towards achieving cultural change. The Bill, if passed, also provides an opportunity to reform the existing KPI and broader performance framework, taking into account the findings and proposals in the ANAO’s report.

3.54 To fulfil part of its responsibility to provide strong leadership and sustained effort in this area, the Committee commits to ongoing engagement with Finance on the prioritised improvement of the framework, irrespective of the passage of the PGPA rules.

3.55 The Committee notes that the ANAO’s pilot project included the examination of approaches being taken in New Zealand and Western Australia, which both have existing frameworks for the audit of KPIs. The Committee suggests that there would be value in Finance and the ANAO engaging with the Australasian Council of Auditors-General (ACAG) during the redevelopment of the Commonwealth’s performance assessment framework. For example, there could be substantial benefits in terms of driving cultural change across governments and services from any promulgation by ACAG of consolidated ‘best practice’ guidance on KPI-focused audits.

3.56 The Committee continues to support the adoption of ongoing audits of agency KPIs, and notes that the relatively small investment of resources required will be far outweighed by long term benefits. As was noted in the recent JCPAA statement to the Parliament on the 2013-14 budget for the ANAO, the Committee will closely monitor funding to the ANAO for its implementation of a full KPI audit program.55 The Committee’s view is that additional funding should be provided for this important purpose in the

55 JCPAA, Statement by the Joint Committee of Public Accounts and Audit on the 2013-14 draft estimates for the Australian National Audit Office and the Parliamentary Budget Office, 14 May 2013.

30 REPORT 439: REVIEW OF AUDITOR-GENERAL’S REPORTS NOS. 11 TO 31 (2012-13)

2014-15 Budget, contingent on the final outcome of the pilot, the PGPA Bill and other related developments.

Recommendation 1

The Joint Committee of Public Accounts and Audit recommends that the Department of Finance and Deregulation, in consultation with the Australian National Audit Office, prioritise the review and update of the performance measurement and reporting framework. A goal should be to have clear policy and guidance in place for the 2014-15 financial year that can be used by agencies to produce auditable Key Performance Indicators, irrespective of the passage of the Public Governance, Performance and Accountability Bill 2013.

Recommendation 2

The Joint Committee of Public Accounts and Audit recommends that:

 the Government reinforce the requirements for agencies to

incorporate specific performance monitoring, reporting and evaluation activities into the design and costing of their programs;

 agencies be appropriately funded to carry out these activities;

and

 monitoring be used to provide assurance that these activities

are implemented.

Rob Oakeshott MP Chair June 2013

A

Appendix A - Submissions

1. Australian National Audit Office

2. Australian National Audit Office

3. Department of Resources, Energy and Tourism

4. Department of Resources, Energy and Tourism

32

B

Appendix B - Public Hearing

Wednesday, 19 June 2013 - Canberra

Auditor General’s Report No17: Design and Implementation of the Energy Efficiency Information Grants Program

Australian National Audit Office

Mr Ian McPhee, Auditor-General

Mr Brian Boyd, Executive Director, Performance Audit Services Group

Mr Michael Shiel, Senior Director, Performance Audit Services Group

Department of Resources, Energy and Tourism

Dr Subho Banerjee, Deputy Secretary

Mr Greg Divall, First Assistant Secretary

Prof Gary Richards, Assistant Secretary

34

Auditor General’s Report No28: The Australian Government Performance Measurement and Reporting Framework - Pilot Project to Audit Key Performance Indicators

Australian National Audit Office

Mr Ian McPhee, Auditor-General

Mr Steve Chapman, Deputy Auditor-General

Mr Michael White, Executive Director, Performance Audit Services Group

Department of Finance and Deregulation

Dr Stein Helgeby, Acting Secretary

Mr Lembit Suur, First Assistant Secretary, Financial Framework

Mr Gareth Hall, Assistant Secretary, Resource Framework and Reporting Branch

C

Appendix C -

Department of Finance and Deregulation response to Recommendations 1 to 3 of JCPAA Report 430: Review of Auditor-General's Reports Nos. 47 (2010-11) to 9 (2011-12) and Reports Nos. 10 to 23 (2011-12)

36

Received 5 Feb 2013

37 37

38 38

39 39

40 40

41 41

42