Note: Where available, the PDF/Word icon below is provided to view the complete and fully formatted document
Assessment of doctors' performance: Australia.



Download PDFDownload PDF

5th International Medical Workforce Conference, Sydney 2000 Session 2: Assessment of Doctors’ Performance in Practice - Australia _______________________________________________________________________________________________________________________________________

87

ASSESSMENT OF DOCTORS’ PERFORMANCE - AUSTRALIA

Author: Professor Bruce Barraclough Chairman, Australian Council for Safety and Quality in Health Care

INTRODUCTION The system, its design and processes, resources and financing are among the strongest determinants of clinical performance.

We have known for many years that anything up to 90% of problems in any production process are due to the system itself.

This was the message that W Edwards Demming gave the Japanese manufacturing industry in the 1950s which enabled them to produce a better motor car.

Many have found it very difficult to accept that this applies also to health care. Health care is made up of multiple interlinked processes, each of which can be extremely complex. It is reasonable, therefore, to assume that the system in which health care is provided is by far the most important issue to be addressed when trying to improve the outcomes of the care provided.

Donabedian’s definition of health care - “a technical task and interpersonal exchange taking place within amenities of care” - appropriately describes the interaction between doctor and patient, but fails to recognise the complexities of modern health care. This interaction is only one step in a series of complex processes. It is therefore difficult to assess an individual’s performance in isolation from the system in which that performance is given.

Much of the product of performance assessment is only valid for the system in which that performance was given and may not be able to be used either as comparative data or for benchmarking from hospital to hospital, health area to health area or country to country. It may, however, be useful to compare an individual’s performance over time within the same system.

Systems of performance assessment are not well developed.

We do not appear to be collecting appropriate data and that which we are collecting is often incomplete, inaccurate, unreliable and irrelevant.

The better identification, collection, review and use of appropriate data must be a first step in allowing us to effectively assess performance of clinicians. To do this we will need to change the culture of most first world health systems, namely, from one of judgement and blame to one of learning for quality improvement.

5th International Medical Workforce Conference, Sydney 2000 Session 2: Assessment of Doctors’ Performance in Practice - Australia _______________________________________________________________________________________________________________________________________

88

LICENSING, REGISTRATION, ACCREDITATION In Australia, licensing, registration and accreditation are complex issues, made more complex by the design of the health system, how it is funded, and the political structure of the country.

Health care at a general practice level and at a specialist consultation level is largely funded by the Commonwealth Government through Medicare rebates to patients which address some or all of doctors’ charges. This system of Medicare rebates also applies to diagnostic procedures performed in private hospitals or outside hospitals.

The public hospital system, accounting for some 70% of available beds and services, is managed at a State and Territory level, funded by the States with some of these funds transferred from the Commonwealth.

Access to the public hospital system is available to any citizen without means testing. These hospitals are also able to be accessed by patients who have bought private health insurance when, if they elect to be private patients in a public hospital, their insurance will cover their hospital costs and Medicare will cover a proportion, if not all of their doctor costs.

The other 30% or so of hospital services are provided in private hospitals or day surgery units. Patients can choose to pay for this care or to take out private insurance to cover the cost of the hospital care. This insurance also supplements the Medicare rebates for their doctors’ costs.

Clinicians working in general practice may work as solo general practitioners, in partnerships and groups, or as employees of health centres or other doctors.

Clinicians working in hospitals tend to work in both the public and private system with a minority of specialists in any discipline working only in the private or only in the public system.

Regulation of the medical profession is based on self-regulation. The most visible components are the State and Territory medical boards1.

Registration of doctors is by the Medical Registration Board or its equivalent in each State and Territory, with most registration boards recognising general registration as a medical practitioner with only two States (South Australia and Queensland) having a specialist register.

Medical boards are designed to protect the public by registration requirements but also by disciplining doctors who engage in unprofessional conduct and by monitoring impaired doctors. The boards are funded by the profession. There is increasingly uniform registration

5th International Medical Workforce Conference, Sydney 2000 Session 2: Assessment of Doctors’ Performance in Practice - Australia _______________________________________________________________________________________________________________________________________

89

requirements in each State and Territory, although there is also now a movement towards all States having specialist registers.

The Australian Medical Council (AMC) is the peak body in relation to registration and the chairs of each registration board serve on the Council. It is funded by the State and Territory medical boards. The AMC is also responsible for examination and assessment of overseas trained medical graduates who wish to work within the Australian system. The AMC will often use the medical colleges as appropriate points of reference to determine whether overseas trained applicants have qualifications that are equivalent to locally educated clinicians. The AMC has an accreditation process applied to the 11 Australian medical schools and a working group is developing a similar process for colleges.

There are similar licensing boards for a number of allied health care groups and the nursing profession.

An accreditation process for general practice has recently been developed, but it is not strongly in place at present. Hospital accreditation has been refined over the last twenty years and most hospitals, public or private, achieve accreditation either through the Australian Council for Health Care Standards (ACHS), the International Standardisation Organisation process or through a number of commercial groups offering accreditation processes. Of those seeking accreditation, the majority of hospitals are accredited using the ACHS process.

The ACHS accreditation process is focussed on hospital processes, and until relatively recently has not looked closely at clinical performance. However, it has recently introduced clinical performance measures - clinical indicators - into the accreditation process.

The indicators have been developed with the assistance of the various medical colleges (professional associations of specialists) with approximately 150 draft indicators covering 90 areas of hospital practice2.

The development of a set of indicators passes through a number of stages to ensure their validity, and a process is in place to assess their reliability, reproducibility and responsiveness.

The qualitative information available indicates a willingness on the part of health care facilities to address the indicators and alter the process of care.

It is anticipated that the quantitative information being collected in aggregate form will subsequently demonstrate an improvement in the outcome of care3. Hospital-specific information currently remains confidential to the ACHS and the particular health care facility.

5th International Medical Workforce Conference, Sydney 2000 Session 2: Assessment of Doctors’ Performance in Practice - Australia _______________________________________________________________________________________________________________________________________

90

There is little doubt that the indicators will be useful to individual hospitals to review their performance over time, but as yet they have not demonstrated any significant ability to be used for comparison of hospitals, units or doctors. This is because of limitations of the data through variations due to chance, low frequency procedures, observational error etc4.

There are also hospital-wide indicators that have been collected for some time and these include numbers of unplanned and unexpected readmissions, unplanned return to the operating room during the same admission, extended length of stay and pulmonary embolism5. Once again, given the differences in the systems of care, demographics and socio-economic status, it is doubtful if these bulk indicators can easily be used for comparative purposes.

Each hospital has an appointments process and a credentialling process, although in most hospitals, the credentialling process is not iterative or robust.

There are well-established quality assurance programmes in hospitals and less well established quality assurance processes in general practice and in specialist practice outside hospitals.

Most clinicians in Australia will belong to a college or professional association that has a continuing professional development programme. Each college has different requirements to demonstrate the maintenance of professional standards.

For example, the Royal Australasian College of Surgeons requires its Fellows to participate in a programme of continuing professional development. Each year they are required to record their involvement in a range of continuing medical education activities, audit a part of their practice which also requires peer review of that audit, and evidence of being credentialled at an approved hospital or day surgery centre.

This annual activity report is required for each surgeon to obtain revalidation every three years.

The validity of the process is checked by a close review of the information given by a random sample of Fellows each year. This process has considerable teeth if the credentialling process is done well.

The College has recently published a credentialling document to allow hospitals to make that process more robust6. The production of a certificate of maintenance of professional standards is usually required by hospitals as part of the credentialling process. If a surgeon is not taking part in the continuing professional development programme, and is not being credentialled at an approved hospital, he or she will not be able to work.

5th International Medical Workforce Conference, Sydney 2000 Session 2: Assessment of Doctors’ Performance in Practice - Australia _______________________________________________________________________________________________________________________________________

91

Different processes apply to other clinicians who do not need the resources of hospitals in order to provide medical care. The quality of audit and peer review, credentialling and even the accreditation of hospitals and practices is patchy and at all levels, it needs to become more robust and transparent. Consumer involvement in these processes at this time is relatively limited.

The continuing education needs of younger doctors working in hospitals and the relatively few career medical officers are attended to by State based postgraduate medical education councils. In this area and within the specialist training programme run by the Colleges, the need for quality education and training and personal development is often in direct conflict with the service commitments required from hospitals.

PERFORMANCE ASSESSMENT PROFILE AND MEASUREMENT Community expectations, fuelled by media hungry for headlines, are of a level of performance that should be exemplary at all times. A perfect outcome every time seems to be the catchcry, even for the most aged and ill patients. The hype surrounding the scientific basis of medicine and the breakthrough mentality that superficially indicates that anything can be cured, may well have been largely responsible for this attitude.

Neither individuals nor systems have performance levels that can match these expectations and so there is increasing focus on measurement of performance. Indeed, there is a very strong move towards report cards for hospital activity and doctor activity. Even in the era of casemix funding, appropriate risk stratification to allow direct comparison is not readily available for most medical activities and the number of adverse events related to any particular type of procedure or patient or illness within each unit are not enough to allow valid comparison4. Even if all hospitals are equally good, then there will be 50% above the mean and 50% below. There will always be someone who appears worst. 12.25% of hospitals will be above average in three successive periods3.

To compare apples with apples is appropriate, to use raw data as comparison is fraudulent, offensive and inaccurate.

There seems to be little understanding in political circles of the difficulties associated with providing appropriate comparative data. An individual’s performance within any system of care is underpinned by the level of competence (knowledge) achieved during medical school education and postgraduate education programmes.

There is nothing to indicate that the level of competence of medical practitioners in this country is less than any other first world country. With this basic level of competence, maintained by continuing professional development activity, performance is therefore largely

5th International Medical Workforce Conference, Sydney 2000 Session 2: Assessment of Doctors’ Performance in Practice - Australia _______________________________________________________________________________________________________________________________________

92

determined by the system in which that competence is applied as well as by a number of personal attributes that tend to increase or decrease the potential for human error7.

Although the public and media perception is that there should be 100% good results, this is not the case. We know from the Quality in Australian Health Care Study published in 1995, and subsequent review of that data by the Australian Patient Safety Foundation, that approximately 10% of people seeking care will have an adverse event during that episode of care. This situation must be improved8.

Early data from the United Kingdom, New Zealand and the United States is said to suggest similar levels of concern. Further review of the data by various workers at the Australian Patient Safety Foundation suggests that up to 80% of the adverse events are preventable.

The benchmarks we use to assess performance of hospitals and doctors are themselves of variable quality and significance. We tend to use the results in publications from specialised and focussed units, which are among the very best in the world, as benchmarks, but we have little valid information about overall systems’ performance in any country. The Quality in Australian Health Care Study is still one of the few publications attempting to report on the level of adverse events across the system.

Even taking into account a number of methodological problems with that study, it remains the best source of information about performance in the Australian context.

There are a vast number of quality assurance programmes in hospitals that are undertaken by individual clinicians attempting to improve the situation9. An audit of anaesthetic deaths in Australia, which is both voluntary and confidential, has achieved information about anaesthetic mortality and morbidity over approximately 25 years that has led the way in improving systems in which and by which patients are anaesthetised and has contributed to a reduction in anaesthetic related deaths.

A similar audit of surgical mortality in the state of New South Wales is in its infancy and a similar programme is about to be set up in Victoria. Much of this activity is protected from legal action by privilege and results, except in aggregated form, remain confidential.

The majority of work done by clinicians in relation to audit, quality assurance and peer review attracts little financial or physical support for data collection, management or review and virtually no funding by governments or hospitals. The rate limiting factor in the terms of establishing national audits with a view to establishing national benchmarks, is the lack of continued long term funding of such activity.

5th International Medical Workforce Conference, Sydney 2000 Session 2: Assessment of Doctors’ Performance in Practice - Australia _______________________________________________________________________________________________________________________________________

93

If performance is to be improved, funding will need to be applied to allow appropriate data collection and review10.

Currently poor performance is only recognised and action taken when patients have been harmed and this action is either through the legal system, through the health care complaints commission of each State or Territory or through disciplinary action following assessment by medical registration boards.

Until recent initiatives by the New South Wales Medical Board, assessment of performance then recommendation for retraining, has not been a prominent response of the system.

DOES ASSESSMENT OF PERFORMANCE IMPROVE THE QUALITY OF HEALTH CARE? Many workers, including Brent James from InterMountain Healthcare in Utah, have demonstrated that when clinicians are provided with data about the performance of clinical pathways that they are following, problems can be solved and outcomes improved11.

Much of the improvement in performance is through improvement to the system in which the performance is given.

Many industries have shown that repetitive tasks are done with greater certainty and fewer problems if performed repeatedly under supervision. The airline industry makes use of simulators to help achieve this.

There is no doubt that in anaesthesia and surgery, simulators, when developed, should improve the ability to train, re-educate and even credential clinicians performing such tasks.

We know that the more complex the process, the greater the probability of poor performance, that system malfunction causes harm, that individual integrity and competence are important, but that responsibility and causation are complex and multi-layered.

There is no doubt that collective responsibility extends to all decision makers.

Our current system discourages and obstructs a positive approach to error reduction. Efforts to improve our system are ineffective because of the fear of discipline, fear of liability, fear of impact on public image and the fear of performance appraisal. While assigning blame fulfils a fundamental human need, it does not enhance safety. Malpractice stress is a major risk factor in clinician decision making and there is little doubt that defensive medicine practices are increasing the cost and probably reducing the quality of care.

The medical workforce feels under threat and therefore has lowered morale.

5th International Medical Workforce Conference, Sydney 2000 Session 2: Assessment of Doctors’ Performance in Practice - Australia _______________________________________________________________________________________________________________________________________

94

To embark on a lifetime in this industry is less attractive. There has been a dramatic increase in medical indemnity premiums, with some practitioners now paying over $40,000 a year. If however, the true cost of claims was borne by the individual groups responsible, obstetricians and orthopaedic surgeons in New South Wales would be paying over $90,000 a year each.

The legal system, and the lack of an alternative method for people to achieve redress of perceived wrongs and compensation, has been largely responsible for this increase in medical indemnity costs12. Relatively few of the total number of adverse events occurring as part of the provision of care are either actionable or have any action taken about them.

WORKFORCE IMPLICATIONS OF PERFORMANCE ASSESSMENT AND OTHER QUALITY IMPROVEMENT ACTIVITIES Performance assessment activities themselves are unlikely to increase workforce requirements. Although systems changes in response to appropriate performance assessment may lead to the need for a dramatically increased workforce, support for part time workers may alter the skill mix. The feminisation of the medical workforce has in some specialties increased the amount of part time work. System changes have not yet been made to fully recognise this change.

The Australian Medical Association has recently introduced its Safe Hours Project for junior doctors, and this will no doubt have implications for the whole medical workforce. Safety and quality activities in relation to skill mix, team work, team training, hours worked, support structures and time for up-skilling, maintenance of professional development programme and even retraining, will all have an impact.

For the first time the Royal Australasian College of Surgeons has had trouble attracting enough appropriately qualified young clinicians into a training programme (neurosurgery). Previously, the limiting factor in relation to surgical workforce has been the number of appropriately resourced training posts. In the fields of intensive care and radiotherapy, the number putting themselves forward to take part in those specialty training programmes has fallen short of requirements for some years.

There is no good evidence about the reasons for this. But one assumes that the high level of commitment required for both training and practice, the amount of on call work and the amount of emergency work, coupled with the ability to achieve only average medical incomes, is probably a set of perceptions that bright young people take into account when determining their future.

5th International Medical Workforce Conference, Sydney 2000 Session 2: Assessment of Doctors’ Performance in Practice - Australia _______________________________________________________________________________________________________________________________________

95

RECENT DEVELOPMENTS IN RELATION TO SAFETY, QUALITY AND IMPROVING PERFORMANCE IN THE AUSTRALIAN HEALTH SYSTEM The Australian Council for Safety and Quality in Health Care was formed in January this year by the Australian Health Ministers. Its role is to lead national efforts to promote systemic improvements in the safety and quality of health care in Australia with a particular focus on minimising the likelihood and effects of error. Its first report Safety First was presented to health ministers in July12. The health ministers endorsed the terms of reference of the Council, agreed in principle to provide $50 million for a five year national programme of work to be led by the Council, noting the intention of the Council to report on an annual basis on progress and planned action. They agreed to make available immediately $5 million of direct funds for the first year of this national programme of work and agreed to make the report publicly available. In that report the financial and personal cost of adverse events was highlighted. It was noted that health care, as a high risk high reliability industry, lagged significantly behind similar industries in its attention to safety. It noted that existing efforts to improve the safety of health care were valuable but insufficient.

The Council has taken a safety first approach as all stakeholders, whether they be carers or providers of care, managers or funders of care can identify with the benefits that can flow from an improvement in safety, which ultimately must lead to improvements in quality.

In that report and in subsequent meetings, the Council has confirmed that the focus of its work will be in priority areas as follows: . better use of data to identify, learn from and prevent error and system failure; . promoting effective approaches to clinical governance and accountability which

address both the competence of organisations and individuals; . redesigning systems and creating a culture of safety within health care organisations; and . putting consumers first.

Working parties have been established to address each of these priority areas and a draft action plan is being formed. The draft proposals for action in each of these priority areas is as follows:

5th International Medical Workforce Conference, Sydney 2000 Session 2: Assessment of Doctors’ Performance in Practice - Australia _______________________________________________________________________________________________________________________________________

96

Information Systems Working Group Proposal at a glance

That the Council initiate work on a national reporting system which operates at national, sub-national and local levels with the critical minimum elements including: 1. National, mandatory reporting of adverse events which lead to serious injury and death;

2. An agreed approach to routine voluntary incident monitoring primarily focused at a local level within health services (but also including scope for sub national and national reporting and analysis);

3. Develop standards for registers which collect information in specific clinical areas and/or in relation to specific therapeutic products; 4. Ongoing, routine monitoring of rates of adverse events from a representative sample of Australian hospitals and general practice; 5. Analysis of trends and qualitative information of patient complaints data and other

related data sets, for example medico-legal cases, coronial information and ACHS data; 6. The development of a range of tools to support clinical audit processes; 7. Capacity for specific short term projects in key speciality areas which are identified as high risk or ‘cutting edge’; 8. Strategies to improve medical record coding of adverse events and analysis of these data; 9. Strategies to improve routine collection and comparison of Australian Bureau of Statistics and mortality data and hospital morbidity data; 10. Annual public reporting on the national reporting system; 11. Convene a workshop with key stakeholders in early 2001 to further scope the elements and direction for the national reporting system.

5th International Medical Workforce Conference, Sydney 2000 Session 2: Assessment of Doctors’ Performance in Practice - Australia _______________________________________________________________________________________________________________________________________

97

Standards and Accreditation Working Group Proposal at a glance

That the Council initiate work on standards and accreditation with the critical minimum elements including: 1. Commission the development of a standards framework for hospital credentialling processes to form part of organisational accreditation processes;

2. Commission an evaluation of current certification and re-validation programs and identify priority areas for reform; 3. Organise a national consultative workshop to identify priority areas for reform of legislative and regulatory frameworks; 4. Commission work on the development of model legislation for statutory immunity; 5. Develop best practice standards for ensuring organisational accountability for clinical

safety and quality; 6. Commission a study of peer review practices in conjunction with relevant organisations to determine effectiveness of current approaches and within the context

of a systemic approach to safety, identify areas for improvement and exemplary practice; 7. Develop an agreed approach to national audits and commence audits in several areas of clinical practice; 8. Develop an agreed approach to national benchmarks.

Culture Change Working Group Proposal at a glance

The critical elements involved in a national approach to developing a culture of safety and quality include: 1. Organise a summit with managers, clinicians and other key stakeholders to identify priority areas for organisational improvement;

2. Develop a national ‘collaborative’ approach in at least one agreed priority area (for example medication error) and consider the establishment of a national centre to take forward work of this nature in the future;

3. Commission work on scope and types of technical tools that might be useful in reducing errors; 4. Commission work on undergraduate curriculum development and joint college continuing training modules; 5. Initiate a pro-active communication outreach strategy from the Council, through

conferences (such as the BMJ/IHI) and co-badging relevant events, workshops and other forums, as well as through media activity, identifying local champions and opinion leaders etc.

5th International Medical Workforce Conference, Sydney 2000 Session 2: Assessment of Doctors’ Performance in Practice - Australia _______________________________________________________________________________________________________________________________________

98

Consumer Working Group Proposal at a glance

That the Council initiate work on ‘putting consumers first’ with the critical minimum elements including: 1. An annual consultative workshop with consumer groups to advise on the whole work program of the Council;

2. Commission market research on community perceptions of the safety of health care; 3. Commission work on standards and education to support best practice models of communication between services providers and consumers for preventing and dealing with adverse events;

4. Support a third phase of the national health complaints information project; 5. Support initiatives to improve the public reporting of information about the safety and quality of health care to the community including jointly convening a workshop/seminar with the Australian Medical Writers’ Association on media reporting

of adverse events in health care.

If these initial proposals in relation to priority areas of activity of the Council can be established within the next twelve months, the Council will have gone some way towards developing a culture of safety, providing resources for appropriate data collection, analysis and feedback, some work on national standards in various areas and a system that is informed by the needs of consumers.

The workforce implications of this activity are likely to be an improvement in morale, less unnecessary variation in care, more satisfactory outcomes, a reduction in adverse events, and therefore more effective use of workforce, although in some areas there may be a requirement for an increased workforce. With improved credentialling and the appropriate use of data, performance assessment will be more able to be put in place in an appropriate way and should subsequently lead to further improvements in patient care.

To achieve all this some funding and much significant work will need to take place before any savings and improvements are evident.

5th International Medical Workforce Conference, Sydney 2000 Session 2: Assessment of Doctors’ Performance in Practice - Australia _______________________________________________________________________________________________________________________________________

99

REFERENCES

1. Brean K. J. 1999, “Self Regulation of the Medical Profession in Australia”, Australian Health Consumer. 2: 25-26.

2. Collopy B.T., Rogers L., Woodruff P., Williams J. 2000, “Early Experience with Clinical Indicators in Surgery”, Australia New Zealand Journal Surgery. 70:448-451.

3. Gibberd R. 2000, Report on Surgical Indicators for the Last Six Months of 1999, unpublished work, University of Newcastle and Australian Council of Healthcare Standards.

4. Marshall E.C., Spiegelhalter 1998, “Reliability of League Tables of In Vitro Fertilization Clinics’ Retrospective Analysis of Live Births”, British Medical Journal, 316: 1701-1705.

5. Collopy B.T. 1995, “Extending Facility Accreditation to the Evaluation of Care: The Australian Experience”, International Journal of Health Planning and Management, 10:223-229.

6. Credentials Committees, Surgical Appointments and Complaints Procedures 2000, A Guide by the Royal Australasian College of Surgeons, RACS Melbourne.

7 de Leval M.R., Carthey J., Wright D.J., Farewell V.T., Reason J.T. 2000, “Human Factors and Cardiac Surgery: A Multicentre Study”, Journal of Thoracic and Cardiovascular Surgery, 119:661-672.

8. Wilson R., Runciman W., Gibberd R.W., Harrison B.T., Newley L., Hamilton J.D., 1995, “The Quality in Australian Health Care Study”, Medical Journal of Australia, 163:458-471.

9. Wolf A., Bourke J. 2000, “Reducing Medical Errors: A Practical Guide”, Medical Journal of Australia, 173: 247-251.

10. Eno L.M., Spigelman A.D. 2000, “A Survey of Surgical Audit in Australia: Whither Clinical Governance?”, Journal of Quality in Clinical Practice, 20:2-4.

11. James B. 1997, “Every Defect a Treasure: Learning from Adverse Events in Hospitals”, Medical Journal of Australia, 166:484-487.

12. Wilson L.L., Fulton M. 2000, “Risk Management: How Doctors, Hospitals and MDOs can Limit the Costs of Malpractice Litigation”, Medical Journal of Australia, 172:77-80.

13 Australian Council for Health Care, 2000 Safety First: A Report of the Australian Council for Health Care to Australian Health Ministers. July 2000.