Download PDF

The Parliament of the Commonwealth of .Australia

SENATE STANDING COMMITTEE ON SOCIAL WELFARE

Through a Glass, Darkly

Evaluation in Australian Health and Welfare Services

Volume Two

Papers Commissioned by the Committee

Brought up and ordered to be printed 11 September 1979

Parliamentary Paper No. 160/1979

THE PARLIAMENT OF THE COMMONWEALTH OF AUSTRALIA

THROUGH A GLASS, DARKLY

Evaluation in Australian Health and Welfare Services

Report from the Senate Standing Committee on Social Welfare

VOLUME TWO

Papers commissioned by the Committee

Australian Government Publishing Service Canberra 1979

@ Commonwealth of Australia 1979

ISBN for set of two volumes: 0 642 04401 5 ISBN for Volume Two: 0 642 04407 4

This report is in two volumes: Volume One-Report and appendixes Volume Two--Papers commissioned by the Committee

Printed by Authority by the Commonwealth Government Printer

MEMBERS OF THE COMMI'ITEE

Senator Peter Baume (New South Wales), Chairman Senator N. T. Bonner (Queensland)* Senator R. C. Elstob (South Australia)* Senator D. J. Grimes (Tasmania) Senator J. I. Melzer (Victoria) Senator M. S. Walters (Tasmania)

Former members of the Committee Senator W. W. C. Brown (Victoria) (to 31 May 1978) Senator R. E. McAuliffe (Queensland) (31 May 1978 to 17 August 1978) Senator T. J. Tehan (Victoria) (to 30 June 1978)

Secretary R. G . Thoms on The Senate Parliament House Canberra

*Appo inted 17 August 1978

Reference Evaluation of the adequacy of Australian health and welfare services with par " ticular reference to: 1. standards of performance and provision of health and welfare services;

2. the pattern of current practice in the provision of such services in terms of need and demand; 3. mechanisms for evaluation of the effectiveness and efficiency of health and welfare services; and 4. requirements for ongoing evaluation as an integral part of the development

of health and welfare service programs.

(Referred by the Senate on 2 June 1976)

iii

For now we see through a glass, darkly; but then face to face: now I know in part; but then shall I know even as also I am known.

1 Corinthians 13: 12

iv

Contents

Abbreviations vi

Biographical note . vii

Introduction viii

The views of clients: what people think of welfare services . 1 (Personal paper commissioned by the Committee from Dr Lois Bryson)

The evaluation of health and welfare services in Australia: agencies 23 (Personal paper commissioned by the Committee from the Rev. J. Davoren)

Evaluation in relation to standards of performance 29

(Personal paper commissioned by the Committee from Frances Donovan)

Recent evaluation activity in Australia 41

(Personal paper commissioned by the Committee from Paul F. Gross)

History and development of evaluation in the human services: a view from the United States 60

(Personal paper commissioned by the Committee from Professor Rosemary C. Sarri)

Methods of evaluating effectiveness of human services 75

(Personal paper commissioned by the Committee from Professor Rosemary C. Sarri)

Need and demand in the making of planning decisions in the evaluation of health and welfare services 83

(Personal paper commissioned by the Committee from Mrs Anne Stevenson and Mrs Wendy Chew)

v

ABBREVIATIONS

Certain abbreviations used by the writers of these papers are listed here for convenience:

The views of clients: what people think of welfare services CES Commonwealth Employment Service CSSD Commonwealth Department of Social Security RSL Returned Services League

Evaluation in relation to standards of performance PPBS Planning/programming/budgeting systems

Recent evaluation activity in Australia AGPS Australian Government Publishing Service HHSC Hospitals and Health Services Commission PM and C Department of the Prime Minister and Cabinet PSA Public Service Act (Commonwealth) PSB Commonwealth Public Service Board RCAGA Royal Commission on Australian Government Administration VCOSS Victorian Council of Social Service

vi

BIOGRAPillCAL NOTE

Dr Lois Bryson Lecturer in Sociology

School of Sociology University of New South Wales Address: P.O. Box 1 Kensington, N.S.W. 2033

Mrs Wendy Chew Former Social Worker, Depaul Centre St Vincent's Hospital, Melbourne Personal address: Calder Highway Ravenswood, Victoria 3551

The Rev. J. Davoren Secretary

Australian Catholic Social Welfare Commission Address: Suite 3A, 4th floor 154 Elizabeth Street Sydney, N.S.W. 2000

Frances Donovan Head of School, School of Social Work Preston Institute of Technology Address: Plenty Road Bundoora, Victoria 3083

Paul F. Gross Special Adviser

Social Welfare Policy Secretariat Address: P.O. Box 1, Woden, A.C.T. 2606

Professor Rosemary C. Sarri Professor of Social Work University of Michigan Address: 1065 Frieze Building Ann Arbor, Michigan, U.S.A . 48109

Mrs Anne Stevenson Social Worker Social Work Department Moorabbin Hospital Address: P.O. Box 72 East Bentleigh, Victoria 3165

Vll

Introduction

This volume contains the text of seven commissioned papers prepared for the Committee during the consideration of its reference on evaluation in Australian health and welfare services. It is intended to be read in conjunction with the Committee's report, entitled Through a Glass, Darkly: Evaluation in Australian Health and Welfare Services.

These papers are seen as important. They had their genesis in the fact that there were gaps in the information available to us. These were thought to be of sufficient importance to warrant particular attention, and we set out to commission a number of personal papers from people with appropriate qualifications.

In the Committee's view, these papers represent valuable additions to the body of knowledge available, particularly in some critical areas. We trust that this volume will therefore be of value to persons interested in evaluation theory and practice.

We believe that, read in conjunction with the Committee's report, it will help to improve the understanding of evaluation in the Australian context. We take this opportunity of thanking the distinguished and expert authors who have contributed their time and energies so generously to assist this Committee and, through it, the Parliament and people of Australia in seeking a better under "

standing of a complex and difficult field.

ix

The views of clients: what people think of welfare services

In his report as Chairman of the Commission of Inquiry into Poverty, Professor Henderson pointed out that after collecting evidence about the organisation and provision of community services: The overall picture gained . . . can only be described as one of poor

planning, lack of integration, frustrated, ignoran∑ t and humiliated users and unmet needs. (1975:93) Since that statement was written more reports have been published by the Com " mission and all have confirmed this picture. My brief for this paper is to consider only clients' or users' perceptions, though in general these tally with the judgments of many workers in the welfare field. 1 I shall first look at the question of why the sudden interest in users' points of view, then consider just what these points of view are. Finally I shall try to tease out some of the major implications of these views for the development of welfare services.

It is only in recent years that it has been accepted as important to ascertain what people who use services think of them. Because the interest is recent we have yet to develop methods adequate to the task of eliciting and understanding these views and, even more difficult, ways of incorporating the information received from this source into policy and practice. 2 These are difficult and immediate problems but nonetheless they are based on very modest goals. Were we to be slightly more ambitious we would point to the fact that such solutions take a very narrow and conservative approach to welfare. More radical suggestions might encompass the idea that people should not be cast in the role of clients, consumers or users at all but as citizens actively participating in the organisation

of appropriate community structures which would serve human needs, and eliminate current power distinctions between helpers and helped. Such a long " term goal, however, should not immobilise us, for there is clearly a need to address the immediate problem of ways in which clients' views might be taken into account, since radical change in our present structures is going to be more difficult to achieve. Our short-term goals should be consistent with, and hopefully even promote, these long-term goals.

There are a number of general developments in thought which we must see as providing a basis for this recent client-oriented approach. The major component is undoubtedly ideological but some of the interest has developed from recent theoretical developments. The ideological component was aired in the U.S.A. during the period in the 1960s when the 'war on poverty' was being waged. Recognition of the powerlessness of the poor, and a sense of the injustice of that,

1. Evans reports that 96 per cent of 166 South Australian social workers surveyed in 1974 'consider that existing social benefits and community facilitie s do not adequatel y and fairly meet the needs of those having experienced some hardship and difficulty' (Evans, 1975). 2. For a fuller discussion of this see Shaw (1976).

led to the promotion of 'maximum feasible participation'. Although programs; based on this notion were so unsuccessful that one must doubt that many actually intended participation in any real sense3, this new emphasis on client involvement left a certain limited legacy. Movements towards equality and social justice have: also stressed that the poor and stigmatised should not be treated as second class citizens and receive services inferior to those that are paid for by wealthier ∑ citizens. Interrelated with this is the questioning of the assumption that middle: class perceptions are more 'correct' than working class, and a consequent questioning of professionalism, particularly in the helping professions which are largely middle class in origin and orientation (cf. lllich, 1977; Tomlinson, 1977; Bryson and Thompson, 1976: Chapter 14).

At a theoretical level we can link these ideological strands with the recognition. that knowledge is not value free. When it was assumed knowledge was value free, all one needed to do to understand a problem was to study it 'scientifically'. It is .. now generally accepted that even the most apparently objective and scientific research is full of implicit assumptions and that because of this, political positions are involved. Thus it becomes important to have voices representing various points of view, particularly where differential power is involved. In the past the value free assumption tended to mean accepting, unscrutinised, the status quo.

So we can say that, from (a) a recognition of the powerlessness of people in, receipt of welfare, (b) a seeking after greater equality, and (c) a recognition of the political nature of knowledge, has developed an interest in the collection of views from clients. Having said that however let me hasten to add that, as. with' the participation programs instituted in the name of the war on poverty in the U.S.A., we cannot take this seeking of clients' views at face value to mean a genuine concern to know and act on them. As in the U.S.A. the mainspring for the movement has come from researchers and policy makers and others removedl from the field of practice. The motivation behind research is a complex matter; so it remains to be seen whether clients' views will be treated seriously or merely collected because it has become methodologically and politically fashionable to∑ do so. There is evidence that some agencies do try to incorporate the information produced by clients. The Brotherhood of St Laurence's program, originally the Family Centre Project and latterly the Action Resource Centre, is a case in point. In this project clients' participation was constantly promoted and the program; developed from this. 4 However such experiments are rare. It is not surprising that this particular one was established by the Brotherhood of St Laurence since this has been one of the few innovatory welfare agencies in Australia, but as is clearly seen from the reports on the project, it was fraught with difficulties, so it: seems unlikely that other agencies will readily take up similar challenges.

Users' points of view We now have a great deal of information about consumers' views, far more than I can cover in the space of one paper. What I shall do is attempt to outline the: main trends to be gleaned from the reports. Two studies in addition to the Family Centre Project do however merit specific comment. The first is a study com " missioned by the Australian Government's Commission of Inquiry into Poverty, carried out by H. J. Cunnington and entitled 'Communication between welfare∑ agencies and clients' ( 1977). This is important because it involved interviews with people in receipt of pensions or benefits, about their experiences with the:

3. See Arnstein (1971) for a discussion of the various guises of participation . 4. See Salmon (1974), Liffman (1978) and Benn (1973-76).

2

Commonwealth Department of Social Security and the Queensland Social Welfare Department. Because these two bodies (with the name of the latter varying from State to State) provide virtually all the income maintenance for Australian .citizens, views about their operation are particularly crucial. The other is a study

published under the title of Who Cares? This was carried out by the Institute .of Applied Economic and Social Research of the University of Melbourne and represents a depth study of a number of families 'to find out their experiences and the impact on the family of the whole system of health, education and welfare

services' (1977: 1). The sample was a selected one and for this reason it is difficult to know how far generalisations can be made from the results. Nonetheless it has tremendous value in that it is a depth analysis of the way in which people go .about solving their problems and about what they find helpful and not he'lpful in

the current welfare system. While these two works are of particvlar importance we find that for Australia, post Poverty Inquiry, we do have a wealth of inform " ation about what consumers think though there has been little attempt so far to systematise this, apart from Professor Henderson's summary in his Main

Report (1975:93-108). Here I shall summarise first the negative aspects clients find in the services, under three headings: (a) knowledge of them

(b) stigma and powerlessness (c) effectiveness and appropriateness of administration Much of this discussion will centre on statutory services, partly because of their strategic importance, but mainly because a large proportion of clients do find these services, or aspects of them, to be unsatisfactory. However in principle I intend to adopt a broad definition of welfare service, since it is clear

that users of these services do not make nice distinctions. I shall include income maintenance schemes, provided by the CSSD and various State departments as well as other services that are utilised by people on a non-payment (or reduced payment) basis. I have deliberately excluded more than passing references to health, educational and legal services because to include them would broaden the topic so far that it would be possible only to attempt the most superficial of

discussions. Also it seems that the most problematic services are those concerned with income maintenance particularly those for groups which might be seen as infringing strongly held community values. For example it seems that unemploy " ment and lone motherhood are statuses which present most problems and result in users of services feeling most unhappy about the treatment they receive.

'(a) Knowledge of services Lack of knowledge about services is a constant theme of the research reports. This comes up in specific cases in which people in need explain how they lacked knowledge of where to turn for help. For example in the case of one family,

the husband had been ill for two months and his wife explained, 'we were actually living off his grandmother. We didn't know anything about sickness benefits or social services' (Salmon, 1974:60). Similar cases are documented in the Who Cares? study, for example in one family the self-employed husband became 'ill

and was unable to work for several weeks. They did not know about sickness benefit and did not seek help until they had spent all their money and had no food left' (McCaughey et al., 1977:112). Many other cases are documented in which people have suffered from lack of information about their entitlements (e.g.

3

McCaughey et al., 1977:12; Cunnington, 1977:84; Australian Red Cross Society, N,S.W. Division, 1975:76; Brewer, 1975: 108-9; Brown et al., 1974: Chapter 8). This does seem to vary somewhat with the problem, with age and widows pensions, that most well known of the statutory provisions, followed by unem " ployment benefit.

However even when a scheme is understood well enough to apply for a pension or benefit, the information the client has may be limited. Salmon reports that even among men who have had experience with the Commonwealth Employment Service, 'there is a marked lack of knowledge of their rights and the correct procedure' (1974:74), and Cunnington found that many of the deserted wives and supporting mothers who were in receipt of Commonwealth benefits reported difficulties 'in obtaining additional information on their entitlements to con " cessions, subsidised health benefits etc.' (1977: 83). Brewer points out that although Commonwealth Employment Service staff are:

. . empowered to issue vouchers for public and private transport, survey interviewers did not become aware of a case (out of 166) where this happened not, in fact, of anyone who was even aware of the existence of such a right. (1975:108) Similarly there is provision for unemployed persons under 16 years of age who have officially left school to be paid a special benefit if they are deemed to be in hardship and/ or if they have previously worked for three months or more, 'however the young people interviewed were not aware of this' (Brewer, 1975:109).

This lack of knowledge of entitlement may be partly due to personal difficultie s in finding out or lack of systematic effort, but there is considerable evidence to show that there is often reluctance by agencies to disseminate information. Brewer's findings about the unemployed are a case in point and in 1971 members of Client Power accused the Queensland Department of Children's Services of refusing 'to tell us what we are entitled to or even how it (the department) determines who is eligible and who is not' (1972: 25). The generality of t his practice at least in the past is illustrated by the fact that the Victorian Social W elfare D epartment this year (1978), for the first time, issued a pamphlet which sets out general principles of eligibili ty for financial assistance. That the Queensland Department of Children's Services has apparently overcome this problem in recent years as well may attest to a general improvement in the area of commu nication of information. Cunnington, in summarising his findings about general access to information about statutory entitlements, is not sanguine, however, and points to the fact that:

Wh en information is stored solely in 'the minds of government officials or in confidential manu als of instruction, the costs that clients pay in obtaining that inform ation are very high indeed . (sometimes) so high that clients believe the si,tuation to be hopeless and do not bother to undertake any form of search at all. (1977:133) An additional dimension about ignorance about services is added whe n we looked at the general public's level of knowledge rather than considering only those who are actually involved with welfare services or who have a recognised problem. In 1974 a Melbourne firm of town planners prepared a report on the development of a fringe area of Melbourne, Berwick. As part of that project a survey of 240 households was undertaken which included questions on knowledge about local services. This information led to the conclusion that:

. the most significant finding in all three areas of the survey (pre-school, health and welfare services) was how little people knew about what was and was

4

not available. Although the availability of pre-school services seemed to be well known to those who wanted " to use them, people were often uncertain where medical, dental, family planning and family counselling were to be found. Moreover, many were unable to distinguish between major government departments such as the Commonwealth Employment Service, Commonwealth Social Security Department and

(State) Social Welfare Department. (Loder and Bayley, 1974:46)

A similar finding is reported by Duigan for 280 residents of the suburb of Hindmarsh in Adelaide whom he surveyed. A series of problems were posed to residents about where they would tum to for help with a range of problems, e.g. employment, education, mental illness, poverty. Duigan found that only about

15 per cent 'would know where to tum in the problem situations defined in the questionnaire' (1975:47). Francis notes a similar conclusion for Fremantle residents (1976:40-1). Ignorance about free legal services was demonstrated by Cass and Sackville, again from a general survey in three suburbs of Sydney. Only 54 per cent of those interviewed (548 people) answered Yes to the question 'Is it possible to get cheap legal services' (1975:68). Examples could be proliferated for almost

all types of service . In fact a number of researchers point out that finding out about entitlements can often be largely a matter of chance. Cunnington for example talking of the difficulty experienced by supporting mothers in finding out about supplementary benefits suggests that 'for these people it was only

"luck" that provided the knowledge, of their entitlement to these benefits' (1977:84). The Who Cares? study makes a similar comment about a particular case, suggesting that one lone father had 'only been helped in a piecemeal and often belated manner by services that he was lucky enough to stumble on'

(McCaughey et al., 1977: 124). Such comments about inadequate information with regard to services are matched by recommendations from the various studies pointing to a need for greater publicity about services. Henderson em phasised this in his Main Report of the Commission of Inquiry into Poverty (1975:103-8). This is seen as particularly necessary in relation to statutory entitlements (see Cunnington,

1977: 133-4). However it is a recommendation made in respect of the range of services that are broadly defined as welfare. A study of Rural Poverty in Northern N.S.W. puts it thus: Existing wel£are programmes in housing, health, education, and employment areas

would be made more accessible .to the rural poor who need them most if they were more widely advertised in local newspapers and on local radio and television . (D epartment of Sociology, University of New England, 1974:108)

However, though such an approach may improve the situation it does not get at the cause of the problem. The very next sentence, like much of the other evidence, suggests a much more satisfactory mode of attacking the problem. It points out that 'many survey respondents suggested that there were too many formalities and too many complicated forms associated with the present schemes' (1974:108). The obvious implication is that schemes should be simplified. This

point will be taken up later.

(b) Stigma and powerlessness We move now to the area of recipients' reactions to being welfare clients and the question of stigma. To what extent is receipt of support seen as a right, and therefore not demeaning, or do people feel that the community in general, often

5

including themselves, react negatively to them and the idea of welfare? A more specific question relates to the sort of treatment clients feel they receive from welfare personnel and the attitudes displayed to them. Since client reactions to accepting help of many kinds are tentative, one wonders if the fear that services will be inundated if eligibility is made easy is actually justified. As with lack of information about services, examples of people feeling 'degraded', 'humiliated', 'stigmatised' are numerous-disturbingly so when much of the Australian literature would at least pay lip service to welfare by 'right' rather than by charity.

Cunnington points out that in the society's prestige scale those who are supported by the government are at the bottom. However, there is an internal hierarchy as well with unemployment beneficiaries and deserted wives and supporting mothers at the bottom of the welfare hierarchy, since the:

. unemployed are still looked upon as dole bludgers; the deserted wife has somehow failed or she would not be deseuted; and the supporting mother is still regarded as a 'girl who got herself into trouble '. (Cunnington, 1977: 125) Because of the particularly vulnerable 'minority status' of those two groups I shall look at them in some detail.

'Unemployment benefit is not for the proud and sensitive' (Henderson, 1975, vol. 2: 20). This statement was made by Professor Henderson after surveying client reactions to the service offered by the Commonwealth Employment Service. Clearly many people felt stigmatised then and the situation today with greater pressure of numbers is likely to be worse rather than better. We find numerous examples in Brewer's study of a general sense of stigmatisation:

- People dislike you because you're unemployed and say you're living off the government. - It (unemployment) makes you feel a lesser class of person. People call you a 'bludger and it makes you feel bad. - People put you down because you've been unemployed. You get labelled as lazy

because you haven't got work . . . even if you've been really trying hard to get a job. I was fussy about jobs at first, but I'll take anything now. (Brewer, 1975:56-7) As well as general feelings of loss of self-esteem there are many reports of feeling humiliated by the service received from CES. Salmon reports a reluctance to register as unemployed and suggests this is due to 'the strained relationship between the men and the CES staff. To the men it is a frustrating and humiliating service' (1974:74). Brewer similarly found that there was hesitation by some unemployed people in approaching CES. One woman complained:

You feel intimidated by the CBS in ∑ that you get offered mundane jobs. It's demoralising because you can't get into what you want. You feel humiliated having to go to the CBS and on a lower social scale being unemployed. (Brewer, 1975: 121) The basis of one of the submissions to the Poverty Inquiry was information from twenty-five young men, who though eligible for unemployment benefit had not registered. Two of them gave the following reasons:

Tony feJ.t that the fact that he had " to wait up to an hour for service at the CBS office and that the (officer's) only concern for him was to get his form filled in was lousy. He wanted to keep some pride because he felt that work was important and he shouldn't be treated as second rate because he didn't have any.

Andrew felt that he could find his own work faster than the CBS (had waited 6 months with no results). [He] didn' " t want handouts. (Henderson, 1975, vol. 2:20)

6

In this area of unemployment we are fortunate in having a study comparing judgments of personnel of the CES with the opinions of clients. Long-term unemployment beneficiaries were asked why they were out of work and in more than half the cases this was seen in impersonal terms e.g. 'no vacancies in the locality'. Personnel were much more likely to see causes in individualistic qualities

(see table). Jordan suggests their judgments are based 'on a fairly simple theory of the causes and cures of unemployment. This implicit theory is comparatively individualistic, voluntaristic and moralistic' (1975: 35). The interviewers' perceptions of reasons for unemployment were closer to the unemployed than to the CES staff. This tendency by staff to 'blame the victim' (Ryan 1971) clearly lends support to the views of the unemployed that they are often treated

as somehow 'at fault' and therefore as 'second class citizens'. It should be noted, though, that Jordan's study was carried out during a time of full employment. It may be that today staff are less likely to attribute causation to the personal characteristics of the unemployed. Certainly we need more up-to-date evidence

but since we actually have seen a hardening of attitudes in some directions, in various measures to clamp down on 'dole bludgers', we can, I think, for the moment assume that the unemployed fare little better today. Also the assignment of blame for social problems to individual failings as Ryan (see above) has clearly indicated is widespread and not just limited to the area of unemployment. Burbidge and Brown note this tendency among welfare workers in Brisbane

(1976:95-6) while Fitzgerald notes this tendency in the educational sphere ( 1976:43). Illustrations also emerge in many of the reports written for the Poverty Inquiry (cf. Vinson et al., 1976:passim; Hill, 1975:72; Australian Red Cross Society, 1975:passim).

Reasons for inability to find work, as stated by respondent ":

No vacancies in locality Too old Lack of skills or education Illness, injury, chronic disability Discrimination by employers Transport problems Nervous condition

Personal preferences

%

58 24 20

18 12 10

8

8

Reasons for inability to find work, as stated by Employment Offices*:

24895!79-2

Lack of motivation Shortage of local vacancies Appearance and manner Physical disability Registrant too selective Age

Lack of skill, education, experience Poor work history Alcoholism Lack of confidence, initiative Discrimination by employers Minor psychiatric condition Family obligations

Mental retardation Prefers unemployment

7

%

27

25 23 18 14

13 13 11 7

6

6

4

3

3

3

Reasons for inability to find work, as stated by interviewer ":

Shortage of local vacancies Lack of skill, education, experience Physical disability Registrant too selective Appearance and manner Discrimination by employers Lack of confidence, initiative Age

Family obligations Minor psychiatric condition No economic need to work Lack of motivation Alcoholism Has adjusted to unemployment Poor work history, instability

* Frequency count, not exhaustive Source: Jordan (1975:34-5) .

%

43 20

18 14

12

12

11

11

9

7

7

7

6

5

5

Another group which registers similar complaints are women who are sup " porting mothers (apart from widows). Whereas the bone of contention about unemployment benefit is whether or not a person really wants to work, exemplified in the work test, for these mothers the contentious question is 'are they living with a man on a bona fide domestic basis' (Cunnington, 1977:82). To check on this, inspectors visit from time to time and there are many complaints recorded about the way inquiries are made. Cunnington suggests that from comments made 'it appears the checking up is often carried out in the way a private inquiry agent might seek damning evidence for a divorce case' (1977:82). A Women's Electoral Lobby submission to the Poverty Inquiry reported:

There were many complaints about invasion of privacy. Deserted wives and unmarried mothers were particularly critical of the way in which the Federal Department of Social Services investigates the social and sexual life of women by unannounced visits and interviews with neighbours. (Henderson, 1975, vol. 2:7)

Professor Henderson notes that these negative feelings may be based more on stories from the past than on current happenings though Cunnington sees such practices as continuing. E ven if the stories are exaggerated, the fact that they persist suggests that they are associated with sensitive aspects of social values. Cunnington points out that in Brisbane at the time of his survey there were no training programes for field office staff. He goes on to suggest that:

When this is combined with the fact that field officers are expected to 'catch-out' those who have been receiving benefits under false pretences ∑ It is little wonder that powerless groups such as supporting mothers and deserted wives fare so badly. (Cunnington, 1977:86)

Their sense of powerlessness is illustrated by the fact that although many com " plained about their treatment t'o the survey interviewers 'Not one of those who related experiences which had raised their indignation had even considered making a formal protest' (Cunnington, 1977: 125).

8

Complaints of powerlessness, loss of self-esteem and feeling of stigmatised came from many groups, not just the unemployed and supporting mothers. A group of transients said:

Lack of sympathetic treatment and impersonality of government agencies leads the transient to often avoid use of health and welfare services. We should not be made to feel different in order to accept services. People should not be treated differently because they are poor. (Australian Council of Social Service, 1975:7)

Aboriginal women from Adelaide recorded their feelings thus:

They wanted to know so much I gave up and went away and will never go back even if we were all starving. I hate them, the government just wants people to crawl on their bellies, to get a little bit of money. (Gale and Binnion, 1974:44)

Most of the comments relate to government agencies. Presumably this is because of the volume of cases they deal with. This not only means more people at risk but tends to make services impersonal and impersonality is generally disliked . However people have also felt demeaned in their approach to voluntary agencies. A number of the Brotherhood of St Laurence Family Centre families mention this. One woman explained that, as the situation got worse:

I found myself going to other organisations for help, for example, Salvation Arm y and St Vincent de Paul Benevolent Society. Sometimes they made you feel more degraded because they wouldn't always believe what you were telling them. All they do is ask question after question. (Salmon, 1974:60)

The experiences of some rural. families even more graphically illu strated negative community reactions:

A number of respondents in very poor houses and with mentally ill household me m bers said they would never again request help from charitable organisations because of the condescending and antagonist ic attitudes of people who had called with food and clothing. (University of New England, 1975: 105)

There appears to be loss of self-esteem amon g at least a proportion of people when they make use of available welfare services. While some of this feeling no doubt develops from actual experience this may not always be its basis, as general community attitudes alone suggest that people who cannot make out on their own are 'failures'. People m ay suffer from these general attitudes without having experienced humiliati ng treatment. That t hese atti tudes are at times applied by people to themselves is illustrated by an example from the study of poor familie s The Hav e Nots, in which an intermittently employed unskilled man 'repeated and

accepted the communi ty's label "I suppose I must be lazy" ' (O'Neill and Nairn, 1972: 127). Aborigines have consist ently suffered from negative evaluat ions by whites and they at times too demonstrate an acceptance of the dominant values. A study of Aborigines in Western Australia found:

Psychologically, the Europeans and certain of their own people regard them as incapable, lazy, shiftless, inferior. Most have identified with this usage, constantl y referring to themselves as 'just poor Aborigines' . (Hill, 1975:81)

Stigmatisation is based on the pow er of one person (the labeller) over another; so it comes as no surpri se to find the Who Cares? study suggesting that for mos t people a more acceptabl e basis for help than welfare services is from neighbours and friends 'because it (is) based on reciprocity' . In line with this, the same study gives examples of families who did 'not see pensions and benefits as a right

9

even when they were eligible' (McCaughey et al., 1977:128). Sentiments relating to not wanting to go to official sources and emphasising reciprocity regularly appear in the client studies. For example: I didn't want to deg:mde myself by having to ask outsiders for help, it really broke

my pride. (Salmon, 1974:60) Members agreed 'that being welfare dependent lowered their self image . . " all feel alienated and isolated and often demoralised by being dependent, always the receiver of handouts and rarely the giver. Northside Pensioners' Group. (Australian

Council of Social Service, 1975:6) Additional insight into such attitudes is provided by the reactions of some unemployed people to a request for co-operation in the Brotherhood of St Laurence's surveys of Workers Without Jobs. Brewer reports that a number refused to be interviewed and indicated that they thought of the Brotherhood of St Laurence as a charitable organisation and expressed their own fierce indepen " dence from charity (1975:9).

In clients' reactions we see confirmed what the sociology of deviance tells us about community attitudes towards welfare recipients and the dominant values of individualism. 5 People generally feel bad about asking for help and may in fact be treated badly by others who identify them as therefore less than worthy.

(c) Effectiveness and appropriateness of the administration The literature abounds in clients' complaints about inaccessibility, complexity and cumbersomeness of the administration of services. A repeated complaint is about delay in receipt of financial assistance. One aspect of this is that there are waiting periods before payment of pensions and benefits but most delays result from failure of agencies to get cheques to recipients whose claims have been recognised .

Virtually all studies involving people in receipt of income maintenance payments mentioned instances of delay (e.g. Adams, 1972; Lewis, 1973; Salmon, 1974; Brewer, 1975; Jordan, 1975; Cunnington, 1977) and this is a point emphasised by Henderson (1975). Again Cunnington's figures are the most comprehensive and they are recent. He found that of all beneficiaries, 56.8 per cent stated there was a de:lay of at least two weeks before they received their first payment, while 40.8 per cent experienced delays of one month or more. We find differences by the type of benefit and in this case supporting mothers fare best presumably because most are transferring from a State scheme. Unemployment beneficiaries fare worst with 52 per cent waiting over one month excluding the mandatory waiting period. The delays of course are of little consequence if they don't cause hardship, and for some entitlements , such as age pensions, they generally do not. However for unemploym ent beneficiaries 20 per cent 'needed money desperately ' and the figures were only slightly lower for sickness beneficiaries ( 18 per cent) and deserted wives ( 16 per cent) (Cunnington, 1977: 117). Clearly then the administration of a numb er of the major income maintenance programs needs overhauling if clients' most pressing needs are to be met. If we approach the situation from the point at which clients are waiting for their entitlements we see the desperate situations with which people have to cope and the strain this puts on existing services. One. municipal social worker from Heidelberg, Victoria, suggested to the Poverty Inquiry that at least 50 per cent of those coming would be helped if the time lag for receipt of benefits were closed (Henderson, 1975, vol. 2.58).

S. See for example Ryan (1971), Matza (1969), Goffmao (1963), Rainwater (1974).

10

Another problem for clients is associated with the method of form filling required by most government departments. Cunnington found a great deal of criticism about this among clients of the CSSD with over 30 per cent believing that all the information asked for was not needed (1977: 110). One of the

points on which private agencies were more positively evaluated than government ones was the paperwork involved. As one unemployed person put it: . the distinction between government and non-government welfare agencies is the government gives you forms to fill in, while non-government agencies give help

quickly and without fuss. (Cunnington, 1977:92) A typical comment, made by an Aboriginal woman from Adelaide, is about the Department of Community Welfare: 'It is awful all the questions they ask' (Gale and Binnion, 197 5:44). Into the bargain Gale and Binnion point out that asking questions is often a quite inadequate way of eliciting informati on about a family's circumstances:

D iscussion with various officers showed that their knowledge of people in their distric t was often quite inaccurate. Only regular visits to a home will give a reliable picture of any particular situation. (Gale and Binnion, 1975:44) Waiting at government offices is another complaint which is lent weight by the research evidence. This is a point that need not be laboured, suffice it to quote Cunnington's overall figure from his survey of 239 pensioners and beneficiaries,

which was that 55 per cent claimed they were 'always' kept waiting on visits to the CSSD and the figure was even higher, 68 per cent, for the Queensland Depart " ment of Children's Services (1977:114). Government agencies come in for criticism for a variety of administrative reasons. Cunnington ( 1977) carefully examines other technical aspects of the provision of services and finds many clients dissatisfied. Adams similarly found generally negative reactions to CSSD counter staff in Perth who were found to

affect: ∑ (a) the client's view of the agency's feelings 'towards him (b) whether the client will continue the contact (c) whether others will make contact (d) the client's self-esteem (Adams, 1972: 120) The Who Cares? study makes a particularly damning summary statement about the effectiveness of current welfare services. In relation to forty-two families forming one group of the families in the study it is said:

They received poor welfare. Only three families had an effective service from the CSSD and none at all from the Social Welfare Department. Municipal services were a little better and the voluntary section was the only one to give an effective service to the majority of users. (Quoted by Henderson, 197 5: 100) What comes through loud and clear from these data on the negative aspects of welfare services is that many clients are not being well served. Even if services have been improved in the few years since most of these studies were reported it seems there was such a distance to go that it is unlikely that the problems will have been eliminated. We do know that agencies are attempting to implement some recommendations from the studies 0 but such changes are slow and often difficult to accomplish, so we must guard against any complacency. Also, more radical solutions have yet to be tried. At present any attempted improvements are largely of the nature of relatively minor adjustments to current programs.

6. For example the Victorian Social Welfare Department's new Family and Community Services program was developed at least partly on the basis of the findings of the Who Cares? and other studies.

11

Government departments do come in for the greatest amount of criticism and because of their strategic position anyway, they should be the focus of major efforts towards change. Were statutory agencies to provide an effective and acceptable service the welfare deck would be cleared and we would be able to better understand what are the intractable problems that remain. The guaranteed minimum income scheme which was the basis of the Brotherhood of St Laurence's Family Centre Project was designed to fathom this effect. However while it certainly has helped the families involved and provided valuable insights, without wider social structural supports for families and routinisation of the guaranteed financial arrangement, the effects are not likely to be clearly understood, nor can we tell what changes are likely to be achieved by schemes that involve larger numbers on a different basis.

What comes through all the literature and is explicitly developed in Who Cares? is the fact that it tends to be families' inherent resources and their capacity to utilise services which are the most telling factors. This is borne out by overseas studies as well; in fact some develop a similar tripartite scheme of capacity to cope (cf. Perlman, 1975) to the Who Cares? groups of clients, 'the copers', 'the vulnerable' and 'the passive' (McCaughey et al., 1977:17). This is obviously an unsatisfactory state of affairs since the burden is being placed back on those least able to cope with it. Of such stuff are 'cycles of poverty' obviously made. Before we move on to an overview of what we should see as the implications for future policy of the evidence from clients, we shall consider what sort of services are generally approved, as clearly we need to know this if we are to make recom " mendations that seriously take into account clients' judgments.

What do clients evaluate positively? Already we have seen that users dislike services that are impersonal, inconvenient and confusing and particularly the demeaning process of being checked up on. The corollary presumably is that they like the reverse. The evidence bears this out in part though there are some curious aspects of this.

We find that favourable views are held about social workers who present bureaucracy's human face. Unfortunately not many clients in Cunnington's study had had dealings with a social worker but the vast majority of those had found the experience a largely positive one. The aspects of contact with social workers evaluated most favourably were:

The ability of social workers to heLp solve client problems (89.6 per cent). Always being able to discuss problems freely with social workers (87.5 per cent). The ability of social workers to understand problems put ,to them (85.4 per cent). The way clients were treated as a !Person who matters (85.4 per cent).

The way social workers showed concern for clients' problems (85.4 per cent). The way social workers respected the point of view of the client (85.1 per cent). (Cunnington, 1977: 114) A study by Adams sheds greater light on just what it is that clients appreciate. This was a study of twenty-four clients who had had dealings with social workers from the CSSD in Perth. As with Cunnington's study, Adams found that most clients evaluated positively their experience with the departmental social workers, despite the fact that in many cases little or no progress was made towards solving the basic problems confronting them. Most had financial problems and all the social worker could do was relieve their feelings. Listening was a positive feature mentioned by twenty of the clients. Interestingly, this was not perceived as part

12

of a professional service but as 'personalised'. Adams suggests the 'relationship was one of idealised friendship-similar to a friendship relationship but without the shortcomings' (Adams, 1972 : 127). Adams further suggests that 'willingness, concern and advice' are what are appreciated rather than 'problem solutions'

( 1972: 161). In view of the negative reactions we have seen to the bureaucratic face of the CSSD we must ask whether, for these people at least, negative evalua " tions are overridden by their positiv e experience with the social workers. In fact this was not so. Adams did not find that satisfacti on with the relationship with a social worker enhanced a client's view of the department; in fact he suggests that it tends to highlight the agency's shortcomings. Clients see the situation within a 'them and us' framework and identify the worker as one of 'us' (Adams,

1972: 107); the agency is thus available as a scapegoat (1972: 83), which pro " tects the social worker from criticism for non-solution of the problem. While Adams sees clients as being basically naive and having limited expecta " tions (1972:87) Cunnington's evidence suggests that their interpretation of the social worker's role is shared generally by staff members of the CSSD. He points out that about 50 per cent of social workers' work loads involve advocacy on behalf of clients. He also found that this 'type of activity was resented' by other staff members who were responsible for the determination of eligibility

( 1977: 134). Lewis similarly describes the social worker's role in CSSD in N .S.W. as one of 'client advocate' and noted 'a mutually hostile relationship between social workers and other departmental officers having service tasks, in the central office' (1973:282).

This evidence supports the preference of people to approach friends rather than impersonal organisations though we need to be careful about applying such judgments to all situations. It seems that people are happy enough and do not feel stigmatised in situations in which assistance is clearly seen as a right. Children's allowances and age pensions seem to have reached this status in our society, with education an even clearer example and health services possibly moving towards this. CUnnington points out that as wealthier people are included

as recipients of income maintenance schemes the schemes become more generally accepted (1977: 125). This has great advantages in terms of human dignity though, as Harris points out, 'social right policies tend to be more costly than individual need programs' (Harris, 1976:157).

So we come to the conclusion that services should be universal and accepted as a right or/and very personalised. It is obvious then that some radical guaran " teed minimum income scheme that applied to all persons regardless of their status (though possibly related to the universal criterion of age) which was clearly accepted as a right would overcome many of the negative effects of the current system, including significantly simplifying it. McCaughey and her col " leagues point to additional, related advantages:

Because a guaranteed minimum income applies to all, it exemplifies the communal rather than the individual aspect of society and places emphasis on interdependence rather than independence. (1977: 128)

Although some submissions to the Poverty Inquiry made similar suggestions and Henderson proposed a guaranteed minimum income, he also came to the conclusion that the check on lone mothers to see if they were cohabiting with a male could not really be discontinued in fairness to families with two parents present (197 5, vol. 2: 11). He is possibly right if one is merely thinking about tidying up the present system but were all citizens as individuals rather than as

13

families to be offered the same rights to community support, such eligibility issues would be irrelevant. If future changes in technology permanently alter the structure of the employment market such a scheme may have irresistible

advantages. For the moment it is likely to be seen as too radical and too costly. There has been a great deal of discussion7 about guaranteed minimum income schemes and this is a crucial area for future consideration; however this is not

the context in which to debate the merits and problems of the various schemes. All that need be said in the light of clients’ views is that such schemes would

overcome the problems which many face with the present income arrangements. It is worth noting that such a scheme would not appear to have any disadvantage for those receiving help and who are happy with that help. On occasions, solving problems for one group can raise problems for another. It seems unlikely that this would occur with a guaranteed income scheme.

The other major conclusion that can be drawn from the evidence is that if

clients are to receive services that are not seen as a right and therefore run the

risk of feelings of powerlessness and loss of self-esteem, then they prefer the service to be personalised. Ideally such a relationship will involve a reciprocal aspect. If there is true reciprocity both negative results will be overcome. There are a number of ways this type of service might be approximated. Self-help groups ciearly meet the requirements. Localised, generalised services also more closely meet

these criteria, especially if peeple can act as helpers at times. A less radical and immediately feasible way of assisting clients would be to extend the advocacy

role which social workers at present perform. The role certainly would be useful in the CSSD and State welfare departments. Cunnington suggests formally extending the role of social workers, but as well the service might be even more effectively personalised via members of groups with whom clients identify. This has been suggested for migrant groups (see Cox, 1975) for example, and lone parent groups might act in this way. Lewis points to how effectively the RSL has intervened on behalf of its members (1973:445). If the role of advocate

were to be developed, care would need to be exercised to ensure that clients’ basic problems were actually being solved and the personalised service was not being used merely as a soporific. '

A series of administrative improvements could obviously go some way

towards meeting clients’ criticisms, as would additional development of the appeals system, preferably in conjunction with the advocacy system. Urgent improvement is required in getting money to people quickly and over the

counter payments are clearly the only answer in some cases. This has been

the practice in Britain for many years and is current practice in Queensland

and is provided for in Victoria, which suggests that objections based on

problems of accountability are not insurmountable.

The evidence from clients does provide a great deal of information that

has only recently become available. This gives us something of a new slant

on current services and provides some obvious lines for improvement. How-ever in present circumstances we must be wary about seeing this as providing

the ultimate in assessment of services, for reasons which I now wish to explore.

7. There has been a great deal of discussion of guaranteed minimum income schemes. They would require quite radical alterations' 1n our economic and social structure. Most importantly such a scheme needs

to be grounded' m a collectivist ideology and this' IS not generally accepted m Australia. For discussion of some of the' Issues see Henderson (1975), Priorities Review Staff (1975), Saunders (1976).

14

Problems of eliciting and making .use of client opinions

If we accept the assumption that citizens have a right to welfare services8

and that therefore these should be tailored to people’s requirements, it

becomes obvious that we need to know people’s views of the available ser-

vices. There is also the practical advantage that this may avoid misconcep-tions. Professional staff and policy makers often do see things in the» same

way as users. There are many illustrations of this. For example in a study

of Glebe, community problems were rated quite diflerently by three groups, patients, doctors and general staff of the community health centre (Harper

and Morey, 1977z9). However it also becomes the case that finding out

what people think has its own problems, problems which stem from the

current nature of our social attitudes and which will therefore not neces-

sarily be easy to overcome. Much of the evidence I have been drawing on,

though not all, was collected in interview surveys. This seems an obvious

way to find out what people want. However, as a number of the writers

point out there are processes at work that may inhibit people in offering

opinions in at least two crucial ways. The most obvious way is the mere

inhibition of expression of opinion through fear, deference, feelings of

inadequacy or just never having thought about the topic. Since these factors are likely to operate most forcefully among the least powerful we have to

guard against merely perpetuating their disenfranchisement. The other,

more perplexing, though not unexpected, difficulty is that people are not

apparently used to seeing the institutions in our society as in any way

subject to their control. Therefore they rarely think about alterations that

would make the system more suitable to their interests, even when they

recognise problems. Shaw points out that people unclear about what kind

of service is available (and we have certainly found this to be true of Aus-

tralian clients and potential clients) are likely ‘to be still less clear when

asked to make suggestions about improvements in the service . . . thus

producing an inbuilt conservatism’ (1976:26).9 Take first the point about reluctance or inability to express a point of

View. A study of rural poverty among whites pointed out that ‘major

difficulties in obtaining information from the target group were directly

attributable to their politeness, shyness and inarticulateness’ (University of

New England, 1974:65). A second part of the study found the same for

Aborigines. For instance there were relatively few complaints about housing although it seemed to the interviewers to be particularly unsatisfactory.

They suggest this was due to several factors:

One is the need of the poor to preserve their dignity. ‘We were brought up never

to complain’; ‘people who are always complaining and looking for handouts give us a bad name’. Another is stoic acceptance of that which cannot be improved.

(University of New England, 19742126)

This type of problem can be overcome if better research techniques are

developed. Increased levels of co-operation can be gained for instance by

approaches to the sample which are sponsored by people who are accepted. For example Gale and Binnion (1975) in their study of urban Aborigines

8. The age pension was recommended as ‘a right not a charity’ by the Royal Commission on Age Pensions as far back as 1905—06. The official position on benefits and other measures is less clear and the state

of public attitudes is lass clear again. For an excellent discussion of ‘Values in Australian Income

Security Policies’ see Lewis (1975). 9. This point is further discussed by Marris and Rein (1967: Chapter 6) and by Platt (1971).

15

A

overcame such problems by employing Aboriginal 'introducers'. The build ing up of rapport is also important: again Gale and Binnion demonstrate thoughtful methods. They attempted to achieve general acceptance through publicity in strategic places:

Various Aboriginal clubs were addressed to ensure the co-operation of the community and to make the aims of the study as widely known as possible before interviewing commenced. (Gale and Binnion, 1975:8)

Method of interviewing is also important. Gale and Binnion found 'informal discussion was the most effective means of obtaining information' (1975:8) and utilising this method found that 'the majority of Aboriginals show remarkable insight into their situation' (1975:51). There is obviously an advantage in repeated contact with respondents and we find that studies such as Who Cares? in which families were visited on three occasions

(McCaughey et al., 19 77:5) yield particularly insightful results. The Brotherhood of St Laurence's Family Centre Project clearly testifies to the advantages of long-term contact with people for coming to a depth under standing of their views (Salmon, 1975; Liffman, 1975). If we are seriously committed t o understanding clients' points of view, ways of constantly monitoring them must be used. Reticence is also likely to be reduced as people discover that their views are actually going to be taken into account.

Problems of eliciti ng appropriate information are important and require our careful consideration but a more intractable problem arises from the fact that many accept uncritically the current welfare arrangements. We should not be surprised about this if we merely remember how ill informed people are, but this is a crucial point in determining policy, if real partici pation is envisaged.

Lack of understanding and developed points of view were evident in a general survey of 450 people, from a range of socio-economic groups in Melbourne, carried out from Monash University in 1973. The basic con cern of the survey was to discover people's attitudes to poverty and to remedies for the problems they perceived. They overwhelmingly believed poverty exists in Australia and showed considerable sympathy for the plight of the poor. They supported current welfare programs and even extensions of them, but in only 5 per cent of cases did those interviewed make any innovative suggestions about ways of dealing with the problems they acknowledged. Naturally this survey suffers from being superficial and it is likely that it did not tap people's ideas adequately. Nonetheless it is revealing that, while co-operation was excellent and the respondents related well to

the subject matter, they apparently were not in the habit of questioning current arrangements. Clearly the public is not vitally involved in matters of welfare-these by and large are left to 'the experts'. 10 It is possible that the general public would not have well-developed views a bout welfare because, for a majority, there has not been the need to utilise services. However the evidence reviewed to date suggests that the ignorance is widespread and includes many people who might be considered 'at risk' and indeed have considerable experience with services. The limitation of critical

10. This survey was carried out by some students and staff of the Department of Anthropology and Socio  logy at Monash University . A report on the survey has not yet been published but the study is now in a second phase in which a repe a t survey is being undertaken to ascertain any changes in attitudes during the last five years.

16

appraisal of services is graphically illustrated in Jordan's study of 'long-term unemployed people'. Here it was found that when people were asked if they needed any further help, slightly more than half said no: Again and again people who seemed to be in quite difficult si,tuations said no

they didn't think there was anything the Government could do for them, and many of the responses that were obtained were produced in passive agreement to suggestions made by the interviewer. (Jordan, 1975:39)

The staff of the Commonwealth Employment Office were also 'comparatively unproductive of suggestion s' (Jordan, 1975 :40). This clearly supports the findings of the Monash survey and suggests people do not feel involved in the programs provided nor do they look upon welfare provision as part of their rights as citizens.

Jordan summarises this atti tude among the long-term unemployed thus: Many people seemed to regard such assistance as was available to them as resulting from arbitrary decisions which they did not understand, were beyond their power to influence and probably not to be depended on. They expected basically to have to try 'to solve their own problems in the best way they could. (Jordan, 1975:40)

Conclusion Although we now have a great deal of information about what clients think about services we have yet to work out adequate ways of utilising this information . From my analysis, certain recommendations have been proposed though it must be

borne in mind that because the range of services is so broad, detailed analysis in a variety of specific fields is still needed. All I have been able to do is deal with the main broad points, though these are likely to be basic to all services and we do find that these conclusions are largely in line with those of writers who have studied in more specialised welfare fields.

In summary it can be said that there is a great deal of dissatisfaction particularly about the major statutory services and especially among the unemployed, deserted wives and supporting mothers. There are many clients, usually a majority, who do not register dissatisfaction but when we regularly find rates of over 25 per cent who are dissatisfied, as Cunnington's and other studies show, improvements are

obviously needed. We must also bear in mind that clients tend to have low expecta " tions. Jordan, for example, found that the average response to the question of satis " faction with help received from the Commonwealth Employment Service was on the 'mild approval' side of 'indifference'. However he adds that clients expected 'comparatively little' (1975:41). Long-term unemployed people, such as Jordan studied, are likely to have very difficult problems, but then welfare systems are presumably designed to deal with people's difficult problems. Also, other studies such as Cunnington's ( 1977) and Brewer's (1975) report similar findings and

they were concerned with a more mixed group of unemployed people. From the analysis of dissatisfactions and satisfactions we gain an idea about how services might be altered to be more in line with users' interests. Immediate r.ction can be based on this. We also learn, though, that clients and the general public are not very critical, presumably because they have recognised their

powerlessness to control services. Therefore strategies other than asking the client will be required if we are to break out of a conservative mould and develop innovative services which are more suited to people's needs. Of course, ultimately we may have a society so suited to people's needs that welfare services as we know them today would not be necessary. While such ideas are unashamedly utopian they have a salutary effect if they are viewed as ultimate goals.

17

There is plenty of scope within the present framework to steer services so that they do meet clients' criticisms. It seems that in some circumstances today statutory agencies are more concerned with eligibility requirements than meeting people's needs though the official organisational goals are designed to meet those needs. Excessive concentration on eligibility requirements and bureaucratic pro " cedures often unjustly deprive people of entitlements, make them wary about applying or at least delay the receipt of pensions or benefits. The attitudes of staff are based on fairly generally held ideas about the 'worthy' poor and this. explains why the unemployed and supporting mothers feel particularly badly treated . Harshness and unjustness of attitudes to the poor are pointed to by Tomlinson who asks 'why is there so much surveillance of welfare clients and so comparatively. little tax surveillance' (1975:202). The government is attempting to redress the balance at present, though one must be sceptical of the outcome. Control of the least advantaged in society is easier to maintain.

Eligibility for services is at present based on notions of 'deservingness'. When one stops to think, people don't need money because they are unemployed or supporting mothers or whatever else. They need money because they need to eat,. have shelter and clothing and other accepted requirements for living. At present our schemes spend a disproportionate amount of effort on administrative matters which would be overcome, along with a great deal of their stigmatising effect, were there a guaranteed minimum income scheme. Of course government policies are part of total social configurations and it is not clear that the public is yet ready to accept such a scheme. However we need to continue raising its advantages. In being over concerned with eligibility criteria the point is often lost that what is saved by one department through eliminating payments to 'welfare cheats' might finish up costing the community more at a later stage if a family's situation deteriorates.U

Apart from a general easing and simplifying of administrative procedures, a humanising of services is needed. Social workers in government departments already act in this way so the context obviously does not preclude it. Were depart " mental staff trained and otherwise encouraged to see the organisational goal as helping people this might go a long way towards providing a more acceptable service . At present it seems that there is often:

. . . too much focus on the internal needs of the organisation to undertake its task of administering the Social Services Act, without sufficient attention being paid to " the needs of the environment and the people for whom the service is provided. (Lewis, 1973 :450) If clients are largely uncritical we must also devise other means of attempting improvements to services. We need to have open minds to, and encourage, a variety of suggestions , to canvass these widely and at times to institute trials. We should then find out what clients think of the new services and in this case they can offer opinions on a more secure basis, rather than merely responding to hypothetical situations , which is what usually happens in research interviews.

Flexibility of services is also essential. This allows services to change in response to clients' judgments and to changes in circumstances. Another principle worth pursuing is variety, as this allows people to exercise choice. We know that people have different needs and different views and therefore different requirements. This

point comes out clearly in the reports of clients' reactions, though it is one I have not had space to document here. Choice tends to be something that is available to the wealthier citizens of our society so should be offered also to the less wealthy. 11. For a discussion of this see Tomlinson (1975).

18

When we consider what individuals think of welfare services we can shortly .answer that many are not impressed. We have a society that is affluent enough to spread benefits more evenly and we at least, in principle, have a society that ;believes in greater equality and improved quality of life for all. Ultimately we :must get beyond thinking of consumers, users or clients of services since this

implies a group who is 'acted on' by a more powerful group, whose role it is to make and execute policy. Even if this policy is benevolent such a situation is ∑ not equalitarian. Taking clients' views into account is a step in the right direction 'but it is still a long way from a fully participating society of equals. Utopian _goals are often scoffed at because they are unrealistic, to me they are essential as 'lthey point the way to our destination, however far off that may be.

19

REFERENCES

Arnstein, Sherry R. 1969 'A ladder of citizen participation in the U.S.A.'. Journal of the American Institute of Planners, 35 (4) July.

Australian Council of Social Service 1975 'Consumer groups and their views on welfare services'. Pp. 3-34 in Consumer Views on We/fare Services and Rented Housing. Commission of Inquiry into Poverty. Canberra: AGPS. Australian Red Cross Society

1975 Chronic Poverty City and Country Families. Canberra: AGPS. Benn, C. 1973- Family Centre Project: Progress Reports 1-6. Fitzroy, Vic.: Brotherhood of St Laurence. 76

Brewer, G. 1975 Workers Without Jobs. Fitzroy, Vic.: Brotherhood of St Laurence. Brown, J. W., Roisin Hirschfeld and Diane Smith 1974 Aboriginals and Islanders in Brisbane. Canberra: AGPS. Bryson, L. and F. Thompson

1976 An Australian Newtown: Life and Leadership in a Working Class Suburb. Malmsbury, Vic.: Kibble Books. Burbidge, A. and J. Brown 1976 Welfare Services in Brisbane. Canberra: AGPS.

Carlton Centre 1970 Report No. 4: Social Work and Neighbourhood Needs. University of Melbourne.

Cass, M. and R . Sackville 1975 Legal Needs of the Poor. Canberra: AGPS.

Cox, David 1975 'The role of ethnic groups in migrant welfare' . Pp. 3-149 in Welfare of Migrants. Commission oflnquiry into Poverty. Canberra: AGPS . Cunnington, H. J.

1977 'Communication between welfare agencies and clients' . Pp. 69-147 in The D elivery of We/fare Services. Commission of Inquiry into Poverty. Canberra: AGPS .

Department of Sociology, University of New England 1974 Rural Poverty in Northern Ne w South Wales. Canberra: AGPS.

Duigan, M. G. 1975 A Study of the Hindmarsh (South Australia) Community. Canberra: AGPS.

Evans, E. C . 1975 'The A.A.S.W., social workers and social change'. Australian Social Work 28 (4) D ec.: 5-10.

Fitzgerald , R. T. 1976 Poverty and Education in Australia. Canberra: AGPS .

Francis, R . 1976 'A community study: Fremantle'. Pp. 21-75 in Community Services: Four Studies. Commis " sion of Inquiry into Poverty. Canberra: AGPS.

20

Gale, F. and J. Binnion 1975 Poverty among Aboriginal Families in Adelaide. Canberra: AGPS . Goffman, E. 1963 Stigma: Notes on the Managem ent of Spoiled Identity . Englewood Cliffs , N.J.: Prentice Hall. Hamilton-Smith, E.

1975 'Issues in measurement of "commun ity needs"'. Australian Journal of Social Issues, 10 (1} Feb.: 35-45. Harper, A. C. and C. Morey 1977 Glebe: Community Needs and Service Organisations. Canberra: AGPS . Harris, C. P.

1976 'Income security programmes and the philosophy of social security policy'. Australian Journal of Social Issues, 11 (1): 157-73. Henderson, R. F. 1975 Poverty in Australia : First Main Report, vols 1 and 2. Canberra: AGPS . Hill, K .

1975 A Study of Aboriginal Poverty in Two Country Towns. Canberra: AGPS . Illich, I. et al. 1977 Disabling Professions . London : Marion Boyars. Johnson, T. J.

1972 Profession and Power. London : Macmillan. Jordan, A . 1975 Long Term Unemployed People under Conditions of Full Employment. Canberra: AGPS. Lewis, M argaret T.

1973 Interest of the Consumer in the Philosophy and Ad ministration of the A ustralian Income Securit y Program. Master of Social Work Thesis, University of N.S.W . Lewis, Ma rgaret T. 1975 Values in Australian Income Security Policies. Canberra: A GPS . Loder and Bayly Planning and Engineering Consultants

1974 Berwick Planning Study, Vo lume 2: Final Report. H awthorn. McCaughe y, J., S. Shaver, H . Ferber and O thers 1977 Who Ca res? Fam ily Problems, Co m munity Links and Helping Services. South Me lbourne: Sun Books. Ma rris, P. and M . Rein

1967 D ilemm as of Social Reform. N.Y .: Atherton Press. Matza, D . 1969 Becoming Deviant. Englew ood C liffs, N .J.: Prentice Hall. M embers of Client Power

1972 'An open letter to our politicians'. Australia n Social Work, 25 (1), March: 24-8.

Murrell, T. and J. Ross 1977 'Health care for infants and mothers'. Pp. 69-88 in Planning and Health Care for Infants and Mothers. Commission of Inquiry into Poverty. Canberra: A G PS. O 'Neill, J. and R . Nairn

1972 The Have Nots: A Study of 150 Low Income Fam ilies. Fitzroy, Vic.: Brotherhood of St Laurence. Perlman, R. 1975 Consumers and Social Services. N.Y.: John Wiley and Sons.

Platt, J. 1971 Social Research in Bethnal Green. London: Macmillan. Priorities Review Staff 1975 Possibili ties for Social Welfare in Australia. Canberra: AGPS.

Rainwater, L. 1974 Deviance and Liberty: A Survey of Mod ern Perspectives on D eviant Behaviour. Chicago: A ldine. Ryan, W .

1971 Blaming the Victim. N .Y.: Pantheon Books.

21

Salmon, J. 1974 Resources /or Poor Families: An Experimental Income Supplement Scheme. Canberra: AGPS. Saunders, P.

1976 'A guaranteed minimum income scheme for Australia: Some problems'. Australian Journal .... . of Social Issues, 11 (3) Aug.: 157-73. Shaw, Ian 1976 'Consumer opinion and social policy: a research review'. Journal of Social Policy, S (1):

19-32.

Tinney, P. M. 1975 A Home away from Home? A Comparative Evaluation of Three Melbourne Family Day Care Projects. Fitzroy, Vic.: Brotherhood of St Laurence. Tomlinson, John

1975 'The importance of being worthy'. Australian Journal of Social Issues, 10 (3) Aug.: 197-207. Tomlinson, J. 1977 Is Band-aid Social Work Enough? Darwin: The Wobbly Press. Vinson, T., R . Home} and R. Bonney

1976 A Community Study: Newcastle. Canberra: AGPS.

22

The evaluation of health and welfare services in Australia: agencies

1. The views of agencies 1.1 It is important to say at the outset that it is not possible to make an authoritative statement on the view of agencies on the question of evaluation. Realistically, very little work has been done on the evaluation of welfare agencies and virtually nothing at all on the evaluation of the of evaluation of welfare agencies in this country at least. This paper can therefore be little more than the distillation of the thinking of some administrators of welfare agencies and of the secretary of a co-ordinating agency of welfare services seeking to introduce or improve the evaluation process.

1.2 It is regrettable that this document could not be a closely reasoned state " ment on the basis of a number of completed research exercises. It is important however to note that this is not possible at this stage; the information is just not available even about the comparatively few agencies that have approached

the issue in a thorough-going way.

1.3 Australia certainly has not progressed very far in evaluating its welfare services. The writer is not in a position to make any comment about the state of the art for health services.

1.4 It is now possible to anticipate some of the problems that can arise when evaluation is being considered and so hopefully avoid the pitfalls by learning from other people's experience. It is with this thought in mind that this paper has been prepared. It seeks to look at some of the possible responses from within an agency to the practicalities of an evaluation process and the reactions of board, administration and staff when a program of evaluation is being planned and introduced.

2. The non-government or voluntary agency 2.1 The paper will look only at the non-government welfare organisation. This kind of agency is not easy to define precisely beyond the fact that it is not part of a government instrumentality and it does not operate for profit. Both factors have a bearing on the kind of staff it attracts. The fact of being non-government,

non-profit making plus a particular kind of staff may well influence reactions to requirements of accountability and the evaluation of the services it provides.

2.2 These organisations are usually fairly small with recurring problems of remaining financially viable. Money comes from a variety of sources that are completely independent of one another and with each unaware of the stresses occasioned by the variations in the flow of money from the others. These include Federal, State and local government, other supporting agencies and the agency's own fund-raising efforts. Staff establishments are therefore frequently kept lower

23

24895/79-3

than desirable. There is no guarantee that money available in one year will be provided in the next. Permanency of tenure is unknown and even the benefits of superannuation are uncommon.

3. Gathering of infonnation 3.1 The uncertainty about the income and its paucity usually means that research activities receive a very low priority. The basic information that is required for research is usually left with the clerical staff who cannot rely on the full co-operation of the staff delivering the service. While it is helping activity that is to be counted the counting activity is not seen as important. It is often thought that this time should be utilised in 'helping'. Even after the information has been collected there is no certainty that it will be appropriately analysed and utilised. It may only be used to provide the basis of a report to the governing body. The board may itself not perceive the importance of certain information potentially available to it or it may not even know what information it can obtain on request from the administration.

4. Measuring effectiveness 4.1 There are very few instruments for measuring the effectiveness of work done in the we lfar e field. It will be appreciated that welfare covers a very wide area of activity and is concerned with the well-being of particular members of society.

What well-being means and how well it has been restored or achieved in a particular instance is very hard to measure and to express in quantitive terms. The success or failure of an intervention by a welfare organisation has been very difficult to assess. The agency or the worker may simply fall back on an assumption that one such approach usually works in these cases. The method of ascertaining the validity of that assumption may be little more than the recalling of incidents where it seemed to the staff that good results had been forthcoming in the past. In many situations accurate measuring techniques are yet to be developed.

5. Funds and standards of service 5.1 The welfare agency is required to demonstrate to funding bodies that it is using the money provided in an effective manner. It must also make a case from time to time that it requires extra money so that services can be expanded or improved.

5.2 The case for an increase in funds is not always and perhaps not even usually related to the effectiveness of the work that is currently being carried out. For example, if it were considered desirable to increase the number of staff to provide a service for disadvantaged children in a specific area it is common for applicant agency and funding body alike to focus on the number of disadvantaged children in the area. Even if a similar service to the one proposed is being offered elsewhere the effectiveness of the existing service may not even be considered . It is apparently assumed both by those preparing and assessing submissions that it is not possible to evaluate effectiveness. Evaluating a need will receive attention but evaluating the ability of a service to meet that need will not.

6. Welfare effectiveness and business efficiency 6.1 There are parallels between welfare and business. There are times when one should approach a problem of a welfare organisation in the same way as one

24

would approach a business problem. Often the members of a board of manage " ment of a welfare organisation are businessmen and very familiar with the techniques of business administration but uncomfortable with the thought and jargon of the professional welfare worker. The board needs to know the number of people served. It cannot allow arguments of quality to obscure the fact that few people are benefiting from the service. The board also should be concerned

to establish that the standards are high. There is no point in delivering a poor service no matter how extensive the clientele receiving it. Effecti veness of service and numbers of people served must be considered together. The principles of business administration are valid in welfare administration even if quality control is not as easy to establish in the latter.

7. Justification

7.1 We are entering a stage of history where more and more welfare organisa " tions both government and non-government will be required to justify their existence. They will be required to defend any schemes for expansion with solid information and the nature and extent of problems and the effectiveness of the means they propose to combat them. Money is in short supply and a growing number of peopJe are questioning the value being obtained for the welfare dollar . Such criticism may be ill founded; it is becoming necessary to be able to demon " strate that public money is being used efficiently and effectively. Such a demonstration will be necessary at all levels of the welfare system. Many voluntary organisations, although by no means all, are aware of the need. The question then is how will these organisations respond when evaluation is mandatory.

8. Evaluation and voluntary agencies

8.1 Administration

8.l.a The administration of voluntary agencies is usually very limited in size. Frequentl y the administrator is hampered by the lack of faciliti es to direct the agency in carrying out its function effectively. In the thinking of some organisa " tions, although this is rarely set down explicitly, administration is only a

necessary evil to keep the organisation moving with some degree of unity. Invest " ment in administration in this thinking should be kept to a minimum since this really is not an essential part of the organisation's work. It is required then to obtain the best possible results from its staff but it may not always receive the

support it requires to carry out its task.

8.l.b The administrator of a voluntary welfare organisation is in a difficul t position . He often will appreciate the need for evaluation and yet not be able to introduce proper evaluation into the organisation .

8.l.c In one sense evaluation is an administrative task but it is not necessarily a skill possessed by administrators. Rarely will there be a person already on the administrator's team to whom such a task could be delegated.

8.l.d Administrators in welfare organisations are often overtaxed. Many will not have the time or the inclination to take the initiative in proposing an evaluation process for the organisation that they administer, especially if it will probably cause trouble within the organisation. Perhaps it is true to say that the adminis "

trator will take the initiative to introduce evaluation when he feels that his day-to-day problems are more or less solved. To him evaluation belongs in the

25

category of a luxury; he will usually not see it as a key to help him overcome some of those chronic day-to-day administrative and organisational difficulties that take up so much time. The value of evaluation must be demonstrated to him first.

8.l.e What the administration can do is to improve the effectiveness of the measuring instruments used by the agency. Many welfare organisations routinely collect statistics about their activities but because of their limited resources these instruments are often out of date, irrelevant and not highly prized by the staff who are required to complete them. The organisation will not invest time and money to update this system unless and until it sees the importance of precise measuring instruments.

8.1.f The collection of accurate information is the important first step in evaluation. It is the only way by which an agency can become aware of where it is placing its priorities of staff and money. There is much room for improvement in this regard in many agencies and a general awareness that the present system is unsatisfactory. The next step is to provide them with a model that they can adapt for their organisation. With a better system they will have the beginnings of

a system of quantity and quality assessment.

8.l.g Administrators are usually pragmatists before they are philosophers . It is necessary to convince them that evaluation can be carried out in a limited way with a fair degree of success and that success can be of real assistance to the agency. Argument about the theoretical importance of evaluation as part of accountability is less likely to impress.

8.2 Staff 8.2.a It is difficult to make general statements about the members of staff of welfare agencies. They differ in character from one agency to another. Some are volunteers with only very basic training to prepare them for the task that they are required to carry out. Some agencies have professionally qualified staff, either social workers or welfare officers, and sometimes a psychologist, or even a sociologist. Most welfare organisations will have a mixture of staff, some volunteers, some trained, some not, some professional staff at various levels of training. Most agencies also have some clerical staff whose important role in evaluation and measurement is often not fully appreciated.

8.2.b It is not possible to be dogmatic about the reaction of staff to evaluation. Of course it would differ from person to person and will be appreciably affected by where they stand in the organisation. There will also be differences depending on whether the person is a professional, a volunteer, or a clerical member of staff. The staff are most likely to be threatened by suggestions of evaluation unless a considerable amount of preparatory work has taken place to inform them of the purposes of the evaluation and the way the information obtained in the evaluation process will be used. It is reasonable to assume that many staff will not welcome an evaluation if they think their job or their progress in the organisation could be threatened. It may, however, not be easy for individual members of staff to publicly oppose a proposed program of evaluation. Lack of opposition does not then necessarily mean that staff will give a program their enthusiastic support. 8.2.c Realistically evaluation carried out properly will provide information about weaknesses in systems and in people. It would be short-sighted to forget

26

'the element of threat that this implies. Any high-sounding announcements about team spirit and the morale of the agency as a whole or even the better service to the client is not easy for a staff member to accept if he may end up by joining the ranks of the unemployed and possibly becoming a welfare recipient himself.

8.2.d The paradox of this situation is that while the manual worker can and will resist time and motion studies the welfare worker is not permitted to do so. The expectation is that the welfare worker is basically concerned about the quality of service provided to the client and therefore that he or she should enthusiastically support any evaluation of his or her work.

8.2.e On the other side evaluation has a meaning only if it provides informa " tion and does so accurat ely. If the agency is not working well then staff arrange " ments may have to be altered and possibly some staff dismissed. This is not common in welfare organisations. Dismissals are rare. The assumption seems to

be that there is a degree of dedication that removes the problem. This however does not get around the incompetent person whose contribution can be harmful.

8.2.£ It would certainly be ludicrous to suggest that evaluation would never have any effect on staff arrangements. It would be even more ludicrous to sug " gest before the evaluation took place that no matter what the information that comes forward no staff person's position would in any way be threatened.

8.3 The board 8.3.a As with all other parts of the voluntary social welfare organisation there is no one kind of board that one may describe as typical. Some agencies do not have a board and the function normally carried out by the board is performed

by one or two individuals. Other agencies have very large boards of up to twenty people. They may meet once a month or once in three months. In general they are charged with the oversight of the organisation. They have the final responsi " bility for the activity of their agency. They review the financial administration and must approve large or unusual expenditure.

8.3.b It is probably true to say that the average board of a social welfare organisation is made up of two different types of people. The first are social welfare professionals; the second are business men and wom en. There is a real danger that the differing contribution s of the 'expert s' and the businessm en is not properly integrated and that the agency does not therefor e gain in a coherent way from a carefully formulated policy. Such policy would need to be based on good social welfare thinking and the use of good business techniques. The two

sides for example wopld place a different emphasis on the standards of service and on the number of people served. Clearly both are important as both sides would readily admit. The problem is to achieve a satisfactory balance of the two and yet this becomes a problem when the thinki ng of one side is not properly

valued by the other.

8.3.c Oearly a number of boards are not well designed. The people who are members are not sufficiently aware of their function and of the best method of proceeding. The result is that the board fails to demand an adequate picture of the agency's activities and does not provide the close critical analysis that is necessary for healthy growth or even healthy maintenance.

27

8.3.d Many boards then would be unaware of the possibilities of effective evaluation. Members may too easily assume that existing data collection is as good as it is possible to achieve with the money they are prepared to spend on it. Boards are probably not being challenged by the administration to take a more radical view of the quality and quantity control of the agencies operating under their direction.

9. Conclusion 9.1 Evaluation is gradually becoming an important word in the vocabulary of non-government social welfare organisations. While it may be seen as desirable in theory it is still receiving a very low priority in practice.

9.2 Evaluation suffers from the myth that it is necessarily an expensive opera " tion and that absence of measurements of quality and effectiveness is an insuperable obstacle.

9.3 The non-government social welfare agencies have difficulty in allocating money for evaluation especially when board members are yet to be convinced that evaluation is possible and necessary.

9.4 The reactions of board, administration and staff will predictably each be different. Unless and until at least boards and administrations are convinced that evaluation should be an integral part of every program there is little chance that agencies will take a strong initiative.

9.5 Evaluation may become a significant part of the welfare scene only after it has been routinely built into social welfare legislation. This would be a most important development that could lead to improvement of measuring techniques, better use of the results and a strong impetus to evaluation generally. It could be expected that we would gain in the long run a better standard of service throughout the social welfare and non-governmental alike.

28

Evaluation in relation to standards of performance

Introduction Evaluation is a difficult word to interpret and other papers will be dealing with the question of definition and the purpose of evaluation in general terms. How " ever, there are different philosophical, educational and practical issues which affect the particular uses of evaluation, including evaluation in relation to standards . I shall therefore briefly place my comments about evaluation in a more general context before dealing with the specific topic. This context will affect how I interpret 'standards' and 'performance' .

Significance of 'evidence' in the human services area The ma jor problem with evaluation in the human services area is the character of the evidence. Whatever the purpose, whatever the method some evidence has to be sifted. Therefore one's attitude to evaluation, for whatever purpose, is related to attitudes to 'evidence'. This includes whether one believes that all evidence m ust be 'objective', and that, for example, objective means quantified, or produced by someone 'outside' the participants in the process. In addition, attitudes to what is required are influenc ed by whether one holds a positivist or idealist view in general philosophical terms.

These two terms are used in their technical sense and idealist does not mean 'idealistic' in the everyday use of the word. The idealist position, that social reality is related to the subjective realities of the participants, has been similar to views held by a significant section of social work thinking, i.e. problems must be under " stood from the inside. In contrast, many psychologists, especially those with a behaviourist approach, have tended to favour the positivist view-that is, that tangible behaviour change can, or should, be measured, and that replicability and transferability of findings is possible.

The importance of recognising the significance of fundamental philosophical views can be seen in such a publication as Evaluating the Helping Services 1 where the dichotomy is seen to be between theories of personality, for example the psychodynamic theorists and social behaviourist theorists. With the more recent

tendency to couple health and welfare, the attitudes to experimental methods, replicability, 'hard' facts, which have characterised some secti ons of medical research, have spread to the 'softer' area of 'welfare'.

Relationship of health and welfare The question of the relationsh ip of health and welfare and the problems of boundaries will not be dealt with in this paper, although they affect the question of assessing standards, particularly the question of what one is assessing and who

I. Evaluating the Helping Services. Department of Employment & Industrial Relations: AGPS, Canberra, 1977.

29

should be involved in the assessment process. There is now a tendency to use both words so broadly that they can equally be taken to encompass the other. The literature is extensive on this question and will be dealt with in other papers. For the purpose of this paper I shall be referring to welfare in a broad sense, for example as in the United Nations definition of 'the broad range of socially sponsored activities and programmes directed towards community and individual well-being'. The interest in welfare, although not recent in academic terms, is relatively recent in political terms. The extent of money spent, especially if one uses the term welfare broadly, now places it in the big spending area.

Present emphasis on costs: why evaluate? Welfare is now recognised to be an 'expensive' item, and a related consequence is that at times when money is scarce the need to examine welfare expenditure begins to be stated more firmly. The danger is that evaluation shows signs of only becoming to be regarded as important at the same time as a need to be careful of finance is being stressed. In the same way, questions of accountability receive more prominence at times of financial stringency. 2 The result of this linkage can be that 'hard' facts and 'object ivity' arguments, which are related to discussion of costs and numbers, receive pre-eminence. This is a reality and a 'fact of life', but it also influences the question of what evidence is needed to evaluate standards of performance. Unfortunatel y the arguments on both 'sides', the soft and hard, the objective and subjective, the positi vist and idealist, the psychodynamic and the behaviourist theorists, tend to dichotomise and often become extreme. This often happens because each 'side' states that it needs to redress an over-emphasis on the other aspects. In times of economic crisis, the trend tends obviously to be towards the 'hard' side.

There is undoubtedly an emphasis at present on the need to evaluate to ensure that the 'taxpayers' money' is well spent. This point came out very clearly at a recent conference on Evaluation of Community Health Centres which I attended, where all the purposes, methods of evaluation were presented, but the assessing of costs was inferred to be the only 'realistic' one 'at the present time'. The result was unfortunately that most parti cipants immediately neglected evaluation for any other purpose but 'proving' that the particular service, i.e. Community Health Centres, 'cost' less than 'other' (rather vaguely defined) alternatives, and were asking for help with political, public relations techniques rather than evalu " ation techniques.

It is difficult enough to construct an evaluation process which measures 'before and after' in the traditional health area, but the problem becom es enormous when one tries to prove that a welfare intervention prevents some social dysfunction. Evaluation as an experimental exercise, and as an influence on social policy will be dealt with elsewhere but it is difficult to separate out the different purposes of evaluation. They are all interrelated and one aspect cannot be given priority and all others neglected.

The question of why and what is achieved is linked to how, i.e. the process as well as the outcome. This question is related to whether one stresses 'prevention' or improvem ent of social functioning . The present climate tends to bring about a

2. This is similar in Australia to overseas experience, e.g. 'The Crisis of Accountability', E. Newman & J. Turen, and 'Social Services: The Problems of Accountability': G. Hoshino (ed.) both in Social Administration, S. Slavin, Haworth Press, 1978. An Approach to Measuring effectiveness of Social Services: James H. Ward : in AdminiJtration in Social Work, vol. I, no. 4, 1977.

30

desire to promote a service because it prevents a dysfunction. The classic examples are where parole may be 'justified' on grounds that it 'prevents' recidivism; that giving of financial assistance keeps children out of institutions; that providing Community Health Centres prevents so many people using hospital services, or that Meals on Wheels keeps elderly citizens out of homes.3 Instead of relating the arguments to the provision of 'better' services, that is standards arguments, the

(apparently) more quantifiable prevention arguments are used. I say 'apparently', as these arguments, although often regarded as politically more acceptable, can turn into an Achilles' heel when attempts are made to 'prove' the prevention of a dysfunction, with all the difficulti es of controlling variables. 4

The prevention arguments will be dealt with elsewhere, but even within that particular area the question of performance and standard of service cannot be ignored. For example one needs to know whether parole is irrelevant to recidivism as a method of helping, as such, or whether it is the quality of the service given which brings about the negative results, i.e. if the answer is a null result should one do away with the service (keep prisoners in jail) or do something about 'improving' standards.

Another consequence of the stress on costs is that evaluation may be seen as only relevant when money is short. I should like to emphasise the fundamental point that no matter how much money is available, even if the mythical 'bottomless purse' existed, there would need to be evaluation of standards of performance,

for the sake of the recipient of the service. This is an issue of responsibility as well as accountability. There are problems in ensuring that those providing a service are accountable to the recipients, the clients; especially in some areas of welfare such as parole, child care5, but there still remains the question of commun ity responsibility to the client to see that the 'best possible' service is given. These

are vague general terms and it is because of the vagueness and generality that so many problems arise, but the responsibility to be continually striving towards improvement of standards is an ethical one, i.e. involves a value stance which is vital to all discussion of standards.

This does not mean that one does not recognise as a 'fact of life' that less money usually creates an extra need to make sure that the available money is put to the best possible use. The basic responsibility is still there. There are also short-te rm and long-term issues involved, with the stress on short term in a difficult economic climate. However, the stress on short term, on the 'quick and

dirty', with the long term only tolerated if it's not too expensive. should not be accepted as an immutable political reality. This question of political implication also relates to internal or external evaluation (see p. 36).

Evaluation and data collection As stated before, all purposes of evaluation interrelat e and this interrelationship is evident in the collection of data. In many cases a slight addition to current data is all that is needed to widen the value of the information collection . A related

problem exists where no alterations are made, and data collected for one purpose

3. This issue was the subject of some intere sting attempts at 'proving' prevention in the Victorian Social Welfare Dept, in relation to Family Supportive Grants (Material Aid) and prevention of children going into care. 4. This is dealt with in an intere sting article 'Interpreting Null Findings of Evaluative Studies': Journal

of Social Service Research, vol. 1 (i), Fall, 1977, A. Rosenblatt. 5. The problem of accountabilit y to the client has preoccupied the social work literature recentl y, e.g. Hoshino, op.cit. discusses this.

31

are distorted so that they can be used for other purposes. As an example, quanti " tative data collected for managerial purposes such as accommodation, staffing, may distort conclusions if used to evaluate standards. More rooms, more clerical staff, more clients seen do not necessarily mean a 'better' service. Informational needs are not evaluation needs.

This point came home to me more clearly in a recent discussion when some members of staff in one agency said they were 'already evaluating'. When this was analysed it meant they were already keeping some informational statistics about client numbers and how their time was spent, e.g. office interviews, home visits, etc. Naturally collection of this sort of information is fundamental as a first step, but the evaluation depends on what one does with and learns from these data and other evidence.

Evaluation of standards: what are standards? There are various definitions of standards and one can use the basic economic or management literature or that applied to the welfare field. In the former, not unexpectedly, the emphasis is on quantity as well as quality and standards are seen very much as part of what is usually called the control cycle. 6 In welfare literature, standards are usually seen as measurements by which the quality of services may be assessed. The question of numbers of clients seen is a typical example of the problems of the quantity versus quality issue. One would not assess that the standard of one agency's service was 50 per cent higher than another because 50 per cent more clients were seen per worker in that agency. In fact, if the servke were a counselling one such as marriage guidance for example, the instinctive reaction of practit ioners would be to state that the service standard would be lower with more clients, though not necessarily in direct mathematical

proportion. However, the point remains the more clients were seen and the rela " tivity argument arises, i.e. is it 'better' to give a lesser service to more clients. This is a constant problem in both health and welfare, and involves the question of whether the service also has long-term advantages, or research value in itself. This is the type of dilemma which faces such projects as the Brotherhood of St Laurence (The Family Centre Project) in Victoria. In order to make this decision, which is difficult enough at any time, there needs to be some evidence that the service really is 'better' 7, that the standards really are higher.

The question is therefore not only one of minimum standards but of many gradations, leading through to optimal as an ultimate goal-i.e. 'the best possible service'. This relates again to the short-term, long-term issue. Often the suggestion is that m inimum standards are used as a guideline for short-term evaluation but that optimal standards are kept in mind for long-term planning. Again these arguments are particularly used in times of financial stringency. However, I suggest that these are not exclusive and if evaluation is seen as a continuous process optimal standards are not ignored at any time.

Financial and physical standards are easier to quantify, and fall more happily into the area of 'hard' data. It is quite easy to set down minimum physical standards, for example that day nurseries will have X amount of space, Y toilets, built to certain specifications, etc. It is also relatively easy to state that certain generally accepted qualifications are needed (e.g. triple-certificated nurse in charge). When other qualities are required, for example, concern for children ,

6. e.g. as in J. Litterer: The Analysis of Organisations, J. Wiley & Sons, New York, 1965. 7. The danger is that the 'better' may only be the more interesting to the provider, or fits in with the personal preferences of the practitioner.

32

the non-measurable aspects again raise the bogeys of objectivity, quality rather than quantity. However, even those who stress that costs are a major factor, usually make reference to the need to consider cost-benefit analyses and recognise the need for balance. This issue has come to the fore in the recent popularity of zero base budgeting. ZBB recognises that one looks at both efficiency and effective " ness, and the decisions as to whether current activities be eliminated or reduced

to find higher priority new programs or to reduce the current budget are still based on a decision unit system which allows for alternatives which consider concern about standards and quality. It should be noted that ZBB and PPBS (now largely discredited) have some problems in common. One of these is a non-recognition of the complexity of the 'soft' governmental area (especially the

'softest', the social services), which means that a greater degree of sophistication and dedication is required of all the participants in the process. This non " recognition could cause the collapse of ZBB as much as PPBS. 8

What is meant by performance Performance usually refers to the actual delivery of the service. The Victorian Council of Social Service sees a useful distinction provided by the terms 'per " formance' and 'provision', where the latter relates to the range of services and the forn1er to the particular individualised service. E ven in the area of performance

one has to decide what elements make up the performance; that is, not only the actual service delivery but performance in the area of man agerial, financial and administrative com petency as well. Is the standard of performance rated according to output or outcome, or by the w ay of activity is carri ed out, t he input and

throughput? The criticism of one management consultant is that those concerned with the human side o∑ f production tend to describe them selves, and rate them " selves, in terms of input rather than output or outcome.9 Clients often tend to rate services by the amou nt of individualisation and

attention that is given to their problems, by the way that they receive or perceive the service. 10 This latter aspect of client perception, as well as provider perception , is fundamental in assessing expected standards, i.e. expected by whom?

Who decides on standards This question is discussed extensively in the welfare literature, and is now beginning to be considered even in the more traditional health areas, although the challenge to expertness and 'professionalism' in health is not nearly so wide "

spread. The question asked in welfare is whether the 'success' of the service delivery, the standard reached, is judged by the service provider, the controller of resources (provider of funds), by the recipient or determinant of dema nd (the client or consumer)!\ by some external body, set of values ('professional' or otherwise) or by a combination of some or all.

Just as the goals by which we evaluate effectiveness are set, either explicitl y or implicitly, by various groups and influenced by a complexity of factors, so 8. This point is made by G . Otten in Zero-Based Budgeting: Implication s for Social Services. Adm inis " tration in Social Work, vol. I, no. 4, 1977.

9. W. J. Reddin, Effective Management Job Description s: Personnel Manag em ent, Oc tober & Decem ber 1975 and March 1976. 10. e.g. C . Mayer & N. Timms, The Client Speaks, Routledge & Kegan Paul, London, 1970. A local project with the Social Welfare Department and Mon ash Universit y will be looking at client perceptions

of the service . 11. This three part approach is discussed by D . Donnison et al., Social Policy and Administration, Allen & Unwin, London, 1965.

33

also the performance standards, minimum or optional , are set by a complexity of factors, implicit more often than explicit. A very interesting example can be∑ seen in the expectations of performance of workers in youth clubs, commun ity centres. A local councillor may be extremely critical of a youth worker 'with<

long hair and radical views' wh o allows the memb ers to 'run wild' because of the concept he holds of how a youth worker perform s, which is in turn related to what he sees as the purpose of youth clubs. A gain the 'preventive' issue arises. If the councillor sees the club as 'preventing delinquency', keeping youth 'out of

jail' or 'off the streets ', his assessment of satisfactory performance will be∑ different from the youth wo rker who sees the club's purpose as 'enriching the life' of the youth or 'transfe rring power'.l 2 These examples illus trate that once one goes outside simple physical standard s the suggestion that there is a simple immutable standard, minimum or otherwise, . is obviously false. A ll that can be accepted is that 'standards' in many areas of welfare, in particular, are complex, dynamic, value laden and subjective. What ∑ can be done is to combine a number of factors to evolve a standard acceptable to various groups at a particular period of time. Even if the exercise is a difficult one it has to be done for the basic purpose m entioned earlier, the protection of clients . All that can be done is to protect the client as best we can with all the∑ current available knowledge, the combined practice wisdom 13, and the expectations of those involved in the total process. In addition, there needs to be encourage " ment of process research which shares knowledge of practice and techniques so that 'standards' can be better known. In practice evaluative research on techniques and methods is often easier to achieve, because of specificity, rather than more∑ global research on prevention, or in evaluating particular projects or programs. Yet this area is very often neglected because it appears practice-oriented rather than 'academic', or not relevant to political and policy issues.

This process may seem too complex but unless it is undertaken it makes any pretence of evaluation of performance a disguised polit ical exercise. In practice ∑ it is not as impossible as it is sometimes stated to be, but one of the unnecessarily complicating factors is that it is usually attempted just at the time when resources are lowest, i.e. in times of financial need. If it were built into the mechanisms of delivery at all times, the complexity and stres ses would not be so inhibiting , and sometimes disabling .

The answer to the question of how to evaluate is influenced by practical , technological, historical and philosophical factors, as this paper is attempting to show. T here is an adequate literature on the question but a dearth of practical examples related to the Australian context.l 4 This does not m ean that evaluation ∑

is not taking place, but that it is not shared and suffici ently visibl e.

Evaluation as a routine part of aU service delivery The first and basic point is that evaluation of performance should be planned' from the beginning of any activity. I have not becom e involved in this discussion of what is, or is not, a program; this is again well covered in the literature, especially by American writers such as Tripodi.l 4 I have therefore simply used'

the words service delivery. Any part of this can be evaluated in terms of standard ,

12. This question is one which is preoccupying a number of practitioner s in commun ity work in England, and will be very relevant if there is a resurgence of the view that community centres 'prevent' dysfunc " tioning and community breakdown. 13. Even in the 'scie ntific' areas there is recognition of the problem of scientific 'truth ': e.g. What is this.

thing called Science? A. F. Chalmers, Un iversit y of Queensland Press, 1978. 14. T . Tripodi et al., Differ ential Social Program Evaluation, Peacock, Illinois, 1978.

34

'for example a small part can be looked at, a technique, a section of normal delivery or the total project or program. In one agency the intake process was -evaluated to see whether one worker on intake or all workers on intake provided a more consistent service in terms of diagnosis of the presenting problem.

The interventive process in terms of the systems approach involves evaluation d the appropriateness of the goals, whether the goals were achieved, but also an ∑ evaluation of the alternative process of achieving the goal. The systems approach15 involves clarification of the aims of the activity, the question of legitimation,

identification of who is the client, the target, and a process involving four phases, i.e. the contact and assessment, action, implementation, and termination and ∑ evaluation phases. A distinction is often made between evaluation and monitoring, where it is

regarded as more convenient to use monitoring as the routine assessment or -evaluation of standards, and evaluation is used for the more formal goal, evaluation in a 'before and after' pattern. As has been stated a number of times in this paper the various purposes of evaluation cannot be separated. The goal of

'rehabilitation of prisoners may be achieved by various methods, from classes in prison to home release, and assessing standards includes assessment of the particular methods. Constant evaluation as part of the total process enables one to make statements about preferred methods in terms of satisfaction to all 'groups, i.e. provider, controller of resources, and consumers. In asking for

evaluation as part of a program E. H. Smith quotes from the 'Stanford Evaluation 'Consortium' that 'Evaluation thus becomes a component of the evolving program itself, rather than disinterested monitoring undertaken to provide ammunition to the warring factions in a political struggle. Final reports to outsider s are reduced

in significance, and research findings become not conclusions but updating of 1he system's picture of itself.' 16 Standards, expectations and values are inextricably linked. Some argue that -one must ignore all 'subjectivity' and measure only cost-effectiveness. To use an

extreme example, if some negative reinforcement method 'cured' a patient effectively and quickly, should the fact that the patient preferred a slower and less unpleasant method be taken into account? Should also one take account of 1he fact that the 'unpleasant' method caused unpleasant consequences in terms of

disruption of family life? How far is the client inhibited by fear of consequences, .desire to please authority etc. in expressing preferences?17 The question of involving the client in the process of evaluation is being dealt with elsewhere but it has some interesting variations when one is speaking of

performance as distinct from provision. Many writers insist that clients should play a part in assessing provisions and priorities but that the 'expert' only can .assess standards. One important factor here is the question of accountability and with whom

the 'contract' is made. The example often used is that of the 'unwilling ' client where the service is part of the community's pattern of social control, as in the case of the prisoner, the parolee. To whom is the provider accountable for his 'performance', i.e. whom does he satisfy? 15. The systems approach has been extensively used recently by social work, e.g. A. Pinkus & A. Minaham ,

Social Work Practice: Model and Method, Peacock, Illinois, 1974. P. Boas and J. Crawley, Explorations in Teaching Generic Social Work Theory, P.l.T. Press, Occasional Paper No. I, 1976. 16. Jean and Elery Hamilton-Smith, Four Flats Hawthorn, 1st Progress Report, Melbourne, 1977. 17. This point is made in a commentary on experimentalll'\odels, even 'soft' ones: by B. Jordan, 'A comme nt

on "Theory and Practice in Social Work"', Briti sh Journal of Social Work, vol. 8, no. I, 1978. This article is a rejoinder to another, and both are concerned with 'science' and evidence in assessing practice .

35

If the goal is to make the parolee's adjustment to the community as pleasant and rapid, for him, as possible, i.e. not necessarily to prevent his return to prison, then standards and 'success' will not depend on recidivism statistics, and the type of service delivery and performance will vary, because of the different ultimate goal.

In the case of helping a client and family in relation to terminal illness the standard may be entirely related to client satisfaction with the support given, with no clearly measurable health outcome. The danger is that the easier measurable outcome may be used, e.g. recidivism and parolee, and that such a claim may be tested. If official goals and service delivery goals are different and service delivery performance is geared to the non-official goal, evaluation of that performance could be confused. Over-simplifying goals to make them acceptable politically has been a dangerous exercise, particularly if evaluation is based on these and backfires because of their inappropriateness. The whole question of what is a 'good' evaluation, and how one 'evaluates the evaluation' has not been dealt with in this paper, but is a basic one to all the aspects of evaluation considered . There have been examples of distorted evaluations which then reflect back on what is being evaluated.

Self-evaluation in relation to standards It has been stated that evaluation is part of the service delivery process and this therefore implies that self-eva luation is at least part of the process. It was recently stated, at the conference mentioned earlier, that self-evaluation would not be accepted as legitimate by funding authorities. This was presumably on the grounds that it is branded unreliable (or even dishonest in the sense that self-evaluation consciously distorts). In contrast, the production of quick, hard data and outside evaluation was seen as legitimate and acceptable, even if that data over-simplified or, even worse, distorted the picture, and made unsubstan " tiated assumptions of objectiviry and causation. The emphasis in this paper is obviously on the importance of both inside and outside evaluation.

Need for recognition of Australian patterns There is a need for recognition that Australian conditions have some significance in evaluation of performance. This need has been played down, perhaps because of the desire for security and comparability with overseas data. It is possible to provide quantified or categorised material based on subjective data which is valuable, provided that the original subjectivity is taken into account. Sometimes the fact that data are quantified makes some writers speak as if they then become objective. As an example, one study assessed whether financial aid prevented children going into care, based on statements of clients that if they bad not received the money they would have put their children into care. The problems of this are obvious, and the local attitude to care, different ethnic attitudes , the attitudes to 'authority' etc. would all influence the answers. Because a service is regarded as 'successful' by one group, or because certain requirements for per " formance exist in one place does not mean that they can be transplanted.

This does not mean that there are no accepted international standards, for example, of child care. It means, ho,wever, that they are only useful in general terms and should be constantly examined in terms of Australian experience . One example of this is the need to recognise the problems in setting standard s, and in developing a uniform evaluation process in a country of the size of Australia with its small population and scattered resources.

36

The current emphasis on localisation , regionalisation and decentralisation in welfare has immense problems in setti ng standards in any uniform way. The differences between local needs, and the problems of expecting similar standards when speaking of a highly populated inner suburban area and a service in the country areas, or say in the North-West of Western Australia, are obvious. Too often the funding authorities are centralised and have little understanding of the local scene. Yet evaluation will fail unless the people at the local service delivery area are involved. Cover-up, distortion and breakdown of the system have resulted from unrealistic imposition of uniformity. This is even more a factor in Australia than the U.K. and U.S.A., which provide so many of our models. When these breakdowns occur they are all too often seen only as 'the locals' being difficult, just plain stupid, or dishonest .

Evaluation of performance as a financial control issue

I have not dealt with evaluation of performance related to government financial assistance to voluntary organisat ions but this is currently an important issue. Does a centralised body have a responsibility to the community to monitor/ evaluate standards in agencies to which it gives grants? This is a ma jor issue in

Australia at present where there is a tendency to emphasise the value of the 'volunta ry' sector and to state that certain services are better done by agencies outside 'the Government' but which need government funds to carry out the program. This is another situation where the local, historical, geographical and ideological factors have to be taken into account.

There are two views about this, not unexpectedly influenced by whether one is connected with a voluntary or statutory agency. I am using these words in the generally accepted sense in the welfare literature, not in the more specific sense used in public administration, where statutory usually means set up by statute

rather than in the sense of governmental. 18 One view is that grants should be given without strings or control , as such control would negate the value of the voluntary organisation, and would be an imposition by the 'government bureaucracy'. The other view is that the statutory organisation , using taxpayers' money, has a responsibility to see that standards are maintained. A related

problem is that there is a tendency to believe that analysis of the activities of welfare organisations, particularly if they are voluntary, and especially if they employ volunteers, is somehow inappropriate and threat ening. The belief seems to be that it 'takes away the heart', will discoura ge voluntary effort. I believe that accountability and responsibility has to be accepted as mu ch by voluntary organisations as statutor y, and that goodwill of the providers does not protect

the client from poor service. 19

The general question is not altered by the statut ory or voluntary name , the same groups need to participate. If either group, statutory or voluntary, or one particular section of either, see themselves as having all the responsibility then they are claiming inappropriate right s. The voluntary organisations cannot expect to get money and not accept any government responsibility for evaluation, but

18. This confusion of terminology and of use of volunteer and voluntary is dealt w ith in Voluntary Organisa " tions: A Ca se Study, F. Do novan, P.I.T. Press, Occasional Paper no. 3, 1977. 19. P. Nokes, The Professional Task in Welfare Practice , Routledge & Kegan Paul, London, 1967. Also dealt with in Voluntary Organisations, op.cit.

37

at the same time governments cannot expect to say 'we gave the money we decide expectations and standards'. This sort of argument has sometimes been inappropriately used by the 'businessman' in community chests.2o

The problem of evaluation of standards in the past has been that: (a) Costs, standards and performance have been arbitrarily separated. (b) The difficulty of the process has meant that it has been shelved or subordinated to the (apparently!) easier evaluation of costs. (c) There has been insufficient preparation for the exercise and demands

have come at an inappropriate time, i.e. financial stringency. (d) There has been insufficient awareness of the need for involvement of a wide variety of people, including clients.

(e) There has been insufficient awareness of the problems of Australia, especially in relation to decentralisation and localisation of services.

(f) There has been insufficient awareness of the need for service delivery personnel to be trained in such aspects as managerial and administrative skills 21 (which includes evaluation skills).

(g) There have been too few consultants and others available (for money as well as skill reasons) 22 to help staff with evaluation. 'Management con " sultants' are often unable to work in the human services area, particularly welfare.

(h) There has been insufficient use made of the evaluation which has taken place as part of the normal service delivery process. When one looks at these problems from the other side of the coin, the neces " sary conditions for adequate evaluation can be seen. This paper has not illustrated the specific types of activities needed. There are overseas examples in the litera " ture, but in specific terms it is very difficult to get many examples of activities in Australia, in evaluation of standards. Some examples can be given from direct experience but there may be others in Australia not widely publicised or recorded. As an example, the Social Welfare Department (Victoria) is attempting to evaluate standards of performance in its Family Support Service Programs, incor " porating many of the required processes for successful evaluation. Another interesting example was the recent wide publication of the adoption principles, processes and guidelines of the Victorian Social Welfare Department, asking for public comment. Another example is the work of the Foundation for Multi " disciplinary Education in Community Health in South Australia. The difficulty of getting material is one indication of the problems listed above. However, if one lists general conditions they can be applied to the current situation .

To enable adequate standards of performance to be evaluated requires a num " ber of factors: 1. The involvement of all personnel who take part in the service delivery level, as well as the participation of higher and centralised levels. This includes

clients , providers of the service as well as funding source.

20. In the early days of community chests in Australia it was interesting to see the emphasis on 'Who pays the piper calls the tune'. The inappropriateness of this concept to welfare is still not always reco go ised. 21. 'Management Training for Public Welfare Agencies: Why the need Remains Unmet', M. Egan &

M. Bendick, Administration in Social Work, vol. I, no. 4, 1977. 22. M. Egan & M. Bendick, op.cit .

38

2. The clarification of the goals and aims of the service so that they can be interrelated with the standards of performance expected to achieve these goals. 3. A clarification of the processes wh,ich will be necessary to carry out the

goals and achieve the standards-both minimum and optimal. 4. Clarification of the decision-making process in relation to methods and processes as well as goals. 5. There needs to be recognition that a wide variety of skills are needed and

that there has been little attempt up to now to develop these skills or use them adequately in the health and welfare field. 6. There needs to be recognition that evaluation must involve self-evaluation as well as outside evaluation, particularly relating to standards, i.e. staff

must be 'trusted' to see evaluation as part of service delivery and as a routine part of their service with balances and checks by outside personnel where appropriate. 7. There needs to be a balance between local differences and general principles ,

and all of these to be incorporated into the Australian pattern. 8. There needs to be a sharing of knowledge and a development of visibility of standards. It is the responsibility of those involved in the process to keep up with all available knowledge, and to revise constantly as a result of

the feedback process. 9. All evaluation of standards of performance needs to recognise the accept " ance of values and the dynamic nature of standards . Machinery needs to be built in for constant revision and constant balancing of different views.

This is one of the most difficult processes but if it is not recognised the result is often rigid and inflexible standards that become discredited and therefore circumvented or ignored. All these difficult processes take time, money, knowledge, commitment; com " modities which are not so favourably regarded in periods of financial crisis and cost stringency and which are not as popular as simple 'solutions'. There are at last beginning to be hopeful signs that the problem is being tackled, although it is patchy and not, as yet, co-ordinated or sufficiently recognised.

39

24895!79-4

Recent evaluation activity in Australia

Summary of this papel' 1. The INTRODUCTION indicates the restrictions placed on interpretation of 'evaluation' in the writing of this paper, and summarises in a single table (Table 1) the types of evaluation activity that have been undertaken in Australia since about 1973.

2. Section I evaluates the adequacy of these evaluation activities in a review of three dimensions of the evaluations (their objectives, comprehensiveness and relevance). This section suggests a number of problems with past evaluations : " the non-specificity of the objectives of many of the evaluations and the

resulting overlap that occurs in some evaluation reports; " the limited criteria used in past evaluations; " the relevance of the reports, as affected by the questions asked in past evaluations, the timing of evaluation study and any resulting report; the

organisation of evaluation research; the adequacy of the data available ; the method of presentation and dissemination of the results of evaluation ; and the adequacy of existing review processes in the public sector (par " ticularly Parliamentary oversight processes).

3. Section II summarises the major conclusions and suggests how program evaluation of social welfare expenditures might be improved. The major recom " mendations relate to: " identifying the types of evaluation required in social welfare;

" enhancing Parliament's role in ensuring accountability in the public sector ; " enhancing the Audit Office role in program evaluation (or program audit); " ensuring that in individual departments and statutory authorities program evaluation becomes an integral, identifiable part of program management;

" enhancing the role of agencies with responsibilities for funding evaluation research.

41

Contents Introduction 43

I. ADEQUACY OF PAST EVALUATIONS (1.1) Criteria for deciding 'adequacy' . 44

(1.2) Objectives and purposes of past evaluations 44

(1.3) Comprehensiveness of past evaluations . 45

(1.4) Relevance of past evaluations 46

A. Answering some questions of the policy makers 46

B. Timing of the evaluation study 47

C. Location, organisation and staffing of the evaluation. 48

D. Appropriateness of available data to the evaluation question posed 49 E. Methods of presentation and dissemination of evaluation reports 49 F. Parliamentary oversight of public sector expenditures 50

II. MAJOR CONCLUSIONS AND RECOMMENDATIONS 51

Table 1: Summary of recent evaluations of social welfare structures, process and outcome 55 Table 2: Criteria regularly used in contemporary evaluations of government programs 58 Table 3: Some issues barely touched by past evaluations 59

42

Introduction 1. According to Webster's dictionary, 'evaluation' is equated to 'assessment', 'appraisal' and 'estimation'. Many past studies could thus lay claim to be called 'evaluations'. For the purposes of keeping this paper within acceptable limits , it is necessary to restrict a review of 'recent evaluation activity in Australia' to

include those examples which illustrate the breadth rather than the depth of the contemporary evaluation process. A second limit will be to focus mainly on the evaluation of social welfare projects and programs which are financed, adminis " tered or staffed by public sector agencies, since there are fewer data on interna l evaluations carried out in the private sector or in the voluntary agency.

Seven broad types of evaluation 2. For the purposes of this paper, evaluation activities will be reviewed under seven broad categories, viz. " under the auspices of the Commonwealth Parliament , either by its Com "

mittees or by Parliament acting as the corporate whole; " initiated by central management agencies (such as Treasury, Finance, Auditor-General or PSB); " initiated within Commonwealth government departments and statutory

authorities; " initiated by recent commissions or committees of review and inquiry; " initiated by State or local government authorities; " initiated by voluntary agencies; or

" initiated by independent academic researchers.

3. For the reader who is unconcerned with detail, Table 1 summarises the major objectives of projects which fall under these seven broad headings. Table 1 also suggests four categories of evaluation, i.e. whether the major concern was with: " an assessment of the community's need for a new service or a modification

of existing services; " the structure of an existing department, a project or a program (how it was organised, financed, staffed, or legislated for); " the process of delivering an existing service (i.e. how and by whom

the client or patient was initially contacted; what type of service wa s given; what happened to the client or patient on discharge or exit from his/her contact with the service; what administrative, budget and financial control processes were used in admission or entry, servicin g, charging or follow -up of the client or patient); " the outcome of an existing or alternative service (i.e. was the service effective

in changing the health, dependency or needs status of the client , did it do so in a cost-efficient or cost-effective manner, and wa s the service provided acceptable to both client and provider?).

4. Some other important dimensions of evaluation are not shown in Table 1. For example, Table 1 says very little about: " the comprehensiveness of the criteria used to evaluate needs, structure, process or outcome;

" whether the evaluation was descriptive or analytical;

43

" whether it was based on subjective or objective data; " whether any statistical analysis carried out was methodologically valid; or " whether the content and format of the final evaluation report was such that it led to a decision to contract, expand or improve an administrative structure,

a program or project in some identifiable way.

5. Some of these matters are briefly discussed in the following section.

I. Adequacy of past evaluation s (1.1) Criteria for deciding adequacy of an evaluation activity

6. There are no universally agreed-upon criteria for deciding whether one evaluation is more useful than another, but in reviewing the scope and outcome of past social welfare evaluation activities in Australia, it is convenient to review the activities listed in Table 1 under the following broad headings:

" objecti ves and purpose of the evaluation (i.e . what was the objective of the evaluation effort?); " comprehensiveness of evaluation processes; and " relevance of the evaluation report in improving the service/program /project

under review.

(1.2) Objectives and purposes of past evaluations 7. Table 1 suggests that a significant proportion of past evaluations of social welfare services focused on structure and process evaluation . In general, the largest proportion of the evaluation effort has been directed at:

" the adequacy of administrative structures for health and welfare services (e.g. Royal Commission on Australian Government Administration, Bland, Bailey, Toose); " program overlap in the health-welfare area (e.g. RCAGA Health-Welfare

Task Force, Bailey, Holmes); " The adequacy and cost-efficiency of existing programs (e.g. the anti-poverty programs reviewed by Henderson; Galbally report on migrant services; Medi " bank Review Committee; pathology services); and " evaluations of the need for radical changes in income security or social

welfare services delivery (e.g. the Asprey, Woodhouse, Hancock and Henderson reports; Family Services Committee).

8. As a result of this preoccupation with evaluations of services, benefit levels and organisations, there is some partial agreement in the reports of different commissions, inquiries and committees: " there is considerable overlap in the administration, financing and staffing of

a range of social welfare services, both between public and private sector providers and between different levels of government (e.g. RCAGA Health " Welfare Task Force; Bailey); " it seems possible to collapse a plethora of welfare programs into fewer

programs, but only if political resolve and adequate financing are mustered to support such a rationalisation (RCAGA Health-Welfare Task Force; Bailey);

44

" if management or statistical data are collected in the administration of the program, the data often lack a comparable baseline or they are in a format that positively inhibits their use for systematic evaluation of the effectiveness or efficiency of similar programs or projects (various Medibank Review Committee reports; unpublished W. D. Scott Final Report of the Joint Management Review Committee to the Steering Committee on Health Data for Policy Evaluation, June 1978; VCOSS Submission to Senate Committee,

19 July 1978).

9'. Yet despite these common findings of many of the recent inquiries, there has been very little change in the structure or process of social welfare services. At this point, it is perhaps relevant to ask: " whether the objectives or terms of reference of the various inquiries are

sufficiently specific to the needs of those decision makers who would be expected to act on the evaluation reports 1 ; " whether future evaluation activities should be designed to address specific areas of concern that have not yet been adequately answered by all these

reviews and evaluations (these topics are addressed in Section II, infra).

(I.3) Comprehensiveness of past evaluations 10. In the main, the evaluations summarised in Table 1 have been concerned with evaluation criteria such as: " performance efficiency (is the administration of the program or service

efficient in its use of financial and human resources?); " adequacy (did the program or service provide an adequate level of service or benefit to the client?); " acceptability (is the program or service acceptable to consumers, providers

and politicians?); and " are there organisational overlaps in service provision?

11. By contrast, Table 2 lists a number of criteria which are regularly applied in contemporary public sector program evaluation. It seems fair to say that very few of the evaluations in Table 1 enable us to answer such crucial questions as: " whether existing programs are responding to objectively determined social

needs (the Henderson reports used objective data to assess income poverty and other correlates but there is still a question of whether income or expenditure data more accurately measure income poverty 2); " whether existing programs are meeting a social need in an effective manner "

i.e. did they achieve what was intended in their enabling legislation?; and " whether existing programs are cost-effective in achieving their outcome-or are there more efficient ways of obtaining the same outcome?

12. Table 3 poses some of the same questions in specific social welfare contexts. But having noted that the evaluations listed in Table 1 were not conspicuous for their use of the most comprehensive range of evaluation criteria, it is still a long

1. The terms of at least one recent inquiry required the investigator to ' ... make a comprehensive review (and to) ... make other recommendations ... in relation to rationale, efficac y and simplification'. 'Efficacy' is sometimes equated to 'efficiency' (e.g. Grosset-Webster Dictionary), but also is equated to 'effectiveness' . The evaluation of 'efficiency' and 'effectiveness' requires very different approaches and very different evalu " ation criteria-see Section 1.2 infra. 2. N. Podder (1978), The Economic Circumstances of the Poor, AGPS , Australian Government Commission of Inquiry into Poverty.

45

jump to another assertion that the studies would have been far more relevant (and, in some cases, have been acted upon) if they had addressed some of these questions. There is very little evidence that global comprehensive evaluation exercises produce more useful results than simple, well thought out exercises attempting to answer a single, answerable evaluation question.

( 1.4) Relevance of past evaluations to major policy issues 13. To this point, we have noted that the commissions, committees and inquiries have often had fairly broad objectives or terms of reference (paragraph 9, supra) for their work. It has also been suggested (paragraph 10, supra) that the comprehensiveness of the evaluation criteria used in many of these studies has

been limited. One result may be that the results of the evaluation are not usable because they are not sufficiently specific to the information needs of the major decision makers asking a different set of evaluation questions (paragraph 11, supra).3

14. Apart from these fundamental questions of evaluation objectives and com " prehensiveness of evaluation criteria, the relevance of much of the past evaluation activities to the policy maker, both in Parliament and in the government agency, may also have been diminished by a number of related factors, including:

(a) whether the evaluation answers the important questions being raised by the rna jor policy makers; (b) the timing of the evaluation activity in the ongoing development of policies and programs; (c) the location, organisation and staffing of the evaluation activity; (d) the appropriateness of the data available to answer the type of evaluation

question being posed; (e) the method of presentation and dissemination of the evaluation report(s); and (f) the extent to which the results of any one evaluation activity are

systematically reviewed, within a continuous process of evaluation, by Parliament and its Committees, within Commonwealth and State govern " ment agencies, and within the voluntary sector. These factors are briefly discussed in the following paragraphs.

A. ANSWERING SOME QUESTIONS OF THE POUCY MAKERS

15. Regardless of whether a national economy is in growth or in decline, modern program evaluation should address some fundamental questions that all policy makers face in their decision to expand, contract or modify a govern " ment program, viz. do social and economic conditions suggest that the present policies and programs should be reviewed for appropriateness or adequacy? Are existing policies and programs accomplishing their intended goals? Can program

performance be improved even if the program is effective? Are we spending too much in certain functional areas?

3. There is some evidence that some ongoing evaluations of specific services have more sharply focused objectives which may answer some of these specific questions. For example the relative cost-effectivenes s of community health centres compared with other competing 'modalities' of treatment (such as hospital casualty department, doctor's office or after-hours service) ma y be partly answered by the A.C.T. studies by Professor Stephen Leeder. The costs of general practice in a rural community have been identified in B. Simon, M . Hobbs and K. Cullen (1978), 'A Study of General Practice and its Costs in a Rural Com " munity', Unit of Clinical Epidemiology, University of Western Australia, June, 43 pp. mimeo.

46

Adequacy of existing policies and programs for the 1980s 16. Are there new social conditions now in existence in Australia which require a change in policy and program or a new policy or program? This question recurs more frequently in these days of higher unemployment and apparently higher income poverty, even as fundamental changes in the international and national economy force a rethinking of the size and composition of all public expenditures, particularly in health, education and social security. With very few exceptions

(e.g. Barrie, 1978: Health-Welfare Task Force Report, Royal Commi ssion on Australian Government Administration, 1975), the evaluations listed in Table 1 have rarely canvassed the implications of a rapidly changing demography and unstable economic conditions for future health and welfare services policy, i.e.

the evaluations are not futures oriented. Equity of benefits distribution 17. Very few of the evaluations listed in Table 1 have clearly identified who is receiving what from the major social welfare programs. The test of this assertion comes when one attempts to answer questions that seem rather fundamental given that the Commonwealth Government distributed at least $15 thousand million 4

in taxpayers' money for social welfare in 1977-78, viz. " who gets what out of existing social welfare expenditures (whether they be used for income security, service delivery or economic opportunity, e.g. employment and training programs)?

" to what extent do existing social welfare expenditures actually benefit those most in need (e.g., the income-poor, the very sick, the handicapped, and the environmentally and culturally deprived)? Cost-effectiveness of programs 18. Finally, few (if any) of the evaluations summarised in Table 1 give either a coherent or complete picture of whether existing social welfare expenditures are:

" effective in their outcome (i.e. they improve 'health', alleviate temporary distress or reduce welfare dependency); " cost-effective in their outcome (i.e. they achieve their intended effect with the most efficient use of resources). B. TIMING OF THE EVALUATION STUDY

19. A second issue of relevance is whether the duration of the evaluation and the timing of the evaluation report affect the implementation of any report. Many of the reports listed in Table 1 are from inquiries that stretched over years of evaluative research (e.g. the Poverty Inquiry; Toose; Royal Commission on

Australian Government Administration). Others were produced in a short period of intensive effort (e.g. the Review of the Community Health Program, HHSC, 1976; evaluation of relative costs of health centres in the A.C.T.). 20. More to the point, because of differences in the times of initiation and

reporting, many of the inquiries overlap in their final recommendations (e.g. Asprey, Hancock, Henderson, Woodhouse) and someone else must subsequently sort out consequences. Such problems will not be overcome until there is more careful of objectives for government inquiries and a requirement that inquiries with overlapping objectives should identify the consequences of overlaps in their recommendations and reconcile the consequences where possible. 4. This figure includes approximately 200 programs covering income maintenance, in-kind benefits and economic opportunity identified by the Social Welfare Policy Secretariat in preliminary studies.

47

C. LOCATION, ORGANISATION AND STAFFING OF THE EVALUATION 21. The evaluations listed in Table 1 have been instigated by different authoritiea, most of which are indicated in paragraph 2, supra. Very few of the evaluations listed in Table 1 have been initiated as part of the program management activities of government departments (exceptions are the Health Commission of N .S.W. studies in item 1). However it is conceivable that much intramural evaluation never comes into the public view; thus Table 1 may be incomplete in this respect. From Table 1 it is unclear whether the method of implementation

of the evaluation study (e.g. by a Royal Commission, a national inquiry, a govern " ment agency, an independent researcher or a parliamentary committee) is an im " portant determinant of whether the evaluation report is acted upon.

22. Of the evaluations listed in Table 1, the following scorecard gives some indi " cations of whether overt action has been taken on the resultant evaluation reports : Item 1: Service need evaluations: Action to improve services is under way in N.S.W.

Item 2: Structure and process of an existi ng service: performance of govern " ment departments: Action has been taken on parts of every report listed, but particularly projects 6-7. Studies of voluntary agencies: Action has been taken on at least one of the two reports.

Specific inquiries into programs: Action to implement a majority of the recommendations would seem to have been taken on four of the fifteen evaluations listed . Ongoing audit review activities by Parliament: Most of these recommenda " tions are made in annual reports and a scorecard of implementation of the recommendations is difficult to construct.

ltem 3: Outcome of an existing service: Effectiveness: Very limited action taken on first two reports, last two still ongoing. Cost-effectiveness: Study report not yet available. Acceptability: Limited action taken in the single project referenced.

Item 4: Outcome of an alternative service delivery system: This feasibility study is still ongoing but it includes most of the elements of evaluation that are required in the prospective analysis of an alternative pro " gram or policy.

23. This scorecard looks impressive at first glance. It is hard to say what action might have been taken on the reports of some of the national inquiries listed under Item 2 had the studies been under different auspices, e.g. if the evaluation had been carried out by:

" a central management agency (e.g. Treasury, Finance, PSB, Audit Office, PM & C);

" an independent evaluation unit within the Parliament (e.g. some equivalent to the U.S. Congressional Budget Office or the Congressional Research Service); " an independent evaluation unit located somewhere between government and

university (e.g. some equivalent to the Brookings Institution, the Urban Institute , and RAND Corporation in the U.S.A., or their U.K. and European equivalents); or

48

" a private consultant group (exemplified in the recent use of the W. D. Scott organisation to assess the adequacy of health financial statistics-see project 8 in Item 2.1 of Table 1 ) .

D. THE APPROPRIATENESS OF AVAILABLE DATA TO

THE EVALUATION QUESTION POSED

24. It is a sad fact that many of the reports listed in Table 1 start their analysis with a caveat about the adequacy of the data available to answer the evaluation question being posed. These caveats are the most conspicuous, common feature of nearly all the reports listed in Table 1 and in other reports omitted for the sake of brevity in this paper.

25. Action has been taken to repair some of the gaps in data in the social welfare area. 5 The recent study of health statistics by a team of public servants and a private consulting firm has produced a consolidated picture of statistics on health expenditures in the last three to four years. Secondly, evaluation studies

have been initiated in hospitals, health centres and community health projects in most States. The results of some of these studies will be available within the next twelve to eighteen months. Thirdly, the new Social Welfare Policy Secre " tariat has already commenced the creation of a data base in the broader area of social welfare expenditures, with an initial focus on 1977-78 and 1978-79

expenditures.

26. In the absence of such data bases, and with an apparent reluctance of many government agencies to systematically review how their program management activities can and should contribute to program evaluation, it is highly unlikely that rapid progress will be made in social welfare program evaluation. The essential components in any action to change this situation are that:

" government' agencies be required, if only to meet PSA section 17 account " ability provisions, to systematically review the adequacy of existing program management procedures, with a view to ultimately automating project reports on service usage and cost data into formal systems of program evaluation; " urgent consideration be given by Parliament to the setting of deadlines

for the production of evaluation reports using a minimal set of evaluation criteria for all programs, as part of the continuing improvement of budget and expenditure control measures in government; and " adequate funds be allocated by government for the development of such

data bases, having firmly in view that the costs of the resulting evaluation system should not exceed its benefits.

E. METHODS OF PRESENTATION AND DISSEMINATION OF EVALUATION REPORT 27. Most of the reports of the evaluations listed in Table 1 average 100-200 pages of fairly solid text and tables. The work of the Commission on Poverty is contained in a series of main reports and in a number of complementary

reports on specific topics. The Royal Commission on Australian Government Administration reported in a major report, a four-volume appendix, and in a series of reports by different Task Forces.

s. Fairly rapid advances in the health area were made during the lifetime of the Hospitals and Health Services Commission, starting with the 1974 report on the adequacy of Au stralia 's health statistics, leading ultimately to the creation of a National Committee on Health and Vital Statistics . Other action includes the contracting for a data base creation to provide relevant health finance statistics and Aboriginal health statistics .

49

28. It is a moot point whether it is possible to collapse the findings of a detaile dl national inquiry into a few summary pages that are digestible by even the most non-enthusiastic reader. It is less moot whether the results of other types of evaluation can be better presented to readers who have little time available to read long, convoluted reports. Third, it is apparent that there is a need for some central information service to disseminate reports on ongoing and completed evaluations, so that overlaps in the funding of evaluative research are minimised. Some redundancy can be often justified in areas where there is a series of evaluative questions to be answered, or where there is no single 'best' evaluatio n " methodology. 29. Very few of the projects listed in Table 1 can be seen to involve duplicate research, the evaluation foci are sufficiently diverse for the duplication to be minimal in all areas except the recent spurt of evaluations of performance of government agencies involved in the health and welfare areas (e.g. RCAG .A Health-Welfare Task Force; Bland; Holmes; Bailey). Even in the areas of income maintenance traversed by the Asprey, Henderson, Woodhouse and Hancock inquiries, there is minimal overlap; indeed these reports suffer from the common limitation that the then available data were not able to identify wlw gets what out of the relevant social welfare programs, e.g. taxation, welfare benefits, compensation, superannuation and the relevant tax expenditure programs. 30. The most obvious action required is to inventor y what is known from past evaluations; this is another major task confronting the Social Welfare Policy Secretariat. A second task is to identify what evaluation activities are to identify what knowledge and data gaps still remain, with dissemination of the findings of such an analysis to those government agencies that fund intramura l and extramural evaluative research. A third priority is to find a method of dissemination of program evaluation reports to a broader, more critical audience in and outside of government, to ensure that future evaluations improve the state of the art in conceptualisation, methodology (i.e. study design and data analysis) and reporting. F. PARLIAMENTARY OVERSIGHT OF PUBLIC SECTOR EXPENDITURES 31. Finally, it seems doubtful that the systematic evaluation of public sector programs will occur in the absence of a continuous program of oversight and' review by Parliament and different Commonwealth government agencies. The House of Representatives Standing Committee on Expenditure and the Public Accounts Committee have the potential to initiate this oversight role, but to date there has been very little overt attempt to

" systematically review the major public sector programs in the social welfare area6; " systematically follow up the reports of past evaluations and inquiries to∑ ensure that some action was taken by appropriate agencies to justify the "

large expenditures involved in national inquiries; " identify the extent of the state of the art and adequacy of the different types. of evaluation and audit in use by modern governments; and " identify future evaluation capacities that Parliament might need to develop

in order to ensure accountability for expenditures . 6. Preliminary studies by the Social Welfare Policy Secretariat have identified at least ninety programs providing income benefit s or subsidies, nearly seventy service programs offering in-kind benefits (by way of health, housing, food and community services) and nearly thirt y programs which enhance economic " opportunity by way of education and training . This list, while not complete, is indicative of the extent of the evaluation effort required to answer the 'who gets what' question, let alone the cost-effectivenes s: questions.

50

D. Major conclusions and recommendations

{II.l) Major conclusions

32. This paper has given an overview of a limited number of past evaluations of social welfare structures, process and outcomes. The review has not been on a project-by-project basis since no useful purpose is served by identifying :specific problems with specific studies. Rather, the intent has been:

" to identify what we still do not know about social welfare programs and expenditures, despite all the evaluation activities of the past; " to identify what action is required to improve the relevance and quality of evaluation in the public sector.

'33. The major conclusions can be stated, perhaps over-simply, in a series of

.(a) Societal and economic conditions are rapidly changing and some of our existing social welfare programs may not be either adequate or relevant to handling the resulting load on our welfare system. Whatever else we do, we ought to acquire a forward-looking capacity in government, and require a 1980s impact assessment in every future activity that involves an evaluation or inquiry of one sort or another.

t(b) From past evaluations (some of which are listed in Table 1), we know something about existing social welfare program overlaps and about poten " tial areas fm consolidation by alternative organisation and financing arrangements. We know very little about who gets what benefits from

existing social welfare expenditures. We know even less about the effective " ness and cost-effectiveness of any of our large expenditure programs in health care financing and income security, social welfare services delivery or economic opportunity. Unless we apply these criteria in program evaluation, it is hardly likely that we can avoid the indiscriminate hacking of programs that is the only action possible in the absence of comparative data on effectiveness, cost-effectiveness and acceptability.

((c) The priorities for program evaluation in the social welfare area would thus seem to include: " specific studies of cost and cost-effectiveness in programs with over " lapping clientele or service outlets; " studies of the equity of benefit and cost incidence of social welfare

expenditures, to see who gains and who loses in the existing patchwork of programs; " evaluation of the managerial performance of all social welfare programs, with a view to assessing the potential impact of a formal program

evaluation system embedded within the program management and reporting system; ∑ " evaluations that permit Parliament and government agencies to make comparisons from year to year to detect trends in specific evaluation

criterion measures (with information gathered in a format that facilitates comparison by Parliamentarians not familiar with rigorous evaluation methodology or having the time to read long-winded reports).

51

(d) The need for a continuous, systematic program review capacity within the Parliament and its Committees is only partially met by the existing Committee system, by the Audit Office efficiency unit or its existing regulatory review and reporting process, or by the budget process. There is a need for Parliament to concern itself with the need to formulate a system of oversight and review which adequately recognises:

" the special needs of Parliamentarians for reports which are easily under " stood and from which actions can flow; " that Parliament may need to again consider whether it is time to upgrade its capacities to acquire, produce and disseminate such material

(either through the P.arliamentary research service, through an upgrad " ing of the Committee secretariat services, or through the provision of funds to certain Committees to initiate evaluative research studies by independent authorities) . (e) Within departments and statutory authorities, there may well be staff

with the appropriate training to undertake program evaluation activities of the requisite quality. There are also independent experts in the private sector, both in consultant organisations and in voluntary agencies. There is also a limited amount of independent academic expertise with some knowledge of government operations and the requirements of modem public sector evaluation. What is not clear at this moment is the scope and quality of ongoing evaluation activities of an intramural and extra " mural nature. It seems essential that one organisation, preferably in government, be designated with the responsibility to:

" systematically inventory what is going on; " identify what is known from past evaluation; " identify what are the gaps in both the knowledge and the data required to achieve specific evaluation objectives; " identify what levels of funding are required to both plug the gaps and

produce a report on what is done or ongoing; " identify what gaps exist in evaluation manpower within the Common " wealth public service, with a view to taking immediate action to upgrade evaluation knowledge and skills through appropriate training programs. 7 (f) Within departments, statutory authorities and voluntary agencies, there is

a specific gap in the management information systems that support pro " gram management in the health-welfare areas. Notw ithstanding the past activities of the Hospitals and Health Services Commission and the present activit ies of the National Committee on Health and Vital Statistics to develop the required data base for a national system of health and vital statistics, there is a need for a fundamental rethinking of the role of program management reporting systems in the cycle of program evaluation. There is a pressing need to review the adequacy of data that are already available in the individual branches of different government departments, yet not accessible for program evaluation. 8

7. The Canberra College of Advanced Education is already involved in offering elements of such a training program within Executive Development Courses sponsored by the Public Service Board. It also proposes to offer a series of eight to ten seminars on program evaluation in 1978-79. Planning for these seminars is already well under way . 8. The recent W. D . Scott-Health team study of health financing statistics revealed how a compo site data base can be constructed out of individual data bases that have existed in isolation in the past.

52

(11.2) Major recommendations to improve education

34. To ensure that future evaluation activities plug some of the gaps identified in this paper, it is recommended that the following action be taken:

CLEARER DEFINITION OF PROGRAM OBJECTIVES

35. Ministers with portfolios in the social welfare area should initiate activities within their departments to identify the specific goals of every major program with annual expenditure above a certain limit, with a view to initiating an evaluation of progress towards these objectives to be completed by the 1980-81 Budget.

DEVELOP STATE-OF-THE-ART REPORT ON EVALUATION PRIORITIES IN SOCIAL WELFARE

36. In conjunction with the Auditor-General's Office, the Social Welfare Policy Secretariat should be requested to produce, for discussion in the 1979-80 Budget Cabinet, a report identifying: " what is ongoing in evaluative research within departments and elsewhere;

" what is already known about the major social welfare programs (perhaps defined in terms of programs above a minimum level of annual expenditure); " the major gaps in knowledge and data on social welfare programs; " a formal program of evaluation activities required to plug the knowledge

gaps, particularly the identification of who benefits from social welfare expenditures by all levels of government and the private voluntary sector; " a specific program for evaluation of costs and effectiveness of a number of large social welfare expenditure items, including details of the budget impli "

cations in agencies with responsibilities for funding evaluative research; " an assessment of the adequacy of a range of potential sources of evaluation expertise (e.g. in government, in private consultant firms, and in academic units), including an assessment of the feasibility of a Brookings-like insti "

tution in Australian social welfare policy analysis and research; " an assessment of the feasibility of upgrading existing program management information systems in the relevant departments so as to produce evaluation data to meet specific, time-constrained evaluation objectives.

PARLIAMENTARY INTENT TO REVIEW SPECIFIC PROGRAMS IN 1979-80 BUDGET

37. The Parliament, in anticipation of the 1979-80 budget review process, should appoint a single Committee to bring forward recommendations on a mini " mal set of evaluation criteria that should be applied, in the budget review pro " cess and in other Committees of Parliament, to a selected group of social welfare expenditures in the 1979-80 Budget, having considered criterion mea sures for

which the relevant data can be obtained by that time.

38. The Parliament, on receipt of the report of the Social Welfare Policy Secretariat required in paragraph 36, should attempt, through its relevant com " mittees (e.g. Expenditure, Public Accounts, Standing Committee on Social Wel " fare) to review a minimum number of social welfare programs in each session of Parliament beyond 1979-80. The allocation of responsibilities to each Committee

would be possible when the Secretariat report is available.

53

39. The Parliament should, before that time, give clear indications of its intent to undertake an extensive oversight of social welfare programs. It should com " municate its intent in a clear instruction to the relevant central management agencies on how responsibilities for such evaluation activities are to be shared between:

(a) the Parliament and its Committees; (b) the central management agencies (specifically PSB, Audit Office, Treasury, Finance, PM & C); (c) the line departments with responsibilities for both social welfare program

management and for funding evaluative research; (d) State and local government authorities, and voluntary agencies, for whom the future levels of cost sharing and the continuity of funding may depend on the availability of specific evaluations on the impact of cost-shared

programs9; and (e) co-ordinating units such as the Social Welfare Policy Secretariat.

SPECIFIC ACTION TO UPGRADE EVALUATION CAPACITIES 40. Before the 1979-80 Budget Session the Public Service Board should be required to report to the Parliament on the following matters: (a) the extent of evaluation capacity available in central management

agencies and in departments and statutory authorities; (b) the shortfalls in evaluation manpower; (c) the extent of ongoing training programs to upgrade the quality and num " bers of such manpower; and (d) the budgets required to undertake the types of evaluative task outlined in

the Social Welfare Policy Secretariat report referred to in paragraph 36.

41. In addition, the PSB report should indicate the range of program evaluation activities that are being pursued in accord with specific sections (e.g. section 17) of the Public Service Act and other relevant Acts, clearly identifying the value for money obtained in each of these evaluation activities (many of which will be outside the social welfare area, and will thus complement the recommen " dations of the Social Welfare Policy Secretariat in this area).

9. Voluntary agencies should be consulted orally in advance of announcing any formal requirement for evaluation as a condition of funding. Some welfare agencies have an understandable fear of goals set by theorists operating remote from the harsh world of service delivery-see testimony of Executive Officer of Royal Tasmanian Society for the Blind and Deaf, Canberra, 18 July 1978.

54

"' ... 00 "' _, !

v.

v.

Table 1.

Type of evaluation

1. NEED FOR A NEW

SERVICE

2. STRUCTURE and/or PROCESS of an EXISTING SERVICE

Major focus of evaluation

Health-welfare status of population in a specific region

(2.1) Specific inquiries into per " formance of government dept(s) involved in health and welfare

(2.2) Studies of voluntary agencies

(2.3) Specific inquiries into pro " grams affecting human develop " ment, income security or economic opportunity

Recent evaluations Report produced

N.S.W . Health Commission studies in South-east H . C . of N .S.W . 1977 Region, and in Sutherland area of Sydney (seven volumes)

1. Health-Welfare Task Force, Royal Commission on Australian Govt Administration, 1975 2. Report; of the Royal Commission on Australian Govt Administration, 1974-76 3. Independent (Toose) Inquiry into the Repatri "

ation System 1971-75 4. Administrative Review (Bland) Committee 1976 5. Task Force on Co-ordination in Welfare and Health (Bailey Committee) 1976 6. Committee of Review, School of Public Health

and Tropical Medicine, Univ. of Sydney, 1974-75 (Hospitals and Health Services Commission) 7. Review of the Delivery of Services Financed by the Department of Aboriginal Affairs (Hay) 8. W. D. Scott review of health financial Statistics

1978

1. Voluntary Agencies and Government Financing (Social Welfare Commission) 2. St Vincents Hospital (Sydney)--evaluation of efficacy of family planning services in a large

sample ofNFP clinics across Australia 1975-78 1. National Committee of Inquiry into Compen " sation and Rehabilitation in Australia (Woodhouse Committee) 2; National Superannuation (Hancock)

Committee oflnquiry 1973-76 3. Commission of Inquiry into Poverty (Henderson) 4. Taxation Review Committee (Asprey)

AGPS, 1975

AGPS, 1976

AGPS, June 1975 (vols 1-2) Not available AGPS 1977, 1978 (vols 1-2) HHSC, 1975

AGPS, 1976

unpublished report, 1978

swc, 1976

Johnson eta!., 1978 (report to Common∑ wealth Dept. of Health)

AGPS, 1974 (vols 1-2)

AGPS, June 1974 (interim report), April 1976, March 1977 (part II) AGPS, 1974-77

AGPS, Jan. 1975 (full report)

Vl

0\

Type of evaluation Major focus of evaluation

(2.4) Ongoing audit/review activities of Parliament or by independent statutory authorities

Recent evaluations

S. Report on the Australian Assistance Plan (Social Welfare Commission) 6. A Medical Rehabilitation Program for Australia (Hospitals and Health Services Commission) 7. Hospitals in Australia (Hospitals and Health

Services Commission) 8. Review of Community Health Program (Hos " pitals and Health Services Commission) 9. Family Services Committee (Social Welfare

Commission) 10. Priorities Review Staff, Report on Housing 11. Priorities Review Staff, Report on Early Child " hood Services

Report produced

swc, 1976

HHSC, 1973

HHSC, 1974

HHSC, 1976

The Committee, 1978 (2 vols) PRS, 1975 PRS, 1974

12. Committee of Inquiry into the Care of the AGPS Aged and the Infirm (Holmes) 13. Income Security Review 1975-78 unpublished reports 14. Medibank Review Committees (1976, 1977, unpublished reports

1978)

IS. Miscellaneous studies of usage of community various project reports (see text) health centres in N.S.W., Vic., Tasmania, South Australia 1. Expenditure Committee 2. Public Accounts Committee

3. Senate Standing Committee on Social Welfare (this current inquiry) 4. Auditor-General's Annual Report to Parliament 5. Public Service Board-section 17 efficiency

audit activities 6. Forward Estimates Budget Review Process 7. Review of Community Health Program by backbenchers committee, 1976

V1

-.1

Type of evaluation

3. OliTCOME OF AN

EXISTING SERVICE

4. LIKELY OUTCOME OFAN ALTERNATIV E SERVICE

Major focus of evaluation

(3.1) Effectiveness of an existing program/project/service in achieving its specific objectives

(3.2) Cost-effectiveness of an existing program/project/service in meeting its specific objectives (3.3) Acceptability of an existing program/project /service to con "

or to provider

(4.1) Likely costs and acceptance of alternative systems of health care

Recent evaluations

1. Royal Commission on Australian Government Administration-reviews of counter services in Department of Social Security 1975 2. Commission of Inquiry into Poverty-various

reports on migrants, low income housing, health care 3. Before-and-after study of impact of a health centre on health in a community, InaJa,

Brisbane 4. Study of efficiency and effectiveness of obstetric care in a large teaching hospital (Brisbane, Sydney, Adelaide)

1. Studies of relative costs of health centres in the A.C.T. (treating specific conditions)

1. Royal Commission on Australian Government Administration, research project into access problems of clients to personal services in Tax " ation Office, Dept of Social Security, Aus " tralian Housing Corporation and Australian Legal Aid Office

Report produced

AGPS, 1976, RCAGA Report Appendix vol. two

AGPS, various years

Ongoing

Gordon et al. (1978) on two Brisbane hospitals; report by T. Kiiver on one Sydney hospital, HHSC 1977; ongoing study (with 1978 progress report to HHSC) in Adelaide hospital Ongoing-final report due August 1978

AGPS, 1976, RCAGA Report Appendix vol. two Appendix 2C pp. 191-331 Appendix 2E pp. 366-70

2. RCAGA pilot project development and evalu- ibid. Appendix 2F pp. 371-424 ation of NOW project, Melbourne

1. Evaluation of feasibility of a Prepaid Health Plan in South Australia Progress report to Hospitals and Health Services Commission 1977

Table 2: Criteria regularly used in contemporary evaluations of Government Programs

1. Social need-Is the program needed? Wanted? Used? By whom?

2. The equity effects of the program, particularly its effects on income redistri " bution and equality of opportunity of access (which may be income determined). 10

3. The effectiveness of the program in reaching its stated goals (i.e . did the pro " gram have its intended impact on the health, poverty, suffering or disability of the recipient?).

4. The efficiency of the program. For a given combination of inputs, how rela " tively efficient is the 'production process' that results in the final product or service outputs?

5. Other attributes of program performance and the management process, viz. " comprehensiveness of program services " continuity of services " accessibility of services (financial, spatial, temporal) " co-ordination of program with other programs " consumer access and involvement

6. The availability of resources, particularly financial and human resources, either to begin or to continue a program.

7. The extent to which the program is in accordance with the moral, ethical and social norms of the time (some programs that would have been under a cloud if judged by this criterion in the early 1970s would have included abortion clinics or drug addiction services supported by government funds).

8. The extent to which the program is in accord with constitutional provision 'S and democratic principles.

9. The extent to which the program is in harmony with existing policies or pro " grams (e.g. what effects do income maintenance programs have on the work incentive?).

10. The acceptance of the program by consumers, providers and other influentials.

11. The political feasibility of the program (It is possible for a program to score high on most of the earlier criteria, only to lose out on this criterion. An example is the proposal for a guaranteed income maintenance scheme, or proposals for federal government support of abortion services in family planning clinics).

12. Program cost (listing this criterion last is not to ignore the fact that, in a tight economy, it often becomes the dominant criterion).

10. Marmor distinguishes between 'vertical efficiency' and 'horizontal efficiency' in an income maintenance program. Vertical efficiency is defined as the ratio of benefits received by intended beneficiaries to total benefits distributed . If payments are made only to intended beneficiaries this ratio is I :0. Horizontal efficiency is defined as the ratio of the number of beneficiaries in the target group to the total number in the target group. The lower the proportion of aid going to the target group, the lower this ratio . Obviously, the first ratio pertains to relative distribution amongst the total population (including a target group), while the second relate s to distribution within the target group. See T. Marmor, 'On Comparing Income Maintenance Alternatives', America/ Political Science Review, vol. 65 (1971).

58

Table 3: Some issues barely touched by past evaluations

1. Are there more efficient methods of meeting objectively determined social needs (e.g. providing access to adequate health care or housing through the income security system, rather than through heavily subsidised health insurance or through low income housing agreements)?

2. Is the existing program meeting a social need but not necessarily achieving the highest possible target efficiency (horizontal and vertical equity) of dis " tribution of program benefits or economic opportunity (e.g. a family allowance or an educational allowance with means testing)?

3. Which specific service delivery or benefit payment arrangements can improve administrative efficiency by reducing the present unit costs of service or pay " ment (e.g. 'contracting out' of social services to voluntary and other private sector organisations, or the simplification of taxation and welfare payments systems into a negative income tax or tax credit system so that governments don't have to pay benefits to the poor and process tax returns from low income families) ?11

4. Are the services or programs having the impact intended in their enabling legislation? Do they improve 'health', alleviate a temporary problem or reduce poverty?

5. In terms of different program or service performance measures, what difference does 'comprehensiveness' and 'continuity' of services make to the client, and would less comprehensive, less continuous services produce an equally acceptable impact on the intended client?

6. Can managerial audit or efficiency audit bring about identifiable changes in the performance of program managers in government administration?

7. Would the costs of social welfare to a whole community be contained if there was more effective co-ordination (or even rationalisation) of benefits or services (e.g. what are the economic, health and welfare impacts of a separate system of Repatriation hospitals on benefits payments costs or on communities which already have an adequate number of hospital beds?)?

8. To what extent can the overall benefits payment system (for the aged, disabled, families, unemployed etc.) be directed towards those in greatest need and streamlined so that those who are eligible for more than one benefit do not feel the weight of marginal tax rates which can exceed 100 per cent, thus impeding

'take-up' of benefits?

9. What programs can be devolved for administration by lower levels of govern " ment or by the voluntary sector, and what specific Commonwealth--State financial arrangements are required to initiate such action, even in a time of tight economic constraint?

11. Some of these questions have been raised in some of the reports listed in Table I (e.g. in various reports of the Commission of Inquiry into Poverty) and in the recent Ho spitals & Health Services Commission report on Paying for Health Care. The questions that have not yet been answered by a systematic study are whether such changes will simplify or complicate administrati ve arrangements. It is fair to say that proposals to simplify income security and health insurance arrangements by usiog the taxation system do not attract instant applause, mainly because of perceptions that the administrative arrangements required will not simplify the existing system. These perceptions are rarely buttre ssed by any solid evidence.

59

History and development of evaluation in the human services: a view from the United States

Introduction

When the history of human service organisations during the twentieth century is written, it is quite possible that program evaluation will be identified as the major innovation in the administration and operation of those organisations. Moreover this phenomenon is particularly noteworthy in nearly all industrialised countries of the world during the latter half of the twentieth century. As is the case with many social inventions, program evaluation was proposed and advocated long before it was accepted and implemented to any significant extent. It is worthwhile to consider ( 1) why adoption of this innovation took place so slowly and reluctantly until the 1960s; and (2) why it is viewed so optimistically today as being of critical importance in both public and voluntary sectors and to local, State and federal governments. 1

An historical perspective on evaluation in the U.S. Evaluation of social experiments early in the twentieth century and initially developed side by side with other types of social research. As long ago as 1906 Lester Ward advocated that scientific research should be an integral part of the development of social policy. Similar statements were made by F. Stuart Chapin, who in the 1930s and 1940s actually evaluated some social programs of the 1930s (Chapin, 1947). Advocacy of program evaluation as a governmental re " sponsibility was first presented by Stephen Spender, who in 1935 suggested:

. . . unless policy makers, social planners and the public know the consequence of efforts, planned change and social innovation, broad-scale programs of social action cannot be judged rationally (Spender, 1935). Nothing came of Spender's recommendation even though the 1930s was a period of massive social reform in the United States, and it is now quite apparent in the

1970s that some of the policies and programs should have been evaluated and changes implemented if more effective policies were to be considered. For example, the legislation authorising payments to mothers and dependent children appears to have fostered, unintentionally, increases in the number of single parent house "

holds. Likewise, the implications of the zero order effects of the evaluation of the Cambridge--Sommerville program for the prevention of delinquency were not con " sidered seriously. Recent findings from subsequent re-evaluation s of that program raise serious questions about the efficacy of counselling programs of this type.

I. Robert Milton noted that evaluation was an innovation accepted by society only in the 1960s when the social structural feature of the society at that time demanded greater accountability, rationality, efficiency, and effectiveness . One means for giving serious attention to those ends was through program evaluation and subsequently from the utilisation of the findings from evaluations for decision making.

61

During the 1940s interest in evaluation developed very slowly, probably be " cause of preoccupation with the war, but Kurt Lewin did utilise the opportunity to evaluate a variety of approaches for changing people's attitudes and behaviour regarding food rationing and so forth. Subsequently, in the 1950s, he and several colleagues conducted a large number of field experiments related to public housing, civil liberties, intergroup relations and delinquency prevention. This applied research was clearly a forerunner of the greatly increased activity in pro " gram evaluation in the 1960s. During this same period Newcomb completed his noted study of the effects of an experimental college program (1943) on student behaviour and attitudes.

A number of large scale social experiments were implemented in the 1950s in vocational rehabilitation, public housing, and community organisation. With the rediscovery of poverty, racism, educational disadvantagement, mental illness and retardation in the 1960s there was greatly increased interest in evaluation. The methodologies for evaluation were not well defined. Both design and instrument s were quite inadequate for the task expected to be performed. It was often, how " ever, possible to justify huge expenditures for innovative social programs when it could be said that they would be evaluated 'scientifically'. The by-words became 'innovate, experiment and evaluate'. The utilisation of results, however, received almost no attention. Moreover, because much evaluation was done by social scien " tists trained and interested in basic research, the findings seldom had relevance to policy or practice or if they did the reports were received long after critical policy decisions were made. It often appeared that the mere doing of evaluative research was the important event, rather than obtaining results that could be utilised in decision making regarding resource allocation, choice of technology, level of intervention and so forth.

M any of the social programs initi ated in the 1960s focused on disadvantaged individuals and groups. It was assumed that with training , counselling and other forms of interpersonal assistance the problems of disadvantagement would be over " come. The goals for programs were often ambiguous and non-specific because a variety of persons with different needs were similarly targeted to receive the service to be provided. The stated goals in legislation referred to improved quality of life, better health care, improved quality of education. These goals cannot be concretely operationalised in a vacuum. Not only ends, however, were problem " atical, the means or technologies were also vague-thus evaluation was further confounded. A third factor influencing the situation was the expectation that all programs would have positive not negative or neutral outcomes. Lastly, the pro " grams were presented to legislatures and funding agencies with many claims re the effectiveness and costs. As a result of the latter belief , the public at large came to have very high (and nearly impossible) expectations for programs and client s.

Although evaluation in the 1950s and 1960s was a mixed phenomenon in the United States, this approach to decision making was utilised extensively overseas by international agencies interested in the outcomes of varied approaches to family planning and population control. In India alone more than nineteen World Bank funded experiments were evaluated and similar experiences were ob " served in other Asian and Latin American countries.

During the period of the 1960s federal legislators and bureaucrats in the United States became more interested in ongoing systematic evaluation that was mandated in substantive social legislation. The Civil Liberties Act of 1964 resulted in the national study of elementary and secondary education (the Coleman Report)

62

and that research continues to have important social policy repercussions through " out the United States in 1979-fifteen years later. In fact, the Coleman Report made it eminently clear that program evaluation can have very profound effects " direct and indirect-with many unanticipated consequences that may be negative

as well as positive. Until the Coleman Report and all the related studies linked to it were completed, most tended to accept the assumption that program evaluation was a rational, apolitical tool for administration and policy makers as well as practitioner s. Few considered that the evaluation itself, both as a process and a product, would have profound effects throughout the society. More recently, how " ever, it has been viewed in terms of political processes and values, particularly in assessment of alternative intervention approaches. Overwhelmingly, the results from the evaluation of most of these programs showed '0' order effects or no difference in results between experimental and control groups. However, these

negative findings had little effect on program elimination until post 1968 (Rossi and Wright, 1977) .2

Passage of crime control and criminal justice legislation in 1966 led to the establishment of the Law Enforcement Assistance Administration . This legis " lation mandated evaluation of alternative approaches to crime control , police action. law enforcement, courts and corrections. Initially emphasis was placed

primarily oh 'hardware' approaches to crime control rather than social pro " grams, but since 1974 the latter have grown more rapidly. Evaluative research has come to play a significant role in state and local demonstration projects in diversion and alternative approaches to court intervention; in community " based programs; and in the design and development of remand centres . The impact of evaluation on the traditional models of incarceration in prisons has been negligible thus far. The Juvenile Justice and Delinquency Prevention Act passed in 197 4 has had far-reaching consequences for evaluation . It pro "

vided that states must deinstitutionali se at specified rates and develop community " based programs in order to continue to receive federal funds.

In the past decade demands for program evaluation have grown in new directions . Evaluation has come to be recognised as a social mo vement and a tool for macro-level decision making in social policy, similar to the ways in which 'social indicator' research was utilised a decade ago. Interest in secondary analysis and critiques of evaluation has also become more prevalent . These

appear to have had a very positi ve outcome for improvement in the design of and methodologies for evaluation. A third factor of increasing importance has been the demand for greater accountability by legislator s, agency adm inis " tration , service deliverers , clients and, last but not least, by the public taxpayer.

Some of the problems that have emerged in evaluating service delivery systems today derive from the fact that in the early 1960s social welfari sts overpromised wh at social service programs could do to deal with the problems of poverty and other forms of human suffering and deviance (Gronberg, Street and Suttles,

1978). In their enthusia sm they asserted that a social service strategy coupled with an income transfer strategy were powerful solutions for these problems. Far too few consider ed whether or not such ambitious programs could be implemented. However, President John Kennedy is known to have said to his

aides, after signing a new legislative measure, 'We have made the law. Now

2. Recently man y observers have noted that evaluation findings have increased im pact today vis-a-vis program reduction and elimination . In contrast, when a society is experiencing a period of rapid innovation and reform, relatively few may argue for evaluation.

63

it remains to be seen whether we can get our government to do it.' Observations of Australian State and federal systems during the 1960s and 1970s suggest that a similar phenomenon occurred here also. M any open-ended promises were written into social security and welfare legislation in the U.S. The impact was a significant expansion in the size of social welfare budgets and programs. Costs of human services escalated rapidl y, sufficiently as to cause alarm among many persons. As a result personnel in the various management and budgeting offices asked:

Do human service programs work? If so, how? Which programs are effective in helping persons at risk and which are not? Where will you cut costs? What evaluation tools will provide a good record of a program's strengths and weaknesses? Thus far evaluation results have been insufficient for adequate answers, but increasingly, routinised information processing systems and new assessment methodologies have moved us significantly toward the necessary capability for obtaining the desired answers. Understanding of how to do effective evaluation is now growing rapidly. Many new approaches go far beyond application of experimental and quasi-experimental designs used in other types of social research. These methodologies have vastly improved the potential of evalu " ation products. At the same time evaluators have become increasingly con " cerned about problems of implementation. Examinations have shown that the failure of many programs is due to faulty implementation of the program more than to ineffective treatments. Thus interest has grown in the process of trans " lation of policies into programs and from programs into services delivered as they are expected to be delivered. Williams (1975) has said:

The past contains few clearer messages than that of bridging the gap between policy decisions and workable field operations . . We simply do not know how to implement complex new social programs or major program modifications . Williams goes on to argue that implementation was the Achilles' heel of the Johnson Administration's social policy. He argues that realistic goals must be set and then more attention must be given to the complexity of implementation.

In their examination of 226 federally funded evaluation projects, Bernstein and Freeman (1975) stated that two major questions must be examined in evaluation: 'Has the program been directed at the appropri ate and specified target population or target area? Were the various practices and inter vention efforts undertaken as specified in the program design or derived from the principl es explicated in that design?' They found that in only 22 per cent of the evaluations was there information about implementation. Since that time the problems of implementati on have received more attention by evaluator s.

Contemporary evaluation strat egies involve new assumptions about hum an service organisa tions, whom they serve, how , and why. Perrow ( 1978) has argued for a more realis tic appraisal of whose inter ests are served by human service organisations rather than continuing to assume that only clients' intere sts are served in a planned and rational manner. Other organisational theorists mad e similar arguments earlier but only recently have these been considered seriously with respect to hum an service organisations (B ecker, 1968; Gouldner, 1968; and Blau, 1975).

The contemporary scene Critiques of the social sciences in the late 1960s influenced evaluation research in that attacks from advocates of both conservative and radical crit ics led applied researchers to take the status quo as problematical. Programs, practic es and

64

organisations were to be assessed according to their effectiveness in attaining stated goals. In addition, evaluators were linked more closely to decision makers and attempted to research 'relevant' problems and issues. It is part of a broader effort to bring rationality into the policy making process, although it is clearly recognised that it lies within the context of 'bounded rationality' (Freeman, 1977).

Perceptions of evaluation vary widely today among administrators, policy makers and researchers. Moreover, these have produced variations from system " atic rigorous application of social research techniques to highly subjective inves " tigations or case studies. Obviously any review or examination must take into consideration these variable definitions of evaluation:

( 1) The monitoring of organisational performance to describe behaviour in a systematic manner with reference to: (a) needs and services re target population; (b) policy or procedural compliance;

(c) quality and quantity of staff performance. ( 2) The monitoring of organisational behaviour to assess efficiency and effectiveness in a systematic manner. (3) Applied social research in which scientific procedure is utilised to

develop generalisations about organisational behaviour. ( 4) Systematic inquiry in which logical and unbiased methods are utilised to assess a problem re organisational performance and/or target population. (5) Mechanism to assist in the rational allocation of resources, e.g., benefit "

cost analysis used by management. ( 6) Political activity to achieve a social reform or to stabilise an innovation. (7) Set of techniques through which scientific knowledge and research can be utilised to initiate reform, e.g. demonstration intervention projects . ( 8) A step in the general problem-solving process:

orientation, evaluation and control. (9) A systematic set of procedures to: (a) assess whether behaviour change has occurred in a target population; (b) the causes for the change as far as these can be known (or at least

causation by extraneous factors is excluded); (c) ascertain the cost of the changes; and (d) determine the probability of generalisations to other organisations and populations . Because the term evaluation has been used to describe any of the above behaviours, and because these have very different meaning and imply different procedures, investigators must clarify the definition utilised by the agency in which he or she will do an assessment. Second, it also has become important to clarify the use of concepts such as environmental context, input, throughput, output and outcome since these also are of crucial importance in the evaluation process. 'Environmental context' refers to the socio-political and physical environ "

ment in which the program is implemented and the evaluation takes place. It includes the social conditions and the human services system within which the needs, problems and goals are assessed along with the operational objectives for that system. 'Input' refers to the strategies, plans, materials and design adopted to achieve the objectives . The actual implementation phase is often referred to

65

as 'throughput' and it includes the activitie s and processes performed by the organisation plus the structures and effort necessary for those activities. It has become very important to distinguish 'output' and 'outcomes' as these terms are now commonly used in the organisational and management literature. Output typically is used to describe the behaviour of the target population at the point of organisational termination. Supposedly these behaviours are the ones produced to som e degree at least , by the organisation al intervention. For examp le, the reading performance of a student at the point of completion of a specified course. In contrast 'outcome' refers to the behaviour of the target at some time subsequent to organisational termination. In many instances changes in the output behaviour must be anticipated because of the negative influence of the environment and/ or because of atrophy or deterioration when the treatment is discontinued. In the above example, we might refer to the reading behaviour of that same student one year after the course was completed. If the reading score declined or increased, it might be the result of many factors only one of which was the relative adequacy of the educational program that was completed.

A third task is consideration of who should be responsible for evaluation and where the evaluation unit should be lodged. It can be located within or outside the agency, and a variety of persons might successfully complete the evaluation. As Bernstein and Freeman (1975) point out, these are differenc es in the evaluation product in terms of who is the evaluator, where the unit is located, and how the timing of the evaluation is managed. Private entrepreneur ial organisations are far more likely to adhere rigidly to requirements set forth in a contract or grant whereas evaluations performed by academic organisations are more likely to deal with theoretical issues and knowledge development. Either approach may be satisfactory but the purposes for which the evaluati on is being done should be an important factor in the choice between alternativ es. It is clear that the grantor or agency for whom the evaluation is being completed must assume primary responsibility for indicating the type of evaluation desired, the expectations regarding the product of the evaluation and the autonomy to be given the evaluator in conducting the assessment.

Humaneness and justice v. effectiveness Recent statement s by organisations such as the American Friends Service C om " mittee (1971), the American Association of University Women (1970), by social researchers such as Lerman (1968), as well as numerous judici al decisions, have highlighted consideration of the relationship between effectivene ss and conditions of fairne ss, humaneness, and justice in the operation of human service programs. 3 Several national commissions and organisations have now articulat ed a set of standards which provide behavioural guidelines regarding hum aneness and justice . Enforcement of these standards through the courts is having an impact on program operation and evaluation, for not only is the 'right to treat " ment' at issue, but so are the general social and physical conditions of client s. It is possible for a program to be effective (at least hypothetically) and yet not meet criteria of humaneness, fairness, and justice . The evidence, however, appears to be overwhelming that these three conditions are necessary to, but not suffici ent for, effectiveness as far as organisational goal attainment is concerned. Writers such as David Rothman ( 1979) stress that any time the state intervenes into

3. For example, of far-reaching action following upon court decisions in the area of 'right to treatmen t' see Wyatt v. Stickney 334 F. Supp. 1341 (M.D . Ala 1972); Morales v. Terman 364 F. Supp. 167, 173-4 (E.D . and Texas 1973); and Donaldson v. Conn . 463 F. 2d 567 (1974).

66

the lives of any of its citizens, it has the obligation to do so in ways that are humane, just and fair; following that, consideration can be given to the attain " ment of specific program goals. Evaluation research has focused more attention on the assessment of new social programs, but of equal importance and perhaps of greater frequency have

been the evaluations of ongoing programs. Any review of evaluation experience in recent years in the United States will benefit from separate consideration of the assessment of these two approaches. SOCIAL EXPERIMENTS

Evaluations of new programs, especially those that are aimed at helping the disadvantaged, have implied scepticism about attempts at remediation . They have become part of the social problems definition process since so often it is implied that target populations are intractable with regard to desired behavioural change. Evaluation methodologies have reinforced many pre-existing structural views, for example the negative income tax experiments. The underlying assump " tion was that the society was generally functioning smoothly and that social programs were only needed for 'fine tuning' of an essentially sound engine. However, contemporary levels of youth unemployment, substance abuse and alienation seriously challenge such views.

Much attention in the late 1960s and early 1970s was given to social experi " ments that included experimental field evaluation (Riecken and Boruch, 1974; Cook and Campbell, 1975). As a result considerable knowledge has accrued about the potential and limits-not only of the experiments themselves but also about experimental field evaluation methodologies. The evaluations of the several negative income tax and income support programs have provided substantial information about both expected and unexpected outcomes, and of the problems in attempting to measure and link short-term outputs to longer term outcomes

(Rossi and Lyall, 1976). In the criminal justice field numerous experimental programs have been evaluated. The attempt by Lipton, Martinson and Wilks ( 1975) to review and critique these assessments provoked much public discussion. Moreover, it has been utilised by some policy makers as justification for reducing treatment programs although the evidence for that decision was far from con " clusive. This incident in the criminal justice area highlights a problem often ignored by evaluators who are intent only on a 'rational' investigation of whether or not a given program achieved the formally stated goals. They failed to examine whether or not the absence of the program would result in negative change or whether the overwhelming impact of state intervention was essentially negative. In the case of the criminal justice systems such questions need urgent attention if offenders are to be returned to the society and not be a greater threat than when they entered the programs. Publication of the findings by Lipton, Martinson and Wilks (1975) provided a convenient excuse for many policy makers to eliminate programs in . prison settings and only today are some of the negative consequences being recognised. This overly rationalistic and somewhat naive approach to the study of very complex social experiments was also observed in the studies of Head Start (Cicirelli, 1969), of community action and man " power development and other programs (Rivlin and Timpone, 1975).

Although the results of the evaluation of field experiments thus far requires that one question their validity or utility for policy making, it appears that at least the following benefits have accrued: 1. Substantial knowledge has been accumulated to indicate that changing human

behaviour is a very slow process. Findings from the experiments contribute know " ledge for public policy debates, but one cannot expect major developments from any single experiment. Each has a modest but important role to play.

67

2. The cost in resources of all types is great for systematic social experiments so they should be used carefully and parsimoniously in areas where their potential for knowledge contribution or policy innovation is greatest. 3. The tension between an experiment as a demonstration project and the ex " periment as scientific research is serious and likely to persist. Thus, mechanisms and priorities must be explicitly established to address the likely problems that will emerge.

ONGOING PROGRAMS

Despite the glamour of novel field experiments and their equally dramatic evaluation, experiments are relatively infrequent and not of as critical import " ance to decision makers as are evaluations of ongoing programs. It is far easier to determine the impact of a new experimental program than it is to icarn whether or not an ongoing program really makes a difference, and whether that difference is in accord with the program's stated goals and policies. More powerful research designed to test the efficacy of a program is impossible and as a result these evaluations are subjected to methodological and technical criti " cisms as the long debate over the Coleman report has indicated (Nosteller and Moynihan, 1973; Jencks et al., 1972; Cain and Watts, 1970).

Perhaps the most result from the attempts at systematic evaluation of ongoing programs has been the underestimation of program effects. Many subsequent reanalyses that have deliberately attempted to use additional con " trols and newly developed statistical procedures have observed consistent differ " ences in favour of the experimental groups in several instances. Longitudinal studies of ongoing programs are particularly difficult because of the lack of adequate records and information systems. Likewise any time series studies are problematical because one needs to be able to rule out competing explana " ticns for the observed change. For example, if a community experienced a calamity such as a devastating cyclone, if 'could be expected that considerabl e

unemployment would ensue. However, part of that unemployment might be due to other employment characteristics of the region or to the stage of the economic cycle at that time. Recently one of the more popular versions of evaluation of ongoing pro " grams has been 'benefit-cost analysis'. H owever, it is not evaluation in the same sense as an impact or process study, but rather, it is a mechanism through which the results of studies can be incorporated into recommendations that are more policy relevant. A program may be found to have a significant impact, but is not really relevant to the achievement of certain societal goals. Benefit " cost analysis helps us to determine its relative social value. To do so, it relies a!.most exclusively on available economic data and thus there are serious limits to its utilisation in instances where such data are unavailable or where 0ther social data are of more critical importance (e.g. areas of general education, criminal justice, and so forth). As Rossi and Wright (1977) point out this methodology is useful when:

I. data are available about the impact of a given program; 2. benefits can be reduced to monetary terms; 3. the general benefit level rather than distributional effects is the main criterion ;

4. there is a theoretical and empirical basis for designating benefits, and the theori es have a known validity.

68

Policy development and evaluation Program evaluation is now a social movement much like the 'social indicator' movement of a decade ago. It is an evolving and changing movement with no clear directionality and although it purports to value rationality as a priority

in policy decision making, clearly we are witnessing a form of 'bounded ration " ality'. In examining the effects of policies, programs and practices on their target populations, it assumes that careful information about those phenomena c;hould improve decision making in the short and long run. Far too few evalu "

ators have recognised that they are inevitably proponents or protagonists of particular values and philosophies. Nowhere is this more apparent than in t!'e definition of criteria for output or outcomes. Likewise evaluation method " ologies endorse an incremental view of social change in organisations.

The distinction between evaluation as a research enterprise as contrasted with it as an aspect of organisational administration and development is of critical importance. The mere commissioning of an evaluation is rarely the pro " duct of scientific endeavour, but rather an expression of political decision making. Evaluation plays a role in the struggle for political power within and among organisations. Those who assume that the introduction of this approach will make the organisation less political delude only themselves.

Weiss (197 5) has noted that regardless of the evaluator's perspectives , there are several audiences waiting for the findings and each will use those findings in terms of their own values and objectives . Those concerned with program survival will fit them to that purpose for 'survival is an old and well established art' . Survival, if anything, has become stronger in this half of the twentieth century as bureaucracies have become large, complex and very well-entrenched organisations. They are able to resist nearly all forms of pressure and clearly

evaluation processes -and products are not at the top of the list. If an evaluator underestimates the dynamics of organisational behaviour-the continuous con " flict between pressures for organisational maintenance and goal attainment-he or she will be unable to complete the evaluation properly or have the findings utilised to increase program effectiveness.

Evaluators have the potential to demystify organisational behaviour and to question conventional wisdom, but they must also recognise that decision makers are no less rational than social scientists, but as Weiss (1975) points out, their rationality is political in its objectives. Policy makers are cqncerned with bar "

gaining and trade-otis, with satisfying constituents, and keeping politically advantageous programs alive whether or not they achieve their stated goals. Goals themselves are apt to be diffuse, exaggerated, and even inconsistent because of the governmental processes that shape them.

The relationships between social intervention and evaluation processes are more complex and disturbing than any of the participants imagined. Nonetheless, Weiss encourages evaluators to enter the arena: 'Only with sensitivity to the politics of evaluation research can the researcher be as creative and strategically useful as he should be' (Weiss , 1975:14).

The legislature and evaluation Since this paper is being prepared for the legislative branch, namely a com " mittee of the Parliament, it seems advisable that particular attention be directed to some experiences and observations in the United States Congress with regard

to program evaluation. During the past decade interest in evaluation and

69

utilisation of evaluation products grew substantially at federal, state and muni " cipal levels. At the federal level, however, far more resources have been allo " cated to evaluation and the federal government has influenced developments throughout the country in private as well as public sectors. Most new social legislation mandates program evaluation at the process as well as impact levels. The most impetus for this emphasis on evaluation grew out of the passage in

197 4 of the Congressional Budget and Impoundment Control Act (Salasin, 1978). That statute was passed because of the felt need by Congress to have information for decision making that would be more equivalent to that already available to the Executive Branch of government. In contrast to countries such as Australia with its Parliamentary system of government, the executive and legislative branches are quite independent in the United States and they often do not share information with each other. The 1974 Act established the Con " gressional Budget Office and a Com_mittee on the Budget in both Houses of the Congress. Those units were mandated to evaluate legislative proposals as well as existing programs, and to facilitate the development of greater evaluation capability by the Congress.

Congressional Budget Act contained one provision empowering all

Congressional committees to conduct evaluations in their respective areas of responsibility, or to see that such were completed. Because of the volume of work that full implementation of Title VII required, legislators are now con " sidering mechanisms for selective and periodic evaluation rather than compre " hensive approaches encompassing an∑ programs. 4 Committees are also attempt " ing to develop mechanisms for linking evaluation more directly to the budgetary process since this was one of the early priorities. The establishment of such linkages, however, is extremely difficult because many programs are not pre " tested or piloted. Thus, evaluation typically gets under way after programs are in operati on; so clarification about linkages between means and ends becomes problematic . Timing is also a major problem since results often are not avail " able when needed for legislative decision making. Changed environmental con " ditions and ethical constraints also arise to curtail or limit the obtaining of information that legislators might wish to have.

The most far-reaching proposal to date was found in the proposed legislation , the Program Evaluation Act of 1977. That Act would have put an automatic termination timetable on all federally funded programs. The so-called 'sunset' provisions would require reauthorisation of every governmental program within three sessions of Congress. Moreover, reauthorisation was to be accomp anied by a prescribed program review. Other proposals of this type have also been introduced but as yet not implemented. Nonetheless, it is clear that the volume and intensity of legislative activity in this area is such that some types of control measures are probably inevitable. Declining economic surplus at a time of rapidly increasing demand for human services plus continuing inflation have interacted to make necessary the re-examination of social welfare goals and programs with reference to their outcomes in facilitating the attainment of con " sensual societal

4. Title VII also mandates the General Accounting Office, the investigative arm of the Con gress, to evaluate the results of federal programs, either on the request of Congress or by its ow n initiative. That agency has had a long history of active involvement in evaluation . It has now developed an elaborate plan and schedule for the evaluation of a large number of federall y funded social programs. 5. It is clear already that the volume of evaluation espoused or decreed can never be properly completed

given the resources available . Thus, the need is urgent to be selective as to how, when , where, and wh y evaluation will be done.

70

It now appears probable that all new or amended social legislation in the United States will require periodic and/or selective evaluation prior to renewal if federal funds are involved. The Youth Employment Act of 1977 illustrates this latter approach. That legislation, designed to create jobs for millions of unemployed youth requires evaluation of identification and implementation processes as well as program impact in a variety of w ays. Findings are to be reported to the Secretary of Labor and to Congress within one year. The required

evaluation is largely descriptive and formalistic, but nonetheless, the goals and outcome criteria to be applied in assessment were far more explicit than in other recently passed similar legislation. Other newly introduced social legislation rrquired ten-year plans and specific goal-oriented budgets.

In October 1977 the United States Senate Committee on Human Resources held the first C ongressional Hearing on program evaluation . The chairperson, Senator Harrison Williams, called upon a number of expert witnesses to testify on past and present evaluation activities and upon evaluators' capability for

aiding in public policy decision making. A brief summary of the testimony provides some insight into current thinking about the entire field of endeavour. 6 Most witnesses concurred that well-designed and properly implemented evaluations do aid in decision making. Most, however, favoured formative rather than summative evaluation as more realistic for the critical problems in the human services sector. They also encouraged administrators to be less defensive about evaluation and, in turn, they suggested that evaluators be more modest about the utility of evaluation findings in policy making. It was stressed, however, that studies should be related to the expressed goals and alternative options of decision makers. Also recommended were the development of standards for evaluation as well as standards for organisational practice . Several observers noted that many evaluators do not properly inform organisations about their

practices and ethical procedures to protect human subjects. Timeliness was emphasised by many as being of great importance if one is concerned about utilisation of findings. All of the experts who testified at these hearings cautioned about overly high expectations of what could be accomplished by evaluation. The process of evalua " tion and its products can contribute to more rational decision making, but it

alone will not insure such rationality, nor can it provide the necessary resources essential for sound programs in the beginning. Henry Aaron, then Assistant Secretary of the Department of Health, Education and Welfare for Planning and Evaluation, noted:

evaluation by itself rarely will enable the decision maker to determine whether a social program should continue or 'be terminated. The complexity and multiplicity of social program outcomes, and the different effect they have on such a wide variety of people, are often exhibited in indirect and ambiguous ways that are unmeasurable by most evaluation standards. Another noted authority on evaluation, Marcia Guttentag, stressed the importance of timeliness . She pointed out that all too often there are no relevant findings made available to legislators when they need them. She also criticised the

applicability of traditional social science research models when they are used for evaluation. She commented: Evaluations of human service programs must provide information to decision makers (whether program managers or legisl ators) about the options they are

considering, and must provide this information when ∑ the decision makers want and need it, not when and how the social scientist decides to produce it. 6. For a detailed presentation of all the testimony see Congressional Record, The U.S. Senate Committee on Human Resources Hearings on 'Cost, Management and Utilization of Human Resources Program

Evaluation', 6 and 27 October 1977, Washington, D.C.

24895/79--6 71

The chairperson, Senator Williams, in commenting on the Hearings stressed that legislators should be treated as intelligent persons capable of handling complex information. He cautioned against evaluators doing all of the inter " pretation and development of recommendations prior to a dialogue with decision makers (Salas in, 197 8) .

Although the federal government is leading the way in evaluation in the United States, important events are also occurring in many of the states. A noted economist now a state legislator in Minnesota has aided the development of evaluated capability in that state. He has commented that outstanding politicians are persons of principle, inventors who do not accept the existing options. They create new frameworks that are of mutual advantage to competing groups. They seek not to know what is but rather what could be (Brandl, 1978). If evaluators wish to ma ximise the utilisation of findings, he advised them to do the following:

I. Realise that evaluation will not be comprehensive and can never specify wh at should be done. 2. Evaluate demonstration project s and experiments before major legislation is attempted. 3. Evaluation should place as much emphasis on process and effort as it does on

impact and outcome. 4. Evaluators and politicians should spend more time together to learn to understand and trust each other. 5. Cases for evaluation should be presented !'trenuously. Scientists must persist in

their commitment to the value of explicitness despite political pressure to the contrary. (Brandl, 1978:7) This brief review indicates clearly that legislators are taking a very active role in influencing the direction and content of program evaluation of publicly funded social services . That influence is unlikely to diminish in the near future.

Summar y: two perspectives Any contemporary observer of public social policy and of human service organisations today recognises immediately that evaluation of program and organisational performance is critical to the development of both sectors. In the literature that has grown to encyclopedic proportions there are many views ex " pressed about evaluation, but we lack thus far an overarching general synthesi s of these varying perspectives. Whether or not such a general theory should or will develop remains to be seen. However, two major contrasting modalities seem apparent at present. The first I have identified as 'Post-hoc e valuation for admini " strative and policy decision making'. This approach is the one found more fre " quently in the literature thus far. Evaluation runs parallel to program develop " ment but operates independentl y of it for the most part. The evaluator is select ed by the top decision maker and is responsible to that person or group for whom the product, a report, is to be prepared. The evaluator enters the picture after a decision has been reached to provide a new service, to alter existin g services, or to assess unaltered ongoing services . The evaluator is i:ypically charged with providing 'objectiv e facts', but only to the decision mak er who em ployed him. The program is implemented with tight or loose controls , with little or much ambiguity and so forth, but with no input from the evaluator (Rossi, 197 8). During the period of actual service implem entation the evaluator may have som e small control over intake and delivery of service and of the recording of infor " mation about various facets of the program. He develops his m easures of input,

72

throughput and output relatively independently. He measures processes and impact or outcome, processes the data, prepares a report and delivers it to the decision maker. The evaluator's tasks and responsibility are completed. Then, if events follow a rational course, the program that failed to produce the promised benefits would be reduced or terminated. The program whose benefits outweighed

the costs would be maintained or strengthened. The above rational model has been largely adapted from social science research methodology with minimal consideration as to whether or not such adaptation was appropriate. Needless to say, our earlier review of evaluation experience and literature requires that we question seriously approaches that are essentially static and view both the organisation and evaluation as closed systems.

In contrast to the above approach, a second perspective on evaluation might be identified as 'Evaluation, policy and organisational development strategy '. This approach is explicitly change oriented and dynamic, with the evaluator as an active participant in the development of effective programs and services. The

evaluator enters the scene as early as possible and aids in initial assessment in ways that facilitate decision making at that time. The evaluator seeks to gain a picture of the implicit and explicit goals and priorities from several vantage points and at different levels within and outside the organisation.

The developmental approach incorporates evaluators in the planning stages before a program is launched. This assessment may include a variety of content depending upon the knowledge already available and the time constraint s. Plan " ning studies are a joint product in which the client community may also par " ticipate. Increasingly, clients are requesting opportunities for active participation in evaluation since the consequences may be so critical for them. Multiple coali " tions may develop requiring trade-offs and bargaining so that decisions become a matter of compromise and fine tuning. But, such decisions are likely to be relevant and attended to because all of the participants have an active vested

interest. With a developmental model new questions and problems will emerge as the program develops and services are delivered. Program goals may also change. Causation can be viewed as a circular phenomenon in that procedures for data collection require early feedback to various organisational participants so that

the design and plan can be changed when such is needed. The process is repeated several times as the program grows and develops. Free exchange is encouraged. The evaluator has expertise to contribute to the organisation , but that expertise also aids the evaluator in maintaining a necessary amount of objectivit y for study and analysis.

Obviously, these two approaches to evaluation encompass contrasting views of the organisation, of decision making, and of evaluation . The former is likely to be more useful or appropriate for situations in which summative evaluation is appropriate, whereas the latter is likely to be more useful when formative evaluation of services is called for.

Throughout the paper we have emphasised that evaluation has important political implications since human services involve important social values that are often at the core of the society. It is the means by which the character of a program or organisation is described and analysed. Evaluators as well as decision makers are just beginning to recognise that they must understand and deal with value and policy questions when they study social programs. In turn,

decision makers are coming to accept the contributions that the evaluator can make to the improvement of public social policy making.

73

BIBLIOGRAPHY

Attkisson, C. Clifford et a!. 1978 Evaluation of Human Service Programs. New York: Academic Press. Abt, Clark (ed.) 1977 The Evaluation of Social Programs. Beverly Hills: Sage. Berk, Richard and Peter Rossi

1976 'Doing Good or Worse: Evaluation Research Politically Reexamined', Social Problems, 23: 337-49. Bernstein, Ilene and Howard Freeman 1976 Academic and Entrepreneurial Research. New York: Russell Sage. Blau, Peter

1974 On the Nature of Organizations. New York: Wiley. Brandl, John 1978 'Evaluation and Politics', Evaluation, Special Issue, 6-7. Chapin, F. S.

1947 Experimental Design in Sociological Research. New York: Harper. Cicirelli, V. G. eta!. 1969 The Impact of Head Start. Athens, Ohio: Westinghouse Training Corporation and Ohio University. Coleman, James

1966 Equality of Educational Opportunity. Washington, D.C.: Government Printing Office.

Dodd, Stuart 1934 A Controlled Experiment on Rural Hygiene in Syria, Beruit. American University Publication, Social Science Series No. 7. Epstein, Irwin and Tony Tripodi

1977 Research Techniques for Program Planning, Monitoring and Evaluation. New York: Columbia University Press. Freeman, Howard 1977 'The Present State of Evaluation Research', Evaluation Studies Review Annual, Beverly

Hills : Sage, 17-51.

Glaser, Daniel 1973 Routinizing Evaluation: Getting Feedback on Effectiveness of Crime and Delinquency Programs. Washington, D.C.: Government Printing Office, Stock No. 1724-00319.

Gronberg, Kirsten, David Street and Gerald Suttles 1978 Poverty and Social Change. Chicago: University of Chicago Press.

Haveman, Robert 1977 A Decade of Federal Antipoverty Programs. New York: Academic Press.

Jencks, C. et a!. 1975 Inequality. New York: Basic Books.

Lewin, Kurt 1948 Resolving Social Conflicts. New York: Harper.

Lipton, Douglas, R. Martinson and J. Wilks 1975 The Effectiveness of Correctional Treatment. New York: Praeger.

Newcomb, Theodore 1943 Personality and Social Change. New York: Holt, Rinehart and Winston.

Perrow, Charles 1978 'Demystifying Organizations', in Sarri and Hasenfeld, op cit., 105-22.

Rieckin, Henry and Robert Boruch 1974 Social Experimentation. New York: Academic Press.

Rivlin, Alice and P. M. Timpone (eds) 1975 Ethical and Legal Issues of Social Experimentation. Washington, D.C.: Brookings Institute .

Rossi, Peter and Sonia Wright 1977 'Evaluation Research: An Assessment of Theory, Practice and Politics', Evaluation Quarterly, 1: 1, 5-50.

74

Rossi, Peter and Katherine Lyall 1976 Reforming Public Welfare, New York: Russell Sage. Rothman, David 1978 'The State as Parent', in Rothman et al., Doing Good. Salasin, Susan

1978 'Hearings on the Cost, Management and Utilization of Human Resources Program Evalu " ation', Evaluation, Special Issue, 8-30. Sarri, Rosemary and Y eheskel Hasenfeld 1978 The Management of Human Services. New Y ork: Columbia University Press. Suchnow, Edward

1967 Evaluative Research. New York: Russell Sage. Struening, Elmer and Marcia Guttentag 1975 Handbook of Evaluation Research. Beverly Hills: Sage, vols 1-2. Ward, Lester

1906 Applied Sociology. Boston: Ginn and Co . Weiss, Carol 1975 'Evaluation and Research in the Political Context', in Struening a::1d Guttentag, op. cit., vol. 1, 13-26. We iss, Carol

1972 Evaluation Research: Methods of A ssessing Program Effectiveness. Englewood C liffs, N ew Jersey: Prentice Hall. W illiams, Walter and Richard Elmore 1976 Social Program Implementatio n. New Y ork: Academic Press.

75

Methods of evaluating effectiveness of human services

Social programs and human service organisations are provided in order to improve the well-being of the society as well as its members in terms of their individual needs and goals. Fundamental problems arise because the problems and programs are diverse: education, income maintenance, mental health, housing, criminal justice, legal services, physical illness and disability and many other areas. Second, the persons processed by these organisations are both raw material and product. Thus, all decisions and actions involve moral judgment of people and have consequences for their normative and moral standing (Hasenfeld and English, 1974; Friedson, 1970). Third, human service organisations confront multiple expectations and conflicting demands in a pluralistic society-thus they are characterised by a precarious domain consensus. For example, there is a demand that social security departments respond in a humane way and provide for the needs of those in poverty. At the same time they face demands to reduce their rolls. Because they must accommodate to multiple and often conflicting goals, human service organisations are likely to develop ambiguous and often contradictory goals. When this occurs the issues concerning who should be served and what services should be provided are never fully resolved, and clear goal priorities are not established.

A fourth problem facing these organisations is that they have limited autonomy in relation to their environment in most instances. They are highly dependent for resources upon other organisations or governments and they are subject to extensive regulations by legislative, administrative and even judicial bodies. As a result the services that they provide may reflect the contingencies imposed by external units far more than the needs of clients. They may even become

the captive of these external units and hardly serve the needs of clientele at all as Helfgot ( 1974) has observed. Because of these critical problems assessment of effectiveness and efficiency is difficult and complex.

Defining effectiveness Contemporary definition of effectivene ss and efficiency face another set of problems for human service organisations and it is those problems that we are primarily concerned with in this paper. Man y social welfare programs today are evaluated in terms of standards arbitrarily set by federal or supervisory bodies on the basis of some consensus about desired values or objectives but without reference to whether these are achievable or whether they are related to the client's perceived or actual needs. All too often these standards are not suitable for evaluation purposes. Thus, it is assumed that a program is successful if a certain professional staff to client ratio is achieved or if staff have achieved a certain level of ex∑ pertise with reference to certain technologies. In such cases

77

attention is directed only to program input, not to output. In other situations a case by case review of agency records is utilised to evaluate a program. Such review reveals limited and selective program data-whether staff have met formal requirements or if eligible clients have received services. They provide some information on services, but they do not provide the data needed by adminis " trators for planning, budgeting or handling new problems. All too often agencies

(because they appear to lack other methodologies) resort to counting clients, contacts and so forth and report such as evaluation. In such cases, numbers are presumed to imply effectiveness. Obviously given the crisis facing human service organisations in the 1970s such approaches to defining effectiveness are wholly inadequate.

The solution to the situation of conflicting demands lies in the ability of agencies to move from evaluation at the 'case' level to evaluations at the 'opera " tions' level (Rossi, 1978). Criteria for evaluation of effectiveness must be developed in relation to identified social problems and needs of client groups that are within the scope of the organisational mandate.

Methodological issues Prior to presentation of an illustrative scheme for evaluating the effectiveness of goal attainment in human service organisations, it is advisable to examine some of the more problematic methodological issues that often arise. This is an area in which rapid development has taken place in the past decade; so the reader interested in a more thorough consideration is invited to consult that literature. 1 One of the first methodological issues to be addressed is the difficulty of measuring weak effect from services delivered. Technical problems abound because of the lack of standardisation, lack of clear distinctions between means and goal, poor quality control, and inadequate information about what is provided. Moreover, these effects of services must be isolated so that they can be linked to the policy, not to some other environmental events that could have produced the observed changes.

Until recently everyone had accepted that the ideal design for evaluation was the classic experiment with clients randomly assigned to treatment and control conditions. Or, at least, a quasi-experimental design was considered the pre " ferred model (Campbell and Stanley, 1966; Cook and Campbell, 1975). Despite the elegance of these designs where the researcher has some ability to impose needed controls and to be able to measure reliability and validity in an accept " able manner, they are simply no longer possible for the majority of evaluati on that is being mandated today. The required controls simply cannot be applied in real-life situations for a host of operational and ethical reasons. Moreover, much evaluation is done of ongoing programs where clearly experimental designs do not fit the situation. Because of all these problems we now have a variety of design approaches to consider (Cook and Campbell, 1975; Struening and Guttentag, 1975; Coursey, 1978). Recently a variety of methodological advances have occurred in cross-sectional studies so that they can yield valid estimates of program effects and do not require the controls necessary in experimental designs (Rossi and Wright, 1977).

1. There is a wealth of evaluation literature in textbooks, monographs, and journals. Much of the literatur e has been adapted to the various sectors of the human services. For an overall view of the field, the reader is referred to Struening and Guttentag (1975); Riecken and Boruch (1974): Rossi and Williams (1972); and Anderson and Ball (1978).

78

The of 'effects' is also obscured by a variety of other problems, among which are the following: (1) 'Creaming' under which an agency selects clients for service who are most likely to succeed and dismisses the more problematic situation .

(2) Delivery of services can be measured more readily when the service is some type of concrete object, not a service such as 'psychotherapy'. (3) Uncontrolled treatment variation occurs when front line personnel are given broad discretion about service delivery.

( 4) Overly sophisticated treatment technologie s for poorly trained personnel also render a service ineffective. These are but a few of the examples that could be enumerated but they clearly show some of the situations with which the evaluator must cope.

All of the above mentioned problems are concerned primarily with what is termed 'intern al validity', namely the ability of a design to provide unbiased estimates of service effects. In evaluation one must also be concerned about the extent to which the findings can be generalised to the populatio n for which the policy exists-'external validity'. Modern sampling procedures permit us to solve these problems quite satisfactorily. A third type of validity problem is described by Cook and Campbell (1975)-'construct validity'. This term refers to the ability of the instruments to measure what they are intended to measure. Here careful development of instruments and pre-testing on various populations is essential. Certainly, the experience in recent years with intellig " ence test instruments has posed som e critical problems in this area (Bernstein ,

1976). A fourth validi ty problem is termed 'policy validity', and that term refers to the capability of the service intervention to become the basis for policy change. Statist ical and logical procedures are required for such a determination, but one must also consider the substantive utility of the service and of the findings. Cook (1975) observed that, in the evaluation of Sesame Street, pre-school children who viewed the program learned to rec.ognise two more letters of the alphabet as compared to those children who did not see it. The effect was statistically significan t, but one can question the substantive utility of that amount of learn " ing relative to the resources expended.

Methodological problems also abound because of the turbulent nature of the usual action setting in wh ich the measurement of change must take place. The evaluator may wish to impose controls to facilitate assessment, but these controls ma y then interfere with the delivery of services. Even if the services are not known to produce a positive effect, protection of human subjects requires that service to them be the first consideration. It is also frequently observed that

program means and goals both over time, further confounding evaluation.

An approach to evaluating effectiveness In order to present cogently some of the methodological issues in evaluation , the follow ing procedural plan is outli ned. This plan requires that the evaluator work closely with agency personnel from the beginning of the program or that

a decision be made at some point to take stock of an ongoing program so that innovations can be introduce d. The earlier and closer the involvement of the evaluator the fewer the problems and pitfalls that will have to be coped with later. The reader must note, however, that even with a jointly developed plan,

difficulties will still emerge requiring ongoing flexibility and adaptation.

79

The approach outlined and illustrated below requires clear definitions of goals and objectives against which to assess the outcome of the program {Elkin and Vorwaller, 1972). It encompasses the following steps: ( 1) Identification and description of the social problems within the scope

of the organisation's interest. (2) Formulation of goals to address these problems. (3) Description of the objectives of each service element in quantifiable terms. ( 4) Determination of measures of effectiveness for the objective. (5) Establishment of evaluation standards for each service component. The first step in the planning process is the development of a service profile as is presented in Table 1. This profile is a convenient way to organise and present the elements required in an effectiveness evaluation system. The service definitions contain the parameters of the area to be evaluated, the agency domain. Services must be precisely defined and boundaries should be clearly stated so that all participants know what these are and can participate in decisions re their relevance . In the example, Table 1, joint planning by agencies serving youth and the elderly would need to be involved as well as school personnel if youth were to be trained formally in secondary schools. It is also apparent that several service programs are potentially possible in relation to the service goals and objectives.

Table 1 : Service Program Planning Profile

Service category Youth unemployment Home care services to elderly Program

Training and work placement opportunities in new home care services for the elderly to enable youth to obtain employment and to enable the elderly to obtain home care services in their own residence . Problem description

Youth lack knowledge of how to be home care providers and they lack status to obtain other types of employment. There is an insufficient number of appropriate home care providers for the elderl y. Goals Youth unemployment will be reduced and youth will obtain satisfyin g positions .

Home care services will be extended on a permanent and stable basis for the needy elderly. Training and placement programs can be established on a stable and viable basis. Objecti ves To reduce the rate of youth unemployment in 'n' commu nities by 'n' percenta ge in eighteen months.

To improve home care of elderly persons and to reduce the rate of institutionalization . To improve satisfaction levels of youth re employment. Measures The rate of youth unemployment for program participants will be compared for the specific

commun ities and overall for all youth in the defined age groups. Measures of satisfact ion by both youth and elderly to ascertain their views about home care services. Hospitali sation and institut ionalisation rates for the elderl y participants will be compared with comparable populations not receiving home care services of this type.

When services or programs have been described in profiles, they can be grouped into related categories and classified as either operation or program goal oriented. Study of the organisational structures of agencies usually can pro " vide knowledge of how services are delivered and wh at the alternative possibilities m ay be.

80

Descn"bing the problem In this framework, a problem is a condition that adversely affects the identified populations, in this case youth and elderly persons. The basis of future measure " ment will be a determination of the amount of change that has occurred in the problem; so the definitions should be precisely drawn. When one is working in an ongoing program, the process starts with the analysis of existing programs, but one should always entertain the consideration of alternative definitions of the problem since this has often been observed to be overly ambiguous or erron " eous. Operational goals for resolving these problems must be then determined along with the objectives for achieving these goals. For example, the objective for a family planning program might be stated in terms of helping poor persons to achieve a 'better quality of life'. Such vagueness is often the outcome of an attempt to obtain consensus among policy makers of different backgrounds and constituencies . But, such an objective is impossible to operation alise in reference to a finite social program. Implicated in any effort at delineation and measure " ment of organisational goals is the definition of the problem that is or will be the target for change. The way in which it is defined influences the means of intervention for resolution of the problem. If delinquency is defined prim arily in terms of the individual person rather than in relation to th e peer group, neigh " borhood, or other social situation, the goals, targets and technologies should re " late to individual level phenomena. On the other hand, if the problem is viewed as situational, the physical, social or economic environment would be the targets for change.

Goal statement must reflect the realistic constraints upon the organisation offering the services. If there is broad participation of staff personnel and com " munity persons in goal development, the accuracy and completeness of the goals can be insured. This participation also aids in developing more willingness to accept goals. In cases where goals are problematic, as in youth unemployment, such participation is probably vital to program success and continuance .

Objectives The specific targets are the objectives which represent interim pro " gress toward a goal within a specified time period. These objectives must be comprehensive and cover all of the activities leading to the attainment of a

specific goal. They provide the detail necessary to enable decisions to be m ade, actions to be taken and results to be assessed. Wherever possibl e the objectives should be stated in quantitative terms so that measurement will be facilitated. In the service profile just presented, this situation can be illustrated with reference to the objective 'to improve satisfaction levels of youth regarding em ployment'. Although some might think otherwise, it is possible to measure

this objective in quantitative terms, but it would be important to agree in advance wha t the measures of satisfaction would be.

M easurement and evaluation standards The last steps in the process involve determination of the amount of change that has occurred in specified conditions of the client group. Me asures of effecti ve " ness for services m ust provide data that can be used to ascertain the extent to which the objectives are met; to record the tangible impact of the service in relatio n to the problem ; and to evaluate the adequacy of service efforts to accomplish an objective or produce the desired result . Sound me asurement will

81

possess the following characteristics: be observable, quantifiable and have social consequences. Obviously, some measurement procedures are more desirabl e than others with reference to their cost, level of intrusion, reliability and validity and so forth. The evaluator's expertise is particularly crucial in the selectio n of

Conclusion The evaluation of effectiveness forces us to focus our attention on the results and impact of the service program, not just on processes or operations without reference to the goals. At the same time, because process and outcome are linked, one is able to address the question of 'why' something does or does not work. Such answers are essential today in the human services. Examination of effect iveness inevitably involves consideration of organisational efficiency since unlimited resources are never available; so one must always make choices among alternatives in terms of their relative cost. Here again the knowledge about process and impact is essential for intelligent choice of alternatives as recent literature on cost benefit analysis has indicated.

Interest in the evaluation of effectiveness is of particular concern in human service organisat ions because of general consensus that we must improve the allocatio n of resources as well as the planning and management of social service programs. The utilisati on of evaluation in administration serves the purposes of securing information about what works or of the impact of services on target populations. It aids in determining the agency's accomplishments in the light of the needs of clients, needs of the community, requirements of the funding agency, and the capabilities of the agency staff. It is also useful to plan for staffing and specialisation requirements and to determine the cost and benefits of a program. Of course, these are the optimal utilities that could be generated from an evaluation undertaking . Realistically, the effort will probably produce information useful in at least one area.

The approach adopted in this paper has been to view evaluation not primarily as a research enterprise, but rather as a general methodology to aid decision makers in judging the worthwhileness of some activity regardless of the method employed. Integration of evaluation into all aspects of agency operations and services forces those responsible for the agency to specify more clearly their goals and objectives. It also forces the inclusion of empirical measurement into all aspects of agency planning for operations and services. Thus, the administrator has data on which to base decision and a method for obtaining continuous infor " mation in -order to monitor agency operations. Evaluation also aids in having a 'systems view' of the agency, one in which all of the interrelated sub-units can be examined as to their effectiveness and efficiency. Such a view facilitates long-range as well as short-term planning.

Viewing evaluation with reference to optimising the functioning of human service organisation permits the agency to be operated more on the basis of rational planning, despite all of the problems with that perspective today (Sarri and Hasenfe1d, 1978). A major criticism of evaluat ion research to date has been that it is aimed at proving or disproving program success. Isolat ed evaluation of one agency component tends to serve that purpose, but by consciously integ " rating evaluation into all aspects of the agency's operations , the success or failure of one service operation will not preclude the successful operation of

82

the agency. If evaluators and staff and clients are all involved in an inter " dependent relationship in which each has a particular contribution, the results will go a long way toward increasing the probability that practitioners, adminis " trators and policy makers will seriously take account of the finding from evalua " tion. And that, after all, is what it is all about!

83

REFERENCES

Anderson, S. and Ball, S., The Profession and Practice of Program Evaluation. San Francisco: Jossey Bass, 1978. Attkisson, C. Clifford et a!., Evaluation of Human Service Programs. New York: Academic Press, 1978. Bernstein, Ilene, (ed.) Validity Issues in Evaluative Research. Beverly Hills: Sage, 1978. Campbell, D. and Stanley, J., Experimental and Quasi-experimental Designs for Research. Chicago:

Rand McNally, 1966. Cook, T. and Campbell, D., 'The Design and Conduct of Quasi-experimental and True Experi " ments in Field Settings' in M.D. Dunnette (ed.) Handbook of Industrial Organizational Research. Chicago: Rand McNally, 1975. Coursey, R. (ed.). Program Evaluation/or Mental Health. New York: Grune Stratton, 1978. Elkin, R. and Vorwaller, D., 'Evaluating the Effectiveness of Social Service', Management Controls,

May 1972, 104-11. Friedson, E., The Profession of Medicine. New York: Dodd Meade, 1970. Hasenfeld, Y., and English, R. (eds), Human Service Organisations. Ann Arbor: University of Michigan Press, 1974. Helfgot, J., 'Professional Reform Organisations and the Symbolic Representation of the Poor',

American Social Review, 39, 475-91, 1974. Riecken, H. and Boruch, R ., Social Experimentation. New York: Academic Press, 1974. Rossi, P. and Wright, S., 'Evaluation Research: An Assessment of Theory, Practice, and Politics' , Evaluation Quarterly, 1:1, 5-50, 1977. Rossi, P., 'Some Issues in the Evaluation of Human Services Delivery', in Sarri and Hasenfeld,

op. cit., 1978, 235-61. Sarri, R. and Hasenfeld, Y., The Management of Human Services. New York: Columbia University Press, 1978. Struening, E. and Guttentag, M. (eds), Handbook of Evaluation Research. Beverly Hills, California :

Sage. 1975.

84

Need and demand in the making of

planning decisions in the evaluation of health and welfare services

I Introduction We have taken it that our particular contribution in a discussion of the suggested topic is expected to come from our interest in the consumer experience of health and welfare services. Firstly we must make it clear that although we attempt to direct our professional concern to the consumer experience, our point of view in no way pretends to represent consumers' views, or to be any substitute for a direct expression from consumers of health and welfare services, either as individuals or as groups. A further initial point we wish to make is that at the practice level of service delivery and of evaluation and of planning, consumer participation in any developed form just does not exist. We are therefore in a better position to talk about the space around the topic than we are to discuss any developed body of knowledge or of experience relating to consumers' participation in the areas suggested.

What we have to say comes partly from our experience at the point of inter " action between the 'server' and the 'served' (but still from the 'server' perception), and partly from our attempts to learn from relevant literature and shared experiences of colleagues. This point we refer to as the 'consumer face'.

Although we are aware of many differences between health and welfare services and structures, in our discussion, which is very general, we use the terms inter " changeably, seeing services in both areas as a societal response to human needs that primarily arise from normal life cycle situations. Our view is that such services should be supports provided by a society through community structures of various sorts to ensure that all citizens can comfortably manage their own lives in accordance with the prevailing standards and expectations of the culture within which they live out their lives. We are committed to the value stance that health and welfare services should spring from a positive expression of a society's collective responsibility to provide adequately for the fullest participation of each of its members, no matter what the range of individual differences in abilities and capacities might be.

It seems to us to be important to take this stand clearly at a time when health and welfare programs at all levels of government are being curtailed and at a time when the most vulnerable members of our society are bearing the brunt of the enormous structural changes that are occurring in the economy, not just nationally but globally. There is a danger that 'evaluation' will be used more often as a rationalisation for arbitrary cutting of programs, than as a genuine attempt to develop more effective ways to do what should be done.

We therefore see service delivery aimed at human growth and development rather than as an instrument of social control.

85

We prefer to point to the range and diversity of human characteristics, rather than to seek the 'central norm', and suggest that interpersonal relationships are a crucial part of the 'consumer face' experience, whether the service being delivered is a highly individualised, personalised one such as a one-to-one relationship between a clinician and a patient, or a face-to-face contact at a Department of Social Security counter between a public servant and a pension applicant, or an educative group situation at a Community Health Centre.

n The concept of evaluation Broadly speaking, the term 'evaluation' refers to the process of determining the worth or value of something, 'against some explicit or implicit yardstick'. 1 A perusal of the increasing amount of literature available about evaluation in the service fields (such as health and welfare) reveals a conflicting multitude of purposes for embarking on the evaluation of service programs.2 For example, the purposes in evaluating can range from the legitimate desire to determine the actual costs of a program in terms of money and human effort, to the less worthy and often covert purposes of postponing decisions, avoiding responsibility or cutting back on programs. 3 Also, the literature reveals a host of evaluati on techniques and strategies that are being developed specifically to deal with all stages of a program from a variety of perspectives . Tripodi for instance outlines many different techniques for assessing the effort, effectiveness and efficiency of a program through all its stages from planning to implementation .4 Other writers highlight the importance of developing different techniques to evaluate the actual structures and processes as well as the 'outcomes' of service programs. 5

The multitude of health and welfare services in this country vary greatly in the degree to which their goals and objectives have been spelled out. Many of their aims are very diffuse, and, as stated above, the purposes of those who plan evaluations are also varied. Thus the explicit yardsticks against which pro " grams could be evaluated will also be numerous and diverse. However, it is worth recalling the fundamentally altruistic connotation of the word 'service' " as defined in the Oxford Dictionary it is: 'work done or doing of work for, benefit conferred on, exertion made for another'. 6 The primary purpose for which health and welfare services have been established, the basic rational e for their existence, is presumably to provide something for the benefit of the com " munity served, to meet some community needs, even if many of these needs have not been clearl y articul ated. J. Najman et al. make this point clearly in an unpublished paper on conceptualising and evaluating community health pro " grams. They discuss the importance in evaluating programs of understanding the overt and covert goals involved.

They state that in community health programs the 'superordinate goal' should be 'responsive to community needs', as a community health care approach is 'based upon the principle that health care facilities should meet community needs'. They then go on to discuss the 'subordinate' goals of treatment, liaison, co-ordination, education and prevention. 7 Although their paper is only considering community

health services , we feel that this highlighting of the response to community needs as the superordinate goal should hold true for other more traditional health and welfare service s, if the altruistic base underpinning all services is to be upheld. Therefore other important goals such as running services more efficiently in terms of money and manpower, improving techniques of service delivery , and promoting harmonious working relationships amongst service personnel are really only legitimated by their bearing on the fundamental goal of meeting people's needs or improving in some manner their quality of life.

86

We agree with Carol Weiss's thesis that the most central purpose in evaluating service programs is to assess 'the effects of a program against the goals it set out to accomplish as a means of contributing to subsequent decision making about the program and improving future programming'. 8 And if we take, as argued above, the view that meeting people's needs should be the basic goal underlying health and welfare services, then the most crucial purpose of any evaluation should be to try and determine whether those services are meeting the particular human needs for which they were established. 9 Although hopefully the evaluation will be assessing the more specific objectives of a program (if the service has

actually defined its objectives specifically), these objectives themselves should be defined from the program's broader goal of meeting community need. As Dr L. R. Denton neatly sums it up in an excellent article, 'stating objectives is a matter of specifying what the programs or services are to accomplish in the

meeting of needs or solving of problems'. 10 While we assert that all (health and welfare) services should have been establi shed to meet community needs, it is quite possible that m any were started, and continue to operate, for less legitimate reasons such as public relations, or providing jobs for service professionals, with scant attention paid to th e real needs of potential or actual consumers. (In fact, the authors are personally aware of a number of these situations, as would be many other workers in the health and welfare fields.) But if evaluation is carried out from the perspective of meet " ing community needs, then at least it would uncover the fact that some services

were not even aware of the relevant needs of the catchment population, let alone doing something appropriate towards meeting those needs. Another vital purpose in evaluation, as stated in the quote from Carol Weiss above, is that it is done to enable better decisions in the program pla nning so that future services will be improved. Unfortunately all too often, as E. G . Guba has pointed out, evaluation has been 'something required from on high that takes time and pain to produce but which has very little real significance for action'. 11 This comment was made about the American scene in the 1960s, but to our knowledge would be equally applicable here in the 1970s. If evaluation is under "

taken for the right reasons then it should not m erely be imposed from above, but should involve the workers and consumers at the lower levels of the structure. Only then can the evaluation be seen as an integral part of decision making to ensure the relevancy of future programs. In sum, the concept of community needs and of participation at a variety of levels is absolutely central to the evaluation question: no meaningful planning decisions about evaluating health and we lfare programs can be made without close attention being paid to the real needs of

the community being serviced.

Ill Need and demand

(a) The complexity of these issues 'In many public discussions it is difficult to tell, when someone says that "society needs ... ", whether he means that he needs it, wh ether he me ans society ought to get it in his opinion, wh ether a m ajority of the m embers of society w ant it, or all of them want it. Nor is it clear whether it is "needed" regardless of t he cost to society' .12

The question of defining wh at constitutes health and welfare needs for any com mu nity is a complex matter. The issue has been expressed most succinctl y by Dr L. R. Denton in his paper t o the Canadian Psychological Association on

24895/79-7 87

the evaluation of Community Mental Health Centre programs. While his dis " cussion centres around the field of mental health, he makes it very clear that his comments are applicable to 'any human or social service program', and it is worth quoting extensively from his discussion on the definition of 'ser vice' need:

One might begin by noting a humanitarian social value about which most people in our culture would agree. This is that all persons should be provided equal oppor " tunities to develop and use the capabilities they require for 'normative function' in the economic and social life of the community according to their potential to do so. It follows that any condition or situation that prevents or reduces a person's or group's capability to develop or function 'normatively' may be defined as a service need. The detection of such needs triggers off activi ,ties (programs or services) to meet the need to solve the problem. From this basis services spring, whether they be programs to care for neglected children, the aged, the unemployed or for improving the lot of those with some physical, mental, social or other kind of disadvantage. The program or service aims to reduce the disadvantage 'by increasing or equalising the person's or group's capacity for more normative function in the community's economic and social life . . . Any . . service need may also be defined as any type of condition or situation that limits a person's or group's capability to develop or function normatively.l 3 It follows that to detect or such needs would involve obtaining data at many levels about the community. To begin with, information would be needed about those conditions in the community which, as Denton says, 'by their presence or absence reduce, limit or prevent normative function'.l 4 One would also need to understand and collect information about the power structures of the com " munity and the attitudes within these structures to the people with service needs -i.e. the consumers and potential consumers of health and welfare services. And, most importantly, data on needs as felt and perceived by the consumers or potential consumers themselves must be systematically collected. Many of these variables concerning need are interrelated and a multi-pronged approach is neces " sary at both the micro and macro levels of society, in order to arrive at a mean " ingful and workable understanding of need. In fact, we would agree with Elery Hamilton-Smith who suggests that the concept of real or intrinsic need is not amenable to direct measurement, but can only be understood, albeit imperfectly, by considering all of four dimensions of need which are capable of being measured or assessed more directly. 15

These four dimensions of needs were first mooted by Jonathan Bradshaw when he suggested a taxonomy of need based on four separate definition s of need " normative, felt, expressed and comparative. 16 His concise paper has been instru " mental in clarifying the need concept, and Elery Hamilton-Smith has refined these ideas further by suggesting-rightly we believe-that these are not four separate and alternative definitions of ∑ need, but rather 'four dimensions along which one may make measurements which, taken together, may help us to form an improved but still incomplete picture of that abstract notion which could be called "intrinsic need" '. 17

NORMATIVE NEED This dimension is what the expert defines as need in any given situat ion. A desirable standard is laid down by 'experts' and if an individual or group falls short of this desirable standard then he/they are identified and defined as being in need. The Henderson poverty line is a good example of a normative need.18 In no w ay is a normative need absolute-it will vary according to the particular

88

bias, assumptions and values of the expert involved. Often there can be different and conflicting standards laid down by various experts and many of these stan " dards are, according to Elery Hamilton-Smith, 'based on assumption or some " times on what can only be described as hunches, with little or no testing of their validity' .19

The great advantage of the normative approach to need lies in its simplicity " it is relatively easy, for example, to calculate whether a person is below the poverty line-and it is this simplicity that often appeals to bureaucrats and legislators. But in the health and welfare field there are unfortunately many areas

about which there is no common agreement on a norm, such as what constitutes good physical health or mental well-being. Social indicators are one kind of normative statement of need that have grown in popularity in recent years. Social indicators can be described (as Shonfield and Shaw define it) 'as a measurement of social phenomena which ...

is normative and integrated in a self-consistent information system'. 20 They represent an attempt to introduce common definitions and categories at the macro level. The development of such across the board comparable data at the national level would prove very useful to legislators grappling with the issues of national

planning and resource allocation. However, when viewing social indicators as a measurement of community need, it must be remembered that they are subject to all the limitations of other normative statements. Hamilton-Smith warns that 'because of their apparent simplicity but real com " plexity their use is all too likely to lead to decisions being made without a full understanding of the built in assumptions or valuations which are inherent in their design'. 21 They, like all normative definitions, must be supplemented by other more personal definitions at the micro level. For the normative approach can mask the vast range of individual differences and needs. There is no way in which individual perceptions of need or satisfaction can be built into social indicators, or other normative statements.

Another drawback with social indicators (and other normative statements) in relation to measuring need is that they are sluggish and resist dramatic changes, yet human needs are constantly changing. This problem exists also if we consider the use of social indicators for evaluating the role of health and welfare services in meeting human needs. For as Carol Weiss points out, many health and welfare programs only reach a small number of participants and make small improve " ments.22 It may be asking too much to expect a service to change people sufficiently for this to be reflected in the macro social indicators-even allowing for the

inevitable time lag in recording any change.

COMPARATIVE NEED In this dimension, a measure of need is found by studying the characteristics of the consumers of a particular service or group of services , and then comparing these to people with similar characteristics in another area-making weighted comparisons if necessary to take into account the differences between the

populations. The gap between those services that exist in one area and the other is then established as a need in the 'less well served' population. This approach does at least avoid the issue of making normative assumptions in the absence of common agreement, and it is an important aspect to consider in the light of our increasing concern with social inequality and unequal distribution of resources. However, it is too simplistic to use on its own as a measure, for as Bradshaw

sums up: 'this is an attempt to standardise provision , but provision may still not correspond with need'. 23 89 24895/79--8

FELT AND EXPRESSED NEEDS

The other two dimensions of need in Bradshaw's taxonomy are felt needs (people' s own attitudes and feelings about what they want) and expressed needs (often termed 'demand', actual behaviour of people in using or asking for a service " the waiting list for a day care centre, for example). It is the dimensions of felt and expressed need, rather than normative or comparative, that we wish to con " centrate on in this paper, and that we wish to relate specifically to evaluation.

Felt and expressed needs are perhaps the least understood and the most difficul t to assess; yet they may be the most important, given that services are supposed to be serving, or meeting people's needs.

In attempting to reach some understanding of real or intrinsic need it is also important to balance the more objective broad ranging (or macro) normative and comparative views with the more subjective views of the consumers them " selves. For there is ample evidence to suggest-at all levels of society, not just the disadvantaged sector-that professionally determined (i.e. normative) needs are often at variance with the felt or expressed wants of the consumers, and are themselves not free of bias or irrationality. 24 As Tranquada aptly states:

Professionally determined needs are likely to be those priorities for institutions, pro " grams, services or commodities which can rationally be defined and ordered by qualified planners as directly beneficial to the health of a community. Nevertheless, we .are confronted by a range of choices by which to reach 'rational' conclusions and such conclusions may be deflected by the professional from the strictly ration al to the extent that unrelated priorities for economic gain, education or researc h may be included. Even so-called 'rational' conclusions may be biased by ec∑ onomic considerations (which measure would provide the greatest economic return for the investment?) social factors (who is in greatest need, or where are the predictabl e effects on education or employability of a given program?) or purely upon sterile quantitative data which provide information about the numerical mass of unmet pmblems.25

We have emphasised several times this fact that normative need is not free from biases and assumptions often built on an inadequate data base. Yet too many people discount the dimension of felt need for being too 'soft' and 'sub " jective'. But we feel that it is equally necessary to add the consumers' biases and assumptions to the already dominant biases of the policy makers, profession als and administrators . Perceptions from the micro or consumer viewpoint may also have the advantage of being less rigid, and more sensitive in reflecting changes than the normative macro perspective. J. Weston et al. talk of the spectacul ar failure 'of many broad aim service programs'. 26 We feel that this failure may be partly due to the services' lack of understanding of the perspective of the target populations. Need from the consumer's viewpoint has been discounted or at least

not understood, and consumers have not been involved in all stages of service programs.

All the dimensions of need are imperfect , which is all the more reason why the four mu st be taken into account, but at present there is a heavy imbalance towards acceptance of expertly determined needs. Michael Jones observes that the notion of interdependence has probably been one of the more important discoveries in applied social research in the last decade. 'No longer are social problems seen in isolation.' 27 This concept of interdependence applies as certainly to the definition of needs as it does to any other social phenomena.

90

(b) importance of 'felt' needs as distinct from 'demand' It is at the point of contact and of interaction between those delivering a service and those consuming a service that the failure to accord significance to felt and expressed needs becomes evident. The study Who Cares? is an attempt to demon strate the crucial importance of felt needs, and the interdependence of

a multiplicity of factors at the consumer face. 28 The methods used in the study were something of a reaction to the normative approach of the earlier Henderson Poverty Survey. 29 Henderson in his introduction to Who Cares? says: If we were to try to summarise what we see as the special contribution of our work,

we might say it lies in our attempt to present the experience of families, caretakers and welfare workers each from their own perspective. We have been able to docu " ment the crucial fact that families prefer to deal with their problems within their own personal worlds . . and we show that service givers, both those who see themselves this way and those who do not, often unknowingly, operate 'worlds apart' from the families with whom they come in contact.so

Teachers in the welfare and health professions have traditionally acknowledged the need to bridge the gap between theoretical perceptions and the reality of the work place, be it hospital, health centre, welfare agency or children's home, by insisting on a balance of field training for students. One of the most valuable

lessons a student learns in a field situation is that there is often a large gap be " tween theory and practice. A student's fresh eyes often see more clearl y than the established worker in the field how frequently confused and puzzled the consumers are about the response of the service deliverer to their expressed

needs. Often the consumers see the service response as quite irrelev ant to their own definition of their problem. An outstanding example of such irrelevance was the mis-application by welfare administrators and professionals in the 1950s and 1960s of micro-behaviour al and psychiatr ic concepts to the dilemmas faced by

the poor. We are still making such mistakes and because of the failure to invest in the development of the appropriate knowledge and practice base, we will con " tinue to make such mistakes for longer than we need to. Such qualitative and practice based projects as the Action Resource Centre of the Brotherhood of St

Laurence in Melbourne need to be actively pursued at a wide variety of points in the network of service delivery organisations. Jeffers et al. make the point that concepts of 'want', 'need', 'demand' and 'shortage' are often confused and ill-differentiated by economists and health professionals. 31 Demand is not only an unreliable indicator of need because of confusion between experts about the variety of meanings involved, but also be " cause once again the focus of perception and of understanding is pushed away from the reality of the consumer experience. As we have already suggested, when

health and welfare professionals are asked to estimate need they are more likely to rely exclusively upon concepts of normative or comparative need, rather than attempting to include an understanding of felt need in their formulatio ns. This tends to create distorted notions of demand which are ultimately as frustrat ing to the conscientious professionals in the delivery of a service as they are to the consumers, who approach a service expectantly only to be confused by their

experience. A most destructive aspect of such an interaction is that consumers are led by their experience to mistrust the validity of their own judgment and are often coerced into translating their expression of need into a spurious expression of their problem. We suggest that it is only by concentrat ing on the development

of a better understanding of felt and expressed needs that planning and delivery of services appropriately geared to consumer experience can be achieved. Further

91

it is only by involving the consumer in the evaluation of services that the gap in the perceptions of felt and expressed needs and normative needs can be nar " rowed, so that eventually a concept of demand can be based upon an interdepen " dent view shared by professional and consumers alike.

Such gaps exist for a variety of complex reasons. Probably the reason most often quoted despairingly by professionals is consumer ignorance. The difficulties for consumers of understanding complex technical aspects of medical treatment and welfare programs are used, at the worst, as a rationalisation for the imposi " tion of inappropriate services on consumers, or at best for the justification of the delivery of a service that is technically appropriate but fails in efficacy because of the differences in understanding between the server and the served. Professionals who have tackled the application of modern technological know " ledge in underdeveloped countries are well aware of such pitfalls and are geared to pick up and deal with these impediments to their work, but in a situation often wrongly assumed to be monocultural in our own community, the more subtle expressions of the rift are often missed.

At the consumer level the consequence of these perceptual differences is strongly reflected in the notorious reluctance of people, especially the poor, to use certain services and to benefit from new developments. Independence and self-sufficiency are values strongly adhered to not only by the leaders in our society but also by the 'little' people. Social welfare personnel are familiar with such attitudes as barriers to the efficacious delivery of services and there is strong evidence to show that many people, even when under tremendous pressure of need, are reluctant to seek help, at least outside their personal networks. For some it is more legitimate to seek help when the problem relates to ill health, but for most, when a social problem arises values held mitigate strongly against the seeking of outside help. 32

But these difficulties, although they are concerned with consumer behaviour, must be tackled at the professional and organisational level as well as at the consumer level. We have recently seen the beginning of consumer awareness in the fields of health and welfare in the form of self-help and consumer groups. But we suggest that professionals and experts should properly be required to include a responsibility for consumer ignorance and consumer reluctance in their daily professional performances. This implies a far more open and less exclusive style of professional and organisational functioning than we have traditionally pur " sued. It requires a less directive and less self-seeking style of operation.

Other reasons, of equal significance, pertain to problems of organisational functioning. Sociologists interested in studying bureaucracies do not have to go far in the health and welfare fields to find examples of organisations and institutions which may have started out with the best of intentions to achieve the worthiest of goals, but which have become bogged down in self-perpetuating processes which have more to do with the needs of governing bodies or of permanent employees or associated professionals than they have to do with the pursuit of what we have earlier referred to as the supra-ordinate goal of meeting community needs. Kitay refers to the way in which medical practitioners are able to impose their own definition of what a client's needs are by maintaining strict control over their access to the client. He comments:

This is not simply a result of lower class life-style, but of the dominance of the medical practitioner over the system of health care. By providing for treatment rather than preventive care, 'and by making those services both difficult for the poor to obtain and unresponsive to their values, the health care systems of western societies tend to lower the .Jevel of effective functioning among the lower class. 33

92

A more specific and detailed account of a particular welfare agency is given by Scott illustrating the same tendency for the functioning of established interests to result in service delivery unrelated to felt needs and in ineffectiveness in meeting felt and expressed needs. 34 Scott compares the distribution of services for the blind to the demographic characteristics of the blind population of a particular area and points to the comparative narrowness of the program delivered by the agency concerned. He suggests that such difficulties arise for a variety of reasons but emphasises two features of the situation as being especially important:

First, welfare services are characteristically distributed in our society through private philanthropy or government at its federa-l, state and local levels. These programs are. ordinarily incorporated in large scale bureaucratic structures. As such they are subJect to the pressures and forces common to all complex bureaucracies. The preservation of the organisation itself is a vital factor in setting program policy; and standardisation based upon the criteria of efficiency, production, and costs is often applied to services which are to meet highly personal needs. Second, welfare programs must rely upon the public for their support, whether through legislative appropriation or private fund raising efforts. The availability of services

depends, at least in part, upon the kinds of support which the benefactors of welfare are willing to provide. When the benefactors are the body politic, funds will ordinarily be made available for only those programs which the legislators believe are politically tenable to support. When the benefactors come from the private sector of society, the kinds of programs they are willing to support depend upon their personal conception of the nature of the problems of disadvantaged groups, and what they imagine constitutes a desirable and moral solution. In either case,

such conceptions are generally responsiw to broad cultural themes and values ... all

The concept of 'demand' then that is usually relied upon when planning de-cisions are made customarily relates to either questions of expediency connected to the influence of the most powerful lobbying interests (not necessarily represen " tative of felt and expressed needs at the community level), or to the way interested 'experts' narrowly interpret the needs of their clientele. This sets up a self " fulfilling prophecy in a situation where practice in service delivery does not allow for more accurate communication between the server and the served at the con " sumer face. Health and welfare professionals and personnel provide a visible service which often circumscribes their own and their client's presentation of 'need' and 'demand'. Both professional and client s express needs in terms of

the conceptual filter offered by the range of the services delivered. This process needs to be counterbalanced by a strong profession al and organisational commit " ment to the client's view of his circumstances and to a belief in the validity of his expression of his needs.

It would be naive to assume that the practice base at the consumer face is sufficiently well developed to ensure that an immediately accurate stream of information could be from this point through to policy makers. The impoverishment of funding for welfare programs has continually ensured that welfare personnel have been forced to develop their social control function s (e.g. of establishing and administering eligibility criteria for services) to the detriment of the development of their skills in touching the reality level of the 'felt' needs of their clients. The social control aspect of welfare services is well understood by consumers. The often quoted fear of 'losin g the kids to the welfare' is not just

a paranoid fantasy. Generations of poor people know the reality base of this fear. Welfare professionals have been slower than their clients to come to a realistic understanding of their social control function. They are, however, increasingly finding themselves in conflict with their boards of management and administrators

93

if they attempt to forsake their social control function for a more radically therapeutic role aimed at promoting personal growth in their clients . Similarl y the concentration in the health services on treatment of illness rather than on promotion of good health has ensured that this area of practice is an undeveloped and an under-financed one. The comp arative infancy and lack of expertise of only recently emerged primary consumer groups as lobbying influences is a further significant factor operating against adequate representation from this level.

This implies, at the practice level, an integration of the experience of the interpersona l relationship between the server and the served with an evaluation function for both: ... a central fact of human existence (is that) interpersonal relationships matter.

It is through them that we meet our physical, psychological and social needs, and if they are satisfactory it is within them that we develop into people in our own right, and we are able to retain our humanity. It is also interpersonal relation " ships that assumptions, values and attitudes can be questioned and influenced, which is often a precursor to the creation of social policies and significant changes in the way a society looks at, and .Jooks after, those wh o are disadvantaged within it. All this may be stating the obvious rand sound simplisti ,c and naive. But in a world increasingly committed to and dependent on technology and social engineering in dealing with its problems, we are apt to neglect the power that lies in relationships

and the vital part they play in any change program.36

(c) Issues in the measurement of 'felt' and 'expressed' needs By the early 1960s many Australian administrators, policy makers and service staff were becoming increasingly aware of the gaps between services-that is, of the areas of need not really covered by any health or welfare program-and of the irrelevancy of some existing services due largely to the sorts of processes and influences discussed immediately above. More attention was therefore given to finding better methods for identifying the needs of consumers and non " consumers of services. Service personnel were keen to collect more objective data to assist in the co-ordination of services and the avoidance of overlap. Popu " lar service groups such as Apex, Lions and Rotary became involved with local welfare services, and actively promoted the doornock-simplistically applying market research technology to the quest for gauging felt needs.

It was during this same decade that sociological theory finally made some inroads into government policy making. As J. We ston et al. point out, it wa s only in the late 1960s-several decades behind U .S.A. and Britain-that social planners were accepted into the policy making hierarchy in Australia. 37 Sociology as a discipline was at last gaining credibility, and its technique of the survey questionnaire became increasingly popular as a tool for assessing the feelin gs, attitudes and opinions of a wide range of people. Our review of the literature on evaluation bears this out: by far the most popular technique advocated for the measurement of felt needs is the survey questionnaire. 38 Many professionals can probably point to certain localities in Melbourne that have been surveyed to saturation point because of their geographical proximity to a research -oriented institution or because of the interesting socio-economic or ethnic characteristic s of their populations. And in most cases the respondents who give their time and effort to these surveys are not given any meaningful feedback. It would be rare indeed for the results of a survey on health and welfare needs to be written in terms readily understood by the researched population. 39 A more serious problem than the lack of feedback to the surveyed population is the inadequacy of this traditional methodological approach in really tapping the felt needs of the

94

respondents. 40 Over the past decade or so there has developed within the discipline of sociology a substantial theoretically based critique of the mechanistic and normative perspective, which is epitomised by the quick survey technique. 41 However, the implications or even knowledge of this critique have not yet filtered through to many of those responsible for evaluating services . In fact this pheno " menological or ethnomethodological perspective will most likely be strongly resisted by the busy evaluator for the standard survey technique is much faster, easier to administer, and provides data that is easier to interpret. These are

attributes that naturally appeal to the busy administrator or staff member, with little time or money available with which to demon strat e the worth of a service, or to assess the needs of the group for which he has to devise .a program. It is clear to us, however, that the phenomenological or ethnomethodological

perspective contains a compelling truth which would be dangerous to ignore when considering the notion of need. Put most simply, ethnomethodology 'is concerned with the sociology of everyday life: routine habits, relationships " all that we take for granted in our life in society. Its practitioners believe that

any scientific understanding of human actions necessarily begins with and is built upon an understanding of the commonsense meaning and actions of the members performing those actions'. 42 The phenomenologists' quarrel with the more traditional sociological methodology is well summed up by Michael Liffman, when he states that, to the phenomenologist, 'much of social science and its methodology is suspect because the explanations it offers themselves derive from

the assumptions, logic and methods which social scientists impose on the pheno " mena they study. For instance, conclusions based on questionnair e findings assume a consistent and straightforward relationship between people's answers to a highly contrived and artificial series of questions, and the way in which they actually make decisions in their own real world.43 Anyone who has been approached to answer questions in a market research or similar doorknock type of survey will have experienced the frustration of being asked a set of structured questions that do not allow a very meaningful response. This often leaves the

respondents feeling unsatisfied with the superficial answers they have been forced to give, which they know do not reflect the reality or complexity of their feelings and attitudes Conclusions about people's felt needs based solely on survey questionnaire data could thus be very misleading if one accepts the critique of the phenomeno " logists. What is the point of designing programs if most of the data on which

they are based do not contain a true reflection of perceived needs? For a service program cannot really succeed if it has not at least partly sprung from the reality of the potential consumers' needs as felt and experienced by them. This latter perspective we see as an essential counterbalance to the other equally valid

considerations of normative and comparative need. IV Consumer involvement in evaluation: the importance of participation in service delivery and of the opportunity for reciprocal exchange of care and infonnation between people

The call for evaluation in the 1970s has followed the cry for participation in the 1960s. But the unachieved polemic of the 1960s has stayed with us and perhaps it is only in a linking of the two processes that we may have a of moving beyond polemic. We have already referred to the many different notions of need and of evaluation. Beringer refers to the:

. . common proposi∑ tion among social planners and administrators . . . tJhat authentic social development requires inoreased citizen and consumer partici " pation in the planning process. 44

95

He also refers to the ambiguity of the term and arrives at a broad statement of meaning which is useful for our discussion: One can view participation as being the effective involvement of people in a broad range of decisions which have important influences on them both individually and

collectively. 45 He notes a range of effects that are expected to flow from the process of par " ticipation in planning; a sharing of power with the 'have-nots', a development of a vested interest in maintaining a system developed from a participatory base, legitimising of planning processes, reduction of social distances between citizens, early identification of needs, mitigation against feelings of alienation and dis " sociation from the social group, and so on. Beringer goes on to trace the evolution of participation in Australia and suggests that:

One of the greatest impediments to participation in Australia has been the absence of a clear declaration of government plans and policy and generally a limited dissemination of information by responsible authorities. 46

He draws a distinction between representational democracy and participatory democracy and points to the remoteness from the people of many Federal, State and voluntary organisations. He pins his hopes for the development of partici " patory processes to the promotion of regionalisation and sees the sharing and dissemination of basic information as essential to such a development. He sug " gests that:

. . some of the dilemmas encountered in attempting to encourage particip ation can be resolved by recognising and adopting methods that fit the particular organi " sations, whether it 'be a voluntary or statutory agency, or 'confederate' planning body such as a Regional Council for Social Development.47

We suggest that an appreciation of the significance of what Titmuss has called reciprocity is a key factor in the process of encouraging participation. Beringer quotes Alinsky's assertion that: To give people help without them having played a significant part in the action

makes the help itself relatively valueless, and contributes nothing to the development of the individual that you are ostensibly 'helping'. ∑ In the deepest sense it is not giving but taking: taking from their dignity. Denial of the opportunity of participat ion is denial of human dignity and democracy.4s

It is at the level of individual and small group life that service delivery is experienced by the consumer. The development of regionalisation does not necessarily guarantee participation. It could be argued that increased efficiency in organisation and administration of programs could lead to an increase in the social control and coercive functions of services unless the consumer is guaran " teed as a right a participatory role. Expectations of providing such opportunities should be explicitly put upon the organisations and personnel responsible for the delivery of a service within the terms of any government funding provisions . It is pointless to simply make a value statement of the worth of such an exer " cise unless the funding of the service and the costing at the detailed level of the work load expected of personnel make it possible for those working at the consumer face to realistically pursue such goals in their everyday practice.

Titmuss asks a fundamental question 'Why give to strangers?', and suggests that the right to give is a fundamental requirement for adequate functioning in a human society. 49 Reciprocity is generally only popularly understood in the 'lucky' society in Australia by the payment of money in return for material goods. No one, it is asserted, values anything they do not have to pay for. But

96

there are, certainly in the health and welfare fields, and in the field of human relationships generally, exchanges other than material ones that are highly significant in our lives.

The 'right to give' suggested by Titmuss can be seen as an essential require " ment for a human environment, providing opportunities for active exchange between people at various levels of the community of their knowledge and experience to the mutual enrichment of all. Such exchanges are of course significantly affected by the access people have to adequate information and adequate capacities to make such exchanges. This implies a host of privileges and skills not available to all in our community, and suggests the importance of the development of strategies to make such privileges and skills within the reach of more people. Macbeth suggests that there is an unequal control and distribution over communication systems in our society and points to the passive nature of the consumer role in the use of the communication system. 110 He views information as a key commodity for those who would actively participate in the society, and sees a loss in our culture of two-way communication:

A concept of information which adequately expresses the needs and problems of people in Australian urban society must recognise that unless people can make some responses to the information they receive, it is not very meaningful to them. People must be a'ble to question and test the information they receive and to influence, change and create their own information. There is a real need for people to feel that they can participate effectively in the processes of information-a need to talk back and share their knowledge For a social system to work effectively it must have feedback-a properly working two-way flow of information. Unless a system has real feedback it cannot be sensitive to changes, cannot monitor the effects of action on various sections of itself, not be aware of malfunctions, blockages and growth. The directing sector will become frozen in 'traditional' responses which are not sensitive to constantly changing reality.s1

This original piece of local Australian research on communication has much to teach us in the health and welfare fields. As part of their methodology the researchers employed processes they termed 'search conferences'. These were used in conjunction with a household survey as well as a variety of other approaches. The search conferences are of particular interest to us because they we re aimed at tapping the complexity of the situations under consideration. They revealed data which pointed among other things to the importance of personal and small scale systems of communication in people's lives. From this they argue for the importance of what they call a 'counselling' role in the promotion of meaningful communication between people in a social system:

Our society is to the point of complexity whereby m any people do not even know what question it is they need to ask in order to satisfy a need which is so personally complex that they cannot make the need explicit ly known.

A dialectical! inter active process is needed whereby the individu al is a'ble to discuss the questions and the answers that make up the web of subjective desires and experience which give the facts a social and personal context and usefulness. We used to be able to, and to some degree still can, use our personal and informatio n

interactions to under,take this interactive development of the information question, but we are now less effectively able to do so because of such factors as the complexity of information needs and the fragmentation of informal society . The information professional, therefore, must take part in an interactive ('counselling') process as well as a fact gathering process. 52

97

The search Conferences made one message very clear: there is need for the re-emphasis of human scale and human rather than material/organisational pro " cesses in our information systems. 53 Macbeth thus demonstrates that the pro " motion of community participation is essential in a healthy communication system. Participation of the consumer in the development of health and welfare ser " vices is seen to be equally important by Paul Gross:

Oommunity development is too important to be left solely to governments, just as health and welfare reform is too important to be left to the professionals in the field. The consumer is the forgotten component of the health ,team . . . the consumer is the ignored component in social change generally. The question is not whether we can afford to ignore the consumer much longer, but how do we involve him fast. 54

The consumer has been more completely ignored in the area of evaluation than in any other area of the health and welfare industries. For while 'consumer participation' in service programs has become a popular catchcry over the last decade, rarely has this concept included consumer involvement in the monitoring and evaluation of health and welfare programs. As Hessler and Walters point out: 'nowhere does one find the consumer involved in evaluation research in the health field'. 55 In most of the literature on evaluation, the consumer is not credited as having any but the most peripheral role to play in evaluation-and in many cases is just not mentioned at all in this context. 56

If we accept, however, that the fundamental goal of health and welfare services is to meet some needs of the community being served, then recipients of services should have a crucial part in assessing whether or not these services have met their needs. The consumer perspective should be built in at all stages of the services industry, not only at the point of need measurement as advocated above, but also at the point of planning, delivery, and evaluation. Otherwise the same danger exists that the biases of the policy makers, administrators and professionals

(including the program evaluators) will continue to dominate both the judgment of outcomes and the assessment of need. The involvement of consumers is crucial in the planning of evaluation, because as stated at the beginning of this paper, the goals of the service program should spring primarily from a real understand " ing of the clients' needs. As Carol Weiss has pointed out, even when the pro " fessional evaluation of service programs indicates that their impact is minimal or even negative, these programs are often kept going or increased because the administrators and policy makers have invested so much of their personal and professional selves in the programs. 57 In these cases the needs of the consumers or target population are being totally discounted . If the community served had an integral involvement in the evaluation of services, including where possible an active role in written documentation, it would be more difficult for their view " point to be overwhelmingly ignored in favour of the needs of the professio nals, policy makers and administrators.

We are not advocating the use of consumer-oriented evaluation to the exclusion of professional input. As with need measurement, all perspectives should be in " cluded in evaluation, and a variety of methodologies and techniques employed, as has been done so skilfully in the Western Australian study on Communit y Information Needs.5s We would just like to see consumers' comments on services , no matter how simply or unscientifically expressed, as an equally legitimate form of evaluation. As Weiss and Rein so clearly point out, evaluation technique s for broad aim programs (such as most health and welfare services) should be many and varied, and the tight scientific experimental approach is not suitable. 11D

98

Yet there is a danger inherent in the rapidly eme rging evaluation industry. In the quest to upgrade the importance of evaluatio n, sophisticat ed quantifi able models for measuring qualitati ve data are being formulated. This is partly due to the widespread but erroneous belief that to evaluate me ans to measure som e "

thing in a 'scientific' and mostly quantitive way. Such a process is often far removed from the world of the consumer, for whom the services were originally set up. Evaluation models can end up serving the need of the researchers for new fields to conquer60; or the need of the administrators for the credence and

aura of sophistication gained from the application of rigorous scientific methods to the evaluation of programs for which they are responsible. The new evaluation industry will not provide much information about the degree to which people's needs are being met unless a great deal of effort is put into it at the 'client face'. A substantial amount of the professional input ought to be channelled towards assisting consumers (and potential consumers) to articulate their own needs in their own words-not merely encouraging them as in the past to use anachronistic stereotypes borrowed from the professionals of the time.

Of the few examples of consumer involvement in evaluation that we have found in the literature , almost all the consumers in question were articulate, well educated and middle class. 61 These are the people most likely to be selected to participate in formal structures set up by institutions for monitoring or evaluative purposes, such as the new Community Health Councils in Britain. 62 While this is a step in the right direction, many of the recipients of services with the mo st needs will be the less articulate, less powerful members of the community, who

are not used to joining committees or speaking in public. A very conscious effort must therefore be made on the part of service providers to build into the service orientation the perspective of the less articulate consumers. Following the impli " cations of the preceding section on participation, it would seem that the service provider in direct contact with the consumers at the 'client face' is in the best position not only to assess need and demand, but also to involve these consumers in the process of evaluation itself. People assume all too readily that the less

articulate recipients of services are not capable of reporting their experiences clearly or of conceptualising their situation. Our interviewers in the Who Cares? study made the same assumption initially, but through the lengthy process of interviewing, they learnt that these people were capable of a considerable degree

of conceptualisation once they were approached as equals on neutral territory. That is, the interviewers talked with the respondents from the perspective of neutral researchers seeking the respondents' help, not from the powerful position of a health or welfare professional with something to give and little to learn.ss If health and welfare providers approached their clients in a similar reciprocal fashion, with the view that they have as much to learn as to give, then they could incorporate a strong consumer perspective on need measurement and service evaluation.

Methodology of consumer evaluation The most common methodology that has been emp loyed in attempts to understand consumers' attitudes to services is the attitudin al survey. The limitations of the attitudinal survey have been outlin ed above in discussing its use in assessing

need from the consumer perspective, and similar limitations apply to its use in this sphere. E. J. Maynard reached similar conclusions about the inadequacy of the survey technique when he looked at a numb er of consumer evaluation studies of medical services. Among other things, he concludes that: 'researchers have

99

been unrealistic about their expectations of consumers, leading to vague rums and a superficial approach . . . we have underestimated the influence of culture on (consumer) expectations, on criteria and on responses ... survey instrument construction has been based on ∑ assumptions and not on a clear understandin g of the dynamics of consumer evaluation ... finally, consumers often have low expectations, minimal and variable experience and little interest in medical services outside their own individual needs'. 64

These shortcomings are inevitable in an approach which views evaluation (whether involving consumers or not) as an entirely separate research method " ology that is tacked on from above to a service program after it has been operating for a long time. The pressure to produce scientific evaluations en " courages this separatist approach. Ideally, evaluation should be an integral and ongoing part of the service function from the beginning. In the health and welfare fields we do not at this stage have the knowledge base, methodologies or skills for sophisticated research designs. In fact we would question the desirabilit y of investing scarce resources in such a quest. What we need at this stage is more exploratory evaluations, and more monitoring of the ongoing processes of the service situation, involving the consumers and potential consumers in this process wherever possible.

Kamerman, in analysing a number of service programs that attempted to involve consumers in various ways, found that, without consumer involvement right from the planning stage, the programs inevitably failed in the long run. If an ongoing role for consumers was not developed, then programs that were initially successful

seemed to fail eventually. 65 We would extend this participatory principle further : if ways are not found to build in a meaningful and ongoing role for consumers in the evaluation process, then such evaluations will only be of limited value in planning for the provision of services more likely to fulfil community needs.

The survey technique does not represent an integral consumer involvement on an ongoing basis. Our argument is that the obvious place to start is at the 'client face', where the consumer already has an ongoing relationship with the service. Consciousness raising of the clients should be seen as a legitimate part of the providers' functions , broadening the clients' horizons, and encouraging them to articulate their needs, and their assessment of the service in their own terms.

As we have previously suggested, the Brotherhood of St Laurence's Family Centre and its successor, the Action and Resource Centre, is probably the most out " standing example of the involvement of consumers in planning, providing and evaluating services. 66 The consumers in this case are low income earners or welfare recipients , traditionally regarded as one of the mo st powerless and least articulate groups in the community. Through open files , sharing of power and passing on

of skills by the professionals, these families gradually took more control over their own lives and the Resource Centre, developing some skill in providing and assessing services. The low income families in volved in the Action and Resource Centre are continuing to expound their views and needs at conferences , seminars and through written documentation.

There are a few more limited attempts to involve consumers in the planning and evaluating processes through open files for example, or through opening up the decision-making processes to select consumers through their representation on committees of mana gem ent. Two Community Health Centres in Victoria have m ade a concerted attempt to involve consumers in the evaluation of the Centres' work as an integral part of the service delivery process. 67 Both Health Centres introduced an ongoing goal-oriented form of evaluation in which clients actively

100

participate. Firstly, both clients and health professionals each formulate their own goals-i.e. what they both want to achieve out of working together-in simple non-jargonistic terms. Then client and professional each rate the outcome of their relationship-i.e. to what degree have their goals been achieved after a certain period of time? The client's perspective of his/her needs and of the services' effectiveness in meeting these needs is thus built into the recording system alongside the perspective of the health professional. One of these health centres has developed this goal-oriented system of evaluation more fully. 68 The staff at

this Centre have become very enthusiastic about this method because it has greatly increased their understanding of consumer perspectives. It has led them to be more realistic and positive in their work with clients, and to consciously and continually involve them in the planning of their work. It has also led these health professionals to critically re-evaluate their own roles in the light of their

increased understanding of clients' needs, and to adapt their manner of wo rking to meet these needs more reali stically and efficaciously. If such pro " cedures could be developed into a common style of practic e, many existing services wou ld increase the relevance of their service delivery without major structural

changes. For a system such as this is yielding many data on felt needs and service effectiveness in meeting these needs, from the crucial consumer perspective that we have shown to be shamefully under-represented in planning and evaluation spheres.

V Conclusion Consumer participation in the making of planning decisions in the evaluation of health and welfare services should be promoted at national policy level. Such participation as a national goal is surely a basic ideal in a liberal democracy.

What is needed is the conscious establishment of the worth of consumer partici " pation in need measurement, service evaluation and policy development and implementation. Governments in the context of their funding systems should require that there be a professional and agency commitment to the development

of consumer feedback to ensure flexible and responsive service delivery . We bve chosen to emphasise the potential of the face to face service delivery rela " tionship as a vehicle for consumer participation because such potential has been neglected iri the past. 1

By stressing the understanding of need from the micro level we are not reject " ing the importance of macro definitions of need. A many levelled 'micro/macro approach' is necessary but consumer feedback is essential to flesh out the bones of the broader perspective.

101

REFERENCES

1. Carol Weiss, Evaluation Research: Methods of Assessing Program EffectivenesJ , Prentice Hall, Methods of Social Science Series, New Jersey, 1972, p. 1. 2. Michael Key, Peter Hudson, John Armstrong, Evaluation Theory and Community Work, YVFF, Stoke on Trent, 1975.

Carol Weiss, op.cit. T. Tripodi, P. Fellin & I. Epstein, Social Program Evaluation, F. E. Peacock Publishers Inc., U.S.A., 1971. L. R. Denton, 'Noc Nee-What's There? Three Basic Considerations in Evaluating Community Mental Health Centre Programs', Canada's Mental Health, vol. 24, no. 2, June 1976, pp. 19-23. 3. Carol Weiss, op.cit., p. 10. 4. T. Tripodi, P. Fellin & I. Epstein, op.cit. 5. Neville J. McCarthy, Can Quality of Care be Measured?, ANZSERCH Proceedings, Canberra,

May 1974, pp. 7-11. J. Najman, C. Bain et al., Conceptualizing and Evaluating Community Health Programmes Using the Method of Content Analysis. Unpublished paper from Univ. of Queensland, 1978. 6. The Pocket Oxford Dictionary, Oxford University Press, London, fourth edition 1942, revised

1946, p. 753. 7. J. Najman, C. Bain et al., op. cit., pp. 3-7. 8. Carol Weiss, op. cit., p. 4.

9. P. Baume, 'The Politician's Role in the Evaluation Process', ANZSERCH Proceedings, 1977, Neville Hicks (ed.), ANZSERCH, University of Adelaide, p. 177. 10. L. R. Denton, op. cit., p. 22.

11. ibid, p. 20.

12. A. S. Shonfield and S. Shaw (eds), Social Indicators and Social Policy, Social Science Research Council, London, 1972, p. 114. 13. L. R. Denton, op. cit., p. 19.

14. ibid, p. 21.

15. E. Hamilton-Smith, 'Issues in the Measurement of Community Need', Australian Journal of Social Issues, vol. 14, no. I, February 1976, p. 35.

16. J. Bradshaw, 'The Concept of Social Need', New Society, March 1972, pp. 640-3.

17. E. Hamilton-Smith, op. cit., p. 36.

18. Ronald F. Henderson, Alison Harcourt, R. J. A. Harper, People in Poverty: A Melbourne Survey, Cheshire, Melbourne, 1970.

19. E. Hamilton-Smith, op. cit., p. 37.

20. A. S. Shonfield and S. Shaw (eds), op. cit., p. 14.

21. E. Hamilton-Smith, op. cit., p. 43.

22. Carol Weiss, op. cit., p. 59. 23. J. Bradshaw, op. cit., p. 641. 24. Some discussion of this can be found in: Y. Hasenfeld, R. English (eds), Human Service Organi " sations , Ann Arbor, University of Michigan Press, 1974, pp. 477-85.

G. B. Kitay: Health Care and the Perception of Illness : Some Aspects of the Av ailability and Use of Health Services. Unpublished Honours Paper, Victoria University of Wellington, New Zealand, 1974, pp. 1-15. R. E. Tranquada, 'Participation of the Poverty Community in Health Care Planning', Social Science and Medicine, 1973, vol. 27, p. 723. 25. ibid., p. 723. 26. J. Weston et al., 'Evaluation as a Social Process', ANZSERCH Proceedings, Neville Hicks (ed.),

1977, ANZSERCH, University of Adelaide, p. 168. See also: Michael Jones, 'The Politics of Social Policy Evaluation'. Paper to Demystification of Research Workshop, School of Social Work, Preston Institute of Technology, 11 Nov. 1975, p. 6. 27. ibid., p. 6.

102

28. J. McCaughey, S. Shaver, H. Ferber & others, Who Cares? Family Problems, Community Links & Helping Services. Published by Cheshire for The Institute of Applied Economic & Social Research, University of Melbourne, 1977.

29. Ronald F. Henderson, Alison Harcourt, R. J. A. Harper, op. cit. 30. J. McCaughey, S. Shaver, H. Ferber & others, op. cit., Introduction XI. 31. James R. Jeffers, Ph.D., Mario Ph.D., & John C. Bartlett, Ph.D., 'On the Demand Versus Need for Medical Services and the Concept of Shortage'. Article in American Journal of

Public Health, vo!. 61, no. I, Jan. 1971, pp. 46-63. 32. J. McCaughey, S. Shaver, H. Ferber & others, op cit., para. 'Attitudes to Seeking Help', p. 63. 33. G. B. Kitay, op. cit., pp. 24 and 25. 34. Robert A. Scott, 'The Selection of Clients by Social Welfare Agencies: The Case of the Blind'.

Article in Y. Hasenfeld & R. English (eds), Human Service Organisations, Ann Arbor, Institute of Social Research, University of Michigan, 1974. 35. Robert A. Scott, ibid. 36. R. Baker, 'The Interpersonal Process in Generic Social Work: An Introduction', in P. Boas &

J. Crawley (eds). Preston Institute of Technology Occasional Paper No. 2, p. 6. 37. J. Weston et al., op. cit., p. 168. 38. Tripodi, op. cit. Carol Weiss, op. cit.

Sheldon H. Olkon, 'Linking Planning with Evaluation in Community Mental Health', in Com " munity Mental Health Journal, vol. 11, no. 4, 1975, pp. 362-3. 39. E. Hamilton-Smith, 'Social Work Research and Social Change'. Paper to 13th National Conference, Australian Association of Social Workers, Perth, 6-10 August 1973, p. 5. 40. E. Hamilton-Smith, 'Issues in the Measurement of Community Need', Australian Journal of

Social Issues, vol. 10, no. 1, pp. 39-40. L. Schatzman & A. L. Strauss, Field Research: Strategies for a Natural Sociology, Prentice Hall, Englewood Cliffs, N.J., 1971, p. 6. 41. For detailed discussion of the phenomenological perspective: Herbert Blumer, Symbolic Inter "

actionism: Perspective and Method, Prentice Hall, New Jersey, 1969. Schatzman & Strauss, op. cit. Michael Liffman, 'Method or Madness: Research within an Experimental Programme for Poor Families'. Paper to 13th National Conference, Australian Assoication of Social Workers, Perth, 1973, pp. 15ff. 42. Jack D. Douglas (ed.), Understanding Everyday Life, Routledge & Kegan Paul, Great Britain ,

1971, p. 1. 43. Michael Liffman, op. cit., p. 14. 44. Ivan Beringer, 'Participation in Regional and Social Planning'. Article in Australian Social Work, vol. 28, no. 1, p. 15. 45. ibid., p. 16. 46. ibid., p. 21. 47. ibid., p. 23. 48. ibid., p. 17. 49. Richard M. Titmuss, 'The Gift Relationship. From Human Blood to Social Policy', Penguin

Books, 1973, p. 273. 50. J. Macbeth, 'Community Information Needs. Research into Information Needs in Perth, W.A.', School of Social Enquiry, Murdoch University. Paper presented to 48th A.N.Z.A.A.S. Congress, Melbourne, August 1977. 51. ibid., p. 3. 52. ibid., pp. 7 and 8. 53. ibid., p. 9. 54. Paul F. Gross, 'Consumer Participation in Community Health Services: Lessons from Overseas',

in Australian Journal of Social Issues, vol. 10, no. 1, 1975, p. 21. ' 55. R. M. Hessler & M. J. Walters, 'Consumer Evaluation Research Implications for Methodology, Social Policy and the Role of the Sociologist', in The Sociological Quarterly, 18, Winter 1976, p. 76.

103

56. J. Tydeman, 'Health Centre Management and Consumer Participation', in ANZSERCH Proceedings, Neville Hicks (ed.), ANZSERCH, University of Adelaide, May 1977, pp. 43-58. In Tydeman's paper, for example, there is a lot of material presented on roles for consumers in community health centres, but no mention of a possible consumer role in evaluation . See also: Hessler & Walters, op. cit. N . J. McCarthy, op. cit. 57. Carol Weiss, op. cit., pp. 126-8.

58. Jim Macbeth, op. cit. 59. RobertS. Weiss & Martin Rein, 'The Evaluation of Broad Aim Programs: Experimental Design, its Difficulties and an Alternative', in Administrative Science Quarterly, vol. 15, no. 1, March 1970. 60. J. Weston eta!., op. cit., p. 169. 61. Erica Bates, 'The Captive Consumer in the Health Services'. Paper delivered to Sociological

Association of Australia and New Zealand Conference, Latrobe University , 1976. 62. ibid., p. 14. 63. J. McCaughey, S. Shaver, H . Ferber & others, op. cit. pp. 6-9.

64. E. J. Maynard, 'Consumer Evaluation of Health Services', ANZSERCH Proceedings, 1977. Neville Hicks (ed.), ANZSERCH, University of Adelaide, p. 17. 65. S. B. Kamerman, 'Participation, Leadership and Expertise: Imbalance or in Balance', in Social Service Review, no. 48, 1974, pp. 403-11. 66. Michael Liffman, The Family Centre Project: an Experiment in Self-Help, Allen and Unwin,

Sydney, 1978. 67. Depaul Community Health Centre (Fitzroy); Golden Square/Kangaroo Flat Community Health Centre. 68. Depaul Community Health Centre.

(R 78/ 1562) Cat. No. 79 7834 5