Australian Government: Attorney-General's Department
Australian Government: Attorney-General's DepartmentAchieving a Just and Secure Society

AIAL 2007 Administrative Law Forum Address - Secretary's address - 15 June 2007

Effectiveness and efficiency of administrative law: How do we measure it?

INTRODUCTION

The proposition that we should measure the effectiveness and efficiency of administrative law immediately brings to my mind ideas about outcomes, outputs, mission statements, corporate plans, key performance indicators, benchmarking and big folders of computer generated statistics.

Bureaucrats are believed to revel in the universal and endless measurement of miscellaneous details in an effort to assess the value of complex government services.

Chief Justice Spigelman of the New South Wales Supreme Court has referred to this approach as pantometry in a way that didn’t seem to be complimentary.

So I suppose I should ask at the outset whether in fact we really want to measure the effectiveness or the efficiency of administrative law.

In doing so, I won’t waste time discussing the difference between effectiveness and efficiency. For today’s purposes, I will simply say that effectiveness means we are achieving the desired outcome and efficiency means we are doing so with a minimum of cost, effort and fuss.

TWO MORE PRELIMINARY POINTS

I want to make two more preliminary points.

The first is that this address is directed to administrative law. It is not limited to administrative review.

The great bulk of the huge number of administrative decisions made in Australia each year are accepted - or corrected on internal reassessment - without merits or judicial review.

In fact, merits or judicial review of administrative decisions is just the tip of the administrative law iceberg. So any consideration of the effectiveness or efficiency of administrative law has to take the totality of the system into account.

The second point is governments undertake this consideration from a different point of view to the tribunal members and other stakeholders more closely enmeshed in the process of administrative review.

Governments have to find ways to assess the effectiveness and efficiency of policies and services directed at achieving complicated and sometimes intangible objectives. They have to form judgments about the administrative law system as a whole.

However, people directly involved in the provision of services in these difficult areas tend to form judgments on the basis of individual cases.

WHY DO WE WANT TO MEASURE EFFECTIVENESS AND EFFICIENCY?

So why do we want to measure the effectiveness and efficiency of administrative law?

The answer is straight forward from a governmental perspective.

Governments are accountable to the parliament and the people for providing institutions and services that meet the needs of the community in a financially responsible manner.

Put another way, politicians are responsive to pressure from constituents and Treasurers decide between competing claims on the Budget on the basis of the perceived benefits that will be achieved with the requested funding.

Claims for additional or even continuing levels of public funds are determined on the basis of the extent to which an agency is seen to be meeting its agreed objectives and the benefits that will result from that Budget allocation.

These principles can be found in the origins of the Administrative Appeals Tribunal.

In his second reading speech on the Administrative Appeals Tribunal Bill on 6 March 1975, the Attorney-General, Mr Enderby, said:

It is not clear at this stage how many presidential members will be required for the work of the Tribunal and accordingly the Bill does not propose any limit. It may be expected, however, that there would be a sufficient workload in Canberra, Sydney and Melbourne for there to be a full-time presidential member in each of those cities.[1]

The Opposition supported the proposal to establish an effective system of review of decisions made by Ministers and officials which affect the ordinary citizens of this country in their daily, personal and business lives. [2]

The Government still adheres to those same objectives over 30 years later.

COURT AND TRIBUNAL PERSPECTIVE

From the point of view of a court or tribunal, there are probably two main factors at work.

The first is the recognition that the body has to operate within the government framework.

Secondly, it is a natural inclination for the office holders and staff in any publicly funded organisation to want their agency to do a good job in discharging its functions in the best interests of the community.

So it’s not surprising that the objectives of effectiveness and efficiency are entrenched in the vision statement for the Administrative Appeals Tribunal.

The AAT aims to be a leader in administrative review, providing fair, just, economical, informal and quick merits review[3].

Realistically, that can only be established if you know what the organisation is doing and it can only be demonstrated by some form of measurement.

HOW DO WE MEASURE IT?

That brings us to the nub of the issue: how do we measure effectiveness and efficiency?

A lot has been written about the difficulty of making these assessments in relation to courts and tribunals.

At a government level, the Productivity Commission provides some statistics in relation to court administration in its annual Report on Government Services.

The Commission identifies four objectives for court administration. They are:

In addition, all governments aim to provide court administration services in an efficient manner.[4]

These objectives sit quite comfortably with the objectives courts and tribunals have set for themselves.

The objectives of the Federal Court are to decide disputes according to law – promptly, courteously and effectively; to provide an effective registry service to the community; and to manage the resources allotted by Parliament efficiently. [5]

The AAT has expressed its goals in these terms:

PERFORMANCE INDICATORS

The Productivity Commission applies six performance indicators to courts administration:

These indicators do, to my mind, give some reasonable indications about workload and the timeliness and disposition of a court or tribunal’s business.

That is particularly so if we track those statistics over time to detect trends in, for example, the volume of work, the type of cases being heard or the length of the delay in hearing.

The Productivity Commission quite properly stays away from any attempt at assessment of the quality of judgments in individual cases.

Governments recognise the importance of the independence of the judiciary and independent decision makers while acknowledging that the executive has to be able to make assessments of the effectiveness and efficiency of our judicial and tribunal systems as a whole.

ANNUAL REPORTS

Tribunals provide these sorts of statistics in their annual reports.

The AAT publishes details of applications, finalisations, resources by output, completed reviews of decisions and percentage of applications finalised within twelve months.

It also usefully provides comparative data for the two preceding financial years so any significant changes or trends can be identified.[8]

Courts also provide workload statistics.

The last Federal Court annual report contains details of filings, dispositions, judgments, incoming work, matters transferred to and from the court, matters completed, matters on hand and the age of the pending workload.[9]

QUANTITATIVE AND QUALITATIVE ASSESSMENTS

Now I have to say straight away that my views about the usefulness of this information do not meet with universal agreement.

A lot has been written about the difficulty and even the desirability of measuring judicial effectiveness and efficiency.

The debate tends to be framed more in the terminology of quantitative and qualitative measurements rather than effectiveness and efficiency, but the essential issues are the same.

While courts and tribunals provide the government and the public with a considerable amount of information about their activities, it basically comprises data about things that are easily counted or assertions of less tangible outcomes that can’t easily be tested.

In part, this is because judges and tribunal members are to be clearly separated from and not accountable to the executive government for their decisions.

That is one of the vital foundation stones of Australian democracy. It explains why the single outcome agreed between the Government and the Federal Court is expressed simply as Federal Court Business.[10]

But it is also because, in any event, broad and sometimes aspirational outcomes aren’t susceptible to quantitative measurement.

Nonetheless, we have to try to resolve these difficulties.

QUANTITATIVE MEASUREMENT

I will deal with quantitative measurements first.

There’s a well-known management maxim that what gets measured gets done. I agree with that general sentiment.

Monitoring tasks or activities and reporting on them is one of the oldest and most effective ways of understanding what an organisation does and ensuring it achieves its stated objectives.

But when you come to apply that simple principle in a particular situation, a number of subsidiary questions require careful thought.

What do you measure?

What do you do with the data when you have it?

What does that information tell you about the organisation?

If the data changes over time, what caused the change?

What about the things that can’t be measured accurately or at all – intangible things like quality of outcomes? Do we get a skewed result if we leave them out?

I could go on, but these few fundamental questions demonstrate the complexity of the problem.

I want to break these considerations about quantitative measurement into three parts:

What do you measure?

Obviously organisations tend to measure whatever is readily available.

In my view, those details are useful.

Statistics like delays in hearing cases and the number of appearances before finalisation are clearly relevant to the cost and inconvenience of litigation to the parties.

If, over time, they can be reduced, there will be a clear benefit to the community and possibly a reduction in the cost of resolution of those disputes both for the users and for the court or tribunal.

The judgment about what quantitative measures are useful is clearly a matter for the court or tribunal to decide.

Understanding the data

Understanding the data is more difficult.

Ultimately, data is just a bunch of statistics. It’s a bit like being spoken to in a foreign language – it means something but you have to have it translated.

Translating data so it gives you a clear understanding about its relevance to your organisation or business often requires considerable insight and skill.

If the delay in hearings is shortened, that sounds superficially like a good result. But what if it is brought about by a 50% reduction in new lodgements?

Take another example. An increase in the reports of sexual assaults may at first seem like a bad outcome. However, it could be a positive sign indicating, not that more assaults are taking place, but that more victims have enough confidence in the criminal justice system to report it.

Here’s a further illustration. A reduction in the number of young people or drug users coming before the criminal courts may not mean there are less offences being committed.

It may indicate that more offenders are being processed through diversionary programs, hopefully with more positive results than a criminal conviction and a prison term.

How do you use the data?

Then we have to decide how to use the data.

I think the most essential point is to recognise that data is not a divine law of management. It doesn’t prescribe inescapable conclusions.

Data usually only tells part of the story because you can’t collect comprehensive and accurate data about every aspect of an organisation’s activities.

Even if you could, data is just an historical record or a snapshot at a particular point in time. It doesn’t necessarily predict the future.

Data is an aid to decision making. Judgment, common sense and a breadth of vision have a role to play – usually the major role – in putting the data into its proper perspective.

You have to treat data with caution, be aware of its limitations and ensure it doesn’t create perverse incentives.

Chief Justice Spigelman has been relentless in his search for bizarre examples that illustrate the danger of misapplied quantitative measures.

Here are a few of them:

  • When comparative success rates for cardiac surgeons began to be published in New York and Pennsylvania, mortality rates in both States declined significantly because heart surgeons refused to operate on risky cases, who were referred to adjoining States, and
  • English hospitals are judged on whether they admitted 90 percent of emergency patients within four hours. Whenever the annual measurement was due, hospitals cancelled operations and flooded their emergency departments with doctors and nurses.

The Chief Justice has concluded: Distortions arise because the things that can be measured are not the only things that matter. Insofar as external judgments are made on an information base which is too narrow, then the incentives created by performance indicators will operate perversely. The more significant the consequences of the measured results, the greater the perversity. [11]

I agree with the Chief Justice’s concerns. So does Professor, now Justice, Marcia Neave.

Before her appointment to the Victorian Court of Appeal last year, Professor Neave delivered a paper to the National Administrative Law Forum here in Canberra in 1999.

In that presentation, she said: …achieving administrative justice requires value judgments to be made about the trade-offs which should be made between competing objectives, for example speed versus accuracy of decision-making. Performance measurement may result in such choices being made covertly, instead of being clearly articulated. Targets for the performance of some objectives (for example cost objectives) may prevent the achievement of others.

A little later in the same speech, Professor Neave argued that prescriptive performance measures posed a greater risk to the independence of decision-makers than descriptive ones.

She said: Prescriptive performance measures could undermine the goals of administrative justice by imposing a significant degree of political or bureaucratic control over the decision-makers. Their inappropriate use could destroy the substance of independent merits review, while maintaining its illusion. For example, imposition of stringent timelines could result in Tribunal members being forced to rubber-stamp departmental decisions. My argument is not that delay should not be measured, but rather that we need to be careful about the purpose for which this information is used.[12]

As far as I am aware, none of the quantitative measures used by courts and tribunals are prescriptive. They do not pose any risk of political or bureaucratic control.

The final point I want to make about quantitative indicators relates to benchmarking performance against other courts or tribunals.

It is obvious that comparisons are only of any use if they are comparing like with like. If they are not doing that, they can be highly dangerous. Flawed comparisons can lead to all sorts of bad decisions.

We have to be particularly careful with Commonwealth – State comparisons because the data equivalent of the standard rail gauge mismatch is alive and well.

QUALITATIVE MEASUREMENT

Now let’s move on to the even more difficult topic of qualitative measurement.

Professor Neave pointed out the difficulty in trying to assess quality in ….administrative justice, which has intangible objectives such as encouraging compliance with the rule of law, contributing to government accountability by enabling individuals to challenge decisions which affect them, and enhancing participatory democracy.[13]

In 1999, the Productivity Commission used the concept of fitness for purpose as equating to quality. It said: A comprehensive assessment of this requires a range of indicators. Ideally such indicators directly capture the quality of outcomes – that is, whether the service achieves the outcomes of the government. Assessment may also involve seeking the views of clients and others with a legitimate interest in service quality.

But Professor Neave concluded, rightly I think, that this statement provides little assistance on how to measure administrative justice.[14]

Some further measures which have been proposed to measure quality include auditing the accuracy of primary decision making, examining appeal rates and surveying stakeholders for client satisfaction.

Other measures that come to mind are peer pressure (in the sense of establishing a collegiate level of acceptable behaviour and performance) and the leadership of the head of jurisdiction in setting standards for that court or tribunal.

None of these seem to overcome the obvious difficulties of trying to measure the quality of justice.

CLIENT SATISFACTION

I want to come back to the issue of client satisfaction and look at it in a little more detail because it is a valid yardstick for many organisations.

The question is whether it is of any assistance to courts and tribunals.

A starting point is to look at what litigants think about the administrative review process.

Robin Creyke and John McMillan undertook an informative survey of the final outcomes where a court had overturned a government decision and the case was remitted to the agency to be reconsidered according to law.

Their research found that in a surprisingly high proportion of cases the ultimate decision of the agency was favourable to the applicant.

But that didn’t mean the applicants were happy.

The Creyke and McMillan empirical study revealed that The majority of applicant comments were critical of the relevant agency and in some cases of the administrative reconsideration process generally. Some stated that the favourable decision only occurred because the court left the agency with no choice….A frequent complaint was the length of time taken to get a result, both initially from the review body and then from the agency following the review.[15]

My view is that the concept of clients and client satisfaction is inappropriate for courts and tribunals which decide adversarial contests.

The same can be said for regulators like the Australian Securities and Investments Commission and law enforcement agencies.

A former policeman called Malcolm Sparrow has written a text entitled The Regulatory Craft [16] that grapples with the problems facing regulators and how to define their role.

The way he analyses the situation seems to me to have some application to courts and tribunals.

Mr Sparrow argues: When people are arrested or fined or have their license revoked or their property seized, most often they are not pleased. Government does not seek to serve them in that instant. In many cases government creates an experience for them that is by design unpleasant.

Of course, those being arrested, fined or forced into compliance are entitled to be treated fairly and with human dignity. But when law is put into action against them, they receive treatment they did not request, did not pay for directly, will not enjoy, and will not want to repeat.

In this context, the notions of quality governance in widest circulation simply fall short. The notion of customer falls short. Regulators need a broader vocabulary, so they can think in terms not only of customers but of stakeholders, citizens, obligatees, objects or targets of enforcement, beneficiaries, taxpayers and society.[17]

These observations resonate with the following comments by Chief Justice Spigelman:

I have no doubt that the courts serve the people. However, they do not provide services to the people. This distinction is not merely semantic; it is fundamental. The courts do not deliver a ‘service’. Courts administer justice in accordance with the law.[18]

It is only marginally reassuring to note that trying to define and measure the quality of judgment or professional advice is posing problems in other areas as well.

For example, lawyers are often criticised for time costed fees which reward inefficiency. But while they talk about an alternative of value billing, that seems to me not to amount to much more than charging an even higher fee if the client is satisfied with the outcome.

Here’s another illustration. In 2005, Dr Brendan Nelson, the then Minister for Education, Science and Training, endorsed the release of a paper directed at the development of a Research Quality Framework.

In his Foreword, he said: The Australian Government regards the development of an RQF as a high priority, and a unified and consistent approach to assess the quality of research undertaken in this country will continue to inspire community confidence.[19]

However, the paper’s starting point was that the Expert Advisory group recognises that there are no agreed-upon major consistent quality measures of the outputs of research training which could be readily included in an RQF at this stage.[20]

I am afraid that this discussion has done no more than highlight the problems inherent in seeking to measure the quality of administrative review decisions.

In the end, I don’t think bureaucrats can solve this issue. The best result would be for courts and tribunals to define quality measures that are acceptable to them.

But that task does need to be attempted because, to quote Chief Justice Spigelman one last time: Quality is hard to assess. But if we ignore it, we do so at the peril of perverting the decision-making processes which we are seeking to improve.[21]

THE BROADER PICTURE

Now let’s move on to look at the broader picture of administrative law, not just administrative review – that is, the iceberg, not the tip.

Robin Creyke and John McMillan remarked that their study showed the diversity of ways in which judicial review proceedings impact on government administration and define the relationship between government and the community. Individual rulings are frequently followed by other governmental action to amend legislation, change policy, rewrite manuals or alter decision-making procedures and practices.[22]

I am sure there are countless examples where that has happened, that is, the overall quality of administrative decision making across an agency or possibly the whole of government has been improved in response to an adverse finding by a court or tribunal.

Similarly, there are examples of governments introducing improvements in the administrative decision making process in response to public concern or simply because they come to the view that the system can be improved.

Let me give you four instances of this happening.

A few years ago, the Victorian Government decided that one major administrative review tribunal would produce better outcomes than a collection of smaller tribunals with separate procedures, forms and processes.

As far as I know, the decision was not in response to any specific concern or criticism of the existing tribunals. The Government simply came to the view that an amalgamated tribunal would be more efficient and effective.

In her second reading speech, Attorney-General Jan Wade said:

The bill contains a range of measures that will assist VCAT to minimise costs and resolve applications quickly and informally. They will assist VCAT to make the most efficient and effective use of its resources.

While many of these measures are, in varying degrees, now in use by tribunals, common procedures and the consistent approach to them will produce far more beneficial results.[23]

Mrs Wade concluded: VCAT will be the most comprehensive reform in Australia in this area to date. The creation of VCAT demonstrates the government’s commitment to improving the tribunal system in Victoria.[24]

Here is another example. The Minister for Justice and Customs announced an extensive review of the Extradition Act on 22 February this year.

The review was prompted in part by the fact that the current extradition process involves too many decisions which may be subject to review.

The Minister, Senator Ellison, said in his media release:

Current extradition … arrangements are characterised by lengthy delays and limitations of the assistance Australia can provide to our foreign partners….Some cases have taken up to seven years to resolve. Even when a person consents to extradition, they can still spend a lengthy time in prison while the process runs its course.

In other words, the Government recognised that this system as a whole was not producing good quality and fair results and decided to change it.

Another illustration is the increasing introduction of automated decision making processes.

But perhaps the most dramatic example can be seen in the changes in the Immigration Department following the inquiries in to the cases of Cornelia Rau and Vivian Alvarez.

You will all know the details of these cases so I won’t go into them now. I simply refer to them as showing the Government taking positive and drastic action to improve administrative decision-making in immigration matters from the ground up.

CONCLUSION

So, in conclusion, let me try to draw these comments together.

The Government and, I am sure, the community expect courts and tribunals reviewing administrative decisions to be able to demonstrate that they are using public funds efficiently and effectively.

Courts and tribunals no doubt have a similar objective.

Well designed and carefully used quantitative measures of performance will assist in meeting that expectation.

Qualitative measures would help even more but are hard to define and will continue to be so. Ultimately, I think the best solution will be for acceptable qualitative measures to be developed by the courts and tribunals themselves.

At the executive level, governments will continue to make decisions to improve our administrative decision making and merits and judicial review processes wherever they see a need for whole-of-government or systemic improvements.

But, as for administrative law as a whole, we don’t need to measures its effectiveness or efficiency to be satisfied about the value of its contribution to Australian democracy.

It influences the development of government policy. It guides government action at many levels. It sets the framework for millions of fair and accepted government decisions every year.

Administrative law is now permanently entrenched in our system of law and government because it responds to every Australian’s expectation of a fair go.

Robert Cornall AO
15 June 2007

1  Hansard, page 1187
2  John Howard MP, Hansard, House of Representatives, 14 May 1975, page 2,278
3  AAT Annual Report 2005 – 2006, page 6.  These words are also used in the legislation relating to the Migration Review Tribunal, the Refugee Review Tribunal, the Social Security  Appeals Tribunal and the Native Title Tribunal.
4  Report on Government Services 2006, page 6.19
5  Federal Court Annual Report 2005 -2006, page 3
6  AAT Annual Report 2005 – 2006, pages 6 - 9
7  Report on Government Services 2006, page 6.19
8  See, for example, AAT Annual Report 2005 - 2006
9  Federal Court of Australia, Annual Report 2005 – 2006 pages 30 - 33
10  Federal Court Annual Report 2005 – 2006, page 61
11  JJ Spigleman AC, Measuring Court Performance, AIJA Annual Conference, Adelaide, 16 September 2006
12  Professor Marcia Neave, In the Eye of the Beholder – Measuring Administrative Justice, 30 April 1999, Canberra
13  Ibid, page 5
14  Ibid, page 5
15  Robin Creyke and John McMillan, Judicial Review Outcomes – An empirical study (2004) 11 A J Admin L page 89
16  Malcolm K Sparrow, The Regulatory Craft,  Brookings Institution Press, 2000
17  Ibid, pages 2 - 3
18  Ibid, page 5
19  Research Quality Framework: Assessing the quality and impact of research in Australia – Advanced Approaches Paper, endorsed for discussion at the National Stakeholder Forum, Canberra, 2 June 2005
20  Ibid, page 11
21  J  Spigelman, Quality in an Age of Measurement, Quadrant Magazine Law, March 2002, Volume XLVI Number 3
22  Ibid, page 98
23  VicHansard, 9 April 1998, page 973
24  Ibid, page 975