Policy Concepts in 1000 Words: Success and Failure (Evaluation)

(podcast download)

Policy success is in the eye of the beholder. The evaluation of success is political in several ways. It can be party political, when election campaigns focus on the record of the incumbent government. Policy decisions produce winners and losers, prompting disputes about success between actors with different aims. Evaluation can be political in subtler but as-important ways, involving scientific disputes about:

  • How long we wait to evaluate.
  • How well-resourced our evaluation should be.
  • The best way to measure and explain outcomes.
  • The ‘benchmarks’ to use – should we compare outcomes with the past or other countries?
  • How we can separate the effect of policy from other causes, in a complex world where randomised-controlled-trials are often difficult to use.

In this more technical-looking discussion, the trade-off is between the selection of a large mixture of measures that are hard to work with, or a small number of measures that are handpicked and represent no more than crude proxies for success.

Evaluation is political because we set the agenda with the measures we use, by prompting a focus on some aims at the expense of others. A classic example is the aim to reduce healthcare waiting times, which represent a small part of health service activity but generate disproportionate attention and action, partly because outcomes are relatively visible and easy to measure. Many policies are implemented and evaluated using such proxies: the government publishes targets to provide an expectation of implementer behaviour; and, regulatory bodies exist to monitor compliance.

Let’s consider success in terms of the aims of the person responsible for the policy. It raises four interesting issues:

  1. The aims of that policymaker may not be clear. For example, they may not say why they made particular choices, they may have many reasons, their reasons may not be specific enough to be meaningful, and/or they may not be entirely truthful.
  2. Policymaking is a group effort, which magnifies the problem of identifying a single, clear, aim.
  3. Aims are not necessarily noble. Marsh and McConnell describe three types. Process measures success in terms of its popularity among particular groups and its ease of passage through the legislature. Political describes its effect on the government’s popularity. Programmatic describes its implementation in terms of original aims, its effect in terms of intended outcomes, and the extent to which it represented an ‘efficient use of resources’. Elected policymakers may justify their actions in programmatic terms, but be more concerned with politics and process. Or, their aims may be unambitious. We could identify success in their terms but still feel that major problems remain unsolved.
  4. Responsibility is a slippery concept. In a Westminster system, we may hold ministers to be ultimately responsible but, in practice, responsibility is shared with a range of people in various types and levels of government. In multi-level political systems, responsibility may be shared with several elected bodies with their own mandates and claims to pursue distinctive aims.

Traditionally, these responsibility issues were played out in top-down and bottom-up discussions of policy implementation. For the sake of simplicity, the ‘top’ is the policymaker at the heart of central government and we try to explain success or failure according to the extent to which policy implementation met these criteria:

1.   The policy’s objectives are clear, consistent and well communicated and understood.

2.   The policy will work as intended when implemented.

3.   The required resources are committed to the programme.

4.   Policy is implemented by skilful and compliant officials.

5.   Success does not depend on cooperation from many bodies.

6.   Support from influential groups is maintained.

7.   Demographic and socioeconomic conditions, and unpredictable events beyond the control of policymakers, do not significantly undermine the process.

Similar explanations for success are still used and repackaged, such as by the Institute for Government:

  1. Understand the past and learn from failure.
  2. Open up the policy process.
  3. Be rigorous in analysis and use of evidence.
  4. Take time and build in scope for iteration and adaptation.
  5. Recognise the importance of individual leadership and strong personal relationships.
  6. Create new institutions to overcome policy inertia.
  7. Build a wider constituency of support.

Bottom-up studies prompted a shift of analysis, towards a larger number of organisations which made policy as they carried it out – and had legitimate reasons to diverge from the aims set at the ‘top’. Indeed, central governments might encourage a bottom up approach, by setting a broad strategy and accepting that other bodies will implement policy in their own way. However, this is difficult to do in Westminster systems, where government success is measured in terms of ministerial and party manifesto aims.

Examples of success and failure?

Many implementation studies focus on failure, including Pressman and Wildavsky’s ‘How Great Expectations in Washington are Dashed in Oakland’ and Marsh & Rhodes’ focus on the ‘implementation gap’ during the Thatcher Government era (1979-90). In contrast, the IFG report focuses on examples of success, derived partly from a vote by UK political scientists, including: the national minimum wage, Scottish devolution, and privatisation. Note the respondents’ reasons for declaring success, based on a mix of their personal values and their assessment of process, political and programmatic factors.  They declare success in very narrow terms, as the successful delivery in the short term. So, privatisation is a success because the government succeeded in raising money, boosting its popularity and international reputation – not because we have established that the nationalized industries work better in the private sector. Similarly, devolution was a declared a success because it solved a problem (local demand for self-autonomy), not because devolved governments are better at making policy or their policies have improved the lives of the Scottish population (Neil McGarvey and I discuss this here). Individual policy instruments like the smoking ban are often treated in similar ways – we declare instant success when the bill passes and public compliance is high, then consider the longer term successes (less smoking, less secondhand smoke) later.

Further reading and watching: (1) Can a Policy Fail and Succeed at the Same Time?

(2)  http://blogs.lse.ac.uk/politicsandpolicy/archives/34735

Why should you read and watch this case study? I hesitate to describe UK tobacco control as a success because it instantly looks like I am moralising, and because it is based on a narrow set of policymaking criteria rather than an outcome in the population (it is up to you to decide if the UK’s policies are appropriate and its current level of smoking and health marks policy success). However, it represents a way to explore success in terms of several ‘causal factors’ (Peter John) that arise in each 1000 Words post: institutions, networks, socioeconomic conditions and ideas. Long term tobacco control ‘success’ happened because:

  • the department of health took the policy lead (replacing trade and treasury departments);
  • tobacco is ‘framed’ as a pressing public health problem, not an economic good;
  • public health groups are consulted at the expense of tobacco companies;
  • socioeconomic conditions (including the value of tobacco taxation, and public attitudes to tobacco control) are conducive to policy change;
  • and, the scientific evidence on the harmful effects of smoking and secondhand smoking are ‘set in stone’ within governments.

The ‘take home’ message here is that ‘success’ depends as much on a policy environment conducive to change as the efficacy of political instruments and leadership qualities of politicians.



Filed under 1000 words, agenda setting, Evidence Based Policymaking (EBPM), public policy, UK politics and policy

16 responses to “Policy Concepts in 1000 Words: Success and Failure (Evaluation)

  1. Pingback: Can a Policy Fail and Succeed at the Same Time? | Paul Cairney: Politics and Policy

  2. Pingback: Scottish Independence: Should You Use the Powers You Have Before You Ask For More? | Paul Cairney: Politics and Policy

  3. Jill Rutter

    Hi — as the author of the Institute for Government report you cite, there is a difference between the reasons the academics give for choosing their successes (though durability was one element) and our definition which is that: “The most successful policies are ones which achieve or exceed their initial goals in such a way that they become embedded; able to survive a change of government; represent a starting point for subsequent policy development or remove the issue from the immediate policy agenda”. I think that is a much better definition than McConnell’s/

    Most tobacco control interventions meet this criterion – though don;t underestimate the importance of tax policy (we wanted to study the long run of tobacco policy but our case studies started out with “reunions” and the long time span meant we focussed on one element). Though when 20% of the population still use a product they know will kill half of them, I am not sure we should proclaim victory. .

    • Thank you for this comment, Jill. Yes, there is a difference between the hasty views of the respondents and the more considered IFG discussion. Still, that’s an interesting definition of success – in both cases (yours and Allan’s) success is largely about process and a pragmatic view on what is possible in a short space of time, not the common sense notion regarding clear victory or the intuitive notion of success measured in terms of longer term outcomes. It chimes with my favourite advice to policymakers: if you want to feel successful at the end of the process, you should lower your expectations.

  4. Pingback: A ‘decisive shift to prevention’: how do we turn an idea into evidence based policy? | Paul Cairney: Politics and Policy

  5. Pingback: Policy Concepts in 1000 Words: Policy change and measurement | Paul Cairney: Politics & Public Policy

  6. Pingback: Can you separate the facts from your beliefs when making policy? | Paul Cairney: Politics & Public Policy

  7. Pingback: Can you separate the facts from your beliefs when making policy? | The Power To Persuade

  8. Pingback: Does it do no harm? How do we know? | First do no harm, Scotland

  9. Pingback: How to present policy analysis to many audiences #POLU9SP | Paul Cairney: Politics & Public Policy

  10. Pingback: Policy success or failure? It’s in the eye of the beholder | The Power To Persuade

  11. Pingback: 12 things to know about studying public policy | Paul Cairney: Politics & Public Policy

  12. Pingback: Policy Concepts in 1000 Words: the intersection between evidence and policy transfer | Paul Cairney: Politics & Public Policy

  13. Pingback: Policy in 500 Words: how much does policy change? | Paul Cairney: Politics & Public Policy

  14. Pingback: Policymaking in the UK: do you really know who is in charge and who to blame? #POLU9UK | Paul Cairney: Politics & Public Policy

  15. Pingback: The theory and practice of evidence-based policy transfer: can we learn how to reduce territorial inequalities? | Paul Cairney: Politics & Public Policy

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s