Policy Analysis in 750 Words: How to deal with ambiguity

This post forms one part of the Policy Analysis in 750 words series. It draws on this 500 Words post, then my interpretation of co-authored work with Drs Emily St Denny and John Boswell (which I would be delighted to share if it gets published). It trails off at the end.

In policy studies, ambiguity describes the ability to entertain more than one interpretation of a policy problem. There are many ways to frame issues as problems. However, only some frames receive high policymaker attention, and policy change relates strongly to that attention. Resolving ambiguity in your favour is the prize.

Policy studies focus on different aspects of this dynamic, including:

1. The exercise of power, such as of the narrator to tell stories and the audience to engage with or ignore them.
2. Policy learning, in which people collaborate (and compete) to assign concrete meaning to abstract aims.
3. A complex process in which many policymakers and influencers are cooperating/ competing to define problems in many policymaking centres.

They suggest that resolving ambiguity affects policy in different ways, to influence the:

• Single policy instrument produced by one centre,
• Accumulation of separate instruments emerging from many centres, or
• Outcome of collaboration across those centres.

The latter descriptions, reflecting multi-centric policymaking, seem particularly relevant to major contemporary policy problems – such as global public health and climate crises – in which cooperation across (and outside of) many levels and types of government is essential.

Resolving ambiguity in policy analysis texts

This context helps us to interpret common (Step 1) advice in policy analysis textbooks: define a policy problem for your client, using your skills of research and persuasion but tailoring your advice to your client’s interests and beliefs. Yet, gone are the mythical days of elite analysts communicating to a single core executive in charge of formulating and implementing all policy instruments. Many analysts engage with many centres producing (or co-producing) many instruments. Resolving ambiguity in one centre does not guarantee the delivery of your aims across many.

Two ways to resolve ambiguity in policy analysis

Classic debates would highlight two different responses:

• ‘Top down’ accounts see this issue through the lens of a single central government, examining how to reassert central control by minimising implementation gaps.

Policy analysis may focus on (a) defining the policy problem, and (b) ensuring the implementation of its solution.

• ‘Bottom up’ accounts identify the inevitability (and legitimacy) of policy influence in multiple centres. Policy analysis may focus on how to define the problem in cooperation with other centres, or to set a strategic direction and encourage other centres to make sense of it in their context.

This terminology went out of fashion, but note the existence of each tendency in two ideal-type approaches to contemporary policy problems:

1. Centralised and formalised approaches.

Seek clarity and order to address urgent policy problems. Define the policy problem clearly, translate that definition into strategies for each centre, and develop a common set of effective ‘tools’ to ensure cooperation and delivery.

Policy analysis may focus on technical aspects, such as how to create a fine-detail blueprint for action, backed by performance management and accountability measures that tie actors to specific commitments.

The tagline may be: ambiguity is a problem to be solved, to direct policy actors towards a common goal.

2. Decentralised, informal, collaborative approaches.

Seek collaboration to make sense of, and address, problems. Reject a single definition of the problem, encourage actors in each centre (or in concert) to deliberate to make sense of problems together, and co-create the rules to guide a continuous process of collective behaviour.

Policy analysis may focus on how to contribute to a collaborative process of sense-making and rule-making.

The tagline may be: ambiguity presents an opportunity to energise policy actors, to harness the potential for innovation arising from deliberation.

Pick one approach and stick with it?

Describing these approaches in such binary terms makes the situation – and choice between approaches – look relatively straightforward. However, note the following issues:

• Many policy sectors (and intersectoral agendas) are characterised by intense disagreement on which choice to make. These disagreements intersect with others (such as when people seek not only transformative policy change to solve global problems, but also equitable process and outcomes).
• Some sectors seem to involve actors seeking the best of both worlds (centralise and localise, formalise and deliberate) without recognising the trade-offs and dilemmas that arise.
• I have described these options as choices, but did not establish if anyone is in the position to make or contribute to that choice.

In that context, resolving ambiguity in your favour may still be the prize, but where would you even begin?

Further reading

Well, that was an unsatisfying end to the post, eh? Maybe I’ll write a better one when some things are published. In the meantime, some of these papers and posts explore some of these issues:

• Policy Concepts in 1000 Words: the Institutional Analysis and Development Framework (IAD) and Governing the Commons – the IAD is your go-to framework to study collaborative governance
• Improving policy implementation through collaborative policymaking
• The future of public health policymaking after COVID-19: lessons from Health in All Policies – identifies these dilemmas in relation to health equity strategies
• The future of education equity policy: ‘neoliberal’ versus ‘social justice’ approaches – identifies equivalent debates in education
• A systematic review of energy systems – tries to discourage people from thinking that we can solve these issues with a vague reference to ‘systems thinking’
• The contested relationship between governance and evidence – relates ‘evidence based’ to ‘collaborative’ approaches to governance (as does Who should be involved in the process of policy analysis?)
• This page  describes a book and many posts on ‘prevention’ policy. We complain that governments use the phrase ‘prevention is better than cure’ without defining prevention, and that they want centralised and decentralised approaches to ‘preventive policymaking’.
• The myth of ‘evidence based policymaking’ in a decentred state argues that policymakers don’t really know what they want, and couldn’t get it if they tried.
• The politics of policy design – tries to make you feel good about never really resolving these dilemmas.

Policymaking in the UK: do you really know who is in charge and who to blame? #POLU9UK

This week, we continue with the idea of two stories of British politics. In one, the Westminster model-style story, the moral is that the centralisation of power produces clear lines of accountability: you know who is in charge and, therefore, the heroes or villains. In another, the complex government story, the world seems too messy and power too diffuse to know all the main characters.

Although some aspects of these stories are specific to the UK, they relate to some ‘universal’ questions and concepts that we can use to identify the limits to centralised power. Put simply, some rather unrealistic requirements for the Westminster story include:

1. You know what policy is, and that it is made by a small number of actors at the heart of government.
2. Those actors possess comprehensive knowledge about the problems and solutions they describe.
3. They can turn policy intent into policy outcomes in a straightforward way.

If life were that simple, I wouldn’t be asking you to read the following blog posts (underlined) which complicate the hell out of our neat story:

You don’t know what policy is, and it is not only made by a small number of actors at the heart of government.

We don’t really know what government policy is. In fact, we don’t even know how to define ‘public policy’ that well. Instead, a definition like ‘the sum total of government action, from signals of intent to the final outcomes’ raises more issues than it settles: policy is remarkably difficult to identify and measure; it is made by many actors inside, outside, and sort of inside/outside government; the boundary between the people influencing and making policy is unclear; and, the study of policy is often about the things governments don’t do.

Actors don’t possess comprehensive knowledge about the problems and solutions they describe

It’s fairly obvious than no-one possesses all possible information about policy problems and the likely effects of proposed solutions. It’s not obvious what happens next. Classic discussions identified a tendency to produce ‘good enough’ decisions based on limited knowledge and cognitive ability, or to seek other measures of ‘good’ policy such as their ability to command widespread consensus (and no radical movement away from such policy settlements). Modern discussions offer us a wealth of discussions of the implications of ‘bounded rationality’, but three insights stand out:

1. Policymakers pay disproportionate attention to a tiny proportion of the issues for which they are responsible. There is great potential for punctuations in policy/ policymaking when their attention lurches, but most policy is made in networks in the absence of such attention.
2. Policymakers combine ‘rational’ and ‘irrational’ ways to make decisions with limited information. The way they frame problems limits their attention to a small number of possible solutions, and that framing can be driven by emotional/ moral choices backed up with a selective use of evidence.
3. It is always difficult to describe this process as ‘evidence-based policymaking’ even when policymakers have sincere intentions.

Policymakers cannot turn policy intent into policy outcomes in a straightforward way

The classic way to describe straightforward policymaking is with reference to a policy cycle and its stages. This image of a cycle was cooked up by marketing companies trying to sell hula hoops to policymakers and interest groups in the 1960s. It is not an accurate description of policymaking (but spirographs are harder to sell).

Instead, for decades we have tried to explain the ‘gap’ between the high expectations of policymakers and the actual – often dispiriting- outcomes, or wonder if policymakers really have such high expectations for success in the first place (or if they prefer to focus on how to present any of their actions as successful). This was a key topic before the rise of ‘multi-level governance’ and the often-deliberate separation of central government action and expected outcomes.

The upshot: in Westminster systems do you really know who is in charge and who to blame?

These factors combine to generate a sense of complex government in which it is difficult to identify policy, link it to the ‘rational’ processes associated with a small number of powerful actors at the heart of government, and trace a direct line from their choices to outcomes.

Of course, we should not go too far to argue that governments don’t make a difference. Indeed, many ministers have demonstrated the amount of damage (or good) you can do in government. Still, let’s not assume that the policy process in the UK is anything like the story we tell about Westminster.

Seminar questions

In the seminar, I’ll ask you reflect on these limits and what they tell us about the ‘Westminster model’. We’ll start by me asking you to summarise the main points of this post. Then, we’ll get into some examples in British politics.

Try to think of some relevant examples of what happens when, for example, minsters seem to make quick and emotional (rather than ‘evidence based’) decisions: what happens next? Some obvious examples – based on our discussions so far – include the Iraq War and the ‘troubled families’ agenda, but please bring some examples that interest you.

In group work, I’ll invite you to answer these questions:

1. What is UK government policy on X? Pick a topic and tell me what government policy is.
2. How did the government choose policy? When you decide what government policy is, describe how it made its choices.
3. What were the outcomes? When you identify government policy choices, describe their impact on policy outcomes.

I’ll also ask you to identify at least one blatant lie in this blog post.

The implementation of policy in Scotland #POLU9SP

There are two classic ways to describe and try to explain policy implementation: top-down and bottom-up (see also the policy cycle).

We can focus on these descriptions of policy implementation to make two points relevant to our discussion so far:

1. You might think that the ‘Scottish policy style’ and ‘Scottish approach’ produce fewer problems of implementation, but they produce different problems.
2. An ‘implementation gap’ reinforces our sense (in the previous lecture) that there hasn’t been that much policy divergence in Scotland since devolution.

Implementation and the Scottish policy style

Based on our discussions so far, you might think that the Scottish Government would suffer fewer problems of implementation than the UK government because:

1. Its public sector landscape often appears to be less fragmented.
2. It is less likely to oversee a ‘top-down’ policy style with unintended consequences (note the potential confusion over the meaning of top-down).
3. Its greater willingness to consult helps it gather information and secure ‘ownership’.

Yet, I found that it (generally) had different, not fewer, problems. For example, you do not guarantee implementation success by relying on local authorities rather than private or third sector bodies. Further, the Scottish Government may have more ‘external conditions’ to take into account, since its policies often overlap with those of the UK government and it often does not control the success of its own policies.

Or, high levels of consultation can help produce unrealistic strategies and inflated expectations when a government gives the impression that: a policy choice represents radical change; it is the key actor (rather than one of many players in a multi-level system); and, it plans to enforce not delegate and negotiate policy delivery.

The Scottish Approach and bottom-up implementation

Indeed, isn’t the newest incarnation of the ‘Scottish approach’ more of a bottom-up than top-down strategy? In other words, it sets a broad framework based on policy outcomes and asks local authorities and community planning partnerships to produce their own strategies to achieve those outcomes.

Consequently, it may not make sense to try to explain an ‘implementation gap’ because some of the top-down conditions for success do not seem to apply, including: there are no clear/ consistent objectives (at least according to my interpretation of that condition), and there is no requirement for compliant officials.

Policy divergence and the implementation gap

Yet, many Scottish Government policies can be analysed usefully through the lens of an ‘implementation gap’, including:

• ‘Free personal care’ for older people. This is an important one, because FPC used to symbolise policy divergence after devolution. Yet, it translated into a less-than-expected reduction in care home costs and, for many people (it is hard to know the number) a replacement of one way of securing free care with another (you should make sure you understand how this happened – see Scottish Politics for more detail). There have also been problems with waiting lists for care, and debate about what counts/ doesn’t count as personal care.
• Housing and homelessness. Over the years, the Scottish Government has promised higher housing standards and lower levels of homelessness but struggled to translate ambitious aims into outcomes (and, it has produced essentially the same strategy on homelessness twice since devolution).
• Fox hunting. You can still hunt foxes if you want (anyway, would there be many people there to stop you if you tried?) and the unintended consequence of policy is that you might now catch the wrong ones.

If we have the time, we might also discuss modern examples such as the Curriculum for Excellence. We might also wonder why some policies seem to have been implemented successfully (can you think of examples?).

In many of these cases, the promise of policy divergence mixes with implementation problems to produce less divergence than we might have expected if we focused simply on the initial choices. This conclusion reinforces the idea that constitutional change in Scotland does not tend to produce radical policy change or major divergence from UK government policy.

Policy Concepts in 1000 Words: Success and Failure (Evaluation)

(podcast download)

Policy success is in the eye of the beholder. The evaluation of success is political in several ways. It can be party political, when election campaigns focus on the record of the incumbent government. Policy decisions produce winners and losers, prompting disputes about success between actors with different aims. Evaluation can be political in subtler but as-important ways, involving scientific disputes about:

• How long we wait to evaluate.
• How well-resourced our evaluation should be.
• The best way to measure and explain outcomes.
• The ‘benchmarks’ to use – should we compare outcomes with the past or other countries?
• How we can separate the effect of policy from other causes, in a complex world where randomised-controlled-trials are often difficult to use.

In this more technical-looking discussion, the trade-off is between the selection of a large mixture of measures that are hard to work with, or a small number of measures that are handpicked and represent no more than crude proxies for success.

Evaluation is political because we set the agenda with the measures we use, by prompting a focus on some aims at the expense of others. A classic example is the aim to reduce healthcare waiting times, which represent a small part of health service activity but generate disproportionate attention and action, partly because outcomes are relatively visible and easy to measure. Many policies are implemented and evaluated using such proxies: the government publishes targets to provide an expectation of implementer behaviour; and, regulatory bodies exist to monitor compliance.

Let’s consider success in terms of the aims of the person responsible for the policy. It raises four interesting issues:

1. The aims of that policymaker may not be clear. For example, they may not say why they made particular choices, they may have many reasons, their reasons may not be specific enough to be meaningful, and/or they may not be entirely truthful.
2. Policymaking is a group effort, which magnifies the problem of identifying a single, clear, aim.
3. Aims are not necessarily noble. Marsh and McConnell describe three types. Process measures success in terms of its popularity among particular groups and its ease of passage through the legislature. Political describes its effect on the government’s popularity. Programmatic describes its implementation in terms of original aims, its effect in terms of intended outcomes, and the extent to which it represented an ‘efficient use of resources’. Elected policymakers may justify their actions in programmatic terms, but be more concerned with politics and process. Or, their aims may be unambitious. We could identify success in their terms but still feel that major problems remain unsolved.
4. Responsibility is a slippery concept. In a Westminster system, we may hold ministers to be ultimately responsible but, in practice, responsibility is shared with a range of people in various types and levels of government. In multi-level political systems, responsibility may be shared with several elected bodies with their own mandates and claims to pursue distinctive aims.

Traditionally, these responsibility issues were played out in top-down and bottom-up discussions of policy implementation. For the sake of simplicity, the ‘top’ is the policymaker at the heart of central government and we try to explain success or failure according to the extent to which policy implementation met these criteria:

1.   The policy’s objectives are clear, consistent and well communicated and understood.

2.   The policy will work as intended when implemented.

3.   The required resources are committed to the programme.

4.   Policy is implemented by skilful and compliant officials.

5.   Success does not depend on cooperation from many bodies.

6.   Support from influential groups is maintained.

7.   Demographic and socioeconomic conditions, and unpredictable events beyond the control of policymakers, do not significantly undermine the process.

Such explanations for success still have some modern day traction, such as in recommendations by the Institute for Government:

1. Understand the past and learn from failure.
2. Open up the policy process.
3. Be rigorous in analysis and use of evidence.
4. Take time and build in scope for iteration and adaptation.
5. Recognise the importance of individual leadership and strong personal relationships.
6. Create new institutions to overcome policy inertia.
7. Build a wider constituency of support.

Alternatively, ‘bottom-up’ studies prompted a shift of analysis, towards a larger number of organisations which made policy as they carried it out – and had legitimate reasons to diverge from the aims set at the ‘top’. Indeed, central governments might encourage a bottom up approach, by setting a broad strategy and accepting that other bodies will implement policy in their own way. However, this is difficult to do in Westminster systems, where government success is measured in terms of ministerial and party manifesto aims.

Examples of success and failure?

Many implementation studies focus on failure, including Pressman and Wildavsky’s ‘How Great Expectations in Washington are Dashed in Oakland’ and Marsh & Rhodes’ focus on the ‘implementation gap’ during the Thatcher Government era (1979-90).

In contrast, the IFG report focuses on examples of success, derived partly from a vote by UK political scientists, including: the national minimum wage, Scottish devolution, and privatisation.

Note the respondents’ reasons for declaring success, based on a mix of their personal values and their assessment of process, political and programmatic factors.  They declare success in very narrow terms, as the successful delivery in the short term.

So, privatisation is a success because the government succeeded in raising money, boosting its popularity and international reputation – not because we have established that the nationalized industries work better in the private sector.

Similarly, devolution was a declared a success because it solved a problem (local demand for self-autonomy), not because devolved governments are better at making policy or their policies have improved the lives of the Scottish population (Neil McGarvey and I discuss this here).

Individual policy instruments like the smoking ban are often treated in similar ways – we declare instant success when the bill passes and public compliance is high, then consider the longer term successes (less smoking, less secondhand smoke) later.

Further reading and watching: (1) Can a Policy Fail and Succeed at the Same Time?

(2)  http://blogs.lse.ac.uk/politicsandpolicy/archives/34735

Why should you read and watch this case study? I hesitate to describe UK tobacco control as a success because it instantly looks like I am moralising, and because it is based on a narrow set of policymaking criteria rather than an outcome in the population (it is up to you to decide if the UK’s policies are appropriate and its current level of smoking and health marks policy success). However, it represents a way to explore success in terms of several ‘causal factors’ (Peter John) that arise in each 1000 Words post: institutions, networks, socioeconomic conditions and ideas. Long term tobacco control ‘success’ happened because:

• the department of health took the policy lead (replacing trade and treasury departments);
• tobacco is ‘framed’ as a pressing public health problem, not an economic good;
• public health groups are consulted at the expense of tobacco companies;
• socioeconomic conditions (including the value of tobacco taxation, and public attitudes to tobacco control) are conducive to policy change;
• and, the scientific evidence on the harmful effects of smoking and secondhand smoking are ‘set in stone’ within governments.

The ‘take home’ message here is that ‘success’ depends as much on a policy environment conducive to change as the efficacy of political instruments and leadership qualities of politicians.

Update September 2019

I have now written up this UK tobacco discussion in this book:

Paul Cairney (2019) ‘The transformation of UK tobacco control’ in (eds) Mallory Compton and Paul ‘t Hart Great Policy Successes: How Governments Get It Right in a Big Way at Least Some of the Time (Oxford: Oxford University Press) Preview PDF

Each chapter is accompanied by a case study, such as the one on UK tobacco by Margot Gagliani

What we can learn from great public policy successes https://t.co/W5EvMuPu7O@paulthart88 & @mec_phd

— Professor Paul Cairney (@CairneyPaul) September 5, 2019

 

Policy Concepts in 1000 Words: The Policy Cycle and its Stages

See also What is Policy? and the Policy concepts in 1000 words series

(podcast download)

The classic way to study policymaking is to break it down into stages. The stages have changed over the years, and vary by country, but the basic ideas remain the same:

1. Descriptive. Let’s simplify a complex world by identifying its key elements.
2. Prescriptive. Let’s work out how to make policy, to translate public demands into government action (or at least to carry out government policy).

A cycle divides the policy process into a series of stages, from a notional starting point at which policymakers begin to think about a policy problem to a notional end point at which a policy has been implemented and policymakers think about how successful it has been before deciding what to do next. The image is of a continuous process rather than a single event. The evaluation stage of policy 1 represents the first stage of policy 2, as lessons learned in the past set the agenda for choices to be made in the future:

• Agenda setting. Identifying problems that require government attention, deciding which issues deserve the most attention and defining the nature of the problem.
• Policy formulation. Setting objectives, identifying the cost and estimating the effect of solutions, choosing from a list of solutions and selecting policy instruments.
• Legitimation. Ensuring that the chosen policy instruments have support. It can involve one or a combination of: legislative approval, executive approval, seeking consent through consultation with interest groups, and referenda.
• Implementation. Establishing or employing an organization to take responsibility for implementation, ensuring that the organization has the resources (such as staffing, money and legal authority) to do so, and making sure that policy decisions are carried out as planned.
• Evaluation. Assessing the extent to which the policy was successful or the policy decision was the correct one; if it was implemented correctly and, if so, had the desired effect.
• Policy maintenance, succession or termination. Considering if the policy should be continued, modified or discontinued.

The cycle is useful in many ways. It is simple and understandable. It can be applied to all political systems. The emphasis on cycles highlights fluid policymaking.  There is also a wide range of important studies (and key debates) based on the analysis of particular stages – such as the top-down versus bottom-up approaches to the study of policymaking.

However, the stages approach is no longer central to policy studies, partly because it does not help explain what it describes, and partly because it oversimplifies a complex world (does it also seem to take the politics out of policymaking? In other words, note the often-fraught politics of seemingly-innocuous stages such as evaluation). The policymaking system may be seen more as a collection of thousands of policy cycles, which interact with each other to produce much less predictable outcomes.  Indeed, many of the theories or concepts outlined in this series serve as replacements for a focus on cycles (see the The Advocacy Coalition Framework and Multiple Streams Analysis in particular).

The prescriptive side of cycles and stages is a bit more interesting, because it may be both unrealistic and useful at the same time. Stages can be used to organise policymaking in a simple way: identify policymaker aims, identify policies to achieve those aims, select a policy measure, ensure that the selection is legitimised by the population or its legislature, identify the necessary resources, implement and then evaluate the policy.  The academic idea is simple and the consequent advice to policy practitioners is straightforward.  It is difficult – but not impossible – to describe a more meaningful, more realistic, analytical model to policymakers (and give advice on how to act) in the same straightforward way.

Related articles

• https://paulcairney.wordpress.com/1000-words/
• For more on the prescriptive side, see How can policy theory have an impact on policymaking? and chapter 2 of The Politics of Evidence-based Policymaking