Tag Archives: practical lessons from policy theory

Evidence-based policymaking: political strategies for scientists living in the real world

Note: I wrote the following discussion (last year) to be a Nature Comment but it was not to be!

Nature articles on evidence-based policymaking often present what scientists would like to see: rules to minimise bias caused by the cognitive limits of policymakers, and a simple policy process in which we know how and when to present the best evidence.[1]  What if neither requirement is ever met? Scientists will despair of policymaking while their competitors engage pragmatically and more effectively.[2]

Alternatively, if scientists learned from successful interest groups, or by using insights from policy studies, they could develop three ‘take home messages’: understand and engage with policymaking in the real world; learn how and when evidence ‘wins the day’; and, decide how far you should go to maximise the use of scientific evidence. Political science helps explain this process[3], and new systematic and thematic reviews add new insights.[4] [5] [6] [7]

Understand and engage with policymaking in the real world

Scientists are drawn to the ‘policy cycle’, because it offers a simple – but misleading – model for engagement with policymaking.[3] It identifies a core group of policymakers at the ‘centre’ of government, perhaps giving the impression that scientists should identify the correct ‘stages’ in which to engage (such as ‘agenda setting’ and ‘policy formulation’) to ensure the best use of evidence at the point of authoritative choice. This is certainly the image generated most frequently by health and environmental scientists when they seek insights from policy studies.[8]

Yet, this model does not describe reality. Many policymakers, in many levels and types of government, adopt and implement many measures at different times. For simplicity, we call the result ‘policy’ but almost no modern policy theory retains the linear policy cycle concept. In fact, it is more common to describe counterintuitive processes in which, for example, by the time policymaker attention rises to a policy problem at the ‘agenda setting’ stage, it is too late to formulate a solution. Instead, ‘policy entrepreneurs’ develop technically and politically feasible solutions then wait for attention to rise and for policymakers to have the motive and opportunity to act.[9]

Experienced government science advisors recognise this inability of the policy cycle image to describe real world policymaking. For example, Sir Peter Gluckman presents an amended version of this model, in which there are many interacting cycles in a kaleidoscope of activity, defying attempts to produce simple flow charts or decision trees. He describes the ‘art and craft’ of policy engagement, using simple heuristics to deal with a complex and ‘messy’ policy system.[10]

Policy studies help us identify two such heuristics or simple strategies.

First, respond to policymaker psychology by adapting to the short cuts they use to gather enough information quickly: ‘rational’, via trusted sources of oral and written evidence, and ‘irrational’, via their beliefs, emotions, and habits. Policy theories describe many interest group or ‘advocacy coalition’ strategies, including a tendency to combine evidence with emotional appeals, romanticise their own cause and demonise their opponents, or tell simple emotional stories with a hero and moral to exploit the biases of their audience.[11]

Second, adapt to complex ‘policy environments’ including: many policymakers at many levels and types of government, each with their own rules of evidence gathering, network formation, and ways of understanding policy problems and relevant socioeconomic conditions.[2] For example, advocates of international treaties often find that the evidence-based arguments their international audience takes for granted become hotly contested at national or subnational levels (even if the national government is a signatory), while the same interest groups presenting the same evidence of a problem can be key insiders in one government department but ignored in another.[3]

Learn the conditions under which evidence ‘wins the day’ in policymaking

Consequently, the availability and supply of scientific evidence, on the nature of problems and effectiveness of solutions, is a necessary but insufficient condition for evidence-informed policy. Three others must be met: actors use scientific evidence to persuade policymakers to pay attention to, and shift their understanding of, policy problems; the policy environment becomes broadly conducive to policy change; and, actors exploit attention to a problem, the availability of a feasible solution, and the motivation of policymakers, during a ‘window of opportunity’ to adopt specific policy instruments.10

Tobacco control represents a ‘best case’ example (box 1) from which we can draw key lessons for ecological and environmental policies, giving us a sense of perspective by highlighting the long term potential for major evidence-informed policy change. However, unlike their colleagues in public health, environmental scientists have not developed a clear sense of how to produce policy instruments that are technically and politically feasible, so the delivery of comparable policy change is not inevitable.[12]

Box 1: Tobacco policy as a best case and cautionary tale of evidence-based policymaking

Tobacco policy is a key example – and useful comparator for ecological and environmental policies – since it represents a best case scenario and cautionary tale.[13] On the one hand, the scientific evidence on the links between smoking, mortality, and preventable death forms the basis for modern tobacco control policy. Leading countries – and the World Health Organisation, which oversees the Framework Convention on Tobacco Control (FCTC) – frame tobacco use as a public health ‘epidemic’ and allow their health departments to take the policy lead. Health departments foster networks with public health and medical groups at the expense of the tobacco industry, and emphasise the socioeconomic conditions – reductions in (a) smoking prevalence, (b) opposition to tobacco control, and (c) economic benefits to tobacco – most supportive of tobacco control. This framing, and conducive policymaking environment, helps give policymakers the motive and opportunity to choose policy instruments, such as bans on smoking in public places, which would otherwise seem politically infeasible.

On the other hand, even in a small handful of leading countries such as the UK, it took twenty to thirty years to go from the supply of the evidence to a proportionate government response: from the early evidence on smoking in the 1950s prompting major changes from the 1980s, to the evidence on passive smoking in the 1980s prompting public bans from the 2000s onwards. In most countries, the production of a ‘comprehensive’ set of policy measures is not yet complete, even though most signed the FCTC.

Decide how far you’ll go to maximise the use of scientific evidence in policymaking

These insights help challenge the naïve position that, if policymaking can change to become less dysfunctional[1], scientists can be ‘honest brokers’[14] and expect policymakers to use their evidence quickly, routinely, and sincerely. Even in the best case scenario, evidence-informed change takes hard work, persistence, and decades to achieve.

Since policymaking will always appear ‘irrational’ and complex’[3], scientists need to think harder about their role, then choose to engage more effectively or accept their lack of influence.

To deal with ‘irrational’ policymakers, they should combine evidence with persuasion, simple stories, and emotional appeals, and frame their evidence to make the implications consistent with policymakers’ beliefs.

To deal with complex environments, they should engage for the long term to work out how to form alliances with influencers who share their beliefs, understand in which ‘venues’ authoritative decisions are made and carried out, the rules of information processing in those venues, and the ‘currency’ used by policymakers when they describe policy problems and feasible solutions.[2] In other words, develop skills that do not come with scientific training, avoid waiting for others to share your scientific mindset or respect for scientific evidence, and plan for the likely eventuality that policymaking will never become ‘evidence based’.

This approach may be taken for granted in policy studies[15], but it raises uncomfortable dilemmas regarding how far scientists should go, to maximise the use of scientific evidence in policy, using persuasion and coalition-building.

These dilemmas are too frequently overshadowed by claims – more comforting to scientists – that politicians are to blame because they do not understand how to generate, analyse, and use the best evidence. Scientists may only become effective in politics if they apply the same critical analysis to themselves.

[1] Sutherland, W.J. & Burgman, M. Nature 526, 317–318 (2015).

[2] Cairney, P. et al. Public Administration Review 76, 3, 399-402 (2016)

[3] Cairney, P. The Politics of Evidence-Based Policy Making (Palgrave Springer, 2016).

[4] Langer, L. et al. The Science of Using Science (EPPI, 2016)

[5] Breckon, J. & Dodson, J. Using Evidence. What Works? (Alliance for Useful Evidence, 2016)

[6] Palgrave Communications series The politics of evidence-based policymaking (ed. Cairney, P.)

[7] Practical lessons from policy theories (eds. Weible, C & Cairney, P.) Policy and Politics April 2018

[8] Oliver, K. et al. Health Research Policy and Systems, 12, 34 (2016)

[9] Kingdon, J. Agendas, Alternatives and Public Policies (Harper Collins, 1984)

[10] Gluckmann, P. Understanding the challenges and opportunities at the science-policy interface

[11] Cairney, P. & Kwiatkowski, R. Palgrave Communications.

[12] Biesbroek et al. Nature Climate Change, 5, 6, 493–494 (2015)

[13] Cairney, P. & Yamazaki, M. Journal of Comparative Policy Analysis

[14] Pielke Jr, R. originated the specific term The honest broker (Cambridge University Press, 2007) but this role is described more loosely by other commentators.

[15] Cairney, P. & Oliver, K. Health Research Policy and Systems 15:35 (2017)

4 Comments

Filed under Evidence Based Policymaking (EBPM), public policy

Debating the politics of evidence-based policy

Joshua Newman has provided an interesting review of three recent books on evidence/ policy (click here). One of those books is mine: The Politics of Evidence-Based Policy Making (which you can access here).

His review is very polite, for which I thank him. I hope my brief response can be seen in a similarly positive light (well, I had hoped to make it brief). Maybe we disagree on one or two things, but often these discussions are about the things we emphasize and the way we describe similar points.

There are 5 points to which I respond because I have 5 digits on my right hand. I’d like you to think of me counting them out on my fingers. In doing so, I’ll use ‘Newman’ throughout, because that’s the academic convention, but I’d also like to you imagine me reading my points aloud and whispering ‘Joshua’ before each ‘Newman’.

  1. Do we really need to ‘take the debate forward’ so often?

I use this phrase myself, knowingly, to keep a discussion catchy, but I think it’s often misleading. I suggest not to get your hopes up too high when Newman raises the possibility of taking the debate forward with his concluding questions. We won’t resolve the relationship between evidence, politics & policy by pretending to reframe the same collection of questions about the prospect of political reform that people have been asking for centuries. It is useful to envisage better political systems (the subject of Newman’s concluding remarks) but I don’t think we should pretend that this is a new concern or that it will get us very far.

Indeed, my usual argument is that researchers need to do something (such as improve how we engage in the policy process) while we wait for political system reforms to happen (while doubting if they will ever happen).

Further, Newman does not produce any political reforms to address the problems he raises. Rather, for example, he draws attention to Trump to describe modern democracies as ‘not pluralist utopias’ and to identify examples in which policymakers draw primarily on beliefs, not evidence. By restating these problems, he does not solve them. So, what are researchers supposed to do after they grow tired of complaining that the world does not meet their hopes or expectations?

In other words, for me, (a) promoting political change and (b) acting during its absence are two sides of the same coin. We go round and round more often than we take things forward.

  1. What debate are we renaming?

Newman’s ‘we’ve heard it before’ argument seems more useful, but there is a lot to hear and relatively few people have heard it. I’d warn against the assumption that ‘I’ve heard this before’ can ever equal ‘we’ve heard it before’ (unless ‘we’ refers to a tiny group of specialists talking only to each other).

Rather, one of the most important things we can do as academics is to tell the same story to each other (to check if we understand the same story, in the same way, and if it remains useful) and to wider audiences (in a way that they can pick up and use without dedicating their career to our discipline).

Some of our most important insights endure for decades and they sometimes improve in the retelling. We apply them to new eras, and often come to the same basic conclusions, but it seems unhelpful to criticise a lack of complete novelty in individual texts (particularly when they are often designed to be syntheses). Why not use them to occasionally take a step back to discuss and clarify what we know?

Perhaps more importantly, I don’t think Newman is correct when he says that each book retells the story of the ‘research utilization’ literature. I’m retelling the story of policy theory, which describes how policymakers deal with bounded rationality in a complex policymaking environment. Policy theory’s intellectual histories often provide very different perspectives – of the policymaker trying to make good enough decisions, rather than the researcher trying to improve the uptake of their research – than the agenda inspired by Weiss et al (see for example The New Policy Sciences).

  1. Don’t just ‘get political’; understand the policy process

I draw on policy theory because it helps people understand policymaking. It would be a mistake to conclude from my book that I simply want researchers to ‘get political’. Rather, I want them to develop useful knowledge of the policy process in which they might want to engage. This knowledge is not freely available; it takes time to understand the discipline and reflect on policy dynamics.

Yet, the payoff can be profound, if only because it helps people think about the difference between two analytically separate causes of a notional ‘evidence policy gap’: (a) individuals making choices based on their beliefs and limited information (which is relatively easy to understand but also to caricature), and (b) systemic or ‘environmental’ causes (which are far more difficult to conceptualise and explain, but often more useful to understand).

  1. Don’t throw out the ‘two communities’ phrase without explaining why

Newman criticises the phrase ‘two communities’ as a description of silos in policymaking versus research, partly because (a) many policymakers use research frequently, and (b) the real divide is often between users/ non-users of research within policymaking organisations. In short, there are more than two communities.

I’d back up his published research with my anecdotal experience of giving talks to government audiences: researchers and analysts within government are often very similar in outlook to academics and they often talk in the same way as academics about the disconnect between their (original or synthetic) research and its use by their ‘operational’ colleagues.

Still, I’m not sure why Newman concludes that the ‘two communities’ phrase is ‘deeply flawed and probably counter-productive’. Yes, the world is more nuanced and less binary than ‘two communities’ suggests. Yes, the real divide may be harder to spot. Still, as Newman et al suggest: ‘Policy makers and academics should focus on bridging instruments that can bring their worlds closer together’. This bullet point from their article seems, to me, to be the point of using the phrase ‘two communities’. Maybe Caplan used the phrase differently in 1979, but to assert its historic meaning then reject the phrase’s use in modern discussion seems less useful than simply clarifying the argument in ways such as:

  • There is no simple policymaker/ academic divide but, … note the major difference in requirements between (a) people who produce or distribute research without taking action, which allows them (for example) to be more comfortable with uncertainty, and (b) people who need to make choices despite having incomplete information to hand.
  • You might find a more receptive audience in one part of government (e.g. research/ analytical) than another (e.g. operational), so be careful about generalising from singular experiences.
  1. Should researchers engage in the policy process?

Newman says that each book, ‘unfairly places the burden of resolving the problem in the hands of an ill-equipped group of academics, operating outside the political system’.

I agree with Newman when he says that many researchers do not possess the skills to engage effectively in the policy process. Scientific training does not equip us with political skills. Indeed, I think you could read a few of my blog posts and conclude, reasonably, that you would like nothing more to do with the policy process because you’d be more effective by focusing on research.

The reason I put the onus back on researchers is because I am engaging with arguments like the one expressed by Newman (in other words, part of the meaning comes from the audience). Many people conclude their evidence policy discussions by identifying (or ‘reframing’) the problem primarily as the need for political reform. For me, the focus on other people changing to suit your preferences seems unrealistic and misplaced. In that context, I present the counter-argument that it may be better to adapt effectively to the policy process that exists, not the one you’d like to see. Sometimes it’s more useful to wear a coat than complain about the weather.

See also:  The Politics of Evidence 

The Politics of Evidence revisited

 

Pivot cover

1 Comment

Filed under Evidence Based Policymaking (EBPM), public policy

5 images of the policy process

Cairney 2017 image of the policy process

A picture tells a thousand words but, in policy studies, those words are often misleading or unclear. The most useful images can present the least useful advice, or capture a misleading metaphor. Images from the most useful theories are useful when you already know the theory, but far more difficult to grasp initially.

So, I present two examples from each, then describe what a compromise image might look like, to combine something that is easy to pick up and use but also not misleading or merely metaphorical.

Why do we need it? It is common practice at workshops and conferences for some to present policy process images on powerpoint and for others to tweet photos of them, generally with little discussion of what they say and how useful they are. I’d like to see as-simple but more-useful images spread this way.

1. The policy cycle

cycle

The policy cycle is perhaps the most used and known image. It divides the policy process into a series of stages (described in 1000 words and 500 words). It oversimplifies, and does not explain, a complex policymaking system. We are better to imagine, for example, thousands of policy cycles interacting with each other to produce less orderly behaviour and less predictable outputs.

For students, we have dozens of concepts and theories which serve as better ways to understand policymaking.

Policymakers have more use for the cycle, to tell a story of what they’d like to do: identify aims, identify policies to achieve those aims, select a policy measure, ensure that the selection is legitimised by the population or its legislature, identify the necessary resources, implement, then evaluate the policy.

Yet, most presentations from policymakers, advisers, and practitioners modify the cycle image to show how messy life really is:

2. The multiple streams metaphor

NASA launch

The ‘multiple streams’ approach uses metaphor to describe this messier world (described in 1000 words and 500 words). Instead of a linear cycle – in which policymakers define problems, then ask for potential solutions, then select one – we describe these ‘stages’ as independent ‘streams’. Each stream – heightened attention to a problem (problem stream), an available and feasible solution (policy stream), and the motive to select it (politics stream) – must come together during a ‘window of opportunity’ or the opportunity is lost.

Many people like MSA because it contains a flexible metaphor which is simple to pick up and use. However, it’s so flexible that I’ve seen many different ways to visualise – and make sense of – the metaphor, including literal watery streams, which suggest that when they come together they are hard to separate.  There is the Ghostbusters metaphor which shows that key actors (‘entrepreneurs’) help couple the streams. There is also Howlett et al’s attempt to combine the streams and cycles metaphors (reproduced here, and criticised here).

However, I’d encourage Kingdon’s space launch metaphor in which policymakers will abort the mission unless every factor is just right.

3. The punctuated equilibrium graph

True et al figure 6.2

Punctuated equilibrium theory is one of the most important approaches to policy dynamics, now backed up with a wealth of data from the Comparative Agendas Project. The image (in True et al, 2007) describes the result of the policy process rather than the process itself. It describes government budgets in the US, although we can find very similar images from studies of budgets in many other countries and in many measures of policy change.

It sums up a profoundly important message about policy change: we find a huge number of very small changes, and a very small number of huge changes. Compare the distribution of values in this image with the ‘normal distribution’ (the dotted line). It shows a ‘leptokurtic’ distribution, with most values deviating minimally from the mean (and the mean change in each budget item is small), but with a high number of ‘outliers’.

The image helps sum up a key aim of PET, to measure and try to explain long periods of policymaking stability, and policy continuity, disrupted by short but intense periods of instability and change. One explanation relates to ‘bounded rationality’: policymakers have to ignore almost all issues while paying attention to some. The lack of ‘macropolitical’ attention to most issues helps explain stability and continuity, while lurches of attention can help explain instability (although attention can fade before ‘institutions’ feel the need to respond).

Here I am, pointing at this graph:

4. The advocacy coalition framework ‘flow diagram’

ACF diagram

The ACF presents an ambitious image of the policy process, in which we zoom out to consider how key elements fit together in a process containing many actors and levels of government. Like many policy theories, it situates most of the ‘action’ in policy networks or subsystems, showing that some issues involve intensely politicized disputes containing many actors while others are treated as technical and processed routinely, largely by policy specialists, out of the public spotlight.

The ACF suggests that people get into politics to turn their beliefs into policy, form coalitions with people who share their beliefs, and compete with coalitions of actors who share different beliefs. This competition takes place in a policy subsystem, in which coalitions understand new evidence through the lens of their beliefs, and exercise power to make sure that their interpretation is accepted. The other boxes describe the factors – the ‘parameters’ likely to be stable during the 10-year period of study, the partial sources of potential ‘shocks’ to the subsystem, and the need and ability of key actors to form consensus for policy change (particularly in political systems with PR elections) – which constrain and facilitate coalition action.

5. What do we need from a new image?

I recommend an image that consolidates or synthesises existing knowledge and insights. It is tempting to produce something that purports to be ‘new’ but, as with ‘new’ concepts or ‘new’ policy theories, how could we accumulate insights if everyone simply declared novelty and rejected the science of the past?

For me, the novelty should be in the presentation of the image, to help people pick up and use a wealth of policy studies which try to capture two key dynamics:

  1. Policy choice despite uncertainty and ambiguity.

Policymakers can only pay attention to a tiny proportion of issues. They use ‘rational’ and ‘irrational’ cognitive shortcuts to make decisions quickly, despite their limited knowledge of the world, and the possibility to understand policy problems from many perspectives.

  1. A policy environment which constrains and facilitates choice.

Such environments are made up of:

  1. Actors (individuals and organisations) influencing policy at many levels and types of government
  2. Institutions: a proliferation of rules and norms followed by different levels or types of government
  3. Networks: relationships between policymakers and influencers
  4. Ideas: a tendency for certain beliefs or ‘paradigms’ to dominate discussion
  5. Context and events: economic, social, demographic, and technological conditions provide the context for policy choice, and routine/ unpredictable events can prompt policymaker attention to lurch at short notice.

The implications of both dynamics are fairly easy to describe in tables (for example, while describing MSA) and to cobble together quickly in a SmartArt picture:

Cairney 2017 image of the policy process

However, note at least three issues with such a visual presentation:

  1. Do we put policymakers and choice at the centre? If so, it could suggest (a bit like the policy cycle) that a small number of key actors are at the ‘centre’ of the process, when we might prefer to show that their environment, or the interaction between many actors, is more important.
  2. Do we show only the policy process or relate it to the ‘outside world’?
  3. There are many overlaps between concepts. For example, we seek to describe the use and reproduction of rules in ‘institutions’ and ‘networks’, while those rules relate strongly to ‘ideas’. Further, ‘networks’ could sum up ‘actors interacting in many levels/ types of government’. So, ideally, we’d have overlapping shapes to denote overlapping relationships and understandings, but it would really mess up the simplicity of the image.

Of course, the bigger issue is that the image I provide is really just a vehicle to put text on a screen (in the hope that it will be shared). At best it says ‘note these concepts’. It does not show causal relationships. It does not describe any substantial interaction between the concepts to show cause and effect (such as, event A prompted policy choice B).

However, if we tried to bring in that level of detail, I think we would quickly end up with the messy process already described in relation to the policy cycle. Or, we would need to provide a more specific and less generally applicable model of policymaking.

So, right now, this image is a statement of intent. I want to produce something better, but don’t yet know what ‘better’ looks like. There is no ‘general theory’ of policymaking, so can we have a general image? Or, like ‘what is policy?’ discussions, do we produce an answer largely to raise more questions?

___

Here I am, looking remarkably pleased with my SmartArt skills

 

5 Comments

Filed under public policy

Practical Lessons from Policy Theories

These links to blog posts (the underlined headings) and tweets (with links to their full article) describe a new special issue of Policy and Politics, published in April 2018 and free to access until the end of May.

Weible Cairney abstract

Three habits of successful policy entrepreneurs

Telling stories that shape public policy

How to design ‘maps’ for policymakers relying on their ‘internal compass’

Three ways to encourage policy learning

How can governments better collaborate to address complex problems?

How do we get governments to make better decisions?

How to navigate complex policy designs

Why advocacy coalitions matter and how to think about them

None of these abstract theories provide a ‘blueprint’ for action (they were designed primarily to examine the policy process scientifically). Instead, they offer one simple insight: you’ll save a lot of energy if you engage with the policy process that exists, not the one you want to see.

Then, they describe variations on the same themes, including:

  1. There are profound limits to the power of individual policymakers: they can only process so much information, have to ignore almost all issues, and therefore tend to share policymaking with many other actors.
  2. You can increase your chances of success if you work with that insight: identify the right policymakers, the ‘venues’ in which they operate, and the ‘rules of the game’ in each venue; build networks and form coalitions to engage in those venues; shape agendas by framing problems and telling good stories, design politically feasible solutions, and learn how to exploit ‘windows of opportunity’ for their selection.

Background to the special issue

Chris Weible and I asked a group of policy theory experts to describe the ‘state of the art’ in their field and the practical lessons that they offer.

Our next presentation was at the ECPR in Oslo:

The final articles in this series are now complete, but our introduction discusses the potential for more useful contributions

Weible Cairney next steps pic

19 Comments

Filed under agenda setting, Evidence Based Policymaking (EBPM), public policy