Debating the politics of evidence-based policy

My new essay on evidence-based policy is now out in the current issue of Public Administration: https://t.co/6qabMyvYa6. Nothing personal, @CairneyPaul @justinparkhurst @ProfStoker @MarkEvansACT… hope to continue the debate!

— Joshua Newman (@_joshuanewman) December 20, 2017

Joshua Newman has provided an interesting review of three recent books on evidence/ policy (click here). One of those books is mine: The Politics of Evidence-Based Policy Making (which you can access here).

His review is very polite, for which I thank him. I hope my brief response can be seen in a similarly positive light (well, I had hoped to make it brief). Maybe we disagree on one or two things, but often these discussions are about the things we emphasize and the way we describe similar points.

There are 5 points to which I respond because I have 5 digits on my right hand. I’d like you to think of me counting them out on my fingers. In doing so, I’ll use ‘Newman’ throughout, because that’s the academic convention, but I’d also like to you imagine me reading my points aloud and whispering ‘Joshua’ before each ‘Newman’.

1. Do we really need to ‘take the debate forward’ so often?

I use this phrase myself, knowingly, to keep a discussion catchy, but I think it’s often misleading. I suggest not to get your hopes up too high when Newman raises the possibility of taking the debate forward with his concluding questions. We won’t resolve the relationship between evidence, politics & policy by pretending to reframe the same collection of questions about the prospect of political reform that people have been asking for centuries. It is useful to envisage better political systems (the subject of Newman’s concluding remarks) but I don’t think we should pretend that this is a new concern or that it will get us very far.

Indeed, my usual argument is that researchers need to do something (such as improve how we engage in the policy process) while we wait for political system reforms to happen (while doubting if they will ever happen).

Further, Newman does not produce any political reforms to address the problems he raises. Rather, for example, he draws attention to Trump to describe modern democracies as ‘not pluralist utopias’ and to identify examples in which policymakers draw primarily on beliefs, not evidence. By restating these problems, he does not solve them. So, what are researchers supposed to do after they grow tired of complaining that the world does not meet their hopes or expectations?

In other words, for me, (a) promoting political change and (b) acting during its absence are two sides of the same coin. We go round and round more often than we take things forward.

2. What debate are we renaming?

Newman’s ‘we’ve heard it before’ argument seems more useful, but there is a lot to hear and relatively few people have heard it. I’d warn against the assumption that ‘I’ve heard this before’ can ever equal ‘we’ve heard it before’ (unless ‘we’ refers to a tiny group of specialists talking only to each other).

Rather, one of the most important things we can do as academics is to tell the same story to each other (to check if we understand the same story, in the same way, and if it remains useful) and to wider audiences (in a way that they can pick up and use without dedicating their career to our discipline).

Some of our most important insights endure for decades and they sometimes improve in the retelling. We apply them to new eras, and often come to the same basic conclusions, but it seems unhelpful to criticise a lack of complete novelty in individual texts (particularly when they are often designed to be syntheses). Why not use them to occasionally take a step back to discuss and clarify what we know?

Perhaps more importantly, I don’t think Newman is correct when he says that each book retells the story of the ‘research utilization’ literature. I’m retelling the story of policy theory, which describes how policymakers deal with bounded rationality in a complex policymaking environment. Policy theory’s intellectual histories often provide very different perspectives – of the policymaker trying to make good enough decisions, rather than the researcher trying to improve the uptake of their research – than the agenda inspired by Weiss et al (see for example The New Policy Sciences).

3. Don’t just ‘get political’; understand the policy process

I draw on policy theory because it helps people understand policymaking. It would be a mistake to conclude from my book that I simply want researchers to ‘get political’. Rather, I want them to develop useful knowledge of the policy process in which they might want to engage. This knowledge is not freely available; it takes time to understand the discipline and reflect on policy dynamics.

Yet, the payoff can be profound, if only because it helps people think about the difference between two analytically separate causes of a notional ‘evidence policy gap’: (a) individuals making choices based on their beliefs and limited information (which is relatively easy to understand but also to caricature), and (b) systemic or ‘environmental’ causes (which are far more difficult to conceptualise and explain, but often more useful to understand).

4. Don’t throw out the ‘two communities’ phrase without explaining why

Newman criticises the phrase ‘two communities’ as a description of silos in policymaking versus research, partly because (a) many policymakers use research frequently, and (b) the real divide is often between users/ non-users of research within policymaking organisations. In short, there are more than two communities.

I’d back up his published research with my anecdotal experience of giving talks to government audiences: researchers and analysts within government are often very similar in outlook to academics and they often talk in the same way as academics about the disconnect between their (original or synthetic) research and its use by their ‘operational’ colleagues.

Still, I’m not sure why Newman concludes that the ‘two communities’ phrase is ‘deeply flawed and probably counter-productive’. Yes, the world is more nuanced and less binary than ‘two communities’ suggests. Yes, the real divide may be harder to spot. Still, as Newman et al suggest: ‘Policy makers and academics should focus on bridging instruments that can bring their worlds closer together’. This bullet point from their article seems, to me, to be the point of using the phrase ‘two communities’. Maybe Caplan used the phrase differently in 1979, but to assert its historic meaning then reject the phrase’s use in modern discussion seems less useful than simply clarifying the argument in ways such as:

• There is no simple policymaker/ academic divide but, … note the major difference in requirements between (a) people who produce or distribute research without taking action, which allows them (for example) to be more comfortable with uncertainty, and (b) people who need to make choices despite having incomplete information to hand.
• You might find a more receptive audience in one part of government (e.g. research/ analytical) than another (e.g. operational), so be careful about generalising from singular experiences.

5. Should researchers engage in the policy process?

Newman says that each book, ‘unfairly places the burden of resolving the problem in the hands of an ill-equipped group of academics, operating outside the political system’.

I agree with Newman when he says that many researchers do not possess the skills to engage effectively in the policy process. Scientific training does not equip us with political skills. Indeed, I think you could read a few of my blog posts and conclude, reasonably, that you would like nothing more to do with the policy process because you’d be more effective by focusing on research.

The reason I put the onus back on researchers is because I am engaging with arguments like the one expressed by Newman (in other words, part of the meaning comes from the audience). Many people conclude their evidence policy discussions by identifying (or ‘reframing’) the problem primarily as the need for political reform. For me, the focus on other people changing to suit your preferences seems unrealistic and misplaced. In that context, I present the counter-argument that it may be better to adapt effectively to the policy process that exists, not the one you’d like to see. Sometimes it’s more useful to wear a coat than complain about the weather.

See also:  The Politics of Evidence 

The Politics of Evidence revisited

 

What do we need to know about the politics of evidence-based policymaking?

Today, I’m helping to deliver a new course – Engaging Policymakers Training Programme – piloted by the Alliance for Useful Evidence and the UCL. Right now, it’s for UCL staff (and mostly early career researchers). My bit is about how we can better understand the policy process so that we can engage in it more effectively.  I have reproduced the brief guide below (for my two 2-hour sessions as part of a wider block). If anyone else is delivering something similar, please let me know. We could compare notes. 

This module will be delivered in two parts to combine theory and practice

Part 1: What do we need to know about the politics of evidence-based policymaking?

Policy theories provide a wealth of knowledge about the role of evidence in policymaking systems. They prompt us to understand and respond to two key dynamics:

1. Policymaker psychology. Policymakers combine rational and irrational shortcuts to gather information and make good enough decisions quickly. To appeal to rational shortcuts and minimise cognitive load, we reduce uncertainty by providing syntheses of the available evidence. To appeal to irrational shortcuts and engage emotional interest, we reduce ambiguity by telling stories or framing problems in specific ways.
2. Complex policymaking environments. These processes take place in the context of a policy environment out of the control of individual policymakers. Environments consist of: many actors in many levels and types of government; engaging with institutions and networks, each with their own informal and formal rules; responding to socioeconomic conditions and events; and, learning how to engage with dominant ideas or beliefs about the nature of the policy problem. In other words, there is no policy cycle or obvious stage in which to get involved.

In this seminar, we discuss how to respond effectively to these dynamics. We focus on unresolved issues:

1. Effective engagement with policymakers requires storytelling skills, but do we possess them?
2. It requires a combination of evidence and emotional appeals, but is it ethical to do more than describe the evidence?
3. The absence of a policy cycle, and presence of an ever-shifting context, requires us to engage for the long term, to form alliances, learn the rules, and build up trust in the messenger. However, do we have and how should we invest the time?

The format will be relatively informal. Cairney will begin by making some introductory points (not a powerpoint driven lecture) and encourage participants to relate the three questions to their research and engagement experience.

Gateway to further reading:

• Paul Cairney and Richard Kwiatkowski (2017) ‘How to communicate effectively with policymakers: combine insights from psychology and policy studies’, Palgrave Communications
• Paul Cairney and Kathryn Oliver (2017) ‘Evidence-based policymaking is not like evidence-based medicine, so how far should you go to bridge the divide between evidence and policy?’ Health Research Policy and Systems (HARPS), DOI: 10.1186/s12961-017-0192-x
• Paul Cairney, Kathryn Oliver, and Adam Wellstead (2016) ‘To Bridge the Divide between Evidence and Policy: Reduce Ambiguity as Much as Uncertainty’, Public Administration Review, Early View (forthcoming) DOI:10.1111/puar.12555 PDF

Part 2: How can we respond pragmatically and effectively to the politics of EBPM?

In this seminar, we move from abstract theory and general advice to concrete examples and specific strategies. Each participant should come prepared to speak about their research and present a theoretically informed policy analysis in 3 minutes (without the aid of powerpoint). Their analysis should address:

1. What policy problem does my research highlight?
2. What are the most technically and politically feasible solutions?
3. How should I engage in the policy process to highlight these problems and solutions?

After each presentation, each participant should be prepared to ask questions about the problem raised and the strategy to engage. Finally, to encourage learning, we will reflect on the memorability and impact of presentations.

• See: How to write theory-driven policy analysis

Powerpoint: Paul Cairney A4UE UCL 2017

How to write theory-driven policy analysis

Writing theory driven policy analysis 10.11.17

(or right click to download this lecture which accompanies my MPP)

Here is a guide to writing theory-driven policy analysis. Your aim is to identify a policy problem and solution, know your audience, and account for the complexity of policymaking.

At first, it may seem like daunting task to put together policy analysis and policy theory. On its own, policy analysis seems difficult but relatively straightforward: use evidence to identify and measure a policy problem, compare the merits of one or more solution, and make a recommendation on the steps to take us from policy to action.

However, policy process research tells us that people will engage emotionally with that evidence, and that policymakers operate in a complex system of which they have very limited knowledge and control.

So, how can we produce a policy analysis paper to which people will pay attention, and respond positively and effectively, under such circumstances? I focus on developing the critical analysis that will help you produce effective and feasible analysis. To do so, I show how policy analysis forms part of a collection of exercises to foster analysis informed by theory and reflection.

Aims of this document:

1. Describe the context. There are two fields of study – theory and analysis – which do not always speak to each other. Theory can inform analysis, but it is not always clear how. I show the payoff to theory-driven policy analysis and the difference between it and regular analysis. Note the two key factors that policy analysis should address: your audience will engage emotionally with your analysis, and the feasibility of your solutions depends on the complexity of the policy environment.
2. Describe how the coursework helps you combine policy theory and policy analysis. Policy analysis is one of four tasks. There is a reflection, to let you ‘show your work’; how your knowledge of policy theory guides your description of a problem and feasible solutions. The essay allows you to expand on theory, to describe how and why policy changes (and therefore what a realistic policy analysis would look like). The blogs encourage new communication skills. In one, you explore how you would expect a policy maker or influencer to sell the recommendations in your policy analysis. In another, you explain complex concepts to a non-academic audience.

Background notes.

I have written this document as if part of a book to be called Teaching Public Policy and co-authored with Dr Emily St Denny.

For that audience, I have two aims: (1) to persuade policy scholars-as-teachers to adopt this kind of coursework in their curriculum; and, (2) to show students how to complete it effectively.

If you prefer shorter advice, see Writing a policy paper and blog post and Writing an essay on politics, policymaking, and policy change.

If you are interested in more background reading, see: The New Policy Sciences (by Paul Cairney and Chris Weible) which describes the need to combine policy theory-driven research with policy analysis; and, Practical Lessons from Policy Theories which describes eight attempts by scholars to translate policy theory into lessons that can be used for policy analysis.

The theories make more sense if you have read the corresponding 1000 Words posts (based on Cairney, 2012). Some of the forthcoming text will look familiar if you read my blog because I am consolidating several individual posts into an overall discussion.

I’m not quite there yet (the chapter is a first draft, a bit scrappy at times, and longer than a chapter should be), so all comments welcome (in the comments bit).

Writing theory driven policy analysis 10.11.17

A 5-step strategy to make evidence count

Let’s imagine a heroic researcher, producing the best evidence and fearlessly ‘speaking truth to power’. Then, let’s place this person in four scenarios, each of which combines a discussion of evidence, policy, and politics in different ways.

1. Imagine your hero presents to HM Treasury an evidence-based report concluding that a unitary UK state would be far more efficient than a union state guaranteeing Scottish devolution. The evidence is top quality and the reasoning is sound, but the research question is ridiculous. The result of political deliberation and electoral choice suggests that your hero is asking a research question that does not deserve to be funded in the current political climate. Your hero is a clown.
2. Imagine your hero presents to the Department of Health a report based on the systematic review of multiple randomised control trials. It recommends that you roll out an almost-identical early years or public health intervention across the whole country. We need high ‘fidelity’ to the model to ensure the correct ‘dosage’ and to measure its effect scientifically. The evidence is of the highest quality, but the research question is not quite right. The government has decided to devolve this responsibility to local public bodies and/ or encourage the co-production of public service design by local public bodies, communities, and service users. So, to focus narrowly on fidelity would be to ignore political choices (perhaps backed by different evidence) about how best to govern. If you don’t know the politics involved, you will ask the wrong questions or provide evidence with unclear relevance. Your hero is either a fool, naïve to the dynamics of governance, or a villain willing to ignore governance principles.        
3. Imagine two fundamentally different – but equally heroic – professions with their own ideas about evidence. One favours a hierarchy of evidence in which RCTs and their systematic review is at the top, and service user and practitioner feedback is near the bottom. The other rejects this hierarchy completely, identifying the unique, complex relationship between practitioner and service user which requires high discretion to make choices in situations that will differ each time. Trying to resolve a debate between them with reference to ‘the evidence’ makes no sense. This is about a conflict between two heroes with opposing beliefs and preferences that can only be resolved through compromise or political choice. This is, oh I don’t know, Batman v Superman, saved by Wonder Woman.
4. Imagine you want the evidence on hydraulic fracturing for shale oil and gas. We know that ‘the evidence’ follows the question: how much can we extract? How much revenue will it produce? Is it safe, from an engineering point of view? Is it safe, from a public health point of view? What will be its impact on climate change? What proportion of the public supports it? What proportion of the electorate supports it? Who will win and lose from the decision? It would be naïve to think that there is some kind of neutral way to produce an evidence-based analysis of such issues. The commissioning and integration of evidence has to be political. To pretend otherwise is a political strategy. Your hero may be another person’s villain.

Now, let’s use these scenarios to produce a 5-step way to ‘make evidence count’.

Step 1. Respect the positive role of politics

A narrow focus on making the supply of evidence count, via ‘evidence-based policymaking’, will always be dispiriting because it ignores politics or treats political choice as an inconvenience. If we:

• begin with a focus on why we need political systems to make authoritative choices between conflicting preferences, and take governance principles seriously, we can
• identify the demand for evidence in that context, then be more strategic and pragmatic about making evidence count, and
• be less dispirited about the outcome.

In other words, think about the positive and necessary role of democratic politics before bemoaning post-truth politics and policy-based-evidence-making.

Step 2. Reject simple models of evidence-based policymaking

Policy is not made in a cycle containing a linear series of separate stages and we won’t ‘make evidence count’ by using it to inform our practices.

You might not want to give up the cycle image because it presents a simple account of how you should make policy. It suggests that we elect policymakers then: identify their aims, identify policies to achieve those aims, select a policy measure, ensure that the selection is legitimised by the population or its legislature, identify the necessary resources, implement and then evaluate. Or, policymakers aided by expert policy analysts make and legitimise choices, skilful public servants carry them out, and, policy analysts assess the results using evidence.

One compromise is to keep the cycle then show how messy it is in practice:

Discussing the challenges of evidence based policy including complexity of the landscape @OECD #evidence4gov pic.twitter.com/1yu3UIKKjO

— Rob Davies (@r0bdavies) June 26, 2017

However, there comes a point when there is too much mess, and the image no longer helps you explain (a) to the public what you are doing, or (b) to providers of evidence how they should engage in political systems. By this point, simple messages from more complicated policy theories may be more useful.

Or, we may no longer want a cycle to symbolise a single source of policymaking authority. In a multi-level system, with many ‘centres’ possessing their own sources of legitimate authority, a single and simple policy cycle seems too artificial to be useful.

Step 3. Tell a simple story about your evidence

People are ‘cognitive misers’ seeking ‘rational’ and ‘irrational’ shortcuts to gather information for action, so you won’t get far if you bombard them with too much evidence. Policymakers already have too much evidence and they seek ways to reduce their cognitive load, relying on: (a) trusted sources of concise evidence relevant to their aims, and (b) their own experience, gut instinct, beliefs, and emotions.

The implication of both shortcuts is that we need to tell simple and persuasive stories about the substance and implications of the evidence we present. To say that ‘the evidence does not speak for itself’ may seem trite, but I’ve met too many people who assume naively that it will somehow ‘win the day’. In contrast, civil servants know that the evidence-informed advice they give to ministers needs to relate to the story that government ministers tell to the public.

Step 4.  Tailor your story to many audiences

In a complex or multi-level environment, one story to one audience (such as a minister) is not enough. If there are many key sources of policymaking authority – including public bodies with high autonomy, organisations and practitioners with the discretion to deliver services, and service users involved in designing services – there are many stories being told about what we should be doing and why. We may convince one audience and alienate (or fail to inspire) another with the same story.

Step 5. Clarify and address key dilemmas with political choice, not evidence

Let me give you one example of the dilemmas that must arise when you combine evidence and politics to produce policy: how do you produce a model of ‘evidence based best practice’ which combines evidence and governance principles in a consistent way? Here are 3 ideal-type models which answer the question in very different ways

The table helps us think through the tensions between models, built on very different principles of good evidence and governance.

In practice, you may want to combine different elements, perhaps while arguing that the loss of consistency is lower than the gain from flexibility. Or, the dynamics of political systems limit such choice or prompt ad hoc and inconsistent choices.

I built a lot of this analysis on the experiences of the Scottish Government, which juggles all three models, including a key focus on improvement method in its Early Years Collaborative.

However, Kathryn Oliver and I show that the UK government faces the same basic dilemma and addresses it in similar ways.

The example freshest in my mind is Sure Start. Its rationale was built on RCT evidence and systematic review. However, its roll-out was built more on local flexibility and service design than insistence on fidelity to a model. More recently, the Troubled Families programme initially set the policy agenda and criteria for inclusion, but increasingly invites local public bodies to select the most appropriate interventions, aided by the Early Intervention Foundation which reviews the evidence but does not insist on one-best-way. Emily St Denny and I explore these issues further in our forthcoming book on prevention policy, an exemplar case study of a field in which it is difficult to know how to ‘make evidence count’.

If you prefer a 3-step take home message:

1. I think we use phrases like ‘impact’ and ‘make evidence count’ to reflect a vague and general worry about a decline in respect for evidence and experts. Certainly, when I go to large conferences of scientists, they usually tell a story about ‘post-truth’ politics.
2. Usually, these stories do not acknowledge the difference between two different explanations for an evidence-policy gap: (a) pathological policymaking and corrupt politicians, versus (b) complex policymaking and politicians having to make choices despite uncertainty.
3. To produce evidence with ‘impact’, and know how to ‘make evidence count’, we need to understand the policy process and the demand for evidence within it.

*Background. This is a post for my talk at the Government Economic Service and Government Social Research Service Annual Training Conference (15th September 2017). This year’s theme is ‘Impact and Future-Proofing: Making Evidence Count’. My brief is to discuss evidence use in the Scottish Government, but it faces the same basic question as the UK Government: how do you combine principles of evidence quality and governance principles? In other words, if you were in a position to design an (a) evidence-gathering system and (b) a political system, you’d soon find major points of tension between them. Resolving those tensions involves political choice, not more evidence. Of course, you are not in a position to design both systems, so the more complicated question is: how do you satisfy principles of evidence and governance in a complex policy process, often driven by policymaker psychology, over which you have little control?  Here are 7 different ‘answers’.

Powerpoint Paul Cairney @ GES GSRS 2017

Policy Concepts in 500 Words: Social Construction and Policy Design

Why would a democratic political system produce ‘degenerative’ policy that undermines democracy? Social Construction and Policy Design (SCPD) describes two main ways in which policymaking alienates many citizens:

1. The Social Construction of Target Populations

High profile politics and electoral competition can cause alienation:

1. Political actors compete to tell ‘stories’ to assign praise or blame to groups of people. For example, politicians describe value judgements about who should be rewarded or punished by government. They base them on stereotypes of ‘target populations’, by (a) exploiting the ways in which many people think about groups, or (b) making emotional and superficial judgements, backed up with selective use of facts.
2. These judgements have a ‘feed-forward’ effect: they are reproduced in policies, practices, and institutions. Such ‘policy designs’ can endure for years or decades. The distribution of rewards and sanctions is cumulative and difficult to overcome.
3. Policy design has an impact on citizens, who participate in politics according to how they are characterised by government. Many know they will be treated badly; their engagement will be dispiriting.

Some groups have the power to challenge the way they are described by policymakers (and the media and public), and receive benefits behind the scenes despite their poor image. However, many people feel powerless, become disenchanted with politics, and do not engage in the democratic process.

SCTP depicts this dynamic with a 2-by-2 table in which target populations are described positively/ negatively and more or less able to respond:

2. Bureaucratic and expert politics

Most policy issues are not salient and politicised in this way. Yet, low salience can exacerbate problems of citizen exclusion. Policies dominated by bureaucratic interests often alienate citizens receiving services. Or a small elite dominates policymaking when there is high acceptance that (a) the best policy is ‘evidence based’, and (b) the evidence should come from experts.

Overall, SCPD describes a political system with major potential to diminish democracy, containing key actors (a) politicising issues to reward or punish populations or (b) depoliticising issues with reference to science and objectivity. In both cases, policy design is not informed by routine citizen participation.

Take home message for students: SCPD began as Schneider and Ingram’s description of the US political system’s failure to solve major problems including inequality, poverty, crime, racism, sexism, and effective universal healthcare and education. Think about how its key drivers apply elsewhere: (1) some people make and exploit quick and emotional judgements for political gain, and others refer to expertise to limit debate; (2) these judgements inform policy design; and, (3) policy design sends signals to citizens which can diminish or boost their incentive to engage in politics.

For more, see the 1000-word and 5000-word versions. The latter has a detailed guide to further reading.

 

 

 

 

Evidence based policymaking: 7 key themes

I looked back at my blog posts on the politics of ‘evidence based policymaking’ and found that I wrote quite a lot (particularly from 2016). Here is a list based on 7 key themes.

1. Use psychological insights to influence the use of evidence

My most-current concern. The same basic theme is that (a) people (including policymakers) are ‘cognitive misers’ seeking ‘rational’ and ‘irrational’ shortcuts to gather information for action, so you won’t get far if you (b) bombard them with information, or (c) call them idiots.

Three ways to communicate more effectively with policymakers (shows how to use psychological insights to promote evidence in policymaking)

Using psychological insights in politics: can we do it without calling our opponents mental, hysterical, or stupid? (yes)

The Psychology of Evidence Based Policymaking: Who Will Speak For the Evidence if it Doesn’t Speak for Itself? (older paper, linking studies of psychology with studies of EBPM)

Older posts on the same theme:

Is there any hope for evidence in emotional debates and chaotic government? (yes)

We are in danger of repeating the same mistakes if we bemoan low attention to ‘facts’

These complaints about ignoring science seem biased and naïve – and too easy to dismiss

How can we close the ‘cultural’ gap between the policymakers and scientists who ‘just don’t get it’?

2. How to use policy process insights to influence the use of evidence

I try to simplify key insights about the policy process to show to use evidence in it. One key message is to give up on the idea of an orderly policy process described by the policy cycle model. What should you do if a far more complicated process exists?

Why don’t policymakers listen to your evidence?

The Politics of Evidence Based Policymaking: 3 messages (3 ways to say that you should engage with the policy process that exists, not a mythical process that will never exist)

Three habits of successful policy entrepreneurs (shows how entrepreneurs are influential in politics)

Why doesn’t evidence win the day in policy and policymaking? and What does it take to turn scientific evidence into policy? Lessons for illegal drugs from tobacco and There is no blueprint for evidence-based policy, so what do you do? (3 posts describing the conditions that must be met for evidence to ‘win the day’)

Writing for Impact: what you need to know, and 5 ways to know it (explains how our knowledge of the policy process helps communicate to policymakers)

How can political actors take into account the limitations of evidence-based policy-making? 5 key points (presentation to European Parliament-European University Institute ‘Policy Roundtable’ 2016)

Evidence Based Policy Making: 5 things you need to know and do (presentation to Open Society Foundations New York 2016)

What 10 questions should we put to evidence for policy experts? (part of a series of videos produced by the European Commission)

3. How to combine principles on ‘good evidence’, ‘good governance’, and ‘good practice’

My argument here is that EBPM is about deciding at the same time what is: (1) good evidence, and (2) a good way to make and deliver policy. If you just focus on one at a time – or consider one while ignoring the other – you cannot produce a defendable way to promote evidence-informed policy delivery.

Kathryn Oliver and I have just published an article on the relationship between evidence and policy (summary of and link to our article on this very topic)

We all want ‘evidence based policy making’ but how do we do it? (presentation to the Scottish Government on 2016)

The ‘Scottish Approach to Policy Making’: Implications for Public Service Delivery

The politics of evidence-based best practice: 4 messages

The politics of implementing evidence-based policies

Policy Concepts in 1000 Words: the intersection between evidence and policy transfer

Key issues in evidence-based policymaking: comparability, control, and centralisation

The politics of evidence and randomised control trials: the symbolic importance of family nurse partnerships

What Works (in a complex policymaking system)?

How Far Should You Go to Make Sure a Policy is Delivered?

4. Face up to your need to make profound choices to pursue EBPM

These posts have arisen largely from my attendance at academic-practitioner conferences on evidence and policy. Many participants tell the same story about the primacy of scientific evidence challenged by post-truth politics and emotional policymakers. I don’t find this argument convincing or useful. So, in many posts, I challenge these participants to think about more pragmatic ways to sum up and do something effective about their predicament.

Political science improves our understanding of evidence-based policymaking, but does it produce better advice? (shows how our knowledge of policymaking clarifies dilemmas about engagement)

The role of ‘standards for evidence’ in ‘evidence informed policymaking’ (argues that a strict adherence to scientific principles may help you become a good researcher but not an effective policy influencer)

How far should you go to secure academic ‘impact’ in policymaking? From ‘honest brokers’ to ‘research purists’ and Machiavellian manipulators (you have to make profound ethical and strategic choices when seeking to maximise the use of evidence in policy)

Principles of science advice to government: key problems and feasible solutions (calling yourself an ‘honest broker’ while complaining about ‘post-truth politics’ is a cop out)

What sciences count in government science advice? (political science, obvs)

I know my audience, but does my other audience know I know my audience? (compares the often profoundly different ways in which scientists and political scientists understand and evaluate EBPM – this matters because, for example, we rarely discuss power in scientist-led debates)

Is Evidence-Based Policymaking the same as good policymaking? (no)

Idealism versus pragmatism in politics and policymaking: … evidence-based policymaking (how to decide between idealism and pragmatism when engaging in politics)

Realistic ‘realist’ reviews: why do you need them and what might they look like? (if you privilege impact you need to build policy relevance into systematic reviews)

‘Co-producing’ comparative policy research: how far should we go to secure policy impact? (describes ways to build evidence advocacy into research design)

The Politics of Evidence (review of – and link to – Justin Parkhurt’s book on the ‘good governance’ of evidence production and use)

5. For students and researchers wanting to read/ hear more

These posts are relatively theory-heavy, linking quite clearly to the academic study of public policy. Hopefully they provide a simple way into the policy literature which can, at times, be dense and jargony.

‘Evidence-based Policymaking’ and the Study of Public Policy

Policy Concepts in 1000 Words: ‘Evidence Based Policymaking’

Practical Lessons from Policy Theories (series of posts on the policy process, offering potential lessons for advocates of evidence use in policy)

Writing a policy paper and blog post 

12 things to know about studying public policy

Can you want evidence based policymaking if you don’t really know what it is? (defines each word in EBPM)

Can you separate the facts from your beliefs when making policy? (no, very no)

Policy Concepts in 1000 Words: Success and Failure (Evaluation) (using evidence to evaluate policy is inevitably political)

Policy Concepts in 1000 Words: Policy Transfer and Learning (so is learning from the experience of others)

Four obstacles to evidence based policymaking (EBPM)

What is ‘Complex Government’ and what can we do about it? (read about it)

How Can Policy Theory Have an Impact on Policy Making? (on translating policy theories into useful advice)

The role of evidence in UK policymaking after Brexit (argues that many challenges/ opportunities for evidence advocates will not change after Brexit)

Why is there more tobacco control policy than alcohol control policy in the UK? (it’s not just because there is more evidence of harm)

Evidence Based Policy Making: If You Want to Inject More Science into Policymaking You Need to Know the Science of Policymaking and The politics of evidence-based policymaking: focus on ambiguity as much as uncertainty and Revisiting the main ‘barriers’ between evidence and policy: focus on ambiguity, not uncertainty and The barriers to evidence based policymaking in environmental policy (early versions of what became the chapters of the book)

6. Using storytelling to promote evidence use

This is increasingly a big interest for me. Storytelling is key to the effective conduct and communication of scientific research. Let’s not pretend we’re objective people just stating the facts (which is the least convincing story of all). So far, so good, except to say that the evidence on the impact of stories (for policy change advocacy) is limited. The major complication is that (a) the story you want to tell and have people hear interacts with (b) the story that your audience members tell themselves.

Combine Good Evidence and Emotional Stories to Change the World

Storytelling for Policy Change: promise and problems

Is politics and policymaking about sharing evidence and facts or telling good stories? Two very silly examples from #SP16

7. The major difficulties in using evidence for policy to reduce inequalities

These posts show how policymakers think about how to combine (a) often-patchy evidence with (b) their beliefs and (c) an electoral imperative to produce policies on inequalities, prevention, and early intervention. I suggest that it’s better to understand and engage with this process than complain about policy-based-evidence from the side-lines. If you do the latter, policymakers will ignore you.

The UK government’s imaginative use of evidence to make policy 

What do you do when 20% of the population causes 80% of its problems? Possibly nothing.

The theory and practice of evidence-based policy transfer: can we learn how to reduce territorial inequalities?

We need better descriptions than ‘evidence-based policy’ and ‘policy-based evidence’: the case of UK government ‘troubled families’ policy

How can you tell the difference between policy-based-evidence and evidence-based-policymaking?

Early intervention policy, from ‘troubled families’ to ‘named persons’: problems with evidence and framing ‘valence’ issues

Key issues in evidence-based policymaking: comparability, control, and centralisation

The politics of evidence and randomised control trials: the symbolic importance of family nurse partnerships

Two myths about the politics of inequality in Scotland

Social investment, prevention and early intervention: a ‘window of opportunity’ for new ideas?

A ‘decisive shift to prevention’: how do we turn an idea into evidence based policy?

Can the Scottish Government pursue ‘prevention policy’ without independence?

Note: these issues are discussed in similar ways in many countries. One example that caught my eye today:

"Changing concepts of 'evidence'" visual summary @fletchermj @PowerToPersuade @PSResearchG #P2P2017 pic.twitter.com/6M9QgTXAyW

— Lisa Carson (@DrLisaCResearch) August 23, 2017

 

All of this discussion can be found under the EBPM category: https://paulcairney.wordpress.com/category/evidence-based-policymaking-ebpm/T

See also the special issue on maximizing the use of evidence in policy

The impact of multi-level policymaking on the UK energy system

In September, we will begin a one-year UKERC project examining current and future energy policy and multi-level policymaking and its impact on ‘energy systems’. This is no mean feat, since the meaning of policy, policymaking (or the ‘policy process’), and ‘system’ are not clear, and our description of the components parts of an energy system and a complex policymaking system may differ markedly. So, one initial aim is to provide some way to turn a complex field of study into something simple enough to understand and engage with.

We do so by focusing on ‘multi-level policymaking’ – which can encompass concepts such as multi-level governance and intergovernmental relations – to reflect the fact that the responsibility for policies relevant to energy are often Europeanised, devolved, and shared between several levels of government. Brexit will produce a major effect on energy and non-energy policies, and prompt the UK and devolved governments to produce relationships, but we all need more clarity on the dynamics of current arrangements before we can talk sensibly about the future. To that end, we pursue three main work packages:

1. What is the ‘energy policymaking system’ and how does it affect the energy system?

Chaudry et al (2009: iv) define the UK energy system as ‘the set of technologies, physical infrastructure, institutions, policies and practices located in and associated with the UK which enable energy services to be delivered to UK consumers’. UK policymaking can have a profound impact, and constitutional changes might produce policy change, but their impacts require careful attention. So, we ‘map’ the policy process and the effect of policy change on energy supply and demand. Mapping sounds fairly straightforward but contains a series of tasks whose level of difficulty rises each time:

1. Identify which level or type of government is responsible – ‘on paper’ and in practice – for the use of each relevant policy instrument.
2. Identify how these actors interact to produce what we call ‘policy’, which can range from statements of intent to final outcomes.
3. Identify an energy policy process containing many actors at many levels, the rules they follow, the networks they form, the ‘ideas’ that dominate discussion, and the conditions and events (often outside policymaker control) which constrain and facilitate action. By this stage, we need to draw on particular policy theories to identify key venues, such as subsystems, and specific collections of actors, such as advocacy coalitions, to produce a useful model of activity.

2. Who is responsible for action to reduce energy demand?

Energy demand is more challenging to policymakers than energy supply because the demand side involves millions of actors who, in the context of household energy use, also constitute the electorate. There are political tensions in making policies to reduce energy demand and carbon where this involves cost and inconvenience for private actors who do not necessarily value the societal returns achieved, and the political dynamics often differ from policy to regulate industrial demand. There are tensions around public perceptions of whose responsibility it is to take action – including local, devolved, national, or international government agencies – and governments look like they are trying to shift responsibility to each other or individuals and firms.

So, there is no end of ways in which energy demand could be regulated or influenced – including energy labelling and product/building standards, emissions reduction measures, promotion of efficient generation, and buildings performance measures – but it is an area of policy which is notoriously diffuse and lacking in co-ordination. So, for the large part, we consider if Brexit provides a ‘window of opportunity’ to change policy and policymaking by, for example, clarifying responsibilities and simplifying relationships.

3: Does Brexit affect UK and devolved policy on energy supply?

It is difficult for single governments to coordinate an overall energy mix to secure supply from many sources, and multi-level policymaking adds a further dimension to planning and cooperation. Yet, the effect of constitutional changes is highly uneven. For example, devolution has allowed Scotland to go its own way on renewable energy, nuclear power and fracking, but Brexit’s impact ranges from high to low. It presents new and sometimes salient challenges for cooperation to supply renewable energy but, while fracking and nuclear are often the most politically salient issues, Brexit may have relatively little impact on policymaking within the UK.

We explore the possibility that renewables policy may be most impacted by Brexit, while nuclear and fracking are examples in which Brexit may have a minimal direct impact on policy. Overall, the big debates are about the future energy mix, and how local, devolved, and UK governments balance the local environmental impacts of, and likely political opposition to, energy development against the economic and energy supply benefits.

For more details, see our 4-page summary

Powerpoint for 13.7.17

Cairney et al UKERC presentation 23.10.17

Policy Concepts in 1000 Words: The Policy Process

We talk a lot about ‘the policy process’ without really saying what it is. If you are new to policy studies, maybe you think that you’ll learn what it is eventually if you read enough material. This would be a mistake! Instead, when you seek a definition of the policy process, you’ll find two common responses:

1. Many will seek to define policy or public policy instead of ‘the policy process’.
2. Some will describe the policy process as a policy cycle with stages.

Both responses seem inadequate: one avoids giving an answer, and another gives the wrong answer!

However, we can combine elements of each approach to give you just enough of a sense of ‘the policy process’ to continue reading the full ‘1000 words’ series:

1. The beauty of the ‘what is policy?’ question …

… is that we don’t give you an answer. It may seem frustrating at first to fail to find a definitive answer, but eventually you’ll accept this problem! The more important outcome is to use the ‘what is policy?’ question to develop analytical skills, to allow you to define policy in more specific circumstances (such as, what are the key elements of policy in this case study?), and ask more useful and specific questions about policy and policymaking. So, look at the questions we need to ask if we begin with the definition, ‘the sum total of government action, from signals of intent to the final outcomes’: does action include statements of intent? Do we include unintended policy outcomes? Are all policymakers in government? What about the things policymakers choose not to do? And so on.

2. The beauty of the policy cycle approach …

… is that it provides a simple way to imagine policy ‘dynamics’, or events and choices producing a never-ending sequence of other events and choices. Look at the stages model to identify many different tasks within one ‘process’, and to get the sense that policymaking is continuous and often ‘its own cause’. It’s not a good description of what actually happens, but it describes what some might like to happen, and used by many governments to describe what they do. Consequently, we can’t simply ignore it, at least without providing a better description, a better plan, and a better way for governments to justify what they do.

There are more complicated but better ways of describing policymaking dynamics

This picture is the ‘policy process’ equivalent of my definition of public policy. It captures the main elements of the policy process described – albeit in different ways – by most policy theories in this series. I present it here to give you enough of an answer – to ‘what is the policy process?’ – to help you ask more questions.

In the middle is ‘policy choice’

At the heart of most policy theory is ‘bounded rationality’, which describes (a) the cognitive limits of all people, and (b) how policymakers overcome such limits to make decisions (in the absence of NZT). In short, they use ‘rational’ and ‘irrational’ shortcuts to action, but these are provocative terms to prompt further reading (on, for example, ‘evidence-based policymaking’).

‘Rational’ describes goal-oriented activity: people may have limits to their attention and ‘information processing’, but they find systematic ways to respond, by setting goals and producing criteria to find the best information. ‘Irrational’ describes aspects of psychology: people draw on habit, emotions, their ‘gut’ or intuition, well-established beliefs, and their familiarity with information to make often-almost-instant decisions.

Surrounding choice is what we’ll call the ‘policy environment’

Environment is a metaphor we’ll use to describe the combination of key elements of the policy process which (a) I describe separately in further 1000 words posts, and (b) policy theories bring together to produce an overall picture of policy dynamics.

There are 5 or 6 key elements. In the picture are 6, reflecting the way Tanya Heikkila and I describe it (and the fact that I had 7 boxes to fill). In real life, I describe 5 because I have 5 digits on each hand. If you are Count Tyrone Rugen you have more choice.

Policy environments are made up of:

1. A wide range of actors (which can be individuals and organisations with the ability to deliberate and act) making or influencing policy at many levels and types of government.
2. Institutions, defined as the rules followed by actors. Some are formal, written down, and easy to identify. Others are informal, reproduced via processes like socialisation, and difficult to spot and describe.
3. Networks, or the relationships between policymakers and influencers. Some are wide open, competitive, and contain many actors. Others are relatively closed, insulated from external attention, and contain few actors.
4. Ideas, or the beliefs held and shared by actors. There is often a tendency for certain beliefs or ‘paradigms’ to dominate discussion, constraining or facilitating the progress of new ‘ideas’ as policy solutions.
5. Context and events. Context describes the policy conditions – including economic, social, demographic, and technological factors – that provide the context for policy choice, and are often outside of the control of policymakers. Events can be routine and predictable, or unpredictable ‘focusing’ events that prompt policymaker attention to lurch at short notice.

This picture is only the beginning of analysis, raising further questions that will make more sense when you read further, including: should policymaker choice be at the centre of this picture? Why are there arrows in the cycle but not in my picture? Should we describe complex policymaking ‘systems’ rather than ‘environments’? How exactly does each element in the ‘policy environment’ or ‘system’ relate to the other?

The answer to the final question can only be found in each theory of the policy process, and each theory describes this relationship in a different way. Let’s not worry about that just now! We’ll return to this issue at the end, when thinking about how to combine the insights of many theories.

 

 

 

Policy in 500 Words: The Policy Process

We talk a lot about ‘the policy process’ without really saying what it is. If you are new to policy studies, maybe you think that you’ll learn what it is eventually if you read enough material. This would be a mistake! Instead, when you seek a definition of the policy process, you’ll find two common responses.

1. Many will seek to define policy or public policy instead of ‘the policy process’.
2. Some will describe the policy process as a policy cycle with stages.

Both responses seem inadequate: one avoids giving an answer, and another gives the wrong answer!

However, we can combine elements of each approach to give you just enough of a sense of ‘the policy process’ to continue reading:

1. The beauty of the ‘what is policy?’ question is that we don’t give you an answer. I give you a working definition to help raise further questions. Look at the questions we need to ask if we begin with the definition, ‘the sum total of government action, from signals of intent to the final outcomes’.
2. The beauty of the policy cycle approach is that it provides a simple way to imagine policy ‘dynamics’, or events and choices producing a sequence of other events and choices. Look at the stages to identify many different tasks within one ‘process’, and to get the sense that policymaking is continuous and often ‘its own cause’.

There are more complicated but better ways of describing policymaking dynamics

This picture is the ‘policy process’ equivalent of my definition of public policy. It captures the main elements of the policy process described (in different ways) by most policy theories. It is there to give you enough of an answer to help you ask the right questions.

In the middle is ‘policy choice’. At the heart of most policy theory is ‘bounded rationality’, which describes (a) the cognitive limits of people, and (b) how they overcome those limits to make decisions. They use ‘rational’ and ‘irrational’ shortcuts to action.

Surrounding choice is what we’ll call the ‘policy environment’, containing: policymakers in many levels and types of government, the ideas or beliefs they share, the rules they follow, the networks they form with influencers, and the ‘structural’ or socioeconomic context in which they operate.

This picture is only the beginning of analysis, raising further questions that will make more sense when you read further, including: should policymaker choice be at the centre of this picture? Why are there arrows (describing the order of choice) in the cycle but not in my picture?

Take home message for students: don’t describe ‘the policy process’ without giving the reader some sense of its meaning. Its definition overlaps with ‘policy’ considerably, but the ‘process’ emphasises modes and dynamics of policymaking, while ‘policy’ emphasises outputs. Then, think about how each policy model or theory tries, in different ways, to capture the key elements of the process. A cycle focuses on ‘stages’ but most theories in this series focus on ‘environments’.