How far should you go to privilege evidence? 2. Policy theories, scenarios, and ethical dilemmas

If you have read Why don’t policymakers listen to your evidence? and What can you do when policymakers ignore your evidence? then join me as we get into the thornier dilemmas in this punchline post. Maybe you already appreciate the importance of bounded rationality and policymaking complexity. Maybe you’ve read the gazillion posts which think through the relationship between ‘evidence based policymaking’ and policy theories and now hope that there’s nothing more to say. Well, you hope in vain.

This post compares some general bland advice, based on these posts, to the dilemmas that you might encounter if you really take key parts of these theories to heart.

My first aim is to compare the ‘how to’ advice that you might take from policy theories versus the grey literature.

Policy concepts describe a wider context in which to produce practical advice:

• If there are so many potential authoritative venues, devote considerable energy to finding where the ‘action’ is.
• Even if you find the right venue, you will not know the unwritten rules unless you study them intensely.
• Some networks are close-knit and difficult to access because bureaucracies have operating procedures that favour some sources of evidence.
• Research advocates can be privileged insiders in some venues and excluded completely in others.
• If your evidence challenges an existing paradigm, you need a persuasion strategy good enough to prompt a shift of attention to a policy problem and a willingness to understand that problem in a new way.
• You can try to find the right time to use evidence to exploit a crisis leading to major policy change, but the opportunities are few and chances of success low.
• And so on.

In that context, theory-informed studies recommend investing your time over the long term – to built up alliances, trust in the messenger, knowledge of the system, and to seek ‘windows of opportunity’ for policy change – but offer no assurances that any of this investment will ever pay off.

The despair does not stop there. More specific theories and studies help us combine practical considerations with the ethical dilemmas that evidence advocates face when trying to be effective in a highly political policymaking environment.

First, refresh your memory of key images of the policy process. Or, if you are on a PC, you can keep the two tabs open for comparison.

Second, consider the following staircase analogy in which the ethical dilemmas – regarding how far you should go to get attention for your evidence – seems to become more problematic with each upwards step:

Step 1: Change levels of attention to issues, not minds.

The narrative policy framework (NPF) suggests that ‘narratives’ – consisting of a setting, characters, plot, and moral – can produce a measurable policy impact, but primarily to reinforce the beliefs of policy actors. The existing beliefs of the audience often seem more important than the skills of the storyteller. Therefore, to maximise the impact of evidence, (a) tell a story which appeals to the biases of your audiences, and (b) employ ‘heresthetic’ strategies in which we try to increase the salience of one belief at the expense of another rather than ask someone to change their belief entirely.

Step 2: Engage only with actors who share your beliefs.

The advocacy coalition framework (ACF) suggests that actors enter politics to turn their beliefs into policy. In highly salient issues, coalition actors romanticise their own cause and demonize their opponents. This competition extends to the use of evidence: each coalition may demand different evidence, or interpret the same evidence differently, to support their own cause. If so, the most feasible strategy may be to treat evidence as a resource to support the coalitions which support your cause, and to engage minimally with competitor coalitions who seek to ignore or discredit your evidence. Only in less salient issues will we find a ‘brokerage’ role for scientists.

Step 3: Exercise power to limit debate and dominate policymaker attention.

Punctuated equilibrium theory (PET) suggests that policy actors frame issues to limit external attention. If they can define a problem successfully as solved, bar the technical details relating to regulation and implementation, they can help reduce external attention and privilege the demand for evidence from scientific experts.

Step 4. Frame evidence to be consistent with objectionable beliefs.

Social construction and policy design theory (SCPD) suggests that, when dealing with salient issues, policymakers exploit social stereotypes strategically, or rely on their emotions, to define target populations as deserving of government benefits or punishments. Some populations can challenge (or exploit the rewards of) their image, but many are powerless to respond. Or, in lower salience issues, there is more scope for bureaucrats and experts to contain discussion to small groups (as in the discussion of PET). In both cases, many social groups become disenchanted with politics because they are punished by government policy and excluded from debate. To find an influential audience for evidence, one may be most effective by framing evidence to be sympathetic to stereotype-led or other forms of misleading political strategy.

The main role of these discussions is to expose the assumptions that we make about the primacy of research evidence and the lengths to which we are willing to go to privilege its use. Policy studies suggest that the most effective ways to privilege research evidence may be to:

• manipulate the order in which we consider issues and make choices
• refuse to engage in debate with our competitors
• frame issues to minimise attention or maximise the convergence between evidence and the rhetorical devices of cynical politicians.

However, they also expose stark ethical dilemmas regarding the consequences for democracy. Put simply, the most effective evidence advocacy strategies may be inconsistent with wider democratic principles and key initiatives such as participatory policymaking.

If so, these discussions prompt us to consider the ways in which we can value research evidence up to a certain point, to produce more ‘co-productive’ strategies which balance efforts to limit participation (to privilege expertise) and encourage it (to privilege deliberative and participatory forms of democracy). This approach is more honest and realistic than the more common story that science is, by its very nature, the antidote to populist or dysfunctional politics.

[If you came here in error, or to continue your adventure, go to page 100]

See also:

• How far should you go to privilege evidence? 3. Use psychological insights to manipulate policymakers (go to page 300 then scroll down to point 3)

EBPM key themes

Policy theories in 1000 words (or short podcasts) and 500 words

How far should you go to privilege evidence? 1. Evidence and governance principles

Read this discussion first, then have a look at this table. It sums up two debates which ‘play out at the same time: epistemological and methodological disagreements on the nature of good evidence; and, practical or ideological disagreements regarding the best way for national policymakers to translate evidence into local policy and practice’.

The table describes three ideal-type approaches to the need to combine two judgements on: (1) how best to produce and share evidence, and (2) the most appropriate way to balance central and local government responsibilities.

It provides a simple way to think about how far you would go to privilege the use of scientific evidence in policy.

For example, let’s say that you like the look of a hierarchy of evidence in which RCTs and their systematic review are at the top. Are you happy to minimise local autonomy (to co-produce and redesign public services) to ensure fidelity to the same policy intervention?

Or, let’s say that you favour governance principles which emphasise localism, respect for user-driven service design, and sharing practitioner experience. Are you happy to dispense with many scientific ways to evaluate policy success?

Or, let’s say that you want good evidence and respectful governance. Does the improvement method give you the best or worst of both worlds?

Reflecting on these choices, I suggest that:

‘A commitment to ‘scaling up evidence-based best practice’ seems like an innocuous and technical exercise that can be built primarily on academic expertise. However, we can identify the role of politics when we highlight the choices that scientists and governments make to gather evidence and ensure policy diffusion. These choices are not simply technical. Rather, they are between fundamentally different approaches to evidence and governance. Policymaking strategies can vary markedly, based on attitudes to hierarchies of evidence and the willingness of central governments to encourage local policymakers to learn and adapt rather than impose the same model’.

Dr Kathryn Oliver and I conclude that:

‘Evidence-based policymaking is not just about the need for policymakers to understand how evidence is produced and should be used. It is also about the need for academics to reflect on the assumptions they make about the best ways to gather evidence and put the results into practice, in a political environment where other people may not share, or even know about, their understanding of the world; and the difference between the identification of evidence on the success of an intervention, in one place and one point in time (or several such instances), and the political choice to roll it out, based on the assumption that national governments are best placed to spread that success throughout the country’

Put more simply, ‘good evidence’ for policy involves a two-sided political choice, on: (1) whose evidence claims should matter, and (2) how much government centralisation you need, to make sure that your knowledge claim counts.

Put more strongly, ignorance of these dilemmas is no excuse for making evidential choices without considering the consequences for governance.

See also:

• How far should you go to privilege evidence? 2. Policy theories, scenarios, and ethical dilemmas (go to page 526)
• How far should you go to privilege evidence? 3. Use psychological insights to manipulate policymakers (go to page 300 then scroll down to point 3)

The evidence policy gap: changing the research mindset is only the beginning

If scientists want to influence policymaking, they need to understand it

“Evidence-based best practice is more political than it looks: a case study of the ‘Scottish Approach’”

‘Evidence-based policymaking is not like evidence-based medicine, so how far should you go to bridge the divide between evidence and policy?’

The UK government’s imaginative use of evidence to make policy

The politics of evidence-based best practice: 4 messages

Policy Concepts in 1000 Words: the intersection between evidence and policy transfer

 

[If you came here in error, or to continue your adventure, go to page 100]

How else can we describe and seek to fill the evidence-policy gap?

There is a common story within science that describes a new era in politics: policymakers do not pay sufficient respect to expertise or attention to good quality evidence.

Common solutions are supply side, to produce better evidence and communicate it more effectively, and demand side, to reform how governments process evidence and improve science literacy among policymakers.

One part of this story emphasises wide academic agreement on how to assess evidence quality – with reference to scientific methods – and technical improvements to communication skills and organisational processes to close the evidence-policy gap. The other part emphasises policymakers irrationality or cynicism and political system dysfunction, producing ‘policy based evidence’ rather than ‘evidence based policy’.

Policy studies help us tell a less alarmist and more consistent story on the use of evidence:

• Policymakers have always combined ‘rational’ and ‘irrational’ cognitive shortcuts to evidence. No person or organisation has the ability to process all information, and cognitive efficiency is necessary to policy choice.
• It would be a mistake to assess evidence quality narrowly in relation to a hierarchy of research methods or exaggerate the extent to which academics adhere to this hierarchy.
• Policymakers identify problems and place certain demands on evidence: to help solve problems quickly, and maintain their image of governing competence, credibility, or political support. Policy relevance and availability are necessary gauges of quality, and an evidence-informed solution must be technically and politically feasible.

Conclusion 1. Reframe supply side solutions on production and communication. Evidence syntheses should combine an inclusive assessment of research quality and policy relevance. Communication should relate to the framing of evidence to influence the ways in which policymakers (a) define problems, and (b) describe the political feasibility of solutions.

• Too many commentators declare policymaking failure in (explicit or implicit) comparison to an ideal-type, such as the policy cycle, in which (a) a core group of policymakers can make choices and process them in a straightforward evidence-informed way, through a series of orderly stages, and therefore (b) researchers know how and when to present evidence.
• Policymaking is better understood as a complex environment or system over which the ‘centre’ has limited control. Policymakers must delegate most responsibilities to many other organisation and networks, and respond to socio-economic conditions and events out of their control. Consequently, the ‘action’ takes place in many different parts of the system, there are many different ‘rules of the game’, and policy often seems to ‘emerge’ locally without central direction. If we think of these dynamics as inevitable features of political systems, not dysfunctions to be solved, we can produce more pragmatic ways to encourage the use of evidence within them.

Conclusion 2. Reframe demand side solutions to take systemic factors into account. Interventions to improve evidence processing or literacy in a small part of government will be ineffective if they do not take into account policymaking scale, and key factors such as the division of policymaking responsibilities across systems, pervasiveness of subsystems, and the limited coordinative capacity of a political system’s ‘centre’.

In the next posts, we consider the practical and ethical dilemmas that arise when you seek the greater use of evidence in policy:

• How far should you go to privilege evidence? 1. Evidence and governance principles (go to page 101)
• How far should you go to privilege evidence? 2. Policy theories, scenarios, and ethical dilemmas (go to page 526)
• How far should you go to privilege evidence? 3. Use psychological insights to manipulate policymakers (go to page 300 then scroll down to point 3)

[If you came here in error, or to continue your adventure, go to page 100]

What can you do when policymakers ignore your evidence? Encourage ‘knowledge management for policy’

Knowledge management for policy impact by Lene Topp, David Mair, Laura Smillie (from the European Commission’s Joint Research Centre, JRC & Paul Cairney (not from the European Commission’s Joint Research Centre, JRC)

Put down that huge blue pen again. Step even further away from the flipchart. The JRC did all those exercises for you, and there are no more sticky circles or fluorescent post-its left.

In this article, we show the payoffs to a wide information-gathering exercise, from expert workshops to literature reviews, to produce advice on how to close the evidence-policy gap. We recommend that relevant organisations should develop teams of researchers, policymakers, and ‘knowledge brokers’ to produce eight key practices:

1. research synthesis, to generate ‘state of the art’ knowledge on a policy problem
2. management of expert communities, to maximise collaboration
3. understanding policymaking, to know when and how to present evidence
4. interpersonal skills, to focus on relationships and interaction
5. engagement, to include citizens and stakeholders
6. effective communication of knowledge
7. monitoring and evaluation, to identify the impact of evidence on policy
8. policy advice, to know how to present knowledge effectively and ethically.

We do so in colour:

We describe the problems that these 8 skills are designed to solve:

Then we scare you with a cautionary tale:

We conclude:

‘In recommending eight skills, we argue that ‘pure scientists’ and ‘professional politicians’ cannot do this job alone. Scientists need ‘knowledge brokers’ and science advisors with the skills to increase policymakers’ demand for evidence. Policymakers need help to understand and explain the evidence and its implications. Brokers are essential: scientists with a feel for policy and policymakers understanding how to manage science and scientists.

Many scientists are in a great position to move into this new profession, providing a more robust form of knowledge-based consultancy, built on a crucial understanding of scientific methods and evidence assessment. However, working between science and policy is hard to manoeuver, and the training we identify is more like a career choice than a quick fix. Science and policy worlds are interconnected, but not always compatible. Therefore, knowledge managers need to professionalise, to develop new skills, and work in teams with a comprehensive set of skills unlikely to be held by one person’.

It already sounds like a career choice, doesn’t it? Still, we need to think further:

• How far should you go to privilege evidence? 1. Evidence and governance principles (go to page 101)
• How far should you go to privilege evidence? 2. Policy theories, scenarios, and ethical dilemmas (go to page 526)
• How far should you go to privilege evidence? 3. Use psychological insights to manipulate policymakers (go to page 300 then scroll down to point 3)

[If you came here in error, or to continue your adventure, go to page 100]

What can you do when policymakers ignore your evidence? Tips from the ‘how to’ literature from the science community

Put down that huge blue pen. Step away from the flipchart.

There is now a large literature on the strategies that people might use to promote the use of evidence in policy, so you no longer have to start from first principles in your workshop.

Thanks to Dr Kathryn Oliver and colleagues, we now have systematic reviews of the peer reviewed academic evidence on the ‘barriers’ between evidence and policy (click here) and the ‘grey literature’, which includes newspaper editorials, blogs, and practitioner reports:

1. Kathryn Oliver and Paul Cairney (2019) ‘The Dos and Don’ts of influencing policy: a systematic review of advice to academics’, Palgrave Communications Open Access (PDF )
2. Paul Cairney and Kathryn Oliver (2018) ‘How should academics engage in policymaking to achieve impact?’ Political Studies Review  (PDF)

The advice from the peer reviewed literature can be summed up as follows:

1. Produce better quality evidence on policy problems and solutions.
2. Improve dissemination strategies to increase policymaker access to research: write more concise and less jargon-filled reports, boost resources for dissemination, and remove paywall obstacles to accessing research.
3. Develop relationships with policymakers, to address the unpredictability of politics, or the importance of timing, serendipity, and ‘windows of opportunity’ to act.
4. Engage directly, in academic-practitioner workshops, or use intermediaries such as ‘knowledge brokers’, to break down communications and cultural barriers associated with the different incentives, rhythms, and language of researcher and policymakers.
5. Encourage policymakers to be more science literate, to appreciate the role of evidence and ways to separate high- and low-quality sources

The grey literature is fairly similar.

Advice for researchers to promote the relevance and use of your research provided by @oliver_kathryn and @CairneyPaul based on a systematic review of the literature #impsci #impsci18 pic.twitter.com/mIXvbGdfkI

— EIC (@implementEIC) July 19, 2018

Kathryn and I summarise key themes and individual recommendations from 78 publications as follows:

1. Do high quality research.

Use specific well-established research designs, methods, or metrics.

2. Make your research relevant and readable.

Provide and disseminate easily-understandable, clear, relevant and high-quality research. Aim for the general but ‘not ignorant’ reader. Produce good stories based, for example, on emotional appeals or humour to expand your audience

3. Understand the policy process, policymaking context, and key actors.

Note the busy and constrained lives of policy actors Maximise your use of established ways to engage, such as in advisory committees. Be pragmatic about what ‘success’ looks like, accepting that research rarely translates into policy options directly

4. Be ‘accessible’ to policymakers: engage routinely, flexibly, and humbly.

As publicly-funded professionals, it is the job of academics to engage with policy and publics. Discuss topics beyond your narrow expertise, as a representative of your discipline or the science profession. Be humble, courteous, professional, and recognise the limits to your skills when giving policy advice. Respect policymakers’ time and expertise.

5. Decide if you want to be an ‘issue advocate’ or ‘honest broker’

There is a commonly-cited ethical dilemma about whether to go beyond providing evidence to recommend specific policy options or remain an ‘honest broker’ explaining the options. If making recommendations, use storytelling to persuade policymakers of a course of action. However, note the consequences of becoming a political actor. David Nutt famously lost his advisory role after publicly criticising government drugs policy, some describe the loss of one’s safety if adopting an activist mindset, and anecdotal conversations describe the risk of losing credibility in government if seen as too evangelical while giving policy advice. However, more common consequences include criticism within one’s peer-group, being seen as an academic ‘lightweight, being used to add legitimacy to a policy position, and the risk of burnout.

6. Build relationships (and ground rules) with policymakers.

Relationship-building activities require major investment and skills, but working collaboratively is often necessary to get evidence into policy. Academics could identify policy actors to provide better insight into policy problems, act as champions for their research, and identify the most helpful policy actors, who may advisors rather than ministers. However, collaboration can also lead to conflict and reputational damage. Therefore, when possible, produce ground rules acceptable to academics and policymakers. Successful engagement may require all parties to agree about processes (ethics, consent, and confidentiality) and outputs (data, intellectual property).

7. Be ‘entrepreneurial’ or find someone who is

Much advice projects an image of a daring, persuasive scientist, comfortable in policy environments and always available when needed. Develop media and marketing skills. If not willing or able to act in this way, hire brokers to act on your behalf.

8. Reflect continuously: should you engage, do you want to, and is it working?

Academics may be a good fit in the policy arena if they enjoy the work or are passionate about the issue. Even so, keep track of when and how you have had impact, and revise your practices continuously.

At first glance, some of this advice appears to be consistent with key reference points in policy studies. For example, multiple streams analysis suggests that ‘policy entrepreneurs’ can make the difference to the uptake of evidence because they recognise three key requirements: tell stories to help draw attention to one way to frame a policy problem; have an evidence-informed and technically/ politically feasible policy solution ready, to attach to a lurch of attention to a problem; and, exploit the temporary willingness and ability of policymakers to select that solution. Yet, MSA only makes sense in relation to the wider policymaking environment which adds a sense of scale to this advice. Put simply, most entrepreneurs fail, and their success depends more on their environment than their skills.

Therefore, to provide more context for – and make more sense of – this ‘how to’ advice, we need to go further:

• What can you do when policymakers ignore your evidence? Encourage ‘knowledge management for policy’ (go to page 568)
• How else can we describe and seek to fill the evidence-policy gap? (go to page 400)
• How far should you go to privilege evidence? 1. Evidence and governance principles (go to page 101)
• How far should you go to privilege evidence? 2. Policy theories, scenarios, and ethical dilemmas (go to page 526)
• How far should you go to privilege evidence? 3. Use psychological insights to manipulate policymakers (go to page 300 then scroll down to point 3)

[If you came here in error, or to continue your adventure, go to page 100]

What can you do when policymakers ignore your evidence?

The first post in this series asks: Why don’t policymakers listen to your evidence? It is based on talks that I have been giving since 2016, mostly to tap into a common story told by people in my audience (and the ‘science community’ more generally) about a new era in politics: policymakers do not pay sufficient respect to expertise or attention to good quality evidence.

It’s not my story, but I think it’s important to respect my audience members enough to (a) try to engage with their question, before (b) inviting them to think differently about how to ask it, and (c) provide different types of solutions according to the changing nature of the question.

Instead of a really long post for (b) and (c), I’ve made it a bit like Ceefax in which you can choose which question to ask or answer:

• Why don’t policymakers listen to your evidence? (go to page 154)
• What can you do when policymakers ignore your evidence? Tips from the ‘how to’ literature from the science community (go to page 650)
• What can you do when policymakers ignore your evidence? Encourage ‘knowledge management for policy’ (go to page 568)
• How else can we describe and seek to fill the evidence-policy gap? (go to page 400)
• How far should you go to privilege evidence? 1. Evidence and governance principles (go to page 101)
• How far should you go to privilege evidence? 2. Policy theories, scenarios, and ethical dilemmas (go to page 526)
• How far should you go to privilege evidence? 3. Use psychological insights to manipulate policymakers (go to page 300 then scroll down to point 3)

Some of this material will appear in work with Dr Kathryn Oliver and with my co-authors on a forthcoming report for Enlightenment 2.0 

See also:

Evidence-based policymaking: political strategies for scientists living in the real world

The Science of Evidence-based Policymaking: How to Be Heard

Evidence based policymaking: 7 key themes

I also do slides, such as:

Paul Cairney FUSE May 2018 

Paul Cairney Victoria May 2018

The Politics of Evidence-Based Policymaking: ANZSOG talks 

Talks and blogs: ANZSOG and beyond

This is me presenting those slides in Cambridge while being very Scottish, enjoying a too-heavy cold, and sucking a lozenge. Please note that I tend to smile a lot and make many sarcastic jokes while presenting, partly to apologise indirectly for all the self-publicity.