What can you do when policymakers ignore your evidence? Tips from the ‘how to’ literature from the science community

Put down that huge blue pen. Step away from the flipchart.

There is now a large literature on the strategies that people might use to promote the use of evidence in policy, so you no longer have to start from first principles in your workshop.

Thanks to Dr Kathryn Oliver and colleagues, we now have systematic reviews of the peer reviewed academic evidence on the ‘barriers’ between evidence and policy (click here) and the ‘grey literature’, which includes newspaper editorials, blogs, and practitioner reports (click here later – this article is still in review).

The advice from the peer reviewed literature can be summed up as follows:

  1. Produce better quality evidence on policy problems and solutions.
  2. Improve dissemination strategies to increase policymaker access to research: write more concise and less jargon-filled reports, boost resources for dissemination, and remove paywall obstacles to accessing research.
  3. Develop relationships with policymakers, to address the unpredictability of politics, or the importance of timing, serendipity, and ‘windows of opportunity’ to act.
  4. Engage directly, in academic-practitioner workshops, or use intermediaries such as ‘knowledge brokers’, to break down communications and cultural barriers associated with the different incentives, rhythms, and language of researcher and policymakers.
  5. Encourage policymakers to be more science literate, to appreciate the role of evidence and ways to separate high- and low-quality sources

The grey literature is fairly similar.

Kathryn and I summarise key themes and individual recommendations from 78 publications as follows:

  1. Do high quality research.

Use specific well-established research designs, methods, or metrics.

  1. Make your research relevant and readable.

Provide and disseminate easily-understandable, clear, relevant and high-quality research. Aim for the general but ‘not ignorant’ reader. Produce good stories based, for example, on emotional appeals or humour to expand your audience

  1. Understand the policy process, policymaking context, and key actors.

Note the busy and constrained lives of policy actors Maximise your use of established ways to engage, such as in advisory committees. Be pragmatic about what ‘success’ looks like, accepting that research rarely translates into policy options directly

  1. Be ‘accessible’ to policymakers: engage routinely, flexibly, and humbly.

As publicly-funded professionals, it is the job of academics to engage with policy and publics. Discuss topics beyond your narrow expertise, as a representative of your discipline or the science profession. Be humble, courteous, professional, and recognise the limits to your skills when giving policy advice. Respect policymakers’ time and expertise.

  1. Decide if you want to be an ‘issue advocate’ or ‘honest broker’

There is a commonly-cited ethical dilemma about whether to go beyond providing evidence to recommend specific policy options or remain an ‘honest broker’ explaining the options. If making recommendations, use storytelling to persuade policymakers of a course of action. However, note the consequences of becoming a political actor. David Nutt famously lost his advisory role after publicly criticising government drugs policy, some describe the loss of one’s safety if adopting an activist mindset, and anecdotal conversations describe the risk of losing credibility in government if seen as too evangelical while giving policy advice. However, more common consequences include criticism within one’s peer-group, being seen as an academic ‘lightweight, being used to add legitimacy to a policy position, and the risk of burnout.

  1. Build relationships (and ground rules) with policymakers.

Relationship-building activities require major investment and skills, but working collaboratively is often necessary to get evidence into policy. Academics could identify policy actors to provide better insight into policy problems, act as champions for their research, and identify the most helpful policy actors, who may advisors rather than ministers. However, collaboration can also lead to conflict and reputational damage. Therefore, when possible, produce ground rules acceptable to academics and policymakers. Successful engagement may require all parties to agree about processes (ethics, consent, and confidentiality) and outputs (data, intellectual property).

  1. Be ‘entrepreneurial’ or find someone who is

Much advice projects an image of a daring, persuasive scientist, comfortable in policy environments and always available when needed. Develop media and marketing skills. If not willing or able to act in this way, hire brokers to act on your behalf.

  1. Reflect continuously: should you engage, do you want to, and is it working?

Academics may be a good fit in the policy arena if they enjoy the work or are passionate about the issue. Even so, keep track of when and how you have had impact, and revise your practices continuously.

At first glance, some of this advice appears to be consistent with key reference points in policy studies. For example, multiple streams analysis suggests that ‘policy entrepreneurs’ can make the difference to the uptake of evidence because they recognise three key requirements: tell stories to help draw attention to one way to frame a policy problem; have an evidence-informed and technically/ politically feasible policy solution ready, to attach to a lurch of attention to a problem; and, exploit the temporary willingness and ability of policymakers to select that solution. Yet, MSA only makes sense in relation to the wider policymaking environment which adds a sense of scale to this advice. Put simply, most entrepreneurs fail, and their success depends more on their environment than their skills.

Therefore, to provide more context for – and make more sense of – this ‘how to’ advice, we need to go further:

  • What can you do when policymakers ignore your evidence? Encourage ‘knowledge management for policy’ (go to page 568)
  • How else can we describe and seek to fill the evidence-policy gap? (go to page 400)
  • How far should you go to privilege evidence? 1. Evidence and governance principles (go to page 101)
  • How far should you go to privilege evidence? 2. Policy theories, scenarios, and ethical dilemmas (go to page 526)
  • How far should you go to privilege evidence? 3. Use psychological insights to manipulate policymakers (go to page 300 then scroll down to point 3)

[If you came here in error, or to continue your adventure, go to page 100]

4 Comments

Filed under Evidence Based Policymaking (EBPM)

What can you do when policymakers ignore your evidence?

The first post in this series asks: Why don’t policymakers listen to your evidence? It is based on talks that I have been giving since 2016, mostly to tap into a common story told by people in my audience (and the ‘science community’ more generally) about a new era in politics: policymakers do not pay sufficient respect to expertise or attention to good quality evidence.

It’s not my story, but I think it’s important to respect my audience members enough to (a) try to engage with their question, before (b) inviting them to think differently about how to ask it, and (c) provide different types of solutions according to the changing nature of the question.

Instead of a really long post for (b) and (c), I’ve made it a bit like Ceefax in which you can choose which question to ask or answer:

  • Why don’t policymakers listen to your evidence? (go to page 154)
  • What can you do when policymakers ignore your evidence? Tips from the ‘how to’ literature from the science community (go to page 650)
  • What can you do when policymakers ignore your evidence? Encourage ‘knowledge management for policy’ (go to page 568)
  • How else can we describe and seek to fill the evidence-policy gap? (go to page 400)
  • How far should you go to privilege evidence? 1. Evidence and governance principles (go to page 101)
  • How far should you go to privilege evidence? 2. Policy theories, scenarios, and ethical dilemmas (go to page 526)
  • How far should you go to privilege evidence? 3. Use psychological insights to manipulate policymakers (go to page 300 then scroll down to point 3)

Some of this material will appear in work with Dr Kathryn Oliver (papers in review) and (assuming they don’t jettison it during the writing process) with my co-authors on a forthcoming report for Enlightenment 2.0 

See also:

Evidence-based policymaking: political strategies for scientists living in the real world

The Science of Evidence-based Policymaking: How to Be Heard

Evidence based policymaking: 7 key themes

I also do slides, such as:

Paul Cairney FUSE May 2018 

Paul Cairney Victoria May 2018

This is me presenting those slides in Cambridge while being very Scottish, enjoying a too-heavy cold, and sucking a lozenge. Please note that I tend to smile a lot and make many sarcastic jokes while presenting, partly to apologise indirectly for all the self-publicity.

6 Comments

Filed under agenda setting, Evidence Based Policymaking (EBPM), Psychology Based Policy Studies, public policy

The Future of Public Bodies

Guest post by Dr Matthew Wood, Lecturer in Politics and Deputy Director of the Crick Centre, University of Sheffield

MattWood

On 1st June 2018 over 30 academics and practitioners from around the world came together at the University of Sheffield to debate the future of arm’s length public bodies, specifically the key challenges they face on accountability and stakeholder engagement. Public bodies are organisations carrying out public work on behalf of the government, but are unelected. Accountability and stakeholder engagement are therefore key for public bodies as ways of assuring public trust and confidence.

The event provided an opportunity to discuss findings from Dr Wood’s three-year ESRC Future Leaders research project on public bodies, and an international survey of accountability in public bodies coordinated by Professor Thomas Schillemans. International experts on public bodies attended from Universities in the Netherlands, Switzerland, Norway, Belgium, and Australia. The event was also attended by representatives from the OECD, UK Cabinet Office, Institute for Government, Department for Environment, Food and Rural Affairs, Department for Education, and practitioners from Dutch public bodies, among others.

Prior to the event, Dr Wood asked academics and practitioners to send through what they thought were the key issues facing public bodies, and made these the subject of debate and discussion over the two hours. This blog summarises the key points of debate, criticism and themes for future research co-produced between all the attendees.

Accountability

What are the emerging challenges for accountability and public confidence in public bodies?

 Brexit was identified as a key challenge for public bodies in the UK, given there will be a large number of regulatory responsibilities transferred from the European level in March 2019, which public bodies are likely to be required to implement. Public bodies are therefore likely to take on a range of tasks previously performed by EU decentralised agencies and the European Commission.

In this context, participants noted an increasing problem related to time pressures and lack of resources to ensure stakeholders are able to hold public bodies properly accountable. For example, the legislative timetable in Parliament is so crowded that there is very little time to fully scrutinise work plans and accounts, with potentially significant implications for public confidence. One emergent theme was therefore the question of how to manage time in a context of constrained resources.

 How do accountability relationships shape decision making in public bodies?

Decision making is shaped by formal requirements for information provision, and informal relationships between public bodies, their Boards and parent departments. The international survey findings point to the potential that stronger disciplining powers for departments to sanction public bodies could have a significant positive impact on working relationships by providing clarity and coherence to relationships.

How are conflicts resolved when public bodies have competing accountabilities in two different directions?

 Public bodies are committed to presenting information in a timely and efficient manner, but often they have competing organisations they have to give account to. For example, public bodies may be asked to report to parliamentary select committees, audit offices, and central departments simultaneously on similar issues. This can create confusion about which relationships to prioritise.

One proposed solution to this problem was to refer back to official rules about which organisations are the main ‘principals’ of the public bodies in question. Public bodies have formal arrangements governing which forums they should engage with first and foremost (often departments) and these ought to be a ‘go to’ source of advice in the event of confusion about lines of reporting.

How can public bodies effectively manage public expectations around taking immediate action to assure accountability during crises, and manage their own approach to adverse social media coverage?

 This part of the discussion centred on the need for departments and public bodies to cooperate in managing public expectations. Rather than participating in ‘blame games’, departments and the staff of public bodies should talk to each other in the event of a crisis and share staff time to develop management strategies. Often, there are no clear ‘effective solutions’, because crises are unpredictable and complex, but focusing on common lines to take and coordinating media responses.

Another key point was that accountability has important temporal dimensions. In essence, there are more pressures for accountability to ‘work’ in the aftermath of emergencies and scandals, so flexibility is important and communicating clearly between relevant operational teams and points of contact in public bodies and governments is crucial.

 How do public managers deal with “information overload” and what are its implications for democracy and accountability within arm’s length agencies?

Information overload is a key issue for public bodies, and across the public sector. This part of the discussion made clear that often public bodies provide large amounts of information to meet accountability demands that overwhelms departments and might be seen as unnecessary. However, there was an appreciation that the demands for public bodies to make sure they are covering all the bases means they often feel encouraged to provide extensive information and justification. One suggestion for resolving this tension was to refer back to guidelines on what needs to be provided and for what purpose.

Stakeholder Engagement

How do public bodies and other expert agencies engage with stakeholders within and outside government?

 Dr Wood’s work suggests that agencies can effectively engage with stakeholders through ‘entrepreneurship’ – going beyond formal organisational responsibilities to be proactive in seeking out opportunities to engage stakeholders. They do this through:

  1. Improving website accessibility/readability;
  2. Pro-actively seeking coverage from traditional and non-traditional media outlets;
  3. Face-to-face events with stakeholders;
  4. Close collaboration with stakeholders through informal working groups;
  5. Training exercises with professional audiences and service users;
  6. Internal learning and reform exercises.

Dr Wood presented a typology of ‘entrepreneurship’ strategies developed by his ESRC-funded research, covering ‘technical’ and ‘insulating’ public bodies that cover less of the six criteria, and ‘networking’ and ‘politicised’ public bodies covering more of them. He presented data suggesting ‘politicised’ public bodies are more likely to be viewed as legitimate within parliamentary debates.

One response to this was that ‘entrepreneurship’ might be more relevant for public bodies with more resources available to them. Another critical view was that entrepreneurial strategies cannot be a substitute or ‘smokescreen’ for formal and legal responsibilities, and public bodies need to be wary of straying too far from their legal remit.

What kinds of stakeholder engagement practices do public bodies create? Which are most effective?

 One key example of good practice for stakeholder engagement was the Electoral Commission. The Commission is good at providing very clear explanations of electoral law, its relevance, and why what it does matters for the public good. It presents information in an accessible but authoritative way, in a similar way to the ‘insulating’ approach presented in Dr Wood’s typology. This suggests that a more constrained strategy, focusing on elements of ‘entrepreneurship’ that are specifically relevant to individual public bodies, could be better for securing legitimacy, than one focused on reaching out to various diverse stakeholders.

How do and with which consequences do agencies balance political responsiveness and agency credibility and reputation?

 The discussion highlighted how public bodies need to provide objective and clear information and have a well-defined approach to communicating their remit, responsibilities, and why their work has public benefit, to key audiences. The reputation of public bodies is forged through a strong sense of public purpose and commitment to serving diverse communities. This challenge is particularly relevant internationally, where the discussion highlighted how international public bodies find communicating their expertise and role more difficult.

 How can governments design ALBs to allow them capacity to manage stakeholder engagement in ways that promote collective public good, and address power inequities and representativeness of affected audiences?

 A key theme running through the final discussion was that public bodies should be confident that, despite being unelected, they carry out and advise on crucial political decisions that require extensive consultation and scrutiny. Since government departments often do not have the time or resources to carry out such detailed work, public bodies provide a key function. A key point was to refer back to official guidance about public bodies’ mission and purpose, and to communicate this in an efficient and effective manner.

Leave a comment

Filed under Uncategorized

Paul Cairney reviews Graham Room’s book on Agile Actors on Complex Terrains (2016)

It’s true. I do.

Policy and Politics Journal

Paul CairneyPaul Cairney

Paul Cairney reviews Graham Room’s Agile Actors on Complex Terrains (Routledge, 2016). Paul is guest editor of our 2018 special issue: Practical Lessons on Policy Theories

Some background context on complexity theory

If used wisely, complexity theory has the potential to make a great contribution to the study of politics and policymaking. It offers a way to think about, and visualise, the interaction between many actors, following many rules, to produce outcomes that we can relate to the properties of complex systems. 

View original post 680 more words

Leave a comment

Filed under Uncategorized

MPP webinar

I’m doing a webinar today to answer questions about the University of Stirling’s MPP. The subject of the Scottish weather is bound to come up, so here is a picture of typical* Scottish weather near me.

2018-07-01 13.17.48

 

 

*there is some debate about the meaning and accuracy of this claim.

Leave a comment

Filed under Uncategorized

When are scientists neutral experts or strategic policy makers?

Great post by Professor Karin Ingold on the potential for different roles for scientists in evidence informed policymaking

Integration and Implementation Insights

Community member post by Karin Ingold

karin-ingold Karin Ingold (biography)

What roles can science and scientific experts adopt in policymaking? One way of examining this is through the Advocacy Coalition Framework (Sabatier and Jenkins-Smith 1993). This framework highlights that policymaking and the negotiations regarding a political issue—such as reform of the health system, or the introduction of an energy tax on fossil fuels—is dominated by advocacy coalitions in opposition. Advocacy coalitions are groups of actors sharing the same opinion about how a policy should be designed and implemented. Each coalition has its own beliefs and ideologies and each wants to see its preferences translated into policies.

I build on the work of Weible and colleagues (2010), who distinguish three types of ‘subsystems’ in which policy is made:

  • a collaborative subsystem, where at least two coalitions exist; they have different opinions, but want to overcome them
  • an adversarial subsystem, where…

View original post 526 more words

1 Comment

Filed under Uncategorized

Evaluation and Governing in the 21st Century

Deirdre Niamh Duffy (2017) Evaluation and Governing in the 21st Century (Palgrave Pivot)

Duffy’s new book engages with the many uses and abuses of evaluation in UK politics. This variation in evaluation practices relates partly to the vagueness of the term. Put simply, evaluation is about measuring the success of public policies. However, such a simple definition is so broad that it can mean everything and therefore nothing.

In that context, Duffy compares the many ways in which people could, do, and should use evaluation to inform policy and policymaking.

In terms of what evaluation could mean, Duffy identifies common definitions of evaluation as:

  • ‘assessment of value/ merit’, involving some sort of combination of the values of the people involved and the methods they use to gather evidence of success
  • “a realistic(ic) ‘science’”, focusing primarily on the allegedly appropriate use of scientific methods to measure success
  • ‘actionable science’, using measures of success to help improve policy.

In terms of what UK governments actually do, Duffy identifies key phases including:

  • A pre-New Labour reluctance to allow outside actors to opine on the success of its policies, combined with a relative lack of sophisticated methodological tools to do so.
  • A New Labour era (from 1997), in which ministers were keen to stress a reliance on evidence based policymaking, to focus on ‘what works’ when identifying promising policies and evaluating their success primarily with reference to technical scientific measures rather than, say, their ideological positions.

One can infer from Duffy’s analysis that there is something to be said for the relative honesty of the pre-Labour position in which governments pretty much told people where to go, and staked their claim as the arbiters of their own success.

In contrast, Labour had a tendency to use the language of evidence to try to depoliticise issues, when there should have been more debate on values and the rationale for policies. It also used this approach to put pressure on delivery organisations – and, by extension, the recipients of policy measures – to do what it wanted. In particular, Duffy focuses on evaluation as the setting of benchmarks, and use of league tables based on proxies of success, to put major pressure on the organisation not doing so well. In part, the government is able to do so by encouraging the fear and shame of the actors leading or working for organisations delivering public services. In this context, ‘evaluation becomes less about EBPM and more about influencing and manipulating behaviour’ (p147).

In terms of what a government should do, Duffy focuses on the potential for evaluation practices to produce positive transformations in policy and practice: ‘evaluation can be reclaimed as part of a transformative project using critical theory’ (p148). Since such transformation would be encouraged via open discussion without a fixed agenda, and without a focus on one best way to use methods to evaluate, it must remain a very general aspiration. Consequently,

For some – particularly those seeking an instrumental guidebook on how to do ‘good evaluation’ – this may seem highly problematic. However, from a critical sociological perspective, it is only through remaining open to potential, as yet unknown emergent transformations that the disciplinary and controlling governing effects of knowledge production processes can be unsettled.

As such, Duffy’s book stands out as a critical theoretical take on the role of evidence in policy making and evaluation. I commend to people who want to broaden their horizons.

duffy

 

 

Leave a comment

Filed under Evidence Based Policymaking (EBPM)