Tag Archives: EBP

Evidence-based policymaking and the ‘new policy sciences’

image policy process round 2 25.10.18

[I wasn’t happy with the first version, so this is version 2]

In the ‘new policy sciences’, Chris Weible and I advocate:

  • a return to Lasswell’s vision of combining policy analysis (to recommend policy change) and policy theory (to explain policy change), but
  • focusing on a far larger collection of actors (beyond a small group at the centre),
  • recognising new developments in studies of the psychology of policymaker choice, and
  • building into policy analysis the recognition that any policy solution is introduced in a complex policymaking environment over which no-one has control.

However, there is a lot of policy theory out there, and we can’t put policy theory together like Lego to produce consistent insights to inform policy analysis.

Rather, each concept in my image of the policy process represents its own literature: see these short explainers on the psychology of policymaking, actors spread across multi-level governance, institutions, networks, ideas, and socioeconomic factors/ events.

What the explainers don’t really project is the sense of debate within the literature about how best to conceptualise each concept. You can pick up their meaning in a few minutes but would need a few years to appreciate the detail and often-fundamental debate.

Ideally, we would put all of the concepts together to help explain policymaker choice within a complex policymaking environment (how else could I put up the image and present is as one source of accumulated wisdom from policy studies?). Peter John describes such accounts as ‘synthetic’. I have also co-authored work with Tanya Heikkila – in 2014 and 2017 to compare the different ways in which ‘synthetic’ theories conceptualise the policy process.

However, note the difficulty of putting together a large collection of separate and diverse literatures into one simple model (e.g. while doing a PhD).

On that basis, I’d encourage you to think of these attempts to synthesise as stories. I tell these stories a lot, but someone else could describe theory very differently (perhaps by relying on fewer male authors or US-derived theories in which there is a very specific reference points and positivism is represented well).

The example of EBPM

I have given a series of talks to explain why we should think of ‘evidence-based policymaking’ as a myth or political slogan, not an ideal scenario or something to expect from policymaking in the real world. They usually involve encouraging framing and storytelling rather than expecting evidence to speak for itself, and rejecting the value of simple models like the policy cycle. I then put up an image of my own and encourage people to think about the implications of each concept:

SLIDE simple advice from hexagon image policy process 24.10.18

I describe the advice as simple-sounding and feasible at first glance, but actually a series of Herculean* tasks:

  • There are many policymakers and influencers spread across government, so find out where the action is, or the key venues in which people are making authoritative decisions.
  • Each venue has its own ‘institutions’ – the formal and written, or informal and unwritten rules of policymaking – so learn the rules of each venue in which you engage.
  • Each venue is guided by a fundamental set of ideas – as paradigms, core beliefs, monopolies of understanding – so learn that language.
  • Each venue has its own networks – the relationships between policy makers and influencers – so build trust and form alliances within networks.
  • Policymaking attention is often driven by changes in socioeconomic factors, or routine/ non-routine events, so be prepared to exploit the ‘windows of opportunity’ to present your solution during heightened attention to a policy problem.

Further, policy theories/ studies help us understand the context in which people make such choices. For example, consider the story that Kathryn Oliver and I tell about the role of evidence in policymaking environments:

If there are so many potential authoritative venues, devote considerable energy to finding where the ‘action’ is (and someone specific to talk to). Even if you find the right venue, you will not know the unwritten rules unless you study them intensely. Some networks are close-knit and difficult to access because bureaucracies have operating procedures that favour some sources of evidence. Research advocates can be privileged insiders in some venues and excluded completely in others. If your evidence challenges an existing paradigm, you need a persuasion strategy good enough to prompt a shift of attention to a policy problem and a willingness to understand that problem in a new way. You can try to find the right time to use evidence to exploit a crisis leading to major policy change, but the opportunities are few and chances of success low.  In that context, policy studies recommend investing your time over the long term – to build up alliances, trust in the messenger, knowledge of the system, and to seek ‘windows of opportunity’ for policy change – but offer no assurances that any of this investment will ever pay off

As described, this focus on the new policy sciences and synthesising insights helps explain why ‘the politics of evidence-based policymaking’ is equally important to civil servants (my occasional audience) as researchers (my usual audience).

To engage in skilled policy analysis, and give good advice, is to recognise the ways in which policymakers combine cognition/emotion to engage with evidence, and must navigate a complex policymaking environment when designing or selecting technically and politically feasible solutions. To give good advice is to recognise what you want policymakers to do, but also that they are not in control of the consequences.

From one story to many?

However, I tell these stories without my audience having the time to look further into each theory and its individual insights. If they do have a little more time, I go into the possible contribution of individual insights to debate.

For example, they adapt insights from psychology in different ways …

  • PET shows the overall effect of policymaker psychology on policy change: they combine cognition and emotion to pay disproportionate attention to a small number of issues (contributing to major change) and ignore the rest (contributing to ‘hyperincremental’ change).
  • The IAD focuses partly on the rules and practices that actors develop to build up trust in each other.
  • The ACF describes actors going into politics to turn their beliefs into policy, forming coalitions with people who share their beliefs, then often romanticising their own cause and demonising their opponents.
  • The NPF describes the relative impact of stories on audiences who use cognitive shortcuts to (for example) identify with a hero and draw a simple moral.
  • SCPD describes policymakers drawing on gut feeling to identify good and bad target populations.
  • Policy learning involves using cognition and emotion to acquire new knowledge and skills.

… even though the pace of change in psychological research often seems faster than the ways in which policy studies can incorporate new and reliable insights.

They also present different conceptions of the policymaking environment in which actors make choices. See this post for more on this discussion in relation to EBPM.

My not-brilliant conclusion is that:

  1. Policy theory/ policy studies has a lot to offer other disciplines and professions, particularly in field like EBPM in which we need to account for politics and, more importantly, policymaking systems, but
  2. Beware any policy theory story that presents the source literature as coherent and consistent.
  3. Rather, any story of the field involves a series of choices about what counts as a good theory and good insight.
  4. In other words, the exhortation to think more about what counts as ‘good evidence’ applies just as much to political science as any other.

Postscript: well, that is the last of the posts for my ANZOG talks. If I’ve done this properly, there should now be a loop of talks. It should be possible to go back to the first one and see it as a sequel to this one!

Or, for more on theory-informed policy analysis – in other words, where the ‘new policy sciences’ article is taking us – here is how I describe it to students doing a policy analysis paper (often for the first time).

Or, have a look at the earlier discussion of images of the policy process. You may have noticed that there is a different image in this post (knocked up in my shed at the weekend). It’s because I am experimenting with shapes. Does the image with circles look more relaxing? Does the hexagonal structure look complicated even though it is designed to simplify? Does it matter? I think so. People engage emotionally with images. They share them. They remember them. So, I need an image more memorable than the policy cycle.

 

Paul Cairney Brisbane EBPM New Policy Sciences 25.10.18

 

 

 

*I welcome suggestions on another word to describe almost-impossibly-hard

2 Comments

Filed under agenda setting, Evidence Based Policymaking (EBPM), Policy learning and transfer, Psychology Based Policy Studies, public policy

Evidence-based policymaking and the ‘new policy sciences’

Circle image policy process 24.10.18

I have given a series of talks to explain why we should think of ‘evidence-based policymaking’ as a myth or political slogan, not an ideal scenario or something to expect from policymaking in the real world. They usually involve encouraging framing and storytelling rather than expecting evidence to speak for itself, and rejecting the value of simple models like the policy cycle. I then put up an image of my own and encourage people to think about the implications of each concept:

SLIDE simple advice from hexagon image policy process 24.10.18

I describe the advice as simple-sounding and feasible at first glance, but actually a series of Herculean* tasks:

  • There are many policymakers and influencers spread across government, so find out where the action is, or the key venues in which people are making authoritative decisions.
  • Each venue has its own ‘institutions’ – the formal and written, or informal and unwritten rules of policymaking – so learn the rules of each venue in which you engage.
  • Each venue is guided by a fundamental set of ideas – as paradigms, core beliefs, monopolies of understanding – so learn that language.
  • Each venue has its own networks – the relationships between policy makers and influencers – so build trust and form alliances within networks.
  • Policymaking attention is often driven by changes in socioeconomic factors, or routine/ non-routine events, so be prepared to exploit the ‘windows of opportunity’ to present your solution during heightened attention to a policy problem.

In most cases, we don’t have time to discuss a more fundamental issue (at least for researchers using policy theory and political science concepts):

From where did these concepts come, and how well do we know them?

To cut a long story short, each concept represents its own literature: see these short explainers on the psychology of policymaking, actors spread across multi-level governance, institutions, networks, ideas, and socioeconomic factors/ events. What the explainers don’t really project is the sense of debate within the literature about how best to conceptualise each concept. You can pick up their meaning in a few minutes but would need a few years to appreciate the detail and often-fundamental debate.

Ideally, we would put all of the concepts together to help explain policymaker choice within a complex policymaking environment (how else could I put up the image and present is as one source of accumulated wisdom from policy studies?). Peter John describes such accounts as ‘synthetic’. I have also co-authored work with Tanya Heikkila – in 2014 and 2017 to compare the different ways in which ‘synthetic’ theories conceptualise the policy process. However, note the difficulty of putting together a large collection of separate and diverse literatures into one simple model (e.g. while doing a PhD).

The new policy sciences

More recently, in the ‘new policy sciences’, Chris Weible and I present a more provocative story of these efforts, in which we advocate:

  • a return to Lasswell’s vision of combining policy analysis (to recommend policy change) and policy theory (to explain policy change), but
  • focusing on a far larger collection of actors (beyond a small group at the centre),
  • recognising new developments in studies of the psychology of policymaker choice, and
  • building into policy analysis the recognition that any policy solution is introduced in a complex policymaking environment over which no-one has control.

This focus on psychology is not new …

  • PET shows the overall effect of policymaker psychology on policy change: they combine cognition and emotion to pay disproportionate attention to a small number of issues (contributing to major change) and ignore the rest (contributing to ‘hyperincremental’ change).
  • The IAD focuses partly on the rules and practices that actors develop to build up trust in each other.
  • The ACF describes actors going into politics to turn their beliefs into policy, forming coalitions with people who share their beliefs, then often romanticising their own cause and demonising their opponents.
  • The NPF describes the relative impact of stories on audiences who use cognitive shortcuts to (for example) identify with a hero and draw a simple moral.
  • SCPD describes policymakers drawing on gut feeling to identify good and bad target populations.
  • Policy learning involves using cognition and emotion to acquire new knowledge and skills.

… but the pace of change in psychological research often seems faster than the ways in which policy studies can incorporate new and reliable insights.

Perhaps more importantly, policy studies help us understand the context in which people make such choices. For example, consider the story that Kathryn Oliver and I tell about the role of evidence in policymaking environments:

If there are so many potential authoritative venues, devote considerable energy to finding where the ‘action’ is (and someone specific to talk to). Even if you find the right venue, you will not know the unwritten rules unless you study them intensely. Some networks are close-knit and difficult to access because bureaucracies have operating procedures that favour some sources of evidence. Research advocates can be privileged insiders in some venues and excluded completely in others. If your evidence challenges an existing paradigm, you need a persuasion strategy good enough to prompt a shift of attention to a policy problem and a willingness to understand that problem in a new way. You can try to find the right time to use evidence to exploit a crisis leading to major policy change, but the opportunities are few and chances of success low.  In that context, policy studies recommend investing your time over the long term – to build up alliances, trust in the messenger, knowledge of the system, and to seek ‘windows of opportunity’ for policy change – but offer no assurances that any of this investment will ever pay off

Then, have a look at this discussion of ‘synthetic’ policy theories, designed to prompt people to consider how far they would go to get their evidence into policy.

Theory-driven policy analysis

As described, this focus on the new policy sciences helps explain why ‘the politics of evidence-based policymaking’ is equally important to civil servants (my occasional audience) as researchers (my usual audience).

To engage in skilled policy analysis, and give good advice, is to recognise the ways in which policymakers combine cognition/emotion to engage with evidence, and must navigate a complex policymaking environment when designing or selecting technically and politically feasible solutions. To give good advice is to recognise what you want policymakers to do, but also that they are not in control of the consequences.

Epilogue

Well, that is the last of the posts for my ANZOG talks. If I’ve done this properly, there should now be a loop of talks. It should be possible to go back to the first one in Auckland and see it as a sequel to this one in Brisbane!

Or, for more on theory-informed policy analysis – in other words, where the ‘new policy sciences’ article is taking us – here is how I describe it to students doing a policy analysis paper (often for the first time).

Or, have a look at the earlier discussion of images of the policy process. You may have noticed that there is a different image in this post (knocked up in my shed at the weekend). It’s because I am experimenting with shapes. Does the image with circles look more relaxing? Does the hexagonal structure look complicated even though it is designed to simplify? Does it matter? I think so. People engage emotionally with images. They share them. They remember them. So, I need an image more memorable than the policy cycle.

 

Paul Cairney Brisbane EBPM New Policy Sciences 25.10.18

 

 

*I welcome suggestions on another word to describe almost-impossibly-hard

2 Comments

Filed under agenda setting, Evidence Based Policymaking (EBPM), Policy learning and transfer, Psychology Based Policy Studies, public policy, Storytelling

Evidence-based policymaking: political strategies for scientists living in the real world

Note: I wrote the following discussion (last year) to be a Nature Comment but it was not to be!

Nature articles on evidence-based policymaking often present what scientists would like to see: rules to minimise bias caused by the cognitive limits of policymakers, and a simple policy process in which we know how and when to present the best evidence.[1]  What if neither requirement is ever met? Scientists will despair of policymaking while their competitors engage pragmatically and more effectively.[2]

Alternatively, if scientists learned from successful interest groups, or by using insights from policy studies, they could develop three ‘take home messages’: understand and engage with policymaking in the real world; learn how and when evidence ‘wins the day’; and, decide how far you should go to maximise the use of scientific evidence. Political science helps explain this process[3], and new systematic and thematic reviews add new insights.[4] [5] [6] [7]

Understand and engage with policymaking in the real world

Scientists are drawn to the ‘policy cycle’, because it offers a simple – but misleading – model for engagement with policymaking.[3] It identifies a core group of policymakers at the ‘centre’ of government, perhaps giving the impression that scientists should identify the correct ‘stages’ in which to engage (such as ‘agenda setting’ and ‘policy formulation’) to ensure the best use of evidence at the point of authoritative choice. This is certainly the image generated most frequently by health and environmental scientists when they seek insights from policy studies.[8]

Yet, this model does not describe reality. Many policymakers, in many levels and types of government, adopt and implement many measures at different times. For simplicity, we call the result ‘policy’ but almost no modern policy theory retains the linear policy cycle concept. In fact, it is more common to describe counterintuitive processes in which, for example, by the time policymaker attention rises to a policy problem at the ‘agenda setting’ stage, it is too late to formulate a solution. Instead, ‘policy entrepreneurs’ develop technically and politically feasible solutions then wait for attention to rise and for policymakers to have the motive and opportunity to act.[9]

Experienced government science advisors recognise this inability of the policy cycle image to describe real world policymaking. For example, Sir Peter Gluckman presents an amended version of this model, in which there are many interacting cycles in a kaleidoscope of activity, defying attempts to produce simple flow charts or decision trees. He describes the ‘art and craft’ of policy engagement, using simple heuristics to deal with a complex and ‘messy’ policy system.[10]

Policy studies help us identify two such heuristics or simple strategies.

First, respond to policymaker psychology by adapting to the short cuts they use to gather enough information quickly: ‘rational’, via trusted sources of oral and written evidence, and ‘irrational’, via their beliefs, emotions, and habits. Policy theories describe many interest group or ‘advocacy coalition’ strategies, including a tendency to combine evidence with emotional appeals, romanticise their own cause and demonise their opponents, or tell simple emotional stories with a hero and moral to exploit the biases of their audience.[11]

Second, adapt to complex ‘policy environments’ including: many policymakers at many levels and types of government, each with their own rules of evidence gathering, network formation, and ways of understanding policy problems and relevant socioeconomic conditions.[2] For example, advocates of international treaties often find that the evidence-based arguments their international audience takes for granted become hotly contested at national or subnational levels (even if the national government is a signatory), while the same interest groups presenting the same evidence of a problem can be key insiders in one government department but ignored in another.[3]

Learn the conditions under which evidence ‘wins the day’ in policymaking

Consequently, the availability and supply of scientific evidence, on the nature of problems and effectiveness of solutions, is a necessary but insufficient condition for evidence-informed policy. Three others must be met: actors use scientific evidence to persuade policymakers to pay attention to, and shift their understanding of, policy problems; the policy environment becomes broadly conducive to policy change; and, actors exploit attention to a problem, the availability of a feasible solution, and the motivation of policymakers, during a ‘window of opportunity’ to adopt specific policy instruments.10

Tobacco control represents a ‘best case’ example (box 1) from which we can draw key lessons for ecological and environmental policies, giving us a sense of perspective by highlighting the long term potential for major evidence-informed policy change. However, unlike their colleagues in public health, environmental scientists have not developed a clear sense of how to produce policy instruments that are technically and politically feasible, so the delivery of comparable policy change is not inevitable.[12]

Box 1: Tobacco policy as a best case and cautionary tale of evidence-based policymaking

Tobacco policy is a key example – and useful comparator for ecological and environmental policies – since it represents a best case scenario and cautionary tale.[13] On the one hand, the scientific evidence on the links between smoking, mortality, and preventable death forms the basis for modern tobacco control policy. Leading countries – and the World Health Organisation, which oversees the Framework Convention on Tobacco Control (FCTC) – frame tobacco use as a public health ‘epidemic’ and allow their health departments to take the policy lead. Health departments foster networks with public health and medical groups at the expense of the tobacco industry, and emphasise the socioeconomic conditions – reductions in (a) smoking prevalence, (b) opposition to tobacco control, and (c) economic benefits to tobacco – most supportive of tobacco control. This framing, and conducive policymaking environment, helps give policymakers the motive and opportunity to choose policy instruments, such as bans on smoking in public places, which would otherwise seem politically infeasible.

On the other hand, even in a small handful of leading countries such as the UK, it took twenty to thirty years to go from the supply of the evidence to a proportionate government response: from the early evidence on smoking in the 1950s prompting major changes from the 1980s, to the evidence on passive smoking in the 1980s prompting public bans from the 2000s onwards. In most countries, the production of a ‘comprehensive’ set of policy measures is not yet complete, even though most signed the FCTC.

Decide how far you’ll go to maximise the use of scientific evidence in policymaking

These insights help challenge the naïve position that, if policymaking can change to become less dysfunctional[1], scientists can be ‘honest brokers’[14] and expect policymakers to use their evidence quickly, routinely, and sincerely. Even in the best case scenario, evidence-informed change takes hard work, persistence, and decades to achieve.

Since policymaking will always appear ‘irrational’ and complex’[3], scientists need to think harder about their role, then choose to engage more effectively or accept their lack of influence.

To deal with ‘irrational’ policymakers, they should combine evidence with persuasion, simple stories, and emotional appeals, and frame their evidence to make the implications consistent with policymakers’ beliefs.

To deal with complex environments, they should engage for the long term to work out how to form alliances with influencers who share their beliefs, understand in which ‘venues’ authoritative decisions are made and carried out, the rules of information processing in those venues, and the ‘currency’ used by policymakers when they describe policy problems and feasible solutions.[2] In other words, develop skills that do not come with scientific training, avoid waiting for others to share your scientific mindset or respect for scientific evidence, and plan for the likely eventuality that policymaking will never become ‘evidence based’.

This approach may be taken for granted in policy studies[15], but it raises uncomfortable dilemmas regarding how far scientists should go, to maximise the use of scientific evidence in policy, using persuasion and coalition-building.

These dilemmas are too frequently overshadowed by claims – more comforting to scientists – that politicians are to blame because they do not understand how to generate, analyse, and use the best evidence. Scientists may only become effective in politics if they apply the same critical analysis to themselves.

[1] Sutherland, W.J. & Burgman, M. Nature 526, 317–318 (2015).

[2] Cairney, P. et al. Public Administration Review 76, 3, 399-402 (2016)

[3] Cairney, P. The Politics of Evidence-Based Policy Making (Palgrave Springer, 2016).

[4] Langer, L. et al. The Science of Using Science (EPPI, 2016)

[5] Breckon, J. & Dodson, J. Using Evidence. What Works? (Alliance for Useful Evidence, 2016)

[6] Palgrave Communications series The politics of evidence-based policymaking (ed. Cairney, P.)

[7] Practical lessons from policy theories (eds. Weible, C & Cairney, P.) Policy and Politics April 2018

[8] Oliver, K. et al. Health Research Policy and Systems, 12, 34 (2016)

[9] Kingdon, J. Agendas, Alternatives and Public Policies (Harper Collins, 1984)

[10] Gluckmann, P. Understanding the challenges and opportunities at the science-policy interface

[11] Cairney, P. & Kwiatkowski, R. Palgrave Communications.

[12] Biesbroek et al. Nature Climate Change, 5, 6, 493–494 (2015)

[13] Cairney, P. & Yamazaki, M. Journal of Comparative Policy Analysis

[14] Pielke Jr, R. originated the specific term The honest broker (Cambridge University Press, 2007) but this role is described more loosely by other commentators.

[15] Cairney, P. & Oliver, K. Health Research Policy and Systems 15:35 (2017)

4 Comments

Filed under Evidence Based Policymaking (EBPM), public policy

The UK government’s imaginative use of evidence to make policy

This post describes a new article published in British Politics (Open Access).

In retrospect, I think the title was too subtle and clever-clever. I wanted to convey two meanings: imaginative as a euphemism for ridiculous/ often cynical and to argue that a government has to be imaginative with evidence. The latter has two meanings: imaginative (1) in the presentation and framing of evidence-informed agenda, and (2) when facing pressure to go beyond the evidence and envisage policy outcomes.

So I describe two cases in which its evidence-use seems cynical, when:

  1. Declaring complete success in turning around the lives of ‘troubled families’
  2. Exploiting vivid neuroscientific images to support ‘early intervention’

Then I describe more difficult cases in which supportive evidence is not clear:

  1. Family intervention project evaluations are of limited value and only tentatively positive
  2. Successful projects like FNP and Incredible Years have limited applicability or ‘scalability’

As scientists, we can shrug our shoulders about the uncertainty, but elected policymakers in government have to do something. So what do they do?

At this point of the article it will look like I have become an apologist for David Cameron’s government. Instead, I’m trying to demonstrate the value of comparing sympathetic/ unsympathetic interpretations and highlight the policy problem from a policymaker’s perspective:

Cairney 2018 British Politics discussion section

I suggest that they use evidence in a mix of ways to: describe an urgent problem, present an image of success and governing competence, and provide cover for more evidence-informed long term action.

The result is the appearance of top-down ‘muscular’ government and ‘a tendency for policy to change as is implemented, such as when mediated by local authority choices and social workers maintaining a commitment to their professional values when delivering policy’

I conclude by arguing that ‘evidence-based policy’ and ‘policy-based evidence’ are political slogans with minimal academic value. The binary divide between EBP/ PBE distracts us from more useful categories which show us the trade-offs policymakers have to make when faced with the need to act despite uncertainty.

Cairney British Politics 2018 Table 1

As such, it forms part of a far wider body of work …

In both cases, the common theme is that, although (1) the world of top-down central government gets most attention, (2) central governments don’t even know what problem they are trying to solve, far less (3) how to control policymaking and outcomes.

See also:

Early intervention policy, from ‘troubled families’ to ‘named persons’: problems with evidence and framing ‘valence’ issues

Why doesn’t evidence win the day in policy and policymaking?

(found by searching for early intervention)

See also:

Here’s why there is always an expectations gap in prevention policy

Social investment, prevention and early intervention: a ‘window of opportunity’ for new ideas?

(found by searching for prevention)

Powerpoint for guest lecture: Paul Cairney UK Government Evidence Policy

3 Comments

Filed under Evidence Based Policymaking (EBPM), POLU9UK, Prevention policy, UK politics and policy

The Politics of Evidence revisited

This is a guest post by Dr Justin Parkhurst, responding to a review of our books by Dr Joshua Newman, and my reply to that review.

I really like that Joshua Newman has done this synthesis of 3 recent books covering aspects of evidence use in policy. Too many book reviews these days just describe the content, so some critical comments are welcome, as is the comparative perspective.

I’m also honoured that my book was included in the shortlist (it is available here, free as an ebook: bit.ly/2gGSn0n for interested readers) and I’d like to follow on from Paul to add some discussion points to the debate here – with replies to both Joshua and Paul (hoping first names are acceptable).

Have we heard all this before?

Firstly, I agree with Paul that saying ‘we’ve heard this all before’ risks speaking about a small community of active researchers who study these issues, and not the wider community. But I’d also add that what we’ve heard before is a starting point to many of these books, not where they end up.

In terms of where we start: I’m sure many of us who work in this field are somewhat frustrated at meetings when we hear people making statements that are well established in the literature. Some examples include:

  • “There can be many types of evidence, not just scientific research…”
  • “In the legal field, ‘evidence’ means something different…”
  • “We need evidence-based policy, not policy-based evidence…”
  • “We need to know ‘what works’ to get evidence into policy…”

Thus, I do think there is still a need to cement the foundations of the field more strongly – in essence, to establish a disciplinary baseline that people weighing in on a subject should be expected to know about before providing additional opinions. One way to help do this is for scholars to continue to lay out the basic starting points in our books – typically in the first chapter or two.

Of course, other specialist fields and disciplines have managed to establish their expertise to a point that individuals with opinions on a subject typically have some awareness that there is a field of study out there which they don’t necessarily know about. This is most obvious in the natural sciences (and perhaps in economics). E.g. most people (current presidents of some large North American countries aside?) are aware that don’t know a lot about engineering, medicine, or quantum physics – so they won’t offer speculative or instinctive opinions about why airplanes stay in the air, how to do bypass surgery, or what was wrong with the ‘Ant-Man’ film. Or when individuals do offer views, they are typically expected to know the basics of the subject.

For the topic of evidence and policy, I often point people to Huw Davies, Isabel Walter, and Sandra Nutley’s book Using Evidence, which is a great introduction to much of this field, as well as Carol Weiss’ insights from the late 70s on the many meanings of research utilisation. I also routinely point people to read The Honest Broker by Roger Pielke Jr. (which I, myself, failed to read before writing my book and, as such, end up repeating many of his points – I’ve apologised to him personally).

So yes, I think there is space for work like ours to continue to establish a baseline, even if some of us know this, because the expertise of the field is not yet widely recognised or established. Yet I think is it not accurate for Joshua to argue we end up repeating what is known, considering our books diverge in key ways after laying out some of the core foundations.

Where do we go from there?

More interesting for this discussion, then, is to reflect on what our various books try to do beyond simply laying out the basics of what we know about evidence use and policy. It is here where I would disagree with Joshua’s point claiming we don’t give a clear picture about the ‘problem’ that ‘evidence-based policy’ (his term – one I reject) is meant to address. Speaking only for my own book, I lay out the problem of bias in evidence use as the key motivation driving both advocates of greater evidence use as well as policy scholars critical of (oversimplified) knowledge translation efforts. But I distinguish between two forms of bias: technical bias – whereby evidence is used in ways that do not adhere to scientific best practice and thus produce sub-optimal social outcomes; and issue bias – whereby pieces of evidence, or mechanisms of evidence use, can obscure the important political choices in decision making, skewing policy choices towards those things that have been measured, or are conducive to measurement. Both of these forms of bias are violations of widely held social values – values of scientific fidelity on the one hand, and of democratic representation on the other. As such, for me, these are the problems that I try to consider in my book, exploring the political and cognitive origins of both, in order to inform thinking on how to address them.

That said, I think Joshua is right in some of the distinctions he makes between our works in how we try to take this field forward, or move beyond current challenges in differing ways. Paul takes the position that researchers need to do something, and one thing they can do is better understand politics and policy making. I think Paul’s writings about policy studies for students is superb (see his book and blog posts about policy concepts). But in terms of applying these insights to evidence use, this is where we most often diverge. I feel that keeping the focus on researchers puts too much emphasis on achieving ‘uptake’ of researcher’s own findings. In my view, I would point to three potential (overlapping) problems with this.

  • First – I do not think it is the role or responsibility of researchers to do this, but rather a failure to establish the right system of evidence provision;
  • Second – I feel it leaves unstated the important but oft ignored normative question of how ‘should’ evidence be used to inform policy;
  • Third – I believe these calls rest on often unstated assumptions about the answer to the second point which we may wish to challenge.

In terms of the first point: I’m more of an institutionalist (as Joshua points out). My view is that the problems around non-use or misuse of evidence can be seen as resulting from a failure to establish appropriate systems that govern the use of evidence in policy processes. As such, the solution would have to lie with institutional development and changes (my final chapter advocates for this) that establish systems which serve to achieve the good governance of evidence.

Paul’s response to Joshua says that researchers are demanding action, so he speaks to them. He wants researchers to develop “useful knowledge of the policy process in which they might want to engage” (as he says above).  Yet while some researchers may wish to engage with policy processes, I think it needs to be clear that doing so is inherently a political act – and can take on a role of issue advocacy by promoting those things you researched or measured over other possible policy considerations (points made well by Roger Pielke Jr. in The Honest Broker). The alternative I point towards is to consider what good systems of evidence use would look like. This is the difference between arguing for more uptake of research, vs. arguing for systems through which all policy relevant evidence can be seen and considered in appropriate ways – regardless of the political savvy, networking, or activism of any given researcher (in my book I have chapters reflecting on what appropriate evidence for policy might be, and what a good process for its use might be, based on particular widely shared values).

In terms of the second and third points – my book might be the most explicit in its discussion of the normative values guiding efforts to improve evidence, and I am more critical than some about the assumption that getting researchers work ‘used’ by policymakers is a de-facto good thing. This is why I disagree with Joshua’s conclusion that my work frames the problem as ‘bridging the gap’. Rather I’d say I frame the problem as asking the question of ‘what does a better system of evidence use look like from a political perspective?’ My ‘good governance of evidence’ discussion presents an explicitly normative framework based the two sets of values mentioned above – those around democratic accountability and around fidelity to scientific good practice – both of which have been raised as important in discussions about evidence use in political processes.

Is the onus on researchers?

Finally, I also would argue against Joshua’s conclusion that my work places the burden of resolving the problems on researchers. Paul argues above that he does this but with good reason. I try not to do this. This is again because my book is not making an argument for more evidence to be ‘used’ per se. (and I don’t expect policy makers to just want to use it either). Rather I focus on identifying principles by which we can judge systems of evidence use, calling for guided incremental changes within national systems.

While I think academics can play an important role in establishing ‘best practice’ ideas, I explicitly argue that the mandate to establish, build, or incrementally change evidence advisory systems lies with the representatives of the people. Indeed, I include ‘stewardship’ as a core principle of my good governance of evidence framework to show that it should be those individuals who are accountable to the public that build these systems in different countries. Thus, the burden lies not with academics, but rather with our representatives – and, indirectly with all of us through the demands we make on them – to improve systems of evidence use.

 

1 Comment

Filed under Evidence Based Policymaking (EBPM), Uncategorized

Debating the politics of evidence-based policy

Joshua Newman has provided an interesting review of three recent books on evidence/ policy (click here). One of those books is mine: The Politics of Evidence-Based Policy Making (which you can access here).

His review is very polite, for which I thank him. I hope my brief response can be seen in a similarly positive light (well, I had hoped to make it brief). Maybe we disagree on one or two things, but often these discussions are about the things we emphasize and the way we describe similar points.

There are 5 points to which I respond because I have 5 digits on my right hand. I’d like you to think of me counting them out on my fingers. In doing so, I’ll use ‘Newman’ throughout, because that’s the academic convention, but I’d also like to you imagine me reading my points aloud and whispering ‘Joshua’ before each ‘Newman’.

  1. Do we really need to ‘take the debate forward’ so often?

I use this phrase myself, knowingly, to keep a discussion catchy, but I think it’s often misleading. I suggest not to get your hopes up too high when Newman raises the possibility of taking the debate forward with his concluding questions. We won’t resolve the relationship between evidence, politics & policy by pretending to reframe the same collection of questions about the prospect of political reform that people have been asking for centuries. It is useful to envisage better political systems (the subject of Newman’s concluding remarks) but I don’t think we should pretend that this is a new concern or that it will get us very far.

Indeed, my usual argument is that researchers need to do something (such as improve how we engage in the policy process) while we wait for political system reforms to happen (while doubting if they will ever happen).

Further, Newman does not produce any political reforms to address the problems he raises. Rather, for example, he draws attention to Trump to describe modern democracies as ‘not pluralist utopias’ and to identify examples in which policymakers draw primarily on beliefs, not evidence. By restating these problems, he does not solve them. So, what are researchers supposed to do after they grow tired of complaining that the world does not meet their hopes or expectations?

In other words, for me, (a) promoting political change and (b) acting during its absence are two sides of the same coin. We go round and round more often than we take things forward.

  1. What debate are we renaming?

Newman’s ‘we’ve heard it before’ argument seems more useful, but there is a lot to hear and relatively few people have heard it. I’d warn against the assumption that ‘I’ve heard this before’ can ever equal ‘we’ve heard it before’ (unless ‘we’ refers to a tiny group of specialists talking only to each other).

Rather, one of the most important things we can do as academics is to tell the same story to each other (to check if we understand the same story, in the same way, and if it remains useful) and to wider audiences (in a way that they can pick up and use without dedicating their career to our discipline).

Some of our most important insights endure for decades and they sometimes improve in the retelling. We apply them to new eras, and often come to the same basic conclusions, but it seems unhelpful to criticise a lack of complete novelty in individual texts (particularly when they are often designed to be syntheses). Why not use them to occasionally take a step back to discuss and clarify what we know?

Perhaps more importantly, I don’t think Newman is correct when he says that each book retells the story of the ‘research utilization’ literature. I’m retelling the story of policy theory, which describes how policymakers deal with bounded rationality in a complex policymaking environment. Policy theory’s intellectual histories often provide very different perspectives – of the policymaker trying to make good enough decisions, rather than the researcher trying to improve the uptake of their research – than the agenda inspired by Weiss et al (see for example The New Policy Sciences).

  1. Don’t just ‘get political’; understand the policy process

I draw on policy theory because it helps people understand policymaking. It would be a mistake to conclude from my book that I simply want researchers to ‘get political’. Rather, I want them to develop useful knowledge of the policy process in which they might want to engage. This knowledge is not freely available; it takes time to understand the discipline and reflect on policy dynamics.

Yet, the payoff can be profound, if only because it helps people think about the difference between two analytically separate causes of a notional ‘evidence policy gap’: (a) individuals making choices based on their beliefs and limited information (which is relatively easy to understand but also to caricature), and (b) systemic or ‘environmental’ causes (which are far more difficult to conceptualise and explain, but often more useful to understand).

  1. Don’t throw out the ‘two communities’ phrase without explaining why

Newman criticises the phrase ‘two communities’ as a description of silos in policymaking versus research, partly because (a) many policymakers use research frequently, and (b) the real divide is often between users/ non-users of research within policymaking organisations. In short, there are more than two communities.

I’d back up his published research with my anecdotal experience of giving talks to government audiences: researchers and analysts within government are often very similar in outlook to academics and they often talk in the same way as academics about the disconnect between their (original or synthetic) research and its use by their ‘operational’ colleagues.

Still, I’m not sure why Newman concludes that the ‘two communities’ phrase is ‘deeply flawed and probably counter-productive’. Yes, the world is more nuanced and less binary than ‘two communities’ suggests. Yes, the real divide may be harder to spot. Still, as Newman et al suggest: ‘Policy makers and academics should focus on bridging instruments that can bring their worlds closer together’. This bullet point from their article seems, to me, to be the point of using the phrase ‘two communities’. Maybe Caplan used the phrase differently in 1979, but to assert its historic meaning then reject the phrase’s use in modern discussion seems less useful than simply clarifying the argument in ways such as:

  • There is no simple policymaker/ academic divide but, … note the major difference in requirements between (a) people who produce or distribute research without taking action, which allows them (for example) to be more comfortable with uncertainty, and (b) people who need to make choices despite having incomplete information to hand.
  • You might find a more receptive audience in one part of government (e.g. research/ analytical) than another (e.g. operational), so be careful about generalising from singular experiences.
  1. Should researchers engage in the policy process?

Newman says that each book, ‘unfairly places the burden of resolving the problem in the hands of an ill-equipped group of academics, operating outside the political system’.

I agree with Newman when he says that many researchers do not possess the skills to engage effectively in the policy process. Scientific training does not equip us with political skills. Indeed, I think you could read a few of my blog posts and conclude, reasonably, that you would like nothing more to do with the policy process because you’d be more effective by focusing on research.

The reason I put the onus back on researchers is because I am engaging with arguments like the one expressed by Newman (in other words, part of the meaning comes from the audience). Many people conclude their evidence policy discussions by identifying (or ‘reframing’) the problem primarily as the need for political reform. For me, the focus on other people changing to suit your preferences seems unrealistic and misplaced. In that context, I present the counter-argument that it may be better to adapt effectively to the policy process that exists, not the one you’d like to see. Sometimes it’s more useful to wear a coat than complain about the weather.

See also:  The Politics of Evidence 

The Politics of Evidence revisited

 

Pivot cover

1 Comment

Filed under Evidence Based Policymaking (EBPM), public policy

What do we need to know about the politics of evidence-based policymaking?

Today, I’m helping to deliver a new course – Engaging Policymakers Training Programme – piloted by the Alliance for Useful Evidence and the UCL. Right now, it’s for UCL staff (and mostly early career researchers). My bit is about how we can better understand the policy process so that we can engage in it more effectively.  I have reproduced the brief guide below (for my two 2-hour sessions as part of a wider block). If anyone else is delivering something similar, please let me know. We could compare notes. 

This module will be delivered in two parts to combine theory and practice

Part 1: What do we need to know about the politics of evidence-based policymaking?

Policy theories provide a wealth of knowledge about the role of evidence in policymaking systems. They prompt us to understand and respond to two key dynamics:

  1. Policymaker psychology. Policymakers combine rational and irrational shortcuts to gather information and make good enough decisions quickly. To appeal to rational shortcuts and minimise cognitive load, we reduce uncertainty by providing syntheses of the available evidence. To appeal to irrational shortcuts and engage emotional interest, we reduce ambiguity by telling stories or framing problems in specific ways.
  2. Complex policymaking environments. These processes take place in the context of a policy environment out of the control of individual policymakers. Environments consist of: many actors in many levels and types of government; engaging with institutions and networks, each with their own informal and formal rules; responding to socioeconomic conditions and events; and, learning how to engage with dominant ideas or beliefs about the nature of the policy problem. In other words, there is no policy cycle or obvious stage in which to get involved.

In this seminar, we discuss how to respond effectively to these dynamics. We focus on unresolved issues:

  1. Effective engagement with policymakers requires storytelling skills, but do we possess them?
  2. It requires a combination of evidence and emotional appeals, but is it ethical to do more than describe the evidence?
  3. The absence of a policy cycle, and presence of an ever-shifting context, requires us to engage for the long term, to form alliances, learn the rules, and build up trust in the messenger. However, do we have and how should we invest the time?

The format will be relatively informal. Cairney will begin by making some introductory points (not a powerpoint driven lecture) and encourage participants to relate the three questions to their research and engagement experience.

Gateway to further reading:

  • Paul Cairney and Richard Kwiatkowski (2017) ‘How to communicate effectively with policymakers: combine insights from psychology and policy studies’, Palgrave Communications
  • Paul Cairney and Kathryn Oliver (2017) ‘Evidence-based policymaking is not like evidence-based medicine, so how far should you go to bridge the divide between evidence and policy?’ Health Research Policy and Systems (HARPS), DOI: 10.1186/s12961-017-0192-x
  • Paul Cairney, Kathryn Oliver, and Adam Wellstead (2016) ‘To Bridge the Divide between Evidence and Policy: Reduce Ambiguity as Much as Uncertainty’, Public Administration Review, Early View (forthcoming) DOI:10.1111/puar.12555 PDF

Part 2: How can we respond pragmatically and effectively to the politics of EBPM?

In this seminar, we move from abstract theory and general advice to concrete examples and specific strategies. Each participant should come prepared to speak about their research and present a theoretically informed policy analysis in 3 minutes (without the aid of powerpoint). Their analysis should address:

  1. What policy problem does my research highlight?
  2. What are the most technically and politically feasible solutions?
  3. How should I engage in the policy process to highlight these problems and solutions?

After each presentation, each participant should be prepared to ask questions about the problem raised and the strategy to engage. Finally, to encourage learning, we will reflect on the memorability and impact of presentations.

Powerpoint: Paul Cairney A4UE UCL 2017

1 Comment

Filed under Evidence Based Policymaking (EBPM), Psychology Based Policy Studies, public policy