This post contains preliminary notes for my keynote speech ‘The politics of evidence-based policymaking’ for the COPOLAD annual conference, ‘From evidence to practice: challenges in the field of drugs policies’ (14th June). I may amend them in the run up to the speech (and during their translation into Spanish).
COPOLAD (Cooperation Programme on Drugs Policies) is a ‘partnership cooperation programme between the European Union, Latin America and the Caribbean countries aiming at improving the coherence, balance and impact of drugs policies, through the exchange of mutual experiences, bi-regional coordination and the promotion of multisectoral, comprehensive and coordinated responses’. It is financed by the EU.
My aim is to draw on policy studies, and the case study of tobacco/ public health policy, to identify four lessons:
- ‘Evidence-based policymaking’ is difficult to describe and understand, but we know it’s a highly political process which differs markedly from ‘evidence based medicine’.
- Actors focus as much on persuasion to reduce ambiguity as scientific evidence to reduce uncertainty. They also develop strategies to navigate complex policymaking ‘systems’ or ‘environments’.
- Tobacco policy demonstrates three conditions for the proportionate uptake of evidence: it helps ‘reframe’ a policy problem; it is used in an environment conducive to policy change; and, policymakers exploit ‘windows of opportunity’ for change.
- Even the ‘best cases’ of tobacco control highlight a gap of 20-30 years between the production of scientific evidence and a proportionate policy response. In many countries it could be 50. I’ll use this final insight to identify some scenarios on how evidence might be used in areas, such as drugs policy, in which many of the ‘best case’ conditions are not met.
‘Evidence-based policymaking’ is highly political and difficult to understand
Evidence-based policymaking (EBPM) is so difficult to understand that we don’t know how to define it or each word in it! People use phrases like ‘policy-based evidence’, to express cynicism about the sincere use of evidence to guide policy, or ‘evidence informed policy’, to highlight its often limited impact. It is more important to try to define each element of EBPM – to identify what counts as evidence, what is policy, who are the policymakers, and what an ‘evidence-based’ policy would look like – but this is easier said than done.
In fact, it is far easier to say what EBPM is not:
It is not ‘comprehensively rational’
‘Comprehensive rationality’ describes, in part, the absence of ambiguity and uncertainty:
- Policymakers translate their values into policy in a straightforward manner – they know what they want and about the problem they seek to solve.
- Policymakers and governments can gather and understand all information required to measure the problem and determine the effectiveness of solutions.
Instead, we talk of ‘bounded rationality’ and how policymakers deal with it. They employ two kinds of shortcut: ‘rational’, by pursuing clear goals and prioritizing certain kinds and sources of information, and ‘irrational’, by drawing on emotions, gut feelings, deeply held beliefs, habits, and what is familiar to them, to make decisions quickly.
It does not take place in a policy cycle with well-ordered stages
‘Policy cycle’ describes the ides that there is a core group of policymakers at the ‘centre’, making policy from the ‘top down’, and pursuing their goals in a series of clearly defined and well-ordered stages, such as: agenda setting, policy formulation, legitimation, implementation, and evaluation.
It does not describe or explain policymaking well. Instead, we tend to identify the role of environments or systems.
When describing less ordered and predictable policy environments, we describe:
- a wide range of actors (individuals and organisations) influencing policy at many levels of government
- a proliferation of rules and norms followed by different levels or types of government
- important relationships (‘networks’) between policymakers and powerful actors (with material resources, or the ability to represent a profession or social group)
- a tendency for certain ‘core beliefs’ or ‘paradigms’ to dominate discussion
- shifting policy conditions and events that can prompt policymaker attention to lurch at short notice.
When describing complex policymaking systems we show that, for example, (a) the same inputs of evidence or policy activity can have no, or a huge, effect, and (b) policy outcomes often ‘emerge’ in the absence of central government control (which makes it difficult to know how, and to whom, to present evidence or try to influence).
It does not resemble ‘evidence based medicine’ or the public health culture
In health policy we can identify an aim, associated with ‘evidence-based medicine’ (EBM), to:
(a) gather the best evidence on the effectiveness of policy interventions, based on a hierarchy of research methods which favours, for example, the systematic review of randomised control trials (RCTs)
(b) ensure that this evidence has a direct impact on healthcare and public health, to exhort practitioners to replace bad interventions with good, as quickly as possible.
Instead, (a) policymakers can ignore the problems raised by scientific evidence for long periods of time, only for (b) their attention to lurch, prompting them to beg, borrow, or steal information quickly from readily available sources. This can involve many sources of evidence (such as the ‘grey literature’) that some scientists would not describe as reliable.
Actors focus as much on persuasion to reduce ambiguity as scientific evidence to reduce uncertainty.
In that context, ‘evidence-based policymaking’ is about framing problems and adapting to complexity.
Framing refers to the ways in which policymakers understand, portray, and categorise issues. Problems are multi-faceted, but bounded rationality limits the attention of policymakers, and actors compete to highlight one ‘image’ at the expense of others. The outcome of this process determines who is involved (for example, portraying an issue as technical limits involvement to experts), who is responsible for policy, how much attention they pay, their demand for evidence on policy solutions, and what kind of solution they favour.
Scientific evidence plays a part in this process, but we should not exaggerate the ability of scientists to win the day with reference to evidence. Rather, policy theories signal the strategies that actors adopt to increase demand for their evidence:
- to combine facts with emotional appeals, to prompt lurches of policymaker attention from one policy image to another (punctuated equilibrium theory)
- to tell simple stories which are easy to understand, help manipulate people’s biases, apportion praise and blame, and highlight the moral and political value of solutions (narrative policy framework)
- to interpret new evidence through the lens of the pre-existing beliefs of actors within coalitions, some of which dominate policy networks (advocacy coalition framework)
- to produce a policy solution that is feasible and exploit a time when policymakers have the opportunity to adopt it (multiple streams analysis).
This takes place in complex ‘systems’ or ‘environments’
A focus on this bigger picture shifts our attention from the use of evidence by an elite group of elected policymakers at the ‘top’ to its use by a wide range of influential actors in a multi-level policy process. It shows actors that:
- They are competing with many others to present evidence in a particular way to secure a policymaker audience.
- Support for particular solutions varies according to which organisation takes the lead and how it understands the problem.
- Some networks are close-knit and difficult to access because bureaucracies have operating procedures that favour particular sources of evidence and some participants over others
- There is a language – indicating which ideas, beliefs, or ways of thinking are most accepted by policymakers and their stakeholders – that takes time to learn.
- Well-established beliefs provide the context for policymaking: new evidence on the effectiveness of a policy solution has to be accompanied by a shift of attention and successful persuasion.
- In some cases, social or economic ‘crises’ can prompt lurches of attention from one issue to another, and some forms of evidence can be used to encourage that shift. However, major policy shifts are rare.
In other words, successful actors develop pragmatic strategies based on the policy process that exists, not the process they’d like to see
We argue that successful actors: identify where the ‘action is’ (in networks and organisations in several levels of government); learn and follow the ‘rules of the game’ within networks to improve strategies and help build up trust; form coalitions with actors with similar aims and beliefs; and, frame the evidence to appeal to the biases, beliefs, and priorities of policymakers.
Tobacco policy demonstrates three conditions for the proportionate uptake of evidence
Case studies allow us to turn this general argument into insights generated from areas such as public health.
There are some obvious and important differences between tobacco and (illegal) drugs policies, but an initial focus on tobacco allows us to consider the conditions that might have to be met to use the best evidence on a problem to promote (what we consider to be) a proportionate and effective solution.
We can then use the experience of a ‘best case scenario’ to identify the issues that we face in less ideal circumstances (first in tobacco, and second in drugs).
With colleagues, I have been examining:
- Why some countries have far more stringent tobacco control policies than others
- Why, although almost all countries in the world are parties to the same global tobacco policy (the WHO Framework Convention on Tobacco Control), the implementation of policy is highly uneven across the globe.
Our studies help us identify the conditions under which scientific evidence, on the size of the tobacco problem and the effectiveness of solutions, translates into a public policy response that its advocates would consider to be proportionate.
- Actors are able to use scientific evidence to persuade policymakers to pay attention to, and shift their understanding of, policy problems.
Although scientific evidence helps reduce uncertainty, it does not reduce ambiguity. Rather, there is high competition to define problems, and the result of this competition helps determine the demand for subsequent evidence.
In tobacco, the evidence on smoking and then passive smoking helped raise attention to public health, but it took decades to translate into a proportionate response, even in ‘leading’ countries such as the UK.
The comparison with ‘laggard’ countries is crucial to show that the same evidence can produce a far more limited response, as policymakers compare the public health imperative with other ‘frames’, relating to their beliefs on personal responsibility, civil liberties, and the economic consequences of tobacco controls.
- The policy environment becomes conducive to policy change.
Public health debates take place in environments more or less conducive to policy change. In the UK, actors used scientific evidence to help reframe the problem. Then, this new understanding helped give the Department of Health a greater role, the health department fostered networks with public health and medical groups at the expense of the industry and, while pursuing policy change, policymakers emphasised the reduced opposition to tobacco control, smoking prevalence, and economic benefits to tobacco,.
In many other countries, these conditions are far less apparent: there are multiple tobacco frames (including economic and civil liberties); economic and trade departments are still central to policy; the industry remains a key player; and, policymakers pay more attention to opposition to tobacco controls (such as bans on smoking in public places) and their potential economic consequences.
Further, differences between countries have largely endured despite the fact that most countries are parties to the FCTC. In other words, a commitment to evidence based ‘policy transfer’ does not necessarily produce actual policy change.
- Actors generate and exploit ‘windows of opportunity’ for major policy change.
Even in favourable policy environments, it is not inevitable that major policy changes will occur. Rather, the UK’s experience of key policy instruments – such as legislation to ban smoking in public places (a major commitment of the FCTC) – shows the high level of serendipity involved in the confluence of three necessary but insufficient conditions:
- high policymaker attention to tobacco as a policy problem
- the production of solutions, introducing partial or comprehensive bans on smoking in public places, that are technically and politically feasible
- the willingness and ability of policymakers to choose the more restrictive solution.
In many other countries, there has been no such window of opportunity, or only an opportunity for a far weaker regulation.
So, this condition – the confluence of three ‘streams’ during a ‘window of opportunity’ – shows the major limits to the effect of scientific evidence. The evidence on the health effects of passive smoking have been available since the 1980s, but they only contributed to comprehensive smoking bans in the UK in the mid-2000s, and they remain unlikely in many other countries.
Comparing ‘best case’ and ‘worst case’ scenarios for policy change
These discussions help us clarify the kinds of conditions that need to be met to produce major ‘evidence based’ policy change, even when policymakers have made a commitment to it, or are pursuing an international agreement.
I provide a notional spectrum of ‘best’ and ‘worst’ case scenarios in relation to these conditions:
- Actors agree on how to gather and interpret scientific evidence.
- Best case: governments fund effective ways to gather and interpret the most relevant evidence on the size of policy problems and the effectiveness of solutions. Policymakers can translate large amounts of evidence on complex situations into simple and effective stories (that everyone can understand) to guide action. This includes evidence of activity in one’s own country, and of transferable success from others.
- Worst case: governments do not know the size of the problem or what solutions have the highest impacts. They rely on old stories that reinforce ineffective action, and do not know how to learn from the experience of other regions (note the ‘not invented here’ issue).
- Actors ‘frame’ the problem simply and/or unambiguously.
- Best case: governments maintain a consensus on how best to understand the cause of a policy problem and therefore which evidence to gather and solutions to seek.
- Worst case: governments juggle many ‘frames’, there is unresolved competition to define the problem, and the best sources of evidence and solutions remain unclear.
- A new policy frame is not undermined by the old way of thinking about, and doing, things
- Best case: the new frame sets the agenda for actors in existing organisations and networks; there is no inertia linked to the old way of thinking about and doing things.
- Worst case: there is a new policy, but it is undermined by old beliefs, rules, pre-existing commitments (for example, we talk of ‘path dependence’ and ‘inheritance before choice’), or actors opposed to the new policy.
- There is a clear ‘delivery chain’ from policy choice to implementation
- Best case: policymakers agree on a solution, they communicate their aims well, and they secure the cooperation of the actors crucial to policy delivery in many levels and types of government.
- Worst case: policymakers communicate an ambiguous message and/ or the actors involved in policy delivery pursue different – and often contradictory – ways to try to solve the same problem.
In international cooperation, it is natural to anticipate and try to minimise at least some of these worst case scenarios. Problems are more difficult to solve when they are transnational. Our general sense of uncertainty and complexity is more apparent when there are many governments involved and we cannot rely on a single authoritative actor to solve problems. Each country (and regions within it) has its own beliefs and ways of doing things, and it is not easy to simply emulate another country (even if we think it is successful and know why). Some countries do not have access to the basic information (for example, on health and mortality, alongside statistics on criminal justice) that others take for granted when they monitor the effectiveness of policies.
Further, these obstacles exist in now-relatively-uncontroversial issues, such as tobacco, in which there is an international consensus on the cause of the problem and the appropriateness and effectiveness of public solutions. It is natural to anticipate further problems when we also apply public health (and, in this case, ‘harm reduction’) measures to more controversial areas such as illegal drugs.