Tag Archives: defining policy

What does it take to turn scientific evidence into policy? Lessons for illegal drugs from tobacco

This post contains preliminary notes for my keynote speech ‘The politics of evidence-based policymaking’ for the COPOLAD annual conference, ‘From evidence to practice: challenges in the field of drugs policies’ (14th June). I may amend them in the run up to the speech (and during their translation into Spanish).

COPOLAD (Cooperation Programme on Drugs Policies) is a ‘partnership cooperation programme between the European Union, Latin America and the Caribbean countries aiming at improving the coherence, balance and impact of drugs policies, through the exchange of mutual experiences, bi-regional coordination and the promotion of multisectoral, comprehensive and coordinated responses’. It is financed by the EU.

My aim is to draw on policy studies, and the case study of tobacco/ public health policy, to identify four lessons:

  1. ‘Evidence-based policymaking’ is difficult to describe and understand, but we know it’s a highly political process which differs markedly from ‘evidence based medicine’.
  2. Actors focus as much on persuasion to reduce ambiguity as scientific evidence to reduce uncertainty. They also develop strategies to navigate complex policymaking ‘systems’ or ‘environments’.
  3. Tobacco policy demonstrates three conditions for the proportionate uptake of evidence: it helps ‘reframe’ a policy problem; it is used in an environment conducive to policy change; and, policymakers exploit ‘windows of opportunity’ for change.
  4. Even the ‘best cases’ of tobacco control highlight a gap of 20-30 years between the production of scientific evidence and a proportionate policy response. In many countries it could be 50. I’ll use this final insight to identify some scenarios on how evidence might be used in areas, such as drugs policy, in which many of the ‘best case’ conditions are not met.

‘Evidence-based policymaking’ is highly political and difficult to understand

Evidence-based policymaking (EBPM) is so difficult to understand that we don’t know how to define it or each word in it! People use phrases like ‘policy-based evidence’, to express cynicism about the sincere use of evidence to guide policy, or ‘evidence informed policy’, to highlight its often limited impact. It is more important to try to define each element of EBPM – to identify what counts as evidence, what is policy, who are the policymakers, and what an ‘evidence-based’ policy would look like – but this is easier said than done.

In fact, it is far easier to say what EBPM is not:

It is not ‘comprehensively rational’

Comprehensive rationality’ describes, in part, the absence of ambiguity and uncertainty:

  • Policymakers translate their values into policy in a straightforward manner – they know what they want and about the problem they seek to solve.
  • Policymakers and governments can gather and understand all information required to measure the problem and determine the effectiveness of solutions.

Instead, we talk of ‘bounded rationality’ and how policymakers deal with it. They employ two kinds of shortcut: ‘rational’, by pursuing clear goals and prioritizing certain kinds and sources of information, and ‘irrational’, by drawing on emotions, gut feelings, deeply held beliefs, habits, and what is familiar to them, to make decisions quickly.

It does not take place in a policy cycle with well-ordered stages

Policy cycle’ describes the ides that there is a core group of policymakers at the ‘centre’, making policy from the ‘top down’, and pursuing their goals in a series of clearly defined and well-ordered stages, such as: agenda setting, policy formulation, legitimation, implementation, and evaluation.

It does not describe or explain policymaking well. Instead, we tend to identify the role of environments or systems.

When describing less ordered and predictable policy environments, we describe:

  • a wide range of actors (individuals and organisations) influencing policy at many levels of government
  • a proliferation of rules and norms followed by different levels or types of government
  • important relationships (‘networks’) between policymakers and powerful actors (with material resources, or the ability to represent a profession or social group)
  • a tendency for certain ‘core beliefs’ or ‘paradigms’ to dominate discussion
  • shifting policy conditions and events that can prompt policymaker attention to lurch at short notice.

When describing complex policymaking systems we show that, for example, (a) the same inputs of evidence or policy activity can have no, or a huge, effect, and (b) policy outcomes often ‘emerge’ in the absence of central government control (which makes it difficult to know how, and to whom, to present evidence or try to influence).

It does not resemble ‘evidence based medicine’ or the public health culture

In health policy we can identify an aim, associated with ‘evidence-based medicine’ (EBM), to:

(a) gather the best evidence on the effectiveness of policy interventions, based on a hierarchy of research methods which favours, for example, the systematic review of randomised control trials (RCTs)

(b) ensure that this evidence has a direct impact on healthcare and public health, to exhort practitioners to replace bad interventions with good, as quickly as possible.

Instead, (a) policymakers can ignore the problems raised by scientific evidence for long periods of time, only for (b) their attention to lurch, prompting them to beg, borrow, or steal information quickly from readily available sources. This can involve many sources of evidence (such as the ‘grey literature’) that some scientists would not describe as reliable.

Actors focus as much on persuasion to reduce ambiguity as scientific evidence to reduce uncertainty.

In that context, ‘evidence-based policymaking’ is about framing problems and adapting to complexity.

Framing refers to the ways in which policymakers understand, portray, and categorise issues. Problems are multi-faceted, but bounded rationality limits the attention of policymakers, and actors compete to highlight one ‘image’ at the expense of others. The outcome of this process determines who is involved (for example, portraying an issue as technical limits involvement to experts), who is responsible for policy, how much attention they pay, their demand for evidence on policy solutions, and what kind of solution they favour.

Scientific evidence plays a part in this process, but we should not exaggerate the ability of scientists to win the day with reference to evidence. Rather, policy theories signal the strategies that actors adopt to increase demand for their evidence:

  • to combine facts with emotional appeals, to prompt lurches of policymaker attention from one policy image to another (punctuated equilibrium theory)
  • to tell simple stories which are easy to understand, help manipulate people’s biases, apportion praise and blame, and highlight the moral and political value of solutions (narrative policy framework)
  • to interpret new evidence through the lens of the pre-existing beliefs of actors within coalitions, some of which dominate policy networks (advocacy coalition framework)
  • to produce a policy solution that is feasible and exploit a time when policymakers have the opportunity to adopt it (multiple streams analysis).

This takes place in complex ‘systems’ or ‘environments’

A focus on this bigger picture shifts our attention from the use of evidence by an elite group of elected policymakers at the ‘top’ to its use by a wide range of influential actors in a multi-level policy process. It shows actors that:

  • They are competing with many others to present evidence in a particular way to secure a policymaker audience.
  • Support for particular solutions varies according to which organisation takes the lead and how it understands the problem.
  • Some networks are close-knit and difficult to access because bureaucracies have operating procedures that favour particular sources of evidence and some participants over others
  • There is a language – indicating which ideas, beliefs, or ways of thinking are most accepted by policymakers and their stakeholders – that takes time to learn.
  • Well-established beliefs provide the context for policymaking: new evidence on the effectiveness of a policy solution has to be accompanied by a shift of attention and successful persuasion.
  • In some cases, social or economic ‘crises’ can prompt lurches of attention from one issue to another, and some forms of evidence can be used to encourage that shift. However, major policy shifts are rare.

In other words, successful actors develop pragmatic strategies based on the policy process that exists, not the process they’d like to see

We argue that successful actors: identify where the ‘action is’ (in networks and organisations in several levels of government); learn and follow the ‘rules of the game’ within networks to improve strategies and help build up trust; form coalitions with actors with similar aims and beliefs; and, frame the evidence to appeal to the biases, beliefs, and priorities of policymakers.

Tobacco policy demonstrates three conditions for the proportionate uptake of evidence

Case studies allow us to turn this general argument into insights generated from areas such as public health.

There are some obvious and important differences between tobacco and (illegal) drugs policies, but an initial focus on tobacco allows us to consider the conditions that might have to be met to use the best evidence on a problem to promote (what we consider to be) a proportionate and effective solution.

We can then use the experience of a ‘best case scenario’ to identify the issues that we face in less ideal circumstances (first in tobacco, and second in drugs).

With colleagues, I have been examining:

Our studies help us identify the conditions under which scientific evidence, on the size of the tobacco problem and the effectiveness of solutions, translates into a public policy response that its advocates would consider to be proportionate.

  1. Actors are able to use scientific evidence to persuade policymakers to pay attention to, and shift their understanding of, policy problems.

Although scientific evidence helps reduce uncertainty, it does not reduce ambiguity. Rather, there is high competition to define problems, and the result of this competition helps determine the demand for subsequent evidence.

In tobacco, the evidence on smoking and then passive smoking helped raise attention to public health, but it took decades to translate into a proportionate response, even in ‘leading’ countries such as the UK.

The comparison with ‘laggard’ countries is crucial to show that the same evidence can produce a far more limited response, as policymakers compare the public health imperative with other ‘frames’, relating to their beliefs on personal responsibility, civil liberties, and the economic consequences of tobacco controls.

  1. The policy environment becomes conducive to policy change.

Public health debates take place in environments more or less conducive to policy change. In the UK, actors used scientific evidence to help reframe the problem. Then, this new understanding helped give the Department of Health a greater role, the health department fostered networks with public health and medical groups at the expense of the industry and, while pursuing policy change, policymakers emphasised the reduced opposition to tobacco control, smoking prevalence, and economic benefits to tobacco,.

In many other countries, these conditions are far less apparent: there are multiple tobacco frames (including economic and civil liberties); economic and trade departments are still central to policy; the industry remains a key player; and, policymakers pay more attention to opposition to tobacco controls (such as bans on smoking in public places) and their potential economic consequences.

Further, differences between countries have largely endured despite the fact that most countries are parties to the FCTC. In other words, a commitment to evidence basedpolicy transfer’ does not necessarily produce actual policy change.

  1. Actors generate and exploit ‘windows of opportunity’ for major policy change.

Even in favourable policy environments, it is not inevitable that major policy changes will occur. Rather, the UK’s experience of key policy instruments – such as legislation to ban smoking in public places (a major commitment of the FCTC) – shows the high level of serendipity involved in the confluence of three necessary but insufficient conditions:

  1. high policymaker attention to tobacco as a policy problem
  2. the production of solutions, introducing partial or comprehensive bans on smoking in public places, that are technically and politically feasible
  3. the willingness and ability of policymakers to choose the more restrictive solution.

In many other countries, there has been no such window of opportunity, or only an opportunity for a far weaker regulation.

So, this condition – the confluence of three ‘streams’ during a ‘window of opportunity’ – shows the major limits to the effect of scientific evidence. The evidence on the health effects of passive smoking have been available since the 1980s, but they only contributed to comprehensive smoking bans in the UK in the mid-2000s, and they remain unlikely in many other countries.

Comparing ‘best case’ and ‘worst case’ scenarios for policy change

These discussions help us clarify the kinds of conditions that need to be met to produce major ‘evidence based’ policy change, even when policymakers have made a commitment to it, or are pursuing an international agreement.

I provide a notional spectrum of ‘best’ and ‘worst’ case scenarios in relation to these conditions:

  1. Actors agree on how to gather and interpret scientific evidence.
  • Best case: governments fund effective ways to gather and interpret the most relevant evidence on the size of policy problems and the effectiveness of solutions. Policymakers can translate large amounts of evidence on complex situations into simple and effective stories (that everyone can understand) to guide action. This includes evidence of activity in one’s own country, and of transferable success from others.
  • Worst case: governments do not know the size of the problem or what solutions have the highest impacts. They rely on old stories that reinforce ineffective action, and do not know how to learn from the experience of other regions (note the ‘not invented hereissue).
  1. Actors ‘frame’ the problem simply and/or unambiguously.
  • Best case: governments maintain a consensus on how best to understand the cause of a policy problem and therefore which evidence to gather and solutions to seek.
  • Worst case: governments juggle many ‘frames’, there is unresolved competition to define the problem, and the best sources of evidence and solutions remain unclear.
  1. A new policy frame is not undermined by the old way of thinking about, and doing, things
  • Best case: the new frame sets the agenda for actors in existing organisations and networks; there is no inertia linked to the old way of thinking about and doing things.
  • Worst case: there is a new policy, but it is undermined by old beliefs, rules, pre-existing commitments (for example, we talk of ‘path dependence’ and ‘inheritance before choice’), or actors opposed to the new policy.
  1. There is a clear ‘delivery chain’ from policy choice to implementation
  • Best case: policymakers agree on a solution, they communicate their aims well, and they secure the cooperation of the actors crucial to policy delivery in many levels and types of government.
  • Worst case: policymakers communicate an ambiguous message and/ or the actors involved in policy delivery pursue different – and often contradictory – ways to try to solve the same problem.

In international cooperation, it is natural to anticipate and try to minimise at least some of these worst case scenarios. Problems are more difficult to solve when they are transnational. Our general sense of uncertainty and complexity is more apparent when there are many governments involved and we cannot rely on a single authoritative actor to solve problems. Each country (and regions within it) has its own beliefs and ways of doing things, and it is not easy to simply emulate another country (even if we think it is successful and know why). Some countries do not have access to the basic information (for example, on health and mortality, alongside statistics on criminal justice) that others take for granted when they monitor the effectiveness of policies.

Further, these obstacles exist in now-relatively-uncontroversial issues, such as tobacco, in which there is an international consensus on the cause of the problem and the appropriateness and effectiveness of public solutions. It is natural to anticipate further problems when we also apply public health (and, in this case, ‘harm reduction’) measures to more controversial areas such as illegal drugs.


Filed under agenda setting, Evidence Based Policymaking (EBPM), Prevention policy, Public health, public policy, tobacco, tobacco policy, UK politics and policy

Can you want evidence based policymaking if you don’t really know what it is?

This is an old post with a new title. It occurred to me that the phrase ‘evidence based policy making’ has a ‘bewitching’ effect. It seems impossible not to want it when you see the words in it. Yet, the problem is that not only do we not really know what it means, but we also do not know what each word means. I try to define them below, but these definitions don’t take us very far. ‘Evidence’ is assertion backed by information. ‘Based’ seems like a metaphor. ‘Policy’ is one of the worst-defined words in politics (up there with power and democracy). Policymaking implies there is a policymaker, but we don’t always know who it is.

This seems like just a semantic discussion, but I think that there is a lot of confusion in the EBPM literature simply because people begin by complaining that we don’t have it without really saying what we don’t have. A focus on things like ‘evidence informed’ doesn’t make things much better, because it is difficult to point to a policy and show how we would know if it related to the evidence, and to give it a mark out of ten. I think what people are often describing is the extent to which they feel that policymakers listen to what they have to say, and act on that basis. In some cases, people are quick to say that a policy is ‘not evidence based’ if policymakers only listen to some of what they have to say and/ or only adopt some of their recommendations (so it is less of a mark out of ten and more of a 0 or 1 code). For me, this is the most they could reasonably hope for – policy made with some regard to a wide range of the available evidence. It is less catchy, but it gives more equal status to the policymaking part of EBPM, in comparison with many accounts of EBPM which suggest that the policy process mangles *the* evidence; that we should judge the process’ merit according to how much the process dilutes scientific evidence.

I was re-reading this sentence …

“The term ‘evidence based policy making’ (EBPM) is in common currency in media and social media. Generally, it is a vague, aspirational term, rather than a good description of the policy process”

… and, before I knew it, I felt the need to define a whole bunch of other things in an end-note. Instead of biting my nails about these definitions, I am going to reproduce them here, and decide only to worry if anyone takes the time to object:

“Unfortunately, it is not the only term to be defined vaguely:

  • Policy. There is no single accepted definition of policy. I use the working definition ‘the sum total of government action, from signals of intent to the final outcomes’ to raise important qualifications: (a) it is problematic to conflate what people say they will do with what they actually do; (b) a policy outcome can be very different from the intention; (c) policy is made routinely through cooperation between elected and unelected policymakers and actors with no formal role in the process; (d) policymaking is also about the power not to do something (Cairney, 2012a: 24-5.
  • It is also important to identify the components or policy instruments that make up policies, including: the level of spending; the use of economic incentives/ penalties; regulations and laws; the use of voluntary agreements and codes of conduct; the provision of public services; education campaigns; funding for scientific studies or advocacy; organisational change; and, the levels of resources/ methods dedicated to policy implementation (2012a: 26).
  • Policymakers. The intuitive definition is ‘people who make policy’, but there are two important distinctions: (1) between elected and unelected people, since actors such as civil servants also make important decisions (2) between people and organisations, with the latter used as a shorthand to refer to a group of people making decisions collectively with reference to a decision-making rule (an ‘institution’). These distinctions are crucial, to remind us that advocates would miss something important if they focused their energies only on elected politicians. They also produce a conceptual problem by blurring the dividing line between the people who make and influence In most respects, this is appropriate, since terms such as ‘policy community’ suggest that policy decisions are made, in some sense, by a collection of people with formal responsibility and informal influence. We just need to be careful to make clear what we mean by ‘policymakers’ in specific discussions.
  • Evidence and scientific evidence. We can define evidence as an argument or assertion backed by information. Scientific evidence therefore describes information produced in a particular way. Some (including me) use ‘scientific’ broadly, to refer to information gathered systematically using recognised methods. Others refer to a specific hierarchy of scientific methods, with randomized control trials (RCTs) and quantitative systematic review/ meta-analysis at the top (see Nutley et al, 2013).
  • Also note the difference between two kinds of evidence-based activity relating to: the size of the problem (for example, the number of smokers and the link between smoking and ill health); and, the effectiveness of the solution (for example, the effect of higher taxes and health warnings on consumption).”

The endnote will appear in The Science of Policymaking

See also Defining policy shows how messed up it seems (as part of the 1000 words series)

‘The strange new world of evidence-free government’ by Zoe Williams

1 Comment

Filed under 1000 words, Evidence Based Policymaking (EBPM), public policy, UK politics and policy