‘Evidence-based policy making’ (EBPM) is a vague, aspirational term, rather than a good description of the policy process. It can be interpreted in very different ways. At one extreme is the naïve view that there can and should be a direct and unproblematic link between ‘the evidence’ and policy decisions and outcomes. At the other is the policy-based-evidence view that politics is so corrupt that no decision is based on an appeal to scientific evidence, or so messy that the evidence gets lost somewhere in the political process. A more useful approach is to argue that EBPM is an ideal-type to compare with the real world, in much the same way as we describe a ‘comprehensively rational’ policymaker. Then, we can draw on the policy literature which provides a wide range of theories to help situate the role of evidence within a complex policymaking system. Evidence may be important but, to identify and gauge its role, we must understand how it fits into the bigger picture in which ‘boundedly rational’ policymakers make choices based on limited information and ambiguity. This takes place in the context of a policy environment which influences how they act and how much control they have over the final outcomes. Images of highly centralised and ‘rational’ policymaking by a small number of actors generally give way to pictures of less predictable or manageable multi-level governance involving many actors. The implication, particularly in Westminster systems, is that we should change how we see the role of evidence: from focusing on its use by policymakers at the ‘top’, or at a notional point of decision, towards explaining how it is used throughout the political system as a whole.
Monthly Archives: February 2014
Evidence Based Policy Making: If You Want to Inject More Science into Policymaking You Need to Know the Science of Policymaking
This post also appears on the ESRC website
The Scottish Centre on Constitutional Change does not simply examine the potential for a major event, Scottish independence, to have a major impact on Scottish politics. It also focuses on policymaking, which often exhibits stability and continuity despite major events. For example, the idea of evidence based policymaking (EBPM) represents a broad aim that the Scottish Government would pursue regardless of its constitutional status. The Scottish Government is also likely to continue its focus on a ‘decisive shift to prevention’ – to pursue ‘early interventions’ in people’s lives and address potential social problems before they produce high demand for acute, responsive public services – because the broad idea generates wide party political and public sector support.
Policymaking is about turning these broad aims into specific policies. In our paper, we set out a framework to think this process through, from identifying an aim, to using evidence to inform the selection of projects to fund and support. Some specific policies may differ under a devolved or independent Scotland, but the thought process stays the same.
First, we set out our aims. A clear definition of prevention and a set of detailed aims aids engagement with the public and the government’s policymaking partners. It helps produce a common set of expectations. It helps maximise the ability of a government to identify the most relevant evidence, when it seeks to learn from pilot projects and international experience.
This is difficult to do with ‘prevention’, which can mean anything from inoculating a whole population against a virus, to providing ‘crisis intervention’ to a small group. It can mean providing education and support or regulating behaviour. It is about identifying the root causes of problems, which is easier for many diseases than ‘wicked’ social problems. It is about how ‘decisive’ you want to be in a short space of time, and how you want to balance aims, such as between reducing inequalities or costs (at least when you can’t do both).
The Scottish Government addresses this problem through the National Performance Framework, which provides broad strategic objectives combined with a set of measures to gauge the success of prevention (and other) policies. It then invites local authorities, in partnership with public bodies, the voluntary and private sector, and service users – through Community Planning Partnerships – to produce Single Outcome Agreements which describe how these objectives will be met in each local area (there are 32).
In this sense, the Scottish Government has two broad aims: to pursue prevention projects and to make policy in a particular way, summed up in terms like ‘co-production’. Prevention policy results from Scottish Government direction, coupled with local plans produced in partnership with a range of participants. These aims can reinforce each other, when cooperation produces a high commitment to co-produced objectives, and/or undermine each other, when a proliferation of plans produces a wide range of meanings of prevention.
Second, we seek evidence to help clarify our aims and inform our decisions. Evidence can be drawn from past experience, pilot projects in our own country or policies pursued in others. The Scottish Government has an interesting double-role: to identify what evidence is relevant, based on its own aims; and, to translate evidence into lessons that are relevant to other participants. A commitment to co-production creates a commitment to understand what drives local policymakers and what information is relevant to them, given the constraints they face (including budget and time) and what drives them to act. It also prompts us to focus as much on the evidence of successful delivery arrangements as successful policy interventions. This is a fundamental issue when we seek to learn lessons from other projects: are we learning about the policy solution or the way in which the solution is understood and used, more or less competently, in particular areas?
Third, we use our judgement. The identification of ‘success’ is a political judgement based on our aims and beliefs. Many projects quickly develop good reputations based on these, rather than ‘scientific’, criteria. We outline a framework to ensure that a sufficient amount of relevant evidence is gathered before a project is deemed successful. However, many projects may not live up to this minimum standard, particularly if we want to learn quickly to address pressing problems. Consequently, evidence-gathering is no substitute for political choice based on limited evidence – and that choice may vary markedly across 32 local areas. This is evidence-based policymaking for the real world.
The full paper, to be presented to the Scottish Government on the 24th February, is here: Cairney St Denny Prevention Paper 21.2.14
Evidence based policymaking (EBPM) is a great idea, isn’t it? Who could object to it, apart from the enemies of science? Well, I’m sort-of going to object in two ways, by arguing: that we partly like it so much because it’s a vague idea and we don’t know what it means; and that when we are clearer on its meaning, one type of EBPM seems very problematic indeed.
Carol Weiss gives us a menu of sensible EBPM options from which to choose. Evidence can be used:
- to inform solutions to a problem identified by policymakers
- as one of many sources of information used by policymakers, alongside ‘stakeholder’ advice and professional and service user experience
- as a resource used selectively by politicians, with entrenched positions, to bolster their case
- as a tool of government, to show it is acting (by setting up a scientific study), or to measure how well policy is working
- as a source of ‘enlightenment’, shaping how people think over the long term.
In other words, these options provide a description of the use of evidence within an often messy political system: scientists may have a role, but they struggle for attention alongside many other people. This is where a separate definition of EBPM comes in, often as a prescription for the policy process: there should be a much closer link between the process in which scientists identify major policy problems with evidence, and the process in which politicians make policy decisions. We should seek to close the ‘evidence-policy gap’. The evidence should come first and we should bemoan the inability of policymakers to act accordingly.
Most policy science-based studies of EBPM would reject this idea on descriptive grounds – as a rather naïve view about the policy process. In that sense, the call for EBPM is a revival of the idea of ‘comprehensive rationality’ in policymaking – which describes an ‘ideal-type’ and, in one sense, an optimal policy process. We assume that the values of society are reflected in the values of policymakers, and that a small number of policymakers control the policy process from its centre. Then, we highlight the conditions that would have to be met to allow those policymakers to use the government machine to turn those aims into policies – we can separate facts from values, organisations can rank a government’s preferences, the policy process is ‘linear’ and separated into clear stages, and analysis of the world is comprehensive. The point of this ideal-type is that it doesn’t exist. Instead, policy theory is about providing more realistic descriptions of the world.
On that basis, we might argue that scientists should quit moaning about the real world and start adapting to it. Stop bemoaning the pathologies of public policy – and some vague notion of the ‘lack of political will’ – and hoping for something better. If the policy process is messy and unpredictable, be pragmatic about how to engage. Balance the desire to produce a direct evidence-policy effect with a realisation that we need to frame the evidence to make it attractive to actors with very different ideas and incentives to act. Accept that policymakers seek many other legitimate sources of information and knowledge, and do not recognise, in the same way, your evidential hierarchies favouring RCTs and systematic reviews.
Even so, should we still secretly fantasise about the idealistic prescriptive side? Is EBPM an ‘ideal’, or something to aspire to even though it is unrealistic? Not if it means something akin to comprehensive rationality. Look again at the assumptions – one of which is that a small number of policymakers control the policy process from its centre. In this scenario, EBPM is about closing the evidence policy gap by providing a clear link between scientists and politicians who centralise policymaking and make policy from the top-down with little role for debate, consultation and other forms of knowledge (one might call this ‘leadership’, often in the face of public opinion). This raises a potentially fundamental tension between EBPM and other sources of ‘good’ policymaking. What if the acceptance of one form of EBPM undermines the other roles of government?
A government may legitimately adopt a ‘bottom up’ approach to policymaking and delivery – consulting widely with a range of interest groups and public bodies to inform its aims, and working in partnership with those groups to deliver policy (perhaps by using long term, ‘co-produced’ outcomes rather than top-down and short term targets to measure success). This approach has important benefits – it generates wide ‘ownership’ of a policy solution and allows governments to generate useful feedback on the effects of policy instruments (which is important since, in practice, it may be impossible to separate the effect of an instrument from the effect of the way in which it was implemented).
If so, it would be difficult to maintain a separate EBPM process in which the central government commissions and receives the evidence which directly informs its aims to be carried out elsewhere. If a government is committed to a bottom-up policy style, it seems inevitable that it would adopt the same approach to evidence – sharing it with a wide range of bodies and ‘co-producing’ a response. If so, the use of evidence becomes much less like a linear and simple process, and much more like a complicated and interactive process in which many actors negotiate the practical implications of scientific evidence – considering it alongside other sources of policy relevant information. This has the potential to take us away from the idea of evidence-driven policy, based on external scientific standards and ‘objective’ evidence based on a hierarchy of methods, towards treating evidence as a resource to be used by actors within political systems who draw on different ideas about the hierarchy of evidential sources. As such, ‘the evidence’ is not a resource that is controlled by the scientists producing the information.
From there, we might ask: is this still EBPM? Well, this takes us back to what it means. If it means that a ‘scientific consensus’ should have super-direct policy effects, then no. If it means that scientists provide information to inform the deliberations of policymakers, who claim a legitimate policymaking role, and engage in other forms of ‘good’ policymaking – by consulting widely and generating a degree of societal, governmental and/or practitioner consensus – then yes.
Full paper (quite old): Cairney PSA 2014 EBPM 28.2.14
“David Cameron gave a major speech today on the referendum on Scottish Independence. It is already being dubbed the ‘phone a friend’ speech because he encouraged people in the rest of the UK to phone their friends in Scotland to remind them how great it is to be a part of the UK. He wants you to phone me. He wants you to pretend that you pressed the wrong button, and then say that we may as well have a bit of a catch up since we’re on the phone anyway. Then he wants you to ask in a vague way how the family is, say ‘mmhmm’ a few times, act interested, then bring it round to how passionate David Cameron feels about the UK. He wants you to tell me that David Cameron has some Scottish ancestry and that his name sounds a bit Scottishy, so that must count for something, right? He wants you to remind me about the Olympics: how British we all felt as well won all those gold medals and, to a lesser extent, how British we felt when we won silver and bronze. He wants to get serious for a minute and to be hard headed about it – to agree that the Olympic success is a symbol of the UK as a whole, with friendships, business, politics and our reputation across the globe all better as a United Kingdom rather than as separate countries. We’re Better Together, aren’t we mate? Then he wants us to bring it down and get a bit emotional, to agree that we all love each other too much to let each other go. Then he wants to make it about him again; for you to tell me that he is passionate about Scotland’s place in the UK. He loves it. He loves Scotland, Great Britain, Scotland and Britain and Northern Ireland, and the UK best of all. Then he wants you to kick off a vague discussion about British values, as long as you don’t get too UKIPy about it and talk about foreigners. Then he wants us to hug, somehow, and tell each other ‘you’re my best mate’. Then I think we’re supposed to promise to keep in touch much more than we used to; maybe go for a coffee next time you’re in Edinburgh or I’m in London.
The only problem is that I’m just off the phone with Alex, who tells me that David Cameron won’t answer his calls. He wants me to tell you that he can’t understand why Cameron won’t come up to Scotland for a coffee and maybe, while he’s there, to have a brilliant debate about Scottish independence on their own, just the two of them, except that everybody will be watching (although in the background I could hear Nicola shouting ‘naw, just tell him to tell Cameron to just keep his big nose out of it’). He’s just spent ten minutes taking the piss out of Cameron for being obsessed with London and only talking about the rest of the UK when he remembers for a second and has a bit of a panic about it. Then he wants me to tell you that we’ve grown apart; that we don’t talk as much as we used to (and, when we do, we don’t agree as much as we used to) because we’ve both been too busy and don’t have as much time for each other as we used to, when we were better friends. He wants me to say that it’s OK we don’t talk much, but that we should stop pretending we’ll keep in touch. We’ll always have the memories, but it’s time to move on. He hopes that we won’t get too silly when we talk about dividing up our assets and keeping things we shared, like the pound and the Queen and the BBC, because we’ll still be friends after all. Then he wants us to hug and say goodbye.”