Evidence Based Policy Making: If You Want to Inject More Science into Policymaking You Need to Know the Science of Policymaking

‘Evidence-based policy making’ (EBPM) is a vague, aspirational term, rather than a good description of the policy process. It can be interpreted in very different ways. At one extreme is the naïve view that there can and should be a direct and unproblematic link between ‘the evidence’ and policy decisions and outcomes. At the other is the policy-based-evidence view that politics is so corrupt that no decision is based on an appeal to scientific evidence, or so messy that the evidence gets lost somewhere in the political process. A more useful approach is to argue that EBPM is an ideal-type to compare with the real world, in much the same way as we describe a ‘comprehensively rational’ policymaker.  Then, we can draw on the policy literature which provides a wide range of theories to help situate the role of evidence within a complex policymaking system. Evidence may be important but, to identify and gauge its role, we must understand how it fits into the bigger picture in which ‘boundedly rational’ policymakers make choices based on limited information and ambiguity. This takes place in the context of a policy environment which influences how they act and how much control they have over the final outcomes. Images of highly centralised and ‘rational’ policymaking by a small number of actors generally give way to pictures of less predictable or manageable multi-level governance involving many actors.  The implication, particularly in Westminster systems, is that we should change how we see the role of evidence: from focusing on its use by policymakers at the ‘top’, or at a notional point of decision, towards explaining how it is used throughout the political system as a whole.

Full paper: Cairney PSA 2014 EBPM 5.3.14 and panel: https://www.psa.ac.uk/conference/2014-conference/use-evidence-impact-and-policymaking

See also: Is Evidence-Based Policymaking the same as good policymaking?

Four obstacles to evidence based policymaking (EBPM)

The Psychology of Evidence Based Policymaking: Who Will Speak For the Evidence if it Doesn’t Speak for Itself?

A ‘decisive shift to prevention’: how do we turn an idea into evidence based policy?

Policy Concepts in 1000 Words: ‘Evidence Based Policymaking’

A ‘decisive shift to prevention’: how do we turn an idea into evidence based policy?

This post also appears on the ESRC website

The Scottish Centre on Constitutional Change does not simply examine the potential for a major event, Scottish independence, to have a major impact on Scottish politics. It also focuses on policymaking, which often exhibits stability and continuity despite major events. For example, the idea of evidence based policymaking (EBPM) represents a broad aim that the Scottish Government would pursue regardless of its constitutional status. The Scottish Government is also likely to continue its focus on a ‘decisive shift to prevention’ – to pursue ‘early interventions’ in people’s lives and address potential social problems before they produce high demand for acute, responsive public services – because the broad idea generates wide party political and public sector support.

Policymaking is about turning these broad aims into specific policies. In our paper, we set out a framework to think this process through, from identifying an aim, to using evidence to inform the selection of projects to fund and support. Some specific policies may differ under a devolved or independent Scotland, but the thought process stays the same.

First, we set out our aims. A clear definition of prevention and a set of detailed aims aids engagement with the public and the government’s policymaking partners. It helps produce a common set of expectations. It helps maximise the ability of a government to identify the most relevant evidence, when it seeks to learn from pilot projects and international experience.

This is difficult to do with ‘prevention’, which can mean anything from inoculating a whole population against a virus, to providing ‘crisis intervention’ to a small group. It can mean providing education and support or regulating behaviour. It is about identifying the root causes of problems, which is easier for many diseases than ‘wicked’ social problems. It is about how ‘decisive’ you want to be in a short space of time, and how you want to balance aims, such as between reducing inequalities or costs (at least when you can’t do both).

The Scottish Government addresses this problem through the National Performance Framework, which provides broad strategic objectives combined with a set of measures to gauge the success of prevention (and other) policies. It then invites local authorities, in partnership with public bodies, the voluntary and private sector, and service users – through Community Planning Partnerships – to produce Single Outcome Agreements which describe how these objectives will be met in each local area (there are 32).

In this sense, the Scottish Government has two broad aims: to pursue prevention projects and to make policy in a particular way, summed up in terms like ‘co-production’. Prevention policy results from Scottish Government direction, coupled with local plans produced in partnership with a range of participants. These aims can reinforce each other, when cooperation produces a high commitment to co-produced objectives, and/or undermine each other, when a proliferation of plans produces a wide range of meanings of prevention.

Second, we seek evidence to help clarify our aims and inform our decisions. Evidence can be drawn from past experience, pilot projects in our own country or policies pursued in others. The Scottish Government has an interesting double-role: to identify what evidence is relevant, based on its own aims; and, to translate evidence into lessons that are relevant to other participants. A commitment to co-production creates a commitment to  understand what drives local policymakers and what information is relevant to them, given the constraints they face (including budget and time) and what drives them to act. It also prompts us to focus as much on the evidence of successful delivery arrangements as successful policy interventions. This is a fundamental issue when we seek to learn lessons from other projects: are we learning about the policy solution or the way in which the solution is understood and used, more or less competently, in particular areas?

Third, we use our judgement. The identification of ‘success’ is a political judgement based on our aims and beliefs. Many projects quickly develop good reputations based on these, rather than ‘scientific’, criteria. We outline a framework to ensure that a sufficient amount of relevant evidence is gathered before a project is deemed successful. However, many projects may not live up to this minimum standard, particularly if we want to learn quickly to address pressing problems. Consequently, evidence-gathering is no substitute for political choice based on limited evidence – and that choice may vary markedly across 32 local areas. This is evidence-based policymaking for the real world.

The full paper, to be presented to the Scottish Government on the 24th February, is here: Cairney St Denny Prevention Paper 21.2.14

Is Evidence-Based Policymaking the same as good policymaking?

Evidence based policymaking (EBPM) is a great idea, isn’t it? Who could object to it, apart from the enemies of science? Well, I’m sort-of going to object in two ways, by arguing: that we partly like it so much because it’s a vague idea and we don’t know what it means; and that when we are clearer on its meaning, one type of EBPM seems very problematic indeed.

Carol Weiss gives us a menu of sensible EBPM options from which to choose. Evidence can be used:

• to inform solutions to a problem identified by policymakers
• as one of many sources of information used by policymakers, alongside ‘stakeholder’ advice and professional and service user experience
• as a resource used selectively by politicians, with entrenched positions, to bolster their case
• as a tool of government, to show it is acting (by setting up a scientific study), or to measure how well policy is working
• as a source of ‘enlightenment’, shaping how people think over the long term.

In other words, these options provide a description of the use of evidence within an often messy political system: scientists may have a role, but they struggle for attention alongside many other people. This is where a separate definition of EBPM comes in, often as a prescription for the policy process: there should be a much closer link between the process in which scientists identify major policy problems with evidence, and the process in which politicians make policy decisions. We should seek to close the ‘evidence-policy gap’. The evidence should come first and we should bemoan the inability of policymakers to act accordingly.

Most policy science-based studies of EBPM would reject this idea on descriptive grounds – as a rather naïve view about the policy process. In that sense, the call for EBPM is a revival of the idea of ‘comprehensive rationality’ in policymaking – which describes an ‘ideal-type’ and, in one sense, an optimal policy process. We assume that the values of society are reflected in the values of policymakers, and that a small number of policymakers control the policy process from its centre. Then, we highlight the conditions that would have to be met to allow those policymakers to use the government machine to turn those aims into policies – we can separate facts from values, organisations can rank a government’s preferences, the policy process is ‘linear’ and separated into clear stages, and analysis of the world is comprehensive. The point of this ideal-type is that it doesn’t exist. Instead, policy theory is about providing more realistic descriptions of the world.

On that basis, we might argue that scientists should quit moaning about the real world and start adapting to it. Stop bemoaning the pathologies of public policy – and some vague notion of the ‘lack of political will’ – and hoping for something better. If the policy process is messy and unpredictable, be pragmatic about how to engage. Balance the desire to produce a direct evidence-policy effect with a realisation that we need to frame the evidence to make it attractive to actors with very different ideas and incentives to act. Accept that policymakers seek many other legitimate sources of information and knowledge, and do not recognise, in the same way, your evidential hierarchies favouring RCTs and systematic reviews.

Even so, should we still secretly fantasise about the idealistic prescriptive side? Is EBPM an ‘ideal’, or something to aspire to even though it is unrealistic?  Not if it means something akin to comprehensive rationality. Look again at the assumptions – one of which is that a small number of policymakers control the policy process from its centre. In this scenario, EBPM is about closing the evidence policy gap by providing a clear link between scientists and politicians who centralise policymaking and make policy from the top-down with little role for debate, consultation and other forms of knowledge (one might call this ‘leadership’, often in the face of public opinion). This raises a potentially fundamental tension between EBPM and other sources of ‘good’ policymaking. What if the acceptance of one form of EBPM undermines the other roles of government?

A government may legitimately adopt a ‘bottom up’ approach to policymaking and delivery – consulting widely with a range of interest groups and public bodies to inform its aims, and working in partnership with those groups to deliver policy (perhaps by using long term, ‘co-produced’ outcomes rather than top-down and short term targets to measure success). This approach has important benefits – it generates wide ‘ownership’ of a policy solution and allows governments to generate useful feedback on the effects of policy instruments (which is important since, in practice, it may be impossible to separate the effect of an instrument from the effect of the way in which it was implemented).

If so, it would be difficult to maintain a separate EBPM process in which the central government commissions and receives the evidence which directly informs its aims to be carried out elsewhere. If a government is committed to a bottom-up policy style, it seems inevitable that it would adopt the same approach to evidence – sharing it with a wide range of bodies and ‘co-producing’ a response. If so, the use of evidence becomes much less like a linear and simple process, and much more like a complicated and interactive process in which many actors negotiate the practical implications of scientific evidence – considering it alongside other sources of policy relevant information. This has the potential to take us away from the idea of evidence-driven policy, based on external scientific standards and ‘objective’ evidence based on a hierarchy of methods, towards treating evidence as a resource to be used by actors within political systems who draw on different ideas about the hierarchy of evidential sources. As such, ‘the evidence’ is not a resource that is controlled by the scientists producing the information.

From there, we might ask: is this still EBPM? Well, this takes us back to what it means. If it means that a ‘scientific consensus’ should have super-direct policy effects, then no. If it means that scientists provide information to inform the deliberations of policymakers, who claim a legitimate policymaking role, and engage in other forms of ‘good’ policymaking – by consulting widely and generating a degree of societal, governmental and/or practitioner consensus – then yes.

Full paper (quite old): Cairney PSA 2014 EBPM 28.2.14

Full book (less old) Paul Cairney (2016) The Politics of Evidence-based Policymaking (London: Palgrave Pivot) PDF (see also)

See also: Policy Concepts in 1000 Words: ‘Evidence Based Policymaking’

Policy Concepts in 1000 Words: Bounded Rationality and Incrementalism

Can the Scottish Government pursue ‘prevention policy’ without independence?

This appears on the ESRC website too

The Scottish Government’s White Paper on independence, Scotland’s Future, reignited arguments about the adequacy of the devolved settlement. While the Scottish Government argues that it could pursue a wide range of new policies under independence, its critics argue that many of the proposed changes – including an expansion of childcare and a devolved response to the so-called ‘bedroom tax’ – could be done by a devolved government.  The debate exposes uncertainty regarding the limits to devolved powers, particularly when the Scottish Government is addressing ‘cross-cutting’ policy issues which often have reserved and devolved elements.

‘Prevention’ policy is a key example. ‘Prevention’ and ‘preventative spending’ are very broad terms to describe ‘early interventions’ in people’s lives, to address potential social problems before they produce high demand for acute, responsive public services. Examples range from disease inoculation, to public health interventions to change unhealthy behaviour, investment in early years education and welfare, measures to reduce crime reoffending, and measures designed to reduce inequalities associated with deprivation and unemployment. Scotland’s Future also places particular emphasis on the production of prevention policies in local areas by a range of public bodies consulting with non-governmental organisations and the public:

‘Scotland has already adopted a distinctive approach in our public services, focused on improving outcomes and building the assets and resilience of people and communities, through prevention and early intervention. It values collaboration by those involved – by people and communities, Third Sector, public providers and businesses. Our approach crucially recognises the importance of designing services with, and for, the people they are there to serve, and of building on our strengths’ (p359).

In 2011, the Scottish Government’s response to the Christie Commission’s report on prevention demonstrated that a great deal can be done by a devolved government. It listed a range of (over 50) interventions and 2011-16 priorities, including a focus on early years (and poverty) investment, class sizes and curriculum reform, employment training, tobacco, drug and alcohol control, ‘inequalities-targeted health checks’, alternatives to short-term custodial sentences, affordable housing, energy assistance and community-based carbon emissions reduction projects. It also announced three new funds, representing £500m ‘investment in preventative spending’ in older people’s services, early years, and reducing reoffending.

In that context, why might the Scottish Government benefit from further devolution or independence? First, the UK Government decides the size of the Scottish Government’s budget and its borrowing capacity. A Scottish Government in control of its own budget could decide to shift resources from some areas to invest in prevention initiatives (some may have high start-up costs, to be recouped over the long term).

Second, the UK Government controls taxation and social security policy, and its reforms have the potential to disrupt ‘joined up’ public service activity in Scotland. For example, housing policy involves a combination of benefits and services and, for example, the Scottish Government has devoted much political energy to reject the ‘bedroom tax’. Previously, the most frequently discussed example was ‘free personal care’ for older people – one part of which involved the Scottish Government paying local authorities to provide personal care at home, which removed the recipients’ entitlement to a UK benefit, Attendance Allowance. The Scottish Government requested that the savings to HM Treasury be passed back to Scotland, but the request was denied by the Treasury and the matter was never resolved fully. In such cases, the Scottish Government may argue that it has less incentive to produce a ‘joined up’ approach, since its spending and delivery may be undermined by UK rules.

The Scottish Government also argues that some post-independence policies will become self-supporting through greater economic activity. For example, it wants to fund its major investment in childcare by borrowing to invest in facilities and training, then recouping the money in more taxes and fewer benefits as more people go to work.

In some cases, further devolution may be enough to make an impact. For example, the Christie Commission (pp p57-8) called for the devolution of ‘employability’ policy:

‘In the course of our meetings across Scotland it has become clear that the interface between reserved and devolved policies on employability (i.e. job search and support services) has compromised the achievement of positive outcomes. Particular concerns were expressed about a ‘one-size fits all’ approach on the part of the Department for Work and Pensions (DWP) and Jobcentre Plus; about the ways in which programmes are contracted out from Whitehall; and about the extent to which DWP and Jobcentre Plus services are coordinated with devolved public services at the local level. We recommend the full devolution of competence for job search and support to the Scottish Parliament to achieve the integration of service provision in the area of employability’.

Scotland’s Future (p108) also talks about the use of ‘employment services … built on the principle of “early intervention” … to prevent individuals from becoming long-term unemployed with all of the associated problems for individuals and for society’.

In other cases, the Scottish Government argument may be that everything is connected; one cannot fully act in one area without the freedom to act in another. For example, to address health, ‘we also need responsibility for our society’s wellbeing and welfare. The solution to ill-health is not in the hands of the NHS alone – it depends on breaking the cycle of poverty, educational underattainment, worklessness, poor mental wellbeing, and, through these, preventable ill-health’ (p136). Similarly, it argues that it cannot ensure joined-up government without being responsible for all government, and so it seeks to ‘streamline the cluttered UK public sector landscape by incorporating the functions of a number of UK public bodies into existing Scottish bodies’ (p361).

The Scottish Government is clearly making a constitutional argument in relation to its policy aims and its policymaking responsibilities. Put simply, the argument is that it can make better policy in a more effective way than its UK counterpart. This is not just about policy choices, but also about how policy is made and delivered even when those choices are not terribly different. This is an argument that deserves debate, since it goes to the heart of all modern discussions of multi-level policymaking and the implications of policy coordination across many levels and types of government.

David Cameron’s Phone Conversation

My post in The Conversation http://theconversation.com/david-cameron-wants-you-to-phone-a-friend-in-scotland-but-theres-a-divorce-hotline-as-well-22968:

“David Cameron gave a major speech today on the referendum on Scottish Independence.  It is already being dubbed the ‘phone a friend’ speech because he encouraged people in the rest of the UK to phone their friends in Scotland to remind them how great it is to be a part of the UK. He wants you to phone me. He wants you to pretend that you pressed the wrong button, and then say that we may as well have a bit of a catch up since we’re on the phone anyway. Then he wants you to ask in a vague way how the family is, say ‘mmhmm’ a few times, act interested, then bring it round to how passionate David Cameron feels about the UK. He wants you to tell me that David Cameron has some Scottish ancestry and that his name sounds a bit Scottishy, so that must count for something, right?  He wants you to remind me about the Olympics: how British we all felt as well won all those gold medals and, to a lesser extent, how British we felt when we won silver and bronze.  He wants to get serious for a minute and to be hard headed about it – to agree that the Olympic success is a symbol of the UK as a whole, with friendships, business, politics and our reputation across the globe all better as a United Kingdom rather than as separate countries. We’re Better Together, aren’t we mate? Then he wants us to bring it down and get a bit emotional, to agree that we all love each other too much to let each other go. Then he wants to make it about him again; for you to tell me that he is passionate about Scotland’s place in the UK. He loves it. He loves Scotland, Great Britain, Scotland and Britain and Northern Ireland, and the UK best of all. Then he wants you to kick off a vague discussion about British values, as long as you don’t get too UKIPy about it and talk about foreigners. Then he wants us to hug, somehow, and tell each other ‘you’re my best mate’. Then I think we’re supposed to promise to keep in touch much more than we used to; maybe go for a coffee next time you’re in Edinburgh or I’m in London.

The only problem is that I’m just off the phone with Alex, who tells me that David Cameron won’t answer his calls. He wants me to tell you that he can’t understand why Cameron won’t come up to Scotland for a coffee and maybe, while he’s there, to have a brilliant debate about Scottish independence on their own, just the two of them, except that everybody will be watching (although in the background I could hear Nicola shouting ‘naw, just tell him to tell Cameron to just keep his big nose out of it’). He’s just spent ten minutes taking the piss out of Cameron for being obsessed with London and only talking about the rest of the UK when he remembers for a second and has a bit of a panic about it. Then he wants me to tell you that we’ve grown apart; that we don’t talk as much as we used to (and, when we do, we don’t agree as much as we used to) because we’ve both been too busy and don’t have as much time for each other as we used to, when we were better friends. He wants me to say that it’s OK we don’t talk much, but that we should stop pretending we’ll keep in touch. We’ll always have the memories, but it’s time to move on. He hopes that we won’t get too silly when we talk about dividing up our assets and keeping things we shared, like the pound and the Queen and the BBC, because we’ll still be friends after all. Then he wants us to hug and say goodbye.”

The Gerry Adams Novelty Mugs on Twitter Test

How far ahead can we make accurate and detailed political predictions? I propose the Gerry Adams Novelty Mugs on Twitter Test. We ask: how many years ago could you have predicted that Gerry Adams would be tweeting about novelty mugs?

https://twitter.com/GerryAdamsSF/status/430801541373915137

We could probably have made that prediction, say, a year ago based on his whimsical twitter style. However, think about the difficulties in going further back, say 5-10 years, to consider the role of the rise of social media and its confluence with Adams’ new position in the political landscape. Then, consider that Adams’ case is relatively simple, compared to the interaction between a wide range of actors, institutions, socioeconomic conditions and events which produce political changes. In short, the test is there to remind us to be wary of people claiming to have the political equivalent of clairvoyance.

See also:

Predicting the future

McBusted has been to the Year 3000 and it predicts a higher income tax in Scotland relative to the rest of the UK

In 50 years, we won’t care about North Sea Oil because we’ll be on solar jet packs