The barriers to evidence based policymaking in environmental policy

Kathryn Oliver and colleagues examined 145 studies of the barriers to the use of evidence in health policy. They identified a range of problems, including a gap between ‘supply and demand’, the inability of scientists to provide evidence in a timely and strategic way, and a tendency for policymakers to use short cuts (including personal experience and an ad hoc reliance on experts) rather than conduct systematic searches for policy relevant evidence.

In my work with Adam Wellstead, I am beginning to identify the same findings in environmental policy. Multiple studies suggest that:

• Current evidence on the nature of environmental problems, or the effectiveness of policy solutions, is incredibly patchy
• The evidence is not ‘packaged’ well (easy to understand, ‘framed’ in a way that is attractive to policymakers, and/ or accompanied by realistic expectations for policy change)
• Scientists do not engage well with policymakers, either in networks, academic-practitioner forums, or by using intermediaries or ‘knowledge brokers’
• Broad differences in academic-policymaking cultures undermine the ability of scientists to engage in politics in a timely manner, or in a way that will maximise the impact of their findings
• Scientists need to adapt to the vagaries of policymaking, or a tendency for policymakers to: address short term issues rather than plan for the long term; rely on personal experience and limited expert advice; misjudge the risks associated with environmental problems; seek simple, easy to understand, stories rather than the results from sophisticated models; and, use science selectively, often to give a gloss of objectivity to their policy choices.

In other words, most policymakers would make hopeless scientists, and most scientists would be rubbish politicians.

This sort of analysis is interesting enough, and tells us something about the ‘science-policy interface’. Yet, it tends not to be well informed by policy theory or political science. This matters because one might look at the list of barriers and conclude that, if we can overcome them, we can make a big difference to the use of evidence in policy.

That would be a mistake.

Yes, it would be good for policymakers to understand the science a bit more, and for scientists to provide punchy one-page reports, but it is also possible that neither exercise would make much difference. Instead, to understand the profound ‘barriers’ to the use of scientific evidence, we need more scientific knowledge about the policy process itself. This is what most of the environmental literature fails to appreciate, and this lack of knowledge produces a tendency for authors to recommend the same things each time – such as more academic-practitioner workshops to identify barriers – without generating a sense of progress. Such studies should also consider, for example, the huge investment of time and energy that other actors invest, to learn how policymakers think about policy problems, and how to form coalitions with other powerful actors. A glossy report passed on by a ‘knowledge broker’, or a few afternoons in workshops looks paltry in comparison.

In other words, most of this literature does not ask the questions – for example, what is the dominant ‘framing’ of the policy problem; who are the most significant winners and losers with regard to the outcomes of policies; and, what is the effect of multi-level governing arrangements on the use of evidence? – that political scientists would take for granted, and the analysis suffers as a result.

I, and we, are in the process of trying to fill some of those gaps – to use the science of evidence based policy making to inform the scientists who may want to be involved in evidence based policy making. There are more details here – https://paulcairney.wordpress.com/ebpm/ – and the draft chapter on environmental policymaking is here: Cairney Palgrave Pivot CHAPTER 4 environment 25.8.15

Barriers/ solutions to the use of evidence in policy (environmental science)

Some examples of the stated ‘barriers’ to the use of evidence in policy, from the environmental science/ policy literature (to match those found in health by Oliver et al, 2014). See here for more details.

Improve the supply of, and/or generate demand for, scientific evidence

(a) the evidence on the problem is not good enough

Until the 1990s, polices to influence climate change behaviour were hindered by the lack of evidence-based indicators of the effect of consumption on the environment. From the 1990s, scientists developed good monitoring tools, but few policymakers became aware of them (Barrett et al, 2005: 38).

Poor policy decisions are often underpinned by poor evidence on ecological history (Hamilton and Penny, 2015)

EU attempts to monitor and advise on the environmental impacts of agricultural practices are hampered by insufficient data (Louwagie et al, 2012: 149-50)

New models should be developed address the paucity of data underpinning policy on climactic vulnerability and adaptation (Malcolm et al, 2014)

The lack of local-area-specific knowledge undermines the effectiveness of otherwise evidence-based land management policies (Molnár, 2014).

Environmental scientists should from evidence based medicine, to producing a database of systematic reviews and policy-relevant synopses (Dicks et al, 2014: 119; Carneiro and da-Silva-Rosa, 2011: 3; Cvitanovic et al, 2013; Cvitanovic et al, 2014; Webb et al, 2012: 203).

(b) the evidence on the solution is not good enough

There is a lack of comprehensive databases of systematic reviews on biodiversity policy. Existing work is presented in a language that is too technical or politically naïve for busy public managers to take on board, and many studies do not provide a clear answer to pressing policy questions (Carneiro and Danton, 2011).

The scientific evidence base on climate change policy interventions is ‘surprisingly weak for such a high profile area’. ‘There is too little systematic climate policy evaluation work in the EU to support systematic evidence- based policy making’ (Haug et al, 2010: 427).

Current performance management practices do not allow us to evaluate the effectiveness of conservation programmes, because organisations measure what is easy to measure (Rissman and Smail, 2015).

(c) the evidence needs to be ‘packaged’ well (easy to understand, framed in a way that is attractive to policymakers, and/ or accompanied by realistic expectations for policy change)

Academics should repackage their work according to the needs of their ‘end user’ – such as by providing pragmatic recommendations or information that helps them predict events and plan ahead – and use ‘knowledge brokers’ (Cvitanovic et al, 2013: 85; Cvitanovic et al, 2014: 35-6).

Policymakers will often not respond to an alleged policy problem if there is not an obvious solution (Lalor and Hickey, 2014: 10-12)

The rise of sophisticated policy assessment tools (such as models) is caused more by technological advance than a demand for information in this form. Simple qualitative stories ‘backed up by illustrative statistics … appeared largely driven by the need to present easily digestible analysis to the decision maker if one wanted the assessment to be instrumentally useful’ (Nilsson et al, 2008: 348).

The carbon capture and storage (CCS) community has a coherent and uniform message for policymakers, which may help explain its major funding successes in the EU (Stephens et al, 2011: 388)

Government reports provide vast amounts of evidence but their links to effective policy are weak, partly because the reports come with unrealistic shopping lists for action (Wellstead and Stedman, 2014: 1000).

Scientists struggle to translate knowledge and concepts about risk to policymakers, stakeholders and the public (Yuen et al, 2013).

(d) engage in networks and academic-practitioner workshops

There is high participant demand to identify best practice in academic-practitioner exchange (or at least to find quick/ easy solutions to gulfs in their relationships), and a belief that regular interaction helps build up trust or ‘social capital’ (Hickey et al, 2013: 539).

To adapt to complex policymaking systems, scientists need to engage in collaborative/ participatory government rather than feed in evidence to the centre (Lalor and Hickey, 2014).

(e) use intermediaries

There is a need for ‘hybrid people’ but an absence of unanimous ‘upper management’ support (in public bodies in Canada and Australia) for ‘knowledge brokers’ (Hickey et al, 2013: 534).

Timing and opportunity

Policymakers value timely and responsive research, but scientists face big time lags in publication (Cvitanovic et al, 2014: 38)

‘Relationships of trust and establishment of expert credibility matter greatly in the acceptance of knowledge claims’ (in international climate change treaty negotiations) (Rowe, 2013: 221)

Despite a new agenda on timely and policy-relevant research (on dryland policies in Africa) the evidence remains ‘sparse’ (Stringer and Dougill, 2013: 328).

Encourage policymaker skills or better government understanding of problems

Governments tend to deal with environmental crises rather than plan for the long term. A lack of government commitment to collecting policy-relevant data produces often undetected policy failures (Clare and Creed, 2014: 243)

Policymakers rely on personal experience and expert advice, not systematic searches of the literature (Carneiro and da-Silva-Rosa, 2011: 1; Cvitanovic, 2013: 85)

Many policy managers do not prioritise scientific evidence and are unaware of advances in adaptation science. Policymakers often have poor knowledge of environmental risks, and their priorities often do not reflect the best evidence (Cvitanovic et al, 2014: 38)

Ministers do not understand the data from the sophisticated policy assessment tools that they, ‘have been so keen to advocate and nurture’ (Nilsson et al, 2008: 350). Rickards et al (2014: 654) provide similar conclusions on scenario planning. As in the nomenclature on evidence-informed policy, they identify ‘scenario methods’ or ‘scenario thinking’

UK government ministers appear unwilling or unable to engage in the systematic review of the evidence on business regulation (Taylor et al, 2013).

These complaints about ignoring science seem biased and naïve – and too easy to dismiss

This is one of two posts on the use of scientific evidence in environmental policy (a broad term which can include climate change, food, land management, and energy policy). In this one I insult some of the people reacting to the Scottish Government’s decision on GM. In the next one, I present preliminary evidence on a systematic review of the use of evidence in environmental policy.

The Scottish Government recently decided to reinforce a moratorium on the use of GM crops in Scotland. It produced a strong response from many individual scientists, and groups representing scientists, which seems predictable enough. That’s what you would expect some scientists to do if you don’t act solely on their advice.

What I didn’t quite expect was a more general, and often hyperbolic, criticism by non-scientist commentators about the failings of policymakers to listen to scientists.

The general problem for non-scientist commentators is that they don’t really know what ‘the evidence’ is. It’s no great insult to predict that most commentators decrying a lack of reliance on ‘the evidence’ have never read any relevant scientific journal article, systematic review, or peer reviewed report on the science in question. In fact, that’s fine – you probably need some training to understand the technical aspects, particularly on method.

What is not fine is a tendency to accept the word of scientists when they tell you that they know the evidence. In general, they don’t. ‘The evidence’ is often remarkably patchy (a point I’ll reinforce in the next post) and highly-disputed within professions, and canny scientific groups or individuals fill those gaps with an appeal to expert-based authority. This practice should be subject to as much critical analysis and scepticism that we afford to politicians. In fact, if you are a good scientist, you welcome the challenge and the debate (with the exception of cases, such as climate change, in which debate may undermine an urgent need to act).

The general problem for the more naïve scientist commentators is that they often read the scientific reports, take a view on what they mean and what should be done, and simply expect policymakers to do the same. They fail to appreciate how scientific evidence fits into the broader picture of politics and policymaking. They fail to articulate what ‘evidence based policy making’ really means, or should mean in practice, when we have equally or more important ways to make decisions – such as representative government (we elect people to state their beliefs and make decisions on our behalf) and participatory/ deliberative democracy (we seek ways to generate a range of perspectives on a policy problem). In effect, they make an appeal to a process in which scientific knowledge or expert beliefs should be privileged above the beliefs of other people (in this case, it is accompanied by a pretentious reference to the Scottish Enlightenment).

In my view*, both of these approaches are counterproductive. The hyperbolic non-scientist criticisms are too easy to dismiss. As soon as you start equating policy decisions with, for example, witch trials (OK – I’ve only seen two people do it), you can be safely ignored as someone with an axe to grind. The narrow scientist criticisms are just as badly off. As soon as you appeal to ‘the evidence’ without showing any appreciation of the policy process or a wider appreciation of policy decisions, you can be safely dismissed as naïve or self-interested.

Update: maybe more than 2 –

SNP's witch-hunt of scientists exposed by @euanmccolm in cracking piece on GM hysteria http://t.co/kASb4GuxAK

— Murdo Fraser (@murdo_fraser) August 19, 2015

Further reading: https://paulcairney.wordpress.com/ebpm/

For a pragmatic scientific account see:

Lawton, J. H. (2007) ‘Ecology, politics and policy’, Journal of Applied Ecology, 44,3, 465–74

*These views are my own, they are stated a bit strongly, apologies to Duncan and Euan, etc.

Why do people seem so down on e-cigs?

There is a useful summary of e-cigarettes here http://www.vox.com/cards/e-cigarettes

Paul Cairney: Politics & Public Policy

It must be very frustrating to give up smoking, with the help of e-cigarettes, only to find that you are no more welcome in public places with a fake cigarette than a real one. UK governments, and many public health advocates, often seem to want to regulate them in the same way, even though the e-cig could be described as a crucial ‘harm reduction’ measure (it’s not exactly healthy, but it’s much better than the other thing you were doing).

Here is a list of historical explanations for this position which won’t make you happy, but can at least distract you while you’re having a sly puff in the toilets:

1. We’ve been here before with tobacco and harm reduction. So many post-war examples – like the idea of smoking a pipe, putting filter tips on cigarettes, ‘low tar’ cigarettes (which is a bit like ‘less shite in your…

View original post 657 more words

Two first drafts of the SNP 2016 manifesto

People have begun to speculate about how the SNP will incorporate a second referendum into its 2016 Scottish Parliament manifesto. So, I’ve had a go at drafting two different versions.

The first one is easy to write but won’t happen.

The second, more realistic one, is harder to write. In particular, it will be hard to get the wording right when the SNP (in effect) promises a referendum in the next manifesto but not this one.

So, we might expect a lot of coverage of the opposition parties trying to pin down Nicola Sturgeon on the detail. Ordinarily this might be sensible, but not while the SNP remains almost immune to that kind of criticism. Maybe it would be better to focus on what the SNP will do, instead, in the next 5 years. Who knows?

The one that won’t be written:

Nicola Sturgeon stated in 2015 that: “Something material would have to change in terms of the circumstances or public opinion before I think it would be appropriate to have a proposal for a referendum.”

Since we gubbed Labour in the 2015 UK General Election and it looks like we’ll do the same in the Scottish Parliament, we believe that material circumstances have changed and we will have a referendum as soon as possible. Loads of Yes supporters joined the SNP straight after the referendum, and they think we were robbed and that loads of people regret voting No. There was also that poll which suggests that we’re hugely popular even though people don’t think we’re doing a brilliant job, so independence is all we want to talk about.

The more likely one that is longer and trickier to write, and doesn’t quite scan:

Nicola Sturgeon stated in 2015 that: “Something material would have to change in terms of the circumstances or public opinion before I think it would be appropriate to have a proposal for a referendum.”

In the period of the next Scottish Parliament, the biggest potential change of circumstances regards Scotland’s membership of the European Union. Nicola Sturgeon has called on David Cameron to ensure that the UK leaves the EU only if all four nations agree. He has rejected this call. If the UK prepares to leave the EU, despite a majority of voters in Scotland voting to remain, it will cause a constitutional crisis. Under those circumstances, we believe that we should hold a referendum on Scottish independence as soon as practicable (this paragraph seems relatively easy to write).

We are heartened by the surge of support in our membership, to over 100,000 members, and in electoral support for the SNP, which produced 56 of 59 SNP MPs. However, this alone does not represent a change of public opinion towards Scottish independence. It does not guarantee that the majority of voters in Scotland would vote Yes in a second referendum on Scottish independence (this one is tricky to get right, and would be used by opponents).

Instead, the electorate also supports us because we present a strong vision to deal with austerity and support essential public services. They do not support the austerity agenda of the Conservative party, and nor do they find Labour to be a credible alternative (bread and butter stuff).

We believe that, in time, support for our positive vision, and a growing belief that the UK parties’ austerity agenda is damaging Scotland’s economy and widening inequalities, will help produce a clear majority in favour of Scottish independence (also bread and butter).

If that change of public opinion becomes clear over the next five years, we will include a firm commitment to a second independence referendum in our manifesto for the 2021 Scottish Parliament election (no, I can’t get that one to work!).

 

What can be done about the UK’s ‘glass floor’?

New research on the ‘glass floor’ presents a striking way to understand socioeconomic inequality in the UK. It also highlights ever-present problems in translating such information into policy: we understand the size of the problem well, speculate on its cause badly, and produce vague calls for government action ineffectively. Our initial shock and enthusiasm for policy change translates into disenchantment with yet another ‘too difficult’ problem.

The UK Government’s Social Mobility and Child Poverty Commission has released new research on the life chances of the British population. It identifies a “’glass floor’ in British society” to reject the idea that people get on in life through hard work and merit. Instead, mediocre and lazy children in the right family will do better than bright and hardworking children in the wrong family.

This is horrible paraphrasing of the report, but you get the idea about how most people might notice the report in a hurry, have their beliefs about the lack of a British meritocracy reinforced, then complain that the government is doing enough about it. There wasn’t quite a public outcry (far from it), but you might be forgiven for thinking that the report gives the government plenty of reason to do something. The big question is: will it do anything new with the information?

I wouldn’t rule it out, but would exercise this note of caution: reports like this don’t speak for themselves or give governments a clear impetus to act. Instead, they form part of a larger pattern in this area (of socio-economic inequalities policy), in which we can speak with much more certainty about the size of the problem than (a) its cause, (b) how we should respond, and (c) who exactly should respond.

The size of the problem

The size of the problem is quantified well (it’s not a simple task to measure cognitive ability, class backgrounds and life chances like this) and easy to understand. For example, the commission’s press release states that:

‘Less able, better-off kids are 35% more likely to become high earners than bright poor kids … children from more advantaged social backgrounds who are assessed at age 5 as having low cognitive ability are nonetheless significantly more likely to become high earners than their high ability peers in lower income households. Children from high income backgrounds who show signs of low academic ability at age 5 are 35% more likely to be high earners as adults than children from poorer families who show early signs of high ability’.

The cause of the problem

This is when things get a bit trickier, because although the chair of the commission, Alan Milburn, describes ‘a social scandal that all too often demography is still destiny in Britain’, the commission is not entirely clear on who or what caused it. There is not one simple message about a single villain. Instead, there are at least two, and both stories are not crystal clear.

First, the author of the report, Dr Abigail McKnight, links the outcomes to the behaviour of certain parents:

“The fact that middle class families are successful in hoarding the best opportunities in the education system and in the labour market is a real barrier to the upward social mobility of less advantaged children.”

The keyword there is ‘hoarding’, which suggests inappropriately selfish behaviour. Yet, the chair of the commission, Alan Milburn, is keen not to blame parents: ‘No one should criticise parents for doing their best for their children. That’s what we all want’.

Instead, Milburn sort of blames the government for its current lack of proportionate action: ‘The government should make its core mission the levelling of the playing field so that every child in the country has an equal opportunity to go as far as their abilities can take them’.

The result is a mixed view about the cause of the problem – perhaps it’s the fault of some hoarding parents (the especially rich ones sending their kids to private schools, getting tutors and securing internships for their children) and not so much others (the ones using their own skills to secure a spot for their child in a good state school) – and maybe the solution is to give other parents some of these skills to ‘level the playing field’ a bit.

The realistic solution

This is when things get even trickier, because the report seems to call for the government to do far more than it will, while giving it the ability to say that it is already doing as much as it should.

In the ‘far more than it will’ column is the call to reduce socio-economic inequalities (through wealth and income redistribution?), remove differences in quality between schools, and remove class-based barriers to University admissions.

In the ‘sort of doing it already’ column is the call for the state to intervene early in people’s lives to, in effect, train disadvantaged parents in how to give their children things like ‘soft skills’ related to forming networks and spotting opportunities.

The ultimate complication

The final, and perhaps trickiest, obstacle is about working out who is in charge of taking the next step, to drive this new policy agenda forward. The final paragraph of the main report is instructive:

‘If politicians are serious about their expressed desire to increase social mobility in the UK they will need to address barriers that are preventing less advantaged children from reaching their full potential and remove barriers that block downward mobility’.

It doesn’t say who the politicians are – perhaps for good reason. In areas such as social and economic inequality, it is increasingly difficult to know who is responsible for policy progress. If it’s mainly about economic redistribution, you can call for action from central government – but, let’s be honest, this won’t get you very far. If it’s mainly about training and encouraging ‘soft skills’ like ‘resilience’, central government might produce a broad strategy document, but its localism agenda suggests that it expects local public bodies to take responsibility for social outcomes.

The overall message is that it takes us seconds to understand the problem and call for government action, but a lot longer to decide what we want them to do, and longer still to find the people likely to do it. By that time, our attention will probably have shifted elsewhere, until the next report comes out and we do it all over again. Maybe this time will be different.

 

 

 

In UK and Scottish politics, should you assume that people are stupid?

Political commentators often make fun of other political commentators when they complain that the public is stupid. Yet, maybe we all do something similar – assume that most people make quick, emotional and habitual decisions to turn a complex world into a series of simple actions. In that sense, the ‘realistic’ political campaigns (and some policies) favoured by such commentators may be based just as much on the ‘stupidity’ of the target audience.

In Scottish politics, there is a lot of fun to be had (if you like that sort of thing) while taking the piss out of Yes supporters who argue that No supporters were manipulated into their decision. For example, if only No supporters could see through media and partisan manipulation they’d come to a different conclusion. The argument here is that such Yes supporters are implying that temporary No supporters are stupid, since they are more open to manipulation than their critics. Only Yes supporters can see through the lies.

In UK politics, the same fun can be had (if you like that sort of thing) with certain Labour supporters who can’t quite believe that they lost the election, or blame it on people who support Labour values in public but vote Conservative in private. There are also some offshoot debates, often led by John Rentoul, about the idea that the ‘wrong people were voting Labour’ under Blair or that current supporters of Jeremy Corbyn deny the truth about why Labour lost the UK General Election. The debate is not quite the same, but you can detect a similar suggestion that left-wing Labour supporters blame current Conservative supporters for not seeing through the cynicism of Tory campaigns.

In both cases, the proposed solution may be a cold dose of realism about the beliefs of the British public: instead of complaining that too many people are naïve or stupid and hold the wrong beliefs (or are ‘delusional’), try to work with their beliefs to produce a campaign that works for them.

You can see why this approach would present some fairly heated debates, since the counterargument is that you should not accept public beliefs when you consider them repugnant (such as in relation to migration) or dangerously misguided (such as in relation to the causes of the financial crisis or the ways in which parties justify austerity measures, often in relation to the misleading analogy of balancing the household books, which exacerbate socio-economic inequalities).

In time, this pragmatic argument may underpin the second referendum on Scottish independence (who knows?).

In the meantime, it is at the heart of the Labour leadership, in which the heroic/left-wing/ ideologue Jeremy Corbyn is pitted against the three more cynical/ pragmatic/ realistic candidates who want to compete with the Conservatives in part by engaging with their arguments rather than dismissing them as stupid and cynical.

What should we do when people are ‘stupid’

In each case, there are two constants that we should always bear in mind:

1. Ideas aren’t stupid. Many people focus too much on the stupidity of arguments or ideas when they should really identify the tendency for people to believe them. I don’t want to get into a big philosophical thing about this, but ideas aren’t simply good/ bad and they don’t spread on their own – they need some people to propose them and others to accept them. Your choice is a mix of (a) giving credit to the people who know how to get good ideas going; and/ or (b) assigning blame to the gullible fools that accept them.
2. People aren’t stupid, but they are open to manipulation. People have a tendency to accept simple stories that chime with their beliefs (and other stuff – Daniel Kahneman’s Thinking Fast and Slow sums up a lot of this kind of argument, but see here and here if you want something more free but less good).

All that remains is to decide what to do about it:

(a) do you start with the assumption that people inevitably have cognitive biases that can be manipulated (particularly when they make snap decisions), to get them to do what you want;

or

(b) do you assume that people can overcome many cognitive biases, try to educate them, or otherwise help them to think more carefully about issues to make well-considered intelligent decisions?

I don’t know the answer to that question ….

All I’ll point out is that I have (hopefully) reframed the initial premise of this post. Now, it’s the fun-makers and realists who think that people are stupid (the a people). Now, our original villains are really the heroes who have faith in the public to think harder and make the right decision next time (the b people).

…except to say that sneering doesn’t help.

A final thought on the strategies of realists is that many of them sneer a lot at the people they think are naïve. This doesn’t just seem rude and annoying – it’s also counterproductive, because politics is often as much about process as outcome. If everyone enters and leaves debates in a respectful way, the losers may be content with a poor outcome. If the winners spend their time sneering at their opponents, and Lording their victories over the losers, it just exposes the very divisions they claim to want to prevent.

Is our understanding of Scottish politics stuck in the past?

The good news is that there has been refreshingly high attention to the substance of Scottish politics lately. Concerns about the operation of Police Scotland, and the new exams system, show that we can talk about more than the next referendum, election, or leadership race.

The bad news is that a lot of this discussion betrays an old fashioned sense of Scottish politics: power is concentrated in central government; and we can hold ministers to account for everything that goes wrong.

Instead, since devolution, successive Scottish governments have been reforming the public landscape, reinforcing the role of non-departmental public bodies, devolving responsibility for public sector performance to local and health authorities, and expecting those bodies to form new relationships with stakeholders, communities, and the users of public services. In this sense, they have been pursuing the idea that we all share responsibility for policy outcomes.

Further, this blurring of central/local accountability comes on top of a more general sense that we don’t know exactly what the Scottish Government is responsible for: the devolved settlement might look clear on paper, but the level of UK/ Scottish Government overlaps and shared responsibilities (which will rise following the new Scotland Act) gives too many people the sense that Scottish ministers are not fully in control and therefore can only take so much of the blame.

These developments help explain why so many SNP ministers seem to be Teflon whenever anything seems to go wrong in government. Of course, most of their ability to remain popular stems from our ongoing fascination with the constitution – but not all of it. Indeed, many people vote SNP because it has developed a hugely impressive image of governing competence (which is the main explanation for its 2011 majority). This image will suffer if it looks like Scottish ministers are making bad decisions which have a direct effect on policy outcomes.

In light of these developments in accountability, it would be right for people to complain that they make it harder to hold central government, and specific ministers, to account in a meaningful way – through parliamentary and media scrutiny, and in Scottish Parliament elections based on the Scottish Government’s record.

It would be wrong, instead, to carry on as normal on the assumption that none of these developments have taken place. In this case, partisan (and some media and a lot of social media) criticisms just don’t hit the mark. They make a general case for SNP ministers to take the blame for something – using argument (a) the buck stops with ministers, or (b) you have been too obsessed with independence to pay attention to government – but don’t present a convincingly specific case which links poor ministerial decisions to poor outcomes.

So, if you see me on twitter arguing that certain criticisms don’t hit the mark, it is not because I am defending the SNP’s record in government. Instead, I am suggesting that too many criticisms of its record are too lazy or scattergun or vague to have any meaningful effect.

The politics of evidence and randomised control trials: the symbolic importance of family nurse partnerships

We await the results of the randomised control trial (RCT) on family nurse partnerships in England. While it looks like an innocuous review of an internationally well-respected programme, and will likely receive minimal media attention, I think it has high-stakes symbolic value in relation to the role of RCTs in British government.

EBM versus EBPM?

We know a lot about the use of evidence in politics – and we hear that politicians play fast and loose with it. We also know that some professions have a very clear idea about what counts as evidence, and that this view is not shared by politicians and policymakers. Somehow, ‘politics’ gets in the way of the good production and use of evidence.

A key example is the ideal of ‘Evidence Based Medicine’ (EBM), which is associated with a hierarchy of evidence in which the status of the RCT is only exceeded by the systematic review of RCTs – particularly when the results of this work are peer reviewed and published in high-status journals or databases.

This contrasts with evidence based policy making (EBPM) in which there are competing notions of evidence value, competing sources of evidence (expertise, policymaker experience, professional opinion, service user feedback, etc.), and a greater sense that policymakers will beg, borrow and steal whatever evidence they can get their hands on quickly to address the specific problem they face – including reports that are not peer reviewed or published in outlets with recognised scientific status.

Policymakers also have to weigh up evidence on policies that are difficult (if not impossible) to compare with each other, and come up with ways of choosing between them – such as by assessing their value for money in relation to the benefits they provide.

A compromise between evidence and politics?

In some cases there may be a decent compromise between these practices. In health, expert bodies such as NICE have become responsible for combining the kinds of evidence consistent with EBM with economic and other methods (often including professional and user feedback) to produce guidance on policy choices. NICE does not quite take the politics out of health and social care choices (and nor should it) but it often acts as a standard which prompts policymakers to accept its advice or explain why they don’t.

There have also been important efforts to encourage the greater use of RCTs more widely in government, such as by the Cabinet Office’s Behavioural Insights Team and academics such as Peter John and Gerry Stoker.

Major obstacles to the uptake of RCTs

Yet, these developments do not guarantee a central role for the RCT in politics more generally (far from it). Rather, many politicians or policymakers exhibit uncertainty or scepticism about:

1. The relevance of RCT evidence – they may argue that (a) an RCT does not answer their question fully or capture the complexity of a policy problem, and (b) that RCT evidence from somewhere else does not apply to their area.
2. Practical and ethical – an RCT could require cooperation across many levels and types of government, and randomisation is a ‘hard political sell’, at least to elected policymakers who (a) rely on an image of certainty when they propose policies (why would you need to test a policy’s value – are you trying something that might fail?), and (b) struggle with the idea of giving a good intervention to one group of people and not another (if the policy works, why don’t you give everyone the benefit?).

These political concerns may combine with academic criticisms about the assumptions behind RCTs (e.g. that you can produce what can meaningfully be called ‘control’ groups in complex social interactions) to produce major obstacles to the uptake of the evidence favoured by key groups of scientists.

The next best thing: importing policies based on RCTs

Perhaps the next best thing to conducting an RCT in the relative dark is to import a programme with an international reputation for well-evidenced and impressive results. That is where the family nurse partnership (FNP) comes in (box 1).

________________________________________________

BOX 1: The Family Nurse Partnership

The FNP began in the US as the Nurse-Family Partnership – designed to engage nurses with first time mothers (deemed to be at relatively high risk of poor life chances) approximately once per month from pregnancy until the child is two. The criteria for inclusion relate to age (teenage), income (low), and partnership status (generally unmarried). Nurses give advice on how mothers can look after their own health, care for their child, minimise the chances of further unplanned pregnancy, and access education or employment. It combines intervention to address the immediate problems of mothers and early intervention to influence the longer term impact on children.

The US’ Coalition for Evidence-Based Policy gave it ‘top tier’ status, which describes ‘Interventions shown in well-designed and implemented randomized controlled trials, preferably conducted in typical community settings, to produce sizable, sustained benefits to participants and/or society’. Identifying three US-based RCTs, it describes common outcomes in at least two, including reductions in pre-natal smoking, child abuse and neglect, and second pregnancies, and improvements in their child’s cognitive function and education attainment (in follow-ups when the children reached 15-19) at a low cost. These trials have been conducted since the first project began in 1977, producing at least 18 peer-reviewed articles, including by its pioneer Professor David Olds, in elite academic journals (such as Journal of the American Medical Association), and at least two which identify new results in non-US studies.

The programme was rolled out in England to 9000 mothers, with reference to its high cost effectiveness and ‘strong evidence base’, which would be enhanced by an RCT to evaluate its effect in a new country. The FNP requires ‘fidelity’ to the US programme (you can only access the progamme if you agree to the licensing conditions) based on evaluation results which showed that the programme was most effective when provided by nurses/ midwives and using a license ‘setting out core model elements covering clinical delivery, staff competencies and organisational standards to ensure it is delivered well’. Fidelity is a requirement because, ‘If evidence-based programmes are diluted or compromised when implemented, research shows that they are unlikely to replicate the benefits’.

________________________________________________

Adopting the FNP doesn’t solve the ‘not invented here’ problem, but it helps reduce many concerns: we import a successful policy (with success demonstrated in multiple RCTs) and conduct an RCT to make sure that a programme that works somewhere else works here. Not everyone gets the programme but, unlike in the US, they still receive ‘universal’ NHS care. The use of an RCT can also be sold politically, as (a) part of the license and (b) the kind of routine evidence gathering/ evaluation that should be present in all policy interventions anyway. This RCT/ programme also relates to a fairly contained group of recipients and healthcare professionals, with not as much need for ‘joined up government’ or ‘health and social care integration’ as in many other initiatives. It is praised by NICE and the Early Intervention Foundation.

In this context, the FNP is almost perfect

That’s what makes it seem so important symbolically. It’s like a trailblazer, showing all that is right with the use of multiple RCTs, to perform meaningful tests to demonstrate the effectiveness of a public policy. It is as much an advert for the value of the RCT as for the value of the programme.

The flip side to this coin is that, if the perfect programme doesn’t produce meaningfully better results than the NHS programme it replaced, some people may get the sense that we went to a lot of bother and expense for very little reward. The idea of a ‘gold standard’ of research may take on a different connotation, particularly during a period of austerity in which governments may be reluctant to invest in new policies and their evaluation when they have to reduce public provision.

Therefore, I expect the release of the RCT results to be political, at least in the sense that they won’t be released without some thought given to how to present the findings in as positive a way as possible. That’s perhaps not in the spirit of the ideal of EBM, but it seems consistent with the reality of EBPM.

Update

The first RCT results were published in October 2015 in the Lancet. A very short summary of these developments is as follows:

• After publishing the results of the RCT, Robling et al argue that ‘Programme continuation is not justified on the basis of available evidence, but could be reconsidered should supportive longer-term evidence emerge’
• David Olds’s reply is that the FNP could be more effective if directed more accurately to the most relevant target population
• The Local Government Association, which recently became responsible for public health (alongside social services), made a broad statement about using the opportunity to look ‘closely at how to achieve maximum effectiveness for the Family Nurse Partnership programme and whether it can be adapted to achieve better value’
• The Early Intervention Foundation commends the use of good evaluation and reinforces its view that there are many well-evidenced programmes from which to choose

Further Reading

This draft paper provides some further reading on the trainspotter’s guide to evidence/ policy, while this link takes you to a draft Palgrave Pivot book with a bibliography on EBPM.

See also: Ruth Kennedy What works. Can we know?