Category Archives: Uncategorized

The role of ‘standards for evidence’ in ‘evidence informed policymaking’

Key points:

  • Maintaining strict adherence to evidence standards is like tying your hands behind your back
  • There is an inescapable trade-off between maintaining scientific distance for integrity and using evidence pragmatically to ensure its impact
  • So, we should not divorce discussions of evidence standards from evidence use

I once spoke with a policymaker from a health unit who described the unintended consequences of their self-imposed evidence standards. They held themselves to such a high standard of evidence that very few studies met their requirements. So, they often had a very strong sense of ‘what works’ but, by their own standards, could not express much confidence in their evidence base.

As a result, their policy recommendations were tentative and equivocal, and directed at a policymaker audience looking for strong and unequivocal support for (often controversial) policy solutions before putting their weight behind them. Even if evidence advocates had (what they thought to be) the best available evidence, they would not make enough of it. Instead, they value their reputations, based on their scientific integrity, producing the best evidence, and not making inflated claims about the policy implications. Let’s wait for more evidence, just to be sure. Let’s not use suboptimal evidence, even if it’s all we have.

Your competitors do not tie their own hands behind their backs in this way

I say this because I have attended many workshops, in the last year, in which we discuss principles for science advice and guidelines or standards for the evidence part of ‘evidence-based’ or ‘evidence-informed’ policymaking.

During such discussions, it is common for people to articulate the equivalent of crossing their fingers and hoping that they can produce rules for the highest evidence standards without the unintended consequences. If you are a fan of Field of Dreams, we can modify the slogan: if you build it (the evidence base), they will come (policymakers will use it sincerely, and we’ll all be happy).*

If you build it

Or, if you are more of a fan of Roger Pielke Jr, you can build the evidence base while remaining an ‘honest broker’, providing evidence without advocacy. Ideally, we’d want to maintain scientific integrity and have a major impact on policy (akin to me wanting to eat chips all day and lose weight) but, in the real world, may settle for the former.

If so, perhaps a more realistic way of phrasing the question would be: what rules for evidence should a small group of often-not-very-influential people agree among themselves? In doing so, we recognise that very few policy actors will follow these rules.

What happens when we don’t divorce a discussion of (a) standards of evidence from (b) the use of evidence for policy impact?

The latter depends on far more than evidence, such as the usual factors we discuss in these workshops, including trust in the messenger, and providing a ‘timely’ message.  Perhaps a high-standard evidence base helps the former (providing a Kite Mark for evidence) and one aspect of the latter (the evidence is there when you demand it). However, policy studies-inspired messages go much further, such as in Three habits of successful entrepreneurs which describes the strategies people use for impact:

  1. They tell simple and persuasive stories to generate demand for their evidence
  2. They have a technically and politically feasible (evidence-based) policy solution ready to chase policy problems
  3. They adapt their strategies to the scale of their policy environments, akin to surfers in large and competitive political systems, but more like Poseidon in less competitive ‘policy communities’ or subnational venues.

In such cases, the availability of evidence becomes secondary to:

  1. the way you use evidence to frame a policy problem, which is often more about the way you connect information to policymaker demand than the quality of the evidence.

Table 1

  1. your skills in being able to spot the right time to present evidence-based solutions, which is not about a mythical policy cycle, and not really about the availability of evidence or speed of delivery.

Table 2

So, when we talk about any guidance for evidence advocates, such as pursued by INGSA, I think you will always find these tensions between evidence quality and scientific integrity on the one hand, and ‘timeliness’ or impact on the other. You don’t address the need for timely evidence simply by making sure that the evidence exists in a database.

I discuss these tensions further on the INGSA website: Principles of science advice to government: key problems and feasible solutions

.

.

*Perhaps you’d like to point out that when Ray Kinsella built it (the baseball field in his cornfield), he did come (the ghost of Shoeless Joe Jackson appeared to play baseball there). I’m sorry to have to tell you this, but actually that was Ray Liotta pretending to be Jackson.

 

Leave a comment

Filed under Evidence Based Policymaking (EBPM), Uncategorized

Professor Grant Jordan

Grant died on Friday. He was my friend and mentor for a long time, and I’m glad that I knew him for so long.

Let me share four short stories with you, to give you a sense of Grant.

The first is of an internationally respected scholar. When Grant and I visited Hokkaido University in 2004, his talk was preceded by a warm and glowing reference by our hosts. One host held aloft Governing Under Pressure and described it as one of the most influential books of his time. It was certainly one of the biggest influences on me and many of my colleagues (as I describe below), and his work was valued by as many colleagues in the US as the UK.

governing under pressure

The second is of his continuous help to students and colleagues. I associate Grant’s working life with one image: his open office door. He kept an office directly across from the departmental office, ensuring that if any student came to us with a problem, he’d be the first to try to solve it. It might not sound like much to say that he knew Aberdeen University’s regulations inside-out, but it symbolised his continuous efforts to make sure that students benefited from them. He offered the same continuous help to many of our colleagues.

The third is of a quietly influential mentor. Some of Grant’s advice was cryptic, but all of it was useful. At key points of my career and intellectual development, he was there to offer pointers and challenge incomplete thought. For example, often his favourite approach was to quote Mandy Rice-Davies (‘he would say that’) to challenge any claims I made from elite interviews, and you can see the effect of such caution on much of my published work.

The fourth is of a funny person. There are odd-sounding times that I’ll remember, such as when Darren Halpin and I met up with Grant and his partner Andrea in Toronto (during an APSA conference), and they were already quite merry when we arrived. Or, when at the football together (Grant was a big Aberdeen FC fan), he would often offer me and my children some suspiciously warm toffees from his pocket. Maybe one of his funniest lines is now one of the most poignant: when I tried to do a decent speech on his career at his retirement dinner, he described it as a speech well suited to his funeral (I guess you had to be there to appreciate the humour!).

Jordan and Richardson’s intellectual legacy

Grant will leave an intellectual legacy. His work with Jeremy Richardson is still at the heart of my understanding of politics and policy. In my first undergraduate year at Strathclyde, Jeremy lectured on British politics and public policy. He presented an image of politics that drew me in (partly via Yes Minster) and an argument – made in partnership with Grant – that I still use most frequently to this day:

  • The size and scope of the state is so large that it is in danger of becoming unmanageable. The same can be said of the crowded environment in which huge numbers of actors seek policy influence. Consequently, the state’s component parts are broken down into policy sectors and sub-sectors, with power spread across government.
  • Elected policymakers can only pay attention to a tiny proportion of issues for which they are responsible. So, they pay attention to a small number and ignore the rest. In effect, they delegate policymaking responsibility to other actors such as bureaucrats, often at low levels of government.
  • At this level of government and specialisation, bureaucrats rely on specialist organisations for information and advice.
  • Those organisations trade that information/advice and other resources for access to, and influence within, the government (other resources may relate to who groups represent – such as a large, paying membership, an important profession, or a high status donor or corporation).
  • Therefore, most public policy is conducted primarily through small and specialist policy communities that process issues at a level of government not particularly visible to the public, and with minimal senior policymaker involvement.

They initially made this argument in the late 1970s and early 1980s, before the rise of ‘Thatcherism’ in the UK. Then, they used it to challenge the idea that Thatcherism marked a radical departure in policymaking. Of course, this new phase of policymaking was important and distinctive, but the same basic argument outlined above still applies, and Jordan went on to do further empirical studies, with colleagues such as William Maloney, to highlight a surprising amount of policymaking stability and continuity. In other words, Jordan and Richardson have shown, and continue to show that the UK generally does not live up to its ‘majoritarian’ image. It’s an argument that I use to this day.

Overall, I was very lucky to have Grant in my life for so long, and I hope he knew how many people shared this combination of admiration for his work and fondness of him.

 

 

1 Comment

Filed under Uncategorized

What’s That?

I apologise in advance.

In the olden days, my former lecturer would describe the wackily short amount of time you would have to explain a problem and possible solution to an elected policymaker: 1 or 2 pages of A4 (scroll to the end).

More recently, I heard a ramped-up version of this pressure: the time it takes to walk them to their car (a variant on the ‘elevator pitch’ idea).

It’s not, enough, though, is it? We need more of a sense of the pressure we face to explain complicated concepts quickly and concisely to different audiences.

So, I suggest this example, borne out of my recent experience as granddad to a 13-month old baby who has reached the stage of pointing randomly at things and saying ‘what’s that?’.

pull along mouse xylophone

I tried and failed to explain that (a) it’s a wooden mouse playing a xylophone, on wheels, which moves when you pull the string along the floor, and (b) the funny thing is, you don’t see many mice playing the xylophone these days.

For this type of audience, there’s only time to reply ‘a toy’ (or, perhaps if you are thinking laterally, ‘would you like a biscuit’?).

So, the new test of conciseness is: what can you explain in the length of time between a child pointing to one thing and saying ‘what’s that’ and the same child pointing to another thing and saying ‘what’s that?’.

Leave a comment

Filed under Uncategorized

My thoughts on how the all-women Wonder Woman screening affects me

Leave a comment

Filed under Uncategorized

What 10 questions should we put to evidence for policy experts?

The European Commission’s Joint Research Centre’s Science Hub is making some videos about evidence and policy, asking 10 questions. Here are my answers. The video will come later.

  1. Who are you?

Paul Cairney, Professor of Politics and Public Policy, University of Stirling. I write about public policy, applying theoretical insight to issues such as ‘the politics of EBPM’.

  1. How did you become interested in evidence for policy?

It was always in the back of my mind because it is the latest version of a long-standing interest (in policy studies) about the absence of ‘comprehensive rationality’: what do policymakers do when they can’t consider all information, and what are the consequences for politics and policy? Do they use ‘irrational’ shortcuts? Does their attention tend to lurch? Does policy become incremental or ‘punctuated’? There are many different answers, explored in this ‘1000 Words series’.

  1. Why is evidence-informed policy important?

It’s part of the broader importance of inclusive policymaking based on a diversity of voices and the generation of knowledge about how the world works (alongside a debate about how it should work).

  1. What is the most common misconception about evidence-informed policy?

I think that many scientists are too quick to dismiss politics – and identify ‘policy based evidence’ driven by ideological and emotional politicians – rather than understand the ever-present limits to the use of evidence in policy. I think many also exaggerate the lack of scientific influence on policy by focusing on the most salient issues.

  1. What are the most common mistakes made by researchers or policymakers?

The classic mistake by researchers is to think that you make a good argument by bombarding people with a lot of information without thinking about how they’ll receive it. An important mistake that policymakers can make is to rely too much on the experts they know and trust, rather than seeking ways to identify diverse and ‘state of the art’ sources of information.

  1. What is the single most important advice to researchers/scientists who want to have policy impact?

Think about your audience and how they demand information: get their attention with a simple story, describe the problem in ways they understand (and think about the world), and show that your solution is technically and politically feasible.

  1. How do you change minds with facts and evidence?

Engage for the long term, recognising your ‘enlightenment’ role. Something dramatic would have to happen to change minds immediately and dramatically – it would be akin to a religious conversion. Or, in politics, it’s about finding a sympathetic audience (different minds) in another policymaking venue or hoping for a change of government. In other words, this is about the power of participants as much as the power of evidence and ideas.

  1. How should you communicate uncertainty about the evidence?

Since I study politics, I’d focus on the political choices here. You can communicate uncertainty in academic journals via ‘limitations’ sections and expect robust challenge on your evidence from your peers. In politics, if you show uncertainty – and your competitor does not – you may be at a disadvantage, and may need to do some soul searching about how much uncertainty you hold back. The rules change as soon as you become a scientist and advocate.

  1. How do you measure the policy impact of evidence?

In ways that are not conducive to ‘impact’ measurement by research bodies! For example, with colleagues, I tracked the influence of evidence on smoking harms on policy. In ‘leading countries’ it took 2-3 decades, and depended on three conditions: (1) key actors ‘frame’ the evidence to set a policy agenda; (2) the policy environment is generally conducive to evidence-informed change; and (3) key actors exploit ‘windows of opportunity’ for each policy change. In most countries, policy change of this scale has not happened. In such cases, we can never say that evidence simply wins the day.

  1. Who or What are your “must-reads”?

I partly took more notice of this topic after reading two articles by Kathryn Oliver and colleagues:

Oliver, K., Innvar, S., Lorenc, T., Woodman, J. and Thomas, J. (2014a) ‘A systematic review of barriers to and facilitators of the use of evidence by policymakers’ BMC health services research, 14 (1), 2. http://www.biomedcentral.com/1472-6963/14/2

Oliver, K., Lorenc, T., & Innvær, S. (2014b) ‘New directions in evidence-based policy research: a critical analysis of the literature’, Health Research Policy and Systems, 12, 34 http://www.biomedcentral.com/content/pdf/1478-4505-12-34.pdf

I was struck by the argument here, that policymakers often fund sophisticated models for evidence-based policymaking but don’t understand or use them:

Nilsson, M., Jordan, A., Turnpenny, J., Hertin, J., Nykvist, B. and Russel, D. (2008) ‘The use and non-use of policy appraisal tools in public policy making: an analysis of three European countries and the European Union’, Policy Sciences, 41, 4, 335-55

It’s also worth reading this account, which shows that policymakers don’t have the same respect for a ‘hierarchy’ of evidence/ methods as many scientists:

Bédard, P. and Ouimet, M. (2012) ‘Cognizance and consultation of randomized controlled trials among ministerial policy analysts’ Review of Policy Research, 29, 5, 625-644

For more information, start with my EBPM page

20170512_095446

1 Comment

Filed under Evidence Based Policymaking (EBPM), Uncategorized

Five advantages of blogging

This is my third ‘hey, let’s blog’ event, so it finally dawned on me to write a blog post about it. See also Fiona Miller’s account of the Stirling event.

I don’t know much about blogging research, so will focus on my personal experience of its advantages. One frequent academic argument against blogging is that it takes you away from more important parts of the job, such as teaching and research. My argument is that it helps you do both things more effectively.

See also the accounts of the disadvantages, which often relate to the ways in which they make you vulnerable to personal abuse on social media (examples 1, 2, 3).

Advantage 1: Clarity

Writing a blog has improved my academic writing. When you blog, you write for a non-specialist audience. You use less jargon or explain its meaning and value. You assume that people will not read your work unless you front-load the ‘reveal’. You need a catchy and tweetable title, to provide a ‘hook’ in the first sentence, and to show your work in a few hundred words (perhaps to encourage people to read more of your work). When you develop these skills, you can use them while writing journal article titles, abstracts, and introductions.

If you like, you can also write a blog post instead of relying on the paper/ powerpoint combo for workshops and conferences, since a 4-paper panel at conferences is usually an endurance test, and a blog post reminds you to say why people should be interested in the paper (e.g. recent examples on evidence/ policy and Scottish independence).

Advantage 2: Timeliness

It can take years for people to read an article you publish in a top journal. Sometimes the article is worth the wait. In other cases, I think it’s best to see this work as part of a package in which the article is one of the last things to appear. There is a good case to be made for taking your time to get articles right, but a less good case to keep it a secret while you do so.

Advantage 3: Exposure

It’s now common to say that we make better links with practitioners and policymakers by making our writing more accessible (short, punchy, and one click away). In my experience, the biggest payoff has been with other academics. Politics colleagues will mention my blog (and textbook) more than my articles. I can also use introductory blog posts to communicate ideas with colleagues in other disciplines – and/ or in other countries – without expecting them to do weeks of homework on the foundational texts. In each case, it works partly because we struggle to find the time to read, and appreciate a short story. Indeed, my articles are one click away on my website, but very, very, very, very few people read them.

However, you don’t need a personal blog. In fact, my most exposureyish posts have been elsewhere, including two in the Guardian’s political science blog (on evidence-based policymaking, and (with Kathryn Oliver) the dilemmas that arise when we seek it), some on the LSE blog (I tried really hard to compare tobacco and alcohol policy – look! There’s a video!), and many in The Conversation.

Advantage 4: Teaching and Learning

Teaching. The most-used page of my website hosts a series of 1000 Word summaries of policy concepts (the ‘policy cycle’ got 26000 hits in 2016). I use them, like a gateway drug, to teach undergraduate and MPP modules: they can get a feel for the concept quickly then do further reading. They now come with podcasts, which I use instead of lectures (for workshops). Other academics also use the podcasts, particularly when their students are new to policy studies (e.g. David P. Carter).

Learning. I also ask my students to write blog posts as part of their coursework, to help them learn how to write in a concise and punchy way for a non-academic audience. In most cases, students excel at this kind of work, as part of a package of assessment in which they learn how to communicate the same insights in many different ways.

Advantage 5: Unexpected benefits

When I started blogging I didn’t really know what it was for. I used to copy and paste my article abstracts, or complain about David Cameron’s handling of Scottish independence. This was at a time in which colleagues at my former University were reticent about self-publicity, and sending round a link to a new journal article via the departmental email was pushing it a bit. Now, self-promotion seems to be part of the job, and we might expect some benefits without really knowing what they’ll be. For example, my links with some very interesting people in places like the European Commission and Alliance for Useful Evidence have arisen largely from blogging.

We all have different things that tickle us in life. For me, the most tickling part of the unexpected benefit of blogging is that I now (almost!) top the following google searches: policy cycle, multiple streams, advocacy coalition framework, punctuated equilibrium theory, the politics of evidence based policymaking, and the psychology of policymaking. I’m also doing my best to push out the other Paul Cairney from the first page of google, but Wikipedia is getting in the way. The more serious point is that a personal blog might need to generate attention through social media first, before it catches fire and rises up the search engine pages.

Leave a comment

Filed under Academic innovation or navel gazing, Folksy wisdom, Uncategorized

The Scottish Government’s holistic education policy: a story of profound success or failure?

The Scottish Government experience of education can give us all a profound lesson, but I’m not yet sure what that lesson will be. The positive lesson might be that you can have a holistic approach to education provision, which has a strategy for childcare, early years, and schools that support further and higher education policy effectively. In particular, its key aim is to address inequality in attainment from a very early age, to solve one driver of unequal access to higher education. More people have a chance of a place at University and higher education remains free.

The negative lesson might be that if you don’t solve the problem at an early stage, your other policies look regressive and reinforce inequalities. Instead of seeing a government committed in a meaningful way to reducing educational inequalities throughout a life course, we see government hubris in one area supporting a vote-chasing and damaging policy in another. Free University education remains a benefit for the higher attainers, and inequalities are reinforced by the lack of financial support for low income students.

In a party political context, we can decide very quickly what lesson to take: for the SNP and its supporters, we are on course for a game changing education policy at all levels. Free tuition fees will become the symbol of its overall success. For their critics, policy is failing at almost every stage and the SNP is saved only by our fixation on the constitution as the beacon for our attention and source of policy obstacles. Every pound spent on free tuition fees for the middle classes is a pound not spent on tackling the worrying levels of attainment inequalities in schools (a point that the Scottish Government often seems to support, with reference to the ‘Heckman curve’ on the greater benefits of spending on high quality education at an early age).

As usual, the truth is likely to be in the middle but, because superficial partisan positions are often so extreme, the middle is a very large space. Without more honesty about what we can generally expect from government policies, and what we can reasonably expect from specific current and future initiatives, this debate will remain a source of poor entertainment, not enlightenment.

What can a government do to reduce educational inequality? What will it do?

The main focus of our ‘game-changer versus hubris’ debate comes from a striking speech by First Minister Nicola Sturgeon on the SNP Government’s aim to abolish inequalities in education attainment. Note how starkly Sturgeon expressed this aim in August 2015:

‘My aim – to put it bluntly – is to close the attainment gap completely. It will not be done overnight – I accept that. But it must be done. After all, its existence is more than just an economic and social challenge for us all. It is a moral challenge. Indeed, I would argue that it goes to the very heart of who we are and how we see ourselves as a nation’.

Sturgeon’s uncompromising language suggests that Scottish governments can and will produce a profound level of influence on socio-economic outcomes.

UK government ministers have abandoned such language partly because they frame the problem increasingly as an individual, not structural, problem. They have no stated ambition to go to the ‘root cause’ of the problem to reduce the socio-economic inequalities driving many attainment inequalities through a far more redistributive tax and benefits system.

It is therefore striking that the SNP-led Scottish Government also has no plans (and a limited ability) to take a ‘root cause’, majorly redistributive fiscal, approach. Instead, we see the use of public services to mitigate the effects of socioeconomic inequalities. This strategy relies heavily on ‘prevention’ policies to intervene as early as possible in people’s lives – through parenting programmes and childcare provision – to improve their chances.

Further, I have not seen another speech like it. Instead, the SNP manifesto in 2016 restated its commitment to free tuition and presented far more modest language on making: ‘significant progress in closing the attainment gap within the lifetime of the next parliament and substantially eliminating it within a decade’.

What can we realistically say about their likely effects?

In that more realistic context, you get the sense that these attainment-reducing initiatives will have limited effect. They include £100m fund to encourage new initiatives and learn from success stories such as the London Challenge, the partial return of testing pupils at key stages in schools, as part of a National Improvement Framework for Scottish education, to ‘ensure that we are making progress in closing the gap in attainment between those in our most and least deprived areas’, and possible reforms to local and regional governance to encourage learning between schools. These school-based measures come on top of substantial plans to increase or maintain childcare entitlement for 3-4 year olds, and for 2 year olds whose guardians meet income-based criteria.

In terms of the effect of attainment strategies on future University entry, we can say that the Scottish Government expects substantial results from schools in 10 years and from its expanded childcare provision (to vulnerable 2 year olds) in 15 years. As described, this does not seem like a holistic or joined up policy anymore, because it involves a gap, between the effect of one policy on another, so large that it seems unreasonable to link the two together.

An early years and attainment strategy this long-term provides almost no cover to its HE policy. Instead, we have free tuition fees in Universities which, in the absence of redistributive fiscal policy, and the long term presence of an attainment gap, reinforces inequalities in education in several ways: a reduced likelihood of University attendance in school leavers from a deprived background; a tendency for HE policy to benefit the middle classes disproportionately, since the debt burden is higher on poorer HE students, and University funding seems to come at the expense of the college places more likely to be filled by students from lower income backgrounds; and a failure to take the Heckman curve seriously enough to prompt a major shift in funding from Universities and schools to early years.

Overall, I expect that we will look back on that one speech – on the ‘moral challenge’ to ‘close the attainment gap completely’ – as an outlier. It is an aim that sounds impressive as a rhetorical device, but it is not backed up by a coherent set of public policies designed to fulfil that end, and – even with the best will in the world – it is not a policy designed to remove the regressive effects of free HE tuition.

 

1 Comment

Filed under public policy, Scottish politics, Uncategorized