The role of ‘standards for evidence’ in ‘evidence informed policymaking’

Key points:

• Maintaining strict adherence to evidence standards is like tying your hands behind your back
• There is an inescapable trade-off between maintaining scientific distance for integrity and using evidence pragmatically to ensure its impact
• So, we should not divorce discussions of evidence standards from evidence use

Monday @OECD ➡️ "Governing better through evidence-informed #policymaking". For Info & agenda see https://t.co/CojUkdg1PV #evidence4gov pic.twitter.com/LmsnZz94sk

— OECD Governance (@OECDgov) June 23, 2017

I once spoke with a policymaker from a health unit who described the unintended consequences of their self-imposed evidence standards. They held themselves to such a high standard of evidence that very few studies met their requirements. So, they often had a very strong sense of ‘what works’ but, by their own standards, could not express much confidence in their evidence base.

As a result, their policy recommendations were tentative and equivocal, and directed at a policymaker audience looking for strong and unequivocal support for (often controversial) policy solutions before putting their weight behind them. Even if evidence advocates had (what they thought to be) the best available evidence, they would not make enough of it. Instead, they value their reputations, based on their scientific integrity, producing the best evidence, and not making inflated claims about the policy implications. Let’s wait for more evidence, just to be sure. Let’s not use suboptimal evidence, even if it’s all we have.

Your competitors do not tie their own hands behind their backs in this way

I say this because I have attended many workshops, in the last year, in which we discuss principles for science advice and guidelines or standards for the evidence part of ‘evidence-based’ or ‘evidence-informed’ policymaking.

During such discussions, it is common for people to articulate the equivalent of crossing their fingers and hoping that they can produce rules for the highest evidence standards without the unintended consequences. If you are a fan of Field of Dreams, we can modify the slogan: if you build it (the evidence base), they will come (policymakers will use it sincerely, and we’ll all be happy).*

Or, if you are more of a fan of Roger Pielke Jr, you can build the evidence base while remaining an ‘honest broker’, providing evidence without advocacy. Ideally, we’d want to maintain scientific integrity and have a major impact on policy (akin to me wanting to eat chips all day and lose weight) but, in the real world, may settle for the former.

If so, perhaps a more realistic way of phrasing the question would be: what rules for evidence should a small group of often-not-very-influential people agree among themselves? In doing so, we recognise that very few policy actors will follow these rules.

What happens when we don’t divorce a discussion of (a) standards of evidence from (b) the use of evidence for policy impact?

The latter depends on far more than evidence, such as the usual factors we discuss in these workshops, including trust in the messenger, and providing a ‘timely’ message.  Perhaps a high-standard evidence base helps the former (providing a Kite Mark for evidence) and one aspect of the latter (the evidence is there when you demand it). However, policy studies-inspired messages go much further, such as in Three habits of successful entrepreneurs which describes the strategies people use for impact:

1. They tell simple and persuasive stories to generate demand for their evidence
2. They have a technically and politically feasible (evidence-based) policy solution ready to chase policy problems
3. They adapt their strategies to the scale of their policy environments, akin to surfers in large and competitive political systems, but more like Poseidon in less competitive ‘policy communities’ or subnational venues.

In such cases, the availability of evidence becomes secondary to:

1. the way you use evidence to frame a policy problem, which is often more about the way you connect information to policymaker demand than the quality of the evidence.

2. your skills in being able to spot the right time to present evidence-based solutions, which is not about a mythical policy cycle, and not really about the availability of evidence or speed of delivery.

So, when we talk about any guidance for evidence advocates, such as pursued by INGSA, I think you will always find these tensions between evidence quality and scientific integrity on the one hand, and ‘timeliness’ or impact on the other. You don’t address the need for timely evidence simply by making sure that the evidence exists in a database.

I discuss these tensions further on the INGSA website: Principles of science advice to government: key problems and feasible solutions

.

.

*Perhaps you’d like to point out that when Ray Kinsella built it (the baseball field in his cornfield), he did come (the ghost of Shoeless Joe Jackson appeared to play baseball there). I’m sorry to have to tell you this, but actually that was Ray Liotta pretending to be Jackson.

 

Professor Grant Jordan

Grant died on Friday. He was my friend and mentor for a long time, and I’m glad that I knew him for so long.

Let me share four short stories with you, to give you a sense of Grant.

The first is of an internationally respected scholar. When Grant and I visited Hokkaido University in 2004, his talk was preceded by a warm and glowing reference by our hosts. One host held aloft Governing Under Pressure and described it as one of the most influential books of his time. It was certainly one of the biggest influences on me and many of my colleagues (as I describe below), and his work was valued by as many colleagues in the US as the UK.

The second is of his continuous help to students and colleagues. I associate Grant’s working life with one image: his open office door. He kept an office directly across from the departmental office, ensuring that if any student came to us with a problem, he’d be the first to try to solve it. It might not sound like much to say that he knew Aberdeen University’s regulations inside-out, but it symbolised his continuous efforts to make sure that students benefited from them. He offered the same continuous help to many of our colleagues.

The third is of a quietly influential mentor. Some of Grant’s advice was cryptic, but all of it was useful. At key points of my career and intellectual development, he was there to offer pointers and challenge incomplete thought. For example, often his favourite approach was to quote Mandy Rice-Davies (‘he would say that’) to challenge any claims I made from elite interviews, and you can see the effect of such caution on much of my published work.

The fourth is of a funny person. There are odd-sounding times that I’ll remember, such as when Darren Halpin and I met up with Grant and his partner Andrea in Toronto (during an APSA conference), and they were already quite merry when we arrived. Or, when at the football together (Grant was a big Aberdeen FC fan), he would often offer me and my children some suspiciously warm toffees from his pocket. Maybe one of his funniest lines is now one of the most poignant: when I tried to do a decent speech on his career at his retirement dinner, he described it as a speech well suited to his funeral (I guess you had to be there to appreciate the humour!).

Jordan and Richardson’s intellectual legacy

Grant will leave an intellectual legacy. His work with Jeremy Richardson is still at the heart of my understanding of politics and policy. In my first undergraduate year at Strathclyde, Jeremy lectured on British politics and public policy. He presented an image of politics that drew me in (partly via Yes Minster) and an argument – made in partnership with Grant – that I still use most frequently to this day:

• The size and scope of the state is so large that it is in danger of becoming unmanageable. The same can be said of the crowded environment in which huge numbers of actors seek policy influence. Consequently, the state’s component parts are broken down into policy sectors and sub-sectors, with power spread across government.
• Elected policymakers can only pay attention to a tiny proportion of issues for which they are responsible. So, they pay attention to a small number and ignore the rest. In effect, they delegate policymaking responsibility to other actors such as bureaucrats, often at low levels of government.
• At this level of government and specialisation, bureaucrats rely on specialist organisations for information and advice.
• Those organisations trade that information/advice and other resources for access to, and influence within, the government (other resources may relate to who groups represent – such as a large, paying membership, an important profession, or a high status donor or corporation).
• Therefore, most public policy is conducted primarily through small and specialist policy communities that process issues at a level of government not particularly visible to the public, and with minimal senior policymaker involvement.

They initially made this argument in the late 1970s and early 1980s, before the rise of ‘Thatcherism’ in the UK. Then, they used it to challenge the idea that Thatcherism marked a radical departure in policymaking. Of course, this new phase of policymaking was important and distinctive, but the same basic argument outlined above still applies, and Jordan went on to do further empirical studies, with colleagues such as William Maloney, to highlight a surprising amount of policymaking stability and continuity. In other words, Jordan and Richardson have shown, and continue to show that the UK generally does not live up to its ‘majoritarian’ image. It’s an argument that I use to this day.

Overall, I was very lucky to have Grant in my life for so long, and I hope he knew how many people shared this combination of admiration for his work and fondness of him.

OLYMPUS DIGITAL CAMERA

 

 

Policy in 500 Words: Multiple Streams Analysis and Policy Entrepreneurs

In a previous post, I ask: if the policy cycle does not exist, what do we do? In this artificial policy cycle world, ‘comprehensively rational’ policymakers combine their values with evidence to define policy problems and their aims, ‘neutral’ bureaucracies produce many possible solutions consistent with those aims, and policymakers select the ‘best’ or most ‘evidence based’ solution, setting in motion a cycle of stages including legitimation, implementation, evaluation, and the choice to maintain or change policy.

In the real world, policymaking is not so simple, and three ‘stages’ seem messed up:

• Defining problems. There is too much going on in the world, and too much information about problems. So, policymakers have to ignore most problems and most ways to understand them. They use ‘rational’ and ‘irrational’ short-cuts to help them pay attention to a manageable number of issues and address problems without fully understanding them. Problems get attention based on how they are ‘framed’: actors use evidence to reduce uncertainty, and persuasion to reduce ambiguity (they focus our minds on one way to understand a problem).
• Producing solutions. When policymaker attention lurches to a problem, it’s too late to produce a new solution that is technically feasible (will it work as intended?) and politically feasible (is it acceptable to enough people in the ‘community’?). While attention lurches quickly, feasible solutions take time to develop.
• Making choices. The willingness and ability of policymakers to select a solution is fleeting, based on their beliefs, perception of the ‘national mood’, and the feedback they receive from interest groups and political parties.

Don’t think of these things as linear ‘stages’. Instead, they are independent ‘streams’ which have to come together during a brief ‘window of opportunity’. All key factors – heightened attention to a problem (problem stream), an available and feasible solution (policy stream), and the motive to select it (politics stream) – must come together at the same time, or the opportunity is lost. If you think of the streams as water, the metaphor suggests that when they come together they are hard to separate.  Instead, a ‘window of opportunity’ is like a space launch in which policymakers will abort the mission unless every factor is just right.

So, what do we do in the absence of a policy cycle?

Policy entrepreneurs’ know how to respond. They use persuasion to frame problems, help develop feasible solutions, wait for the right time to present them, and know how to adapt to their environment to exploit ‘windows of opportunity’.

Take home message for students. It is easy to dismiss the policy cycle, and find better explanations, but don’t stop there. Consider how to turn this insight into action. If policymaking is so messy, how should people respond? Studying ‘entrepreneurs’ helps us identify strategies to influence the policy process, but how could elected policymakers justify such a weird-looking process? Finally, look at many case studies to see how scholars describe MSA. It’s a flexible metaphor, but is there a coherent literature with common themes?

Next steps for reading:

Policy Concepts in 1000 Words: Multiple Streams Analysis – an expanded version of this introductory post

What is a policy entrepreneur? – describes various ways in which policy scholars define ‘entrepreneur’

Three habits of successful policy entrepreneurs – a blog post and paper on how entrepreneurs deal with ‘organized anarchy’

Whatever happened to multiple streams analysis? – introduces an article by Michael Jones and me on MSA studies

Paul Cairney and Michael Jones (2016) ‘Kingdon’s Multiple Streams Approach: What Is the Empirical Impact of this Universal Theory?’ Policy Studies Journal, 44, 1, 37-58 PDF (Annex to Cairney Jones 2016) (special issue of PSJ)

Paul Cairney and Nikos Zahariadis (2016) ‘Multiple streams analysis’ in Zahariadis, N. (eds) Handbook of Public Policy Agenda-Setting (Cheltenham: Edward Elgar) PDF see also

There is also a chapter on MSA and ideas in Understanding Public Policy.

If you are feeling really energetic, you can read the source texts:

Kingdon J (1984) Agendas, Alternatives and Public Policies New York: Harper Collins

Cohen, M., March, J. and Olsen, J. (1972) ‘A Garbage Can Model of Organizational Choice’, Administrative Science Quarterly, 17, 1, 1-25

Picture source: NASA Deep Space Gateway to Open Opportunities for Distant Destinations

 

 

 

 

 

The role of evidence in UK policymaking after Brexit

Join us for free panel discussion on policmaking post-Brexit today at 15:30 with @CairneyPaul & @jameswilsdon et al https://t.co/D9oWn9QNpR

— Humanities & Social Sciences Communications (@HSScomms) June 22, 2017

We are launching a series of papers on evidence and policy in Palgrave Communications. Of course, we used Brexit as a hook, to tap into current attention to instability and major policy change. However, many of the issues we discuss are timeless and about surprising levels of stability and continuity in policy processes, despite periods of upheaval.

In my day, academics would build their careers on being annoying, and sometimes usefully annoying. This would involve developing counterintuitive insights, identifying gaps in analysis, and challenging a ‘common wisdom’ in political studies. Although not exactly common wisdom, the idea of ‘post truth’ politics, a reduction in respect for ‘experts’, and a belief that Brexit is a policymaking game-changer, are great candidates for some annoyingly contrary analysis.

In policy studies, many of us argue that things like elections, changes of government, and even constitutional changes are far less important than commonly portrayed. In media and social media accounts, we find hyperbole about the destabilising and changing impact of the latest events. In policy studies, we often stress stability and continuity.  My favourite old example regards the debates from the 1970s about electoral reform. While some were arguing that first-past-the-post was a disastrous electoral system since it produces swings of government, instability, and incoherent policy change, Richardson and Jordan would point out surprisingly high levels of stability and continuity.

In part, this is because the state is huge, policymakers can only pay attention to a tiny part of it, and therefore most of it is processed as a low level of government, out of the public spotlight.

These insights still have profound relevance today, for two key reasons.

1. The role of experts is more important than you think

This larger process provides far more opportunities for experts than we’d associate with ‘tip of the iceberg’ politics.

Some issues are salient. They command the interest of elected politicians, and those politicians often have firm beliefs that limit the ‘impact’ of any evidence that does not support their beliefs.

However, most issues are not salient. They command minimal interest, they are processed by other policymakers, and those policymakers are looking for information and advice from reliable experts.

Indeed, a lot of policy studies highlight the privileged status of certain experts, at the expense of most members of the public (which is a useful corrective to the story, associated with Brexit, that the public is too emotionally driven, too sceptical of experts, and too much in charge of the future of constitutional change).

So, Brexit will change the role of experts, but expect that change to relate to the venue in which they engage, and the networks of which they are a part, more than the practices of policymakers. Much policymaking is akin to an open door to government for people with useful information and a reputation for being reliable in their dealings with policymakers.

2. Provide less evidence for more impact

If the problem is that policymakers can only pay attention to a tiny proportion of their responsibilities, the solution is not to bombard them with a huge amount of evidence. Instead, assume that they seek ways to ignore almost all information while still managing to make choices. The trick may be to provide just enough information to prompt demand for more, not oversupply evidence on the assumption that you have only one chance for influence.

With Richard Kwiatkoswki, I draw on policy and psychology studies to help us understand how to supply evidence to anyone using ‘rational’ and ‘irrational’ ways to limit their attention, information processing, and thought before making decisions.

Our working assumption is that policymakers need to gather information quickly and effectively, so they develop heuristics to allow them to make what they believe to be good choices. Their solutions often seem to be driven more by their emotions than a ‘rational’ analysis of the evidence, partly because we hold them to a standard that no human can reach. If so, and if they have high confidence in their heuristics, they will dismiss our criticism as biased and naïve. Under those circumstances, restating the need for ‘evidence-based policymaking’ is futile, and naively ‘speaking truth to power’ counterproductive.

Instead, try out these strategies:

1. Develop ways to respond positively to ‘irrational’ policymaking

Instead of automatically bemoaning the irrationality of policymakers, let’s marvel at the heuristics they develop to make quick decisions despite uncertainty. Then, let’s think about how to respond pragmatically, to pursue the kinds of evidence informed policymaking that is realistic in a complex and constantly changing policymaking environment.

2. Tailor framing strategies to policymaker cognition

The usual advice is to minimise the cognitive burden of your presentation, and use strategies tailored to the ways in which people pay attention to, and remember information.

The less usual advice includes:

• If policymakers are combining cognitive and emotive processes, combine facts with emotional appeals.
• If policymakers are making quick choices based on their values and simple moral judgements, tell simple stories with a hero and a clear moral.
• If policymakers are reflecting a ‘group emotion’, based on their membership of a coalition with firmly-held beliefs, frame new evidence to be consistent with the ‘lens’ through which actors in those coalitions understand the world.

3. Identify the right time to influence individuals and processes

Understand what it means to find the right time to exploit ‘windows of opportunity’.

‘Timing’ can refer to the right time to influence an individual, which involves how open they are to, say, new arguments and evidence.

Or, timing refers to a ‘window of opportunity’ when political conditions are aligned. I discuss the latter in a separate paper on effective ‘policy entrepreneurs’.

4. Adapt to real-world organisations rather than waiting for an orderly process to appear

Politicians may appear confident of policy and with a grasp of facts and details, but are (a) often vulnerable and therefore defensive or closed to challenging information, and/ or (b) inadequate in organisational politics, or unable to change the rules of their organisations.

So, develop pragmatic strategies: form relationships in networks, coalitions, or organisations first, then supply challenging information second. To challenge without establishing trust may be counterproductive.

5. Recognise that the biases we ascribe to policymakers are present in ourselves and our own groups.

Identifying only the biases in our competitors may help mask academic/ scientific examples of group-think, and it may be counterproductive to use euphemistic terms like ‘low information’ to describe actors whose views we do not respect. This is a particular problem for scholars if they assume that most people do not live up to their own imagined standards of high-information-led action (often described as a ‘deficit model’ of engagement).

It may be more effective to recognise that: (a) people’s beliefs are honestly held, and policymakers believe that their role is to serve a cause greater than themselves.; and, (b) a fundamental aspect of evolutionary psychology is that people need to get on with each other, so showing simple respect – or going further, to ‘mirror’ that person’s non-verbal signals – can be useful even if it looks facile.

This leaves open the ethical question of how far we should go to identify our biases, accept the need to work with people whose ways of thinking we do not share, and how far we should go to secure their trust without lying about one’s beliefs.

At the very least, we do not suggest these 5 strategies as a way to manipulate people for personal gain. They are better seen as ways to use psychology to communicate well. They are also likely to be as important to policy engagement regardless of Brexit. Venues may change quickly, but the ways in which people process information and make choices may not.

 

We have a window of opportunity to improve Scottish devolution, so let’s start with parliamentary reform

Now is the perfect time to think about maximising the benefits of Scottish devolution. The first independence referendum produced important new constitutional changes, enshrined in the Scotland Act 2016. It now seems unlikely that there will be a second referendum any time soon. So, we have a window of opportunity to take a step back, understand the Scottish Government’s new powers, and consider how the Scottish Parliament can best hold it to account, encourage new voices in politics, and represent the views of the public. In other words, to think about how devolution’s original aims, summed up by the phrase ‘new politics’ (as compared at the time to ‘old Westminster’).

The Scottish Parliament is now a mature institution, supported strongly by the public, and here to stay. In the early years of devolution, it is understandable that there was more concern about public support and the financial cost of enhanced Scottish democracy. Now, it is time to start looking to the future, to note the Parliament’s success to date, and build on examples of good practice to make it as effective as possible as it takes on new responsibilities.

The Commission on Parliamentary Reform has been doing just that. It was set up by the Presiding Officer, Ken Macintosh MSP, with John McCormick its chair and consisting of representatives of all main parties. Although asked to give the Parliament an ‘MOT’, a metaphor which might suggest that the fewer issues raised the better, the Commission came up with a series of measures to turbo boost the Parliament.

To strengthen the role of the Scottish Parliament, it recommends:

• A more assertive role for the Presiding Officer, to control parliamentary business and encourage more effective scrutiny and debate.
• Leaner and stronger committees, led by elected convenors, and more able to set the political agenda rather than simply respond to the government.
• More independent MSPs, trained to be parliamentarians first and representatives of political parties second.

To enhance the Parliament’s equality and diversity principles, it recommends:

• Establishing a vision for an equal and diverse parliament, setting benchmarks for MSP recruitment from under-represented groups, backed up by measures to influence political party recruitment.

To engage in new ways with the public and give ‘the people’ a voice, it recommends:

• That the Parliament becomes a world leader in public engagement, experimenting with new ways to gather views and evidence, identifying the most excluded groups and ways to overcome barriers to engagement, and working with schools to encourage greater knowledge of the Scottish Parliament.

A big part of this shift of thinking should be about the ways in which we describe and appreciate MSPs. If elected politics should not be the part-time occupation of people with independent wealth and a larger income from other jobs, we should make sure it provides the kinds of pay and conditions that we’d take for granted in other parts of the public sector. If politics is a profession that requires particular skills which improve with experience, we should be more hesitant about complaining about a political class full of MSPs that never had a ‘proper job’. If we expect MSPs to work incredibly hard for their constituents and engage fully in Parliament, we should note that they routinely work well over the 35 hours we’d associate with most other public sector jobs, often at the expense of their long term health and life outside work. We should also be realistic about what we can expect from MSPs if the Scottish Parliament is to retain its ‘family friendly’ aims and allow MSPs to balance work and life.

This argument about appreciating MSPs should help us take an interesting story from the report. It doesn’t say that we should trade responsibilities and rights, but that we should place as much emphasis on their rights as we should their responsibilities.

So, it provides a series of recommendations which ask MSPs to reconsider their responsibilities to the Scottish Parliament, to focus a little bit less on partisanship and a bit more on the Parliament as an institution. We should expect MSPs to honour their responsibilities to the Parliament, to engage in parliamentary work as parliamentarians (particularly in committees), not simply representatives of their parties, and to help improve the quality of Scottish policy, not simply criticise policy from the Scottish Government.

It also suggests that MSPs deserve comparable employment rights to any other public employees, including the positive moves – such as parental leave and workplace flexibility – that help us remain effective at work and practice ‘self care’ and care for others. In fact, as a beacon for Scottish democracy, the Scottish Parliament should also be a beacon of progressiveness, turning its founding commitment to a ‘family friendly’ culture into best practice for all MSPs with caring and other personal responsibilities.  Further, MSPs should not have to apologise for being paid fairly while working hard for the Parliament or when claiming legitimate funds to support their work.

It is in that spirit that I’d suggest reading key parts of the report. Yes, the commission makes strong recommendations for important reforms in parliamentary rules and MSP behaviour. However, it also invites us to remember that the Parliament is here to stay, and a lot of the credit for its success should go to the people who work there.

The full report can be found here: https://parliamentaryreform.scot/

Paul Cairney is Professor of Politics and Public Policy at the University of Stirling. From January to June 2017 he was the advisor to the Commission on Parliamentary Reform, but these are his personal thoughts on the report.

 

Three habits of successful policy entrepreneurs

This post is one part of a series – called Practical Lessons from Policy Theories and it summarizes this article (PDF).

‘Policy entrepreneurs’ invest their time wisely for future reward, and possess key skills that help them adapt particularly well to their environments. They are the agents for policy change who possess the knowledge, power, tenacity, and luck to be able to exploit key opportunities. They draw on three strategies:

1. Don’t focus on bombarding policymakers with evidence.

Scientists focus on making more evidence to reduce uncertainty, but put people off with too much information. Entrepreneurs tell a good story, grab the audience’s interest, and the audience demands information.

2. By the time people pay attention to a problem it’s too late to produce a solution.

So, you produce your solution then chase problems.

3. When your environment changes, your strategy changes.

For example, in the US federal level, you’re in the sea, and you’re a surfer waiting for the big wave. In the smaller subnational level, on a low attention and low budget issue, you can be Poseidon moving the ‘streams’. In the US federal level, you need to ‘soften’ up solutions over a long time to generate support. In subnational or other countries, you have more opportunity to import and adapt ready-made solutions.

It all adds up to one simple piece of advice – timing and luck matters when making a policy case – but policy entrepreneurs know how to influence timing and help create their own luck.

Full paper: Three habits of successful policy entrepreneurs

(Note: the previous version was friendlier and more focused on entrepreneurs)

For more on ‘multiple streams’ see:

Paul Cairney and Michael Jones (2016) ‘Kingdon’s Multiple Streams Approach: What Is the Empirical Impact of this Universal Theory?’ Policy Studies Journal, 44, 1, 37-58 PDF (Annex to Cairney Jones 2016) (special issue of PSJ)

Paul Cairney and Nikos Zahariadis (2016) ‘Multiple streams analysis: A flexible metaphor presents an opportunity to operationalize agenda setting processes’ in Zahariadis, N. (eds) Handbook of Public Policy Agenda-Setting (Cheltenham: Edward Elgar) PDF see also

I use a space launch metaphor in the paper. If you prefer different images, have a look at 5 images of the policy process. If you prefer a watery metaphor (it’s your life, I suppose), click Policy Concepts in 1000 Words: Multiple Streams Analysis

For more on entrepreneurs:

So you want to be a policy entrepreneur? https://t.co/y8RG7MOSKV by @MikeMintrom

— Professor Paul Cairney (@CairneyPaul) October 29, 2019