Category Archives: Uncategorized

Evidence based policymaking: 7 key themes

7 themes of EBPM

I looked back at my blog posts on the politics of ‘evidence based policymaking’ and found that I wrote quite a lot (particularly from 2016). Here is a list based on 7 key themes.

1. Use psychological insights to influence the use of evidence

My most-current concern. The same basic theme is that (a) people (including policymakers) are ‘cognitive misers’ seeking ‘rational’ and ‘irrational’ shortcuts to gather information for action, so you won’t get far if you (b) bombard them with information, or (c) call them idiots.

Three ways to communicate more effectively with policymakers (shows how to use psychological insights to promote evidence in policymaking)

Using psychological insights in politics: can we do it without calling our opponents mental, hysterical, or stupid? (yes)

The Psychology of Evidence Based Policymaking: Who Will Speak For the Evidence if it Doesn’t Speak for Itself? (older paper, linking studies of psychology with studies of EBPM)

Older posts on the same theme:

Is there any hope for evidence in emotional debates and chaotic government? (yes)

We are in danger of repeating the same mistakes if we bemoan low attention to ‘facts’

These complaints about ignoring science seem biased and naïve – and too easy to dismiss

How can we close the ‘cultural’ gap between the policymakers and scientists who ‘just don’t get it’?

2. How to use policy process insights to influence the use of evidence

I try to simplify key insights about the policy process to show to use evidence in it. One key message is to give up on the idea of an orderly policy process described by the policy cycle model. What should you do if a far more complicated process exists?

The Politics of Evidence Based Policymaking: 3 messages (3 ways to say that you should engage with the policy process that exists, not a mythical process that will never exist)

Three habits of successful policy entrepreneurs (shows how entrepreneurs are influential in politics)

Why doesn’t evidence win the day in policy and policymaking? and What does it take to turn scientific evidence into policy? Lessons for illegal drugs from tobacco and There is no blueprint for evidence-based policy, so what do you do? (3 posts describing the conditions that must be met for evidence to ‘win the day’)

Writing for Impact: what you need to know, and 5 ways to know it (explains how our knowledge of the policy process helps communicate to policymakers)

How can political actors take into account the limitations of evidence-based policy-making? 5 key points (presentation to European Parliament-European University Institute ‘Policy Roundtable’ 2016)

Evidence Based Policy Making: 5 things you need to know and do (presentation to Open Society Foundations New York 2016)

What 10 questions should we put to evidence for policy experts? (part of a series of videos produced by the European Commission)

3. How to combine principles on ‘good evidence’, ‘good governance’, and ‘good practice’

My argument here is that EBPM is about deciding at the same time what is: (1) good evidence, and (2) a good way to make and deliver policy. If you just focus on one at a time – or consider one while ignoring the other – you cannot produce a defendable way to promote evidence-informed policy delivery.

Kathryn Oliver and I have just published an article on the relationship between evidence and policy (summary of and link to our article on this very topic)

We all want ‘evidence based policy making’ but how do we do it? (presentation to the Scottish Government on 2016)

The ‘Scottish Approach to Policy Making’: Implications for Public Service Delivery

The politics of evidence-based best practice: 4 messages

The politics of implementing evidence-based policies

Policy Concepts in 1000 Words: the intersection between evidence and policy transfer

Key issues in evidence-based policymaking: comparability, control, and centralisation

The politics of evidence and randomised control trials: the symbolic importance of family nurse partnerships

What Works (in a complex policymaking system)?

How Far Should You Go to Make Sure a Policy is Delivered?

4. Face up to your need to make profound choices to pursue EBPM

These posts have arisen largely from my attendance at academic-practitioner conferences on evidence and policy. Many participants tell the same story about the primacy of scientific evidence challenged by post-truth politics and emotional policymakers. I don’t find this argument convincing or useful. So, in many posts, I challenge these participants to think about more pragmatic ways to sum up and do something effective about their predicament.

Political science improves our understanding of evidence-based policymaking, but does it produce better advice? (shows how our knowledge of policymaking clarifies dilemmas about engagement)

The role of ‘standards for evidence’ in ‘evidence informed policymaking’ (argues that a strict adherence to scientific principles may help you become a good researcher but not an effective policy influencer)

How far should you go to secure academic ‘impact’ in policymaking? From ‘honest brokers’ to ‘research purists’ and Machiavellian manipulators (you have to make profound ethical and strategic choices when seeking to maximise the use of evidence in policy)

Principles of science advice to government: key problems and feasible solutions (calling yourself an ‘honest broker’ while complaining about ‘post-truth politics’ is a cop out)

What sciences count in government science advice? (political science, obvs)

I know my audience, but does my other audience know I know my audience? (compares the often profoundly different ways in which scientists and political scientists understand and evaluate EBPM – this matters because, for example, we rarely discuss power in scientist-led debates)

Is Evidence-Based Policymaking the same as good policymaking? (no)

Idealism versus pragmatism in politics and policymaking: … evidence-based policymaking (how to decide between idealism and pragmatism when engaging in politics)

Realistic ‘realist’ reviews: why do you need them and what might they look like? (if you privilege impact you need to build policy relevance into systematic reviews)

‘Co-producing’ comparative policy research: how far should we go to secure policy impact? (describes ways to build evidence advocacy into research design)

The Politics of Evidence (review of – and link to – Justin Parkhurt’s book on the ‘good governance’ of evidence production and use)

20170512_095446

5. For students and researchers wanting to read/ hear more

These posts are relatively theory-heavy, linking quite clearly to the academic study of public policy. Hopefully they provide a simple way into the policy literature which can, at times, be dense and jargony.

‘Evidence-based Policymaking’ and the Study of Public Policy

Policy Concepts in 1000 Words: ‘Evidence Based Policymaking’

Practical Lessons from Policy Theories (series of posts on the policy process, offering potential lessons for advocates of evidence use in policy)

Writing a policy paper and blog post 

12 things to know about studying public policy

Can you want evidence based policymaking if you don’t really know what it is? (defines each word in EBPM)

Can you separate the facts from your beliefs when making policy? (no, very no)

Policy Concepts in 1000 Words: Success and Failure (Evaluation) (using evidence to evaluate policy is inevitably political)

Policy Concepts in 1000 Words: Policy Transfer and Learning (so is learning from the experience of others)

Four obstacles to evidence based policymaking (EBPM)

What is ‘Complex Government’ and what can we do about it? (read about it)

How Can Policy Theory Have an Impact on Policy Making? (on translating policy theories into useful advice)

The role of evidence in UK policymaking after Brexit (argues that many challenges/ opportunities for evidence advocates will not change after Brexit)

Why is there more tobacco control policy than alcohol control policy in the UK? (it’s not just because there is more evidence of harm)

Evidence Based Policy Making: If You Want to Inject More Science into Policymaking You Need to Know the Science of Policymaking and The politics of evidence-based policymaking: focus on ambiguity as much as uncertainty and Revisiting the main ‘barriers’ between evidence and policy: focus on ambiguity, not uncertainty and The barriers to evidence based policymaking in environmental policy (early versions of what became the chapters of the book)

6. Using storytelling to promote evidence use

This is increasingly a big interest for me. Storytelling is key to the effective conduct and communication of scientific research. Let’s not pretend we’re objective people just stating the facts (which is the least convincing story of all).

Combine Good Evidence and Emotional Stories to Change the World

Storytelling for Policy Change: promise and problems

Is politics and policymaking about sharing evidence and facts or telling good stories? Two very silly examples from #SP16

7. The major difficulties in using evidence for policy to reduce inequalities

These posts show how policymakers think about how to combine (a) often-patchy evidence with (b) their beliefs and (c) an electoral imperative to produce policies on inequalities, prevention, and early intervention. I suggest that it’s better to understand and engage with this process than complain about policy-based-evidence from the side-lines. If you do the latter, policymakers will ignore you.

What do you do when 20% of the population causes 80% of its problems? Possibly nothing.

The theory and practice of evidence-based policy transfer: can we learn how to reduce territorial inequalities?

We need better descriptions than ‘evidence-based policy’ and ‘policy-based evidence’: the case of UK government ‘troubled families’ policy

How can you tell the difference between policy-based-evidence and evidence-based-policymaking?

Early intervention policy, from ‘troubled families’ to ‘named persons’: problems with evidence and framing ‘valence’ issues

Key issues in evidence-based policymaking: comparability, control, and centralisation

The politics of evidence and randomised control trials: the symbolic importance of family nurse partnerships

Two myths about the politics of inequality in Scotland

Social investment, prevention and early intervention: a ‘window of opportunity’ for new ideas?

A ‘decisive shift to prevention’: how do we turn an idea into evidence based policy?

Can the Scottish Government pursue ‘prevention policy’ without independence?

Note: these issues are discussed in similar ways in many countries. One example that caught my eye today:

 

All of this discussion can be found under the EBPM category: https://paulcairney.wordpress.com/category/evidence-based-policymaking-ebpm/T

See also the special issue on maximizing the use of evidence in policy

Palgrave C special

Leave a comment

Filed under Uncategorized

How to Navigate Complex Policy Designs

Tanya Krister

This is a guest post by Professor Tanya Heikkila (left) and  Professor Krister Andersson (right), discussing how to use insights from the institutional analysis and development (IAD) framework to think about how to design policy effectively. The full paper has been submitted to the series for Policy and Politics called Practical Lessons from Policy Theories.

Policy design is hard work. Attempts in the United States Congress to repeal and replace, or even revise, the Affordable Care Act (ACA) in the spring and summer or 2017 are good examples of the challenges.  Even setting aside issues of congressional partisanship, key lawmakers and President Trump seemed taken aback by how complex both the ACA and the underlying health care insurance issue are. Lawmakers struggled for several months, and failed, to come up with viable policy options that could make health insurance in the U.S. more cost-efficient, while providing flexibility to states, private firms, and individuals who must comply with the policy.

The recent experience in the US with revising the ACA is illustrative of a larger question:

How do policymakers or analysts navigate and design effective policies around complex collective problems?

Tips from the IAD framework

In general, these tips encourage a policymaker or analyst to take a step back and start with the basics—what we might call the research lessons for sound policy design. Below we offer a summary of three basic lessons.

1. People are capable of solving collective problems from the bottom-up, both outside and within government settings.

Some conventional wisdom has suggested that it’s best to leave policy design up to the “experts,” which might include technocrats, senior elected officials, or even benevolent dictators.

Institutional analysis research has shown, however, that people who are most closely tied to or affected by a policy issue, are not only capable but often best at designing policies. Excluding these groups from the design stage is likely to lead to weakened legitimacy resulting in less compliance.

One might assume that this idea only applies to localized issues, or problems that are small in scope.  Yet, in many formal policy settings or government venues, decision-makers similarly must learn to embrace the wisdom of their collectives, and of the actors affected by the policy issue.

Of course this takes time, because to do this well requires the development of trust, experience, and adequate information gathering. And in the case of a complex decision-making body like the US Congress, policymakers may believe it’s more important to take advantage of a political window of opportunity and push through decisions quickly. Additionally, individual elected officials may believe that their interests are best served by taking a position on an issue that seems to be most politically palatable to their constituents, without thinking in the interest of the whole country (or even their states).

This line of thinking back-fired in the summer of 2017 with the ACA repeal efforts. A more robust approach, as supported by the IAD-logic, would be for elected officials to take the time to build and open dialogue, work directly with key constituents in thinking about the best approach for health care for the country, and spend time with each other, in Congress and in state legislatures, to make to design and/or adapt policy toward more productive ends.

2. Use a framework to navigate the complexity of policy design

We can’t devise, amend, or adapt policy effectively without understanding it. Yet, people have a natural tendency to engage in reactionary and emotional reasoning when they are passionate about an issue. Even when not colored by emotional reasoning, policymakers and analysts also come with their own professional and cultural biases that can lead to ‘blind spots’.

Frameworks can help us guard against the tendency toward biased analyses or a focus on features of a policy that are most obvious. A good framework provides a toolkit for identifying the general factors that policymakers and other stakeholders should consider when developing new policies or trying to understanding existing policies.

The IAD framework, for instance, helps identify:

  • the relevant actors for devising and implementing policy
  • their information, knowledge motivations, and interactions
  • the various types of rules these actors already use
  • the biophysical and community context surrounding the actors
  • the evaluative criteria appropriate for assessing the policy in question.

It also helps us understand what external or broader rules can constrain or enable particular actions.  In other words, it makes us aware of our ‘blind spots’ and enables a deeper understanding of the factors that are important for effective policy design.

3. Stop looking for panaceas. Instead, understand the nuance of policy design

There are no silver bullets to policy designs. General blueprint solutions rarely work and it is important to design contextually appropriate policy interventions.

This requires scrutinizing the design elements of policies (e.g., the types of rules embedded in policies), and how they interact with the incentives and information that different actors use in devising or implementing a policy.

It also involves deep knowledge of the factors that can structure their choices in light of the local context where policies are used.  Policy designs ultimately are more likely to be successful when they acknowledge the autonomy and problem-solving capabilities of people whose behavior the interventions are trying to change.

In the context of the ACA, for instance, policymakers need to be tuned into the ways in which different rules for participating in health exchanges can affect the incentives for participation, which can be critical for keeping insurance costs down.

While these types of recommendations may seem intuitive, we often fail to follow them. Perhaps this is because they require extra time, which may mean we miss our windows of opportunity.

Ultimately, resisting the temptation to rush policy analysis and design, and instead engaging with people who understand the policy, using an analytical framework to mitigate our biases, and paying close attention to the nuances of policy design helps us produce more successful policies.

2 Comments

Filed under Uncategorized

Policy concepts in 1000 or 500 words

Imagine that your audience is a group of scientists who have read everything and are only interested in something new. You need a new theory, method, study, or set of results to get their attention.

Let’s say that audience is a few hundred people, or half a dozen in each subfield. It would be nice to impress them, perhaps with some lovely jargon and in-jokes, but almost no-one else will know or care what you are talking about.

Imagine that your audience is a group of budding scientists, researchers, students, practitioners, or knowledge-aware citizens who are new to the field and only interested in what they can pick up and use (without devoting their life to each subfield). Novelty is no longer your friend. Instead, your best friends are communication, clarity, synthesis, and a constant reminder not to take your knowledge and frame of reference for granted.

Let’s say that audience is a few gazillion people. If you want to impress them, imagine that you are giving them one of the first – if not the first – ways of understanding your topic. Reduce the jargon. Explain your problem and why people should care about how you try to solve it. Clear and descriptive titles. No more in-jokes (just stick with the equivalent of ‘I went to the doctor because a strawberry was growing in my arse, and she gave me some cream for it’).

At least, that’s what I’ve been telling myself lately. As things stand, my most-read post of all time is destined to be on the policy cycle, and most people read it because it’s the first entry on a google search. Most readers of that post may never read anything else I’ve written (over a million words, if I cheat a bit with the calculation). They won’t care that there are a dozen better ways to understand the policy process. I have one shot to make it interesting, to encourage people to read more. The same goes for the half-dozen other concepts (including multiple streams, punctuated equilibrium theory, the Advocacy Coalition Framework) which I explain to students first because I now do well in google search (go on, give it a try!).

I also say this because I didn’t anticipate this outcome when I wrote those posts. Now, a few years on, I’m worried that they are not very good. They were summaries of chapters from Understanding Public Policy, rather than first principles discussions, and lots of people have told me that UPP is a little bit complicated for the casual reader. So, when revising it, I hope to make it better, and by better I mean to appeal to a wider audience without dumping the insights. I have begun by trying to write 500-words posts as, I hope, improvements on the 1000-word versions. However, I am also open to advice on the originals. Which ones work, and which ones don’t? Where are the gaps in exposition? Where are the gaps in content?

This post is 500 words.

https://paulcairney.wordpress.com/1000-words/

https://paulcairney.wordpress.com/500-words/

Leave a comment

Filed under 1000 words, 500 words, Uncategorized

The role of ‘standards for evidence’ in ‘evidence informed policymaking’

Key points:

  • Maintaining strict adherence to evidence standards is like tying your hands behind your back
  • There is an inescapable trade-off between maintaining scientific distance for integrity and using evidence pragmatically to ensure its impact
  • So, we should not divorce discussions of evidence standards from evidence use

I once spoke with a policymaker from a health unit who described the unintended consequences of their self-imposed evidence standards. They held themselves to such a high standard of evidence that very few studies met their requirements. So, they often had a very strong sense of ‘what works’ but, by their own standards, could not express much confidence in their evidence base.

As a result, their policy recommendations were tentative and equivocal, and directed at a policymaker audience looking for strong and unequivocal support for (often controversial) policy solutions before putting their weight behind them. Even if evidence advocates had (what they thought to be) the best available evidence, they would not make enough of it. Instead, they value their reputations, based on their scientific integrity, producing the best evidence, and not making inflated claims about the policy implications. Let’s wait for more evidence, just to be sure. Let’s not use suboptimal evidence, even if it’s all we have.

Your competitors do not tie their own hands behind their backs in this way

I say this because I have attended many workshops, in the last year, in which we discuss principles for science advice and guidelines or standards for the evidence part of ‘evidence-based’ or ‘evidence-informed’ policymaking.

During such discussions, it is common for people to articulate the equivalent of crossing their fingers and hoping that they can produce rules for the highest evidence standards without the unintended consequences. If you are a fan of Field of Dreams, we can modify the slogan: if you build it (the evidence base), they will come (policymakers will use it sincerely, and we’ll all be happy).*

If you build it

Or, if you are more of a fan of Roger Pielke Jr, you can build the evidence base while remaining an ‘honest broker’, providing evidence without advocacy. Ideally, we’d want to maintain scientific integrity and have a major impact on policy (akin to me wanting to eat chips all day and lose weight) but, in the real world, may settle for the former.

If so, perhaps a more realistic way of phrasing the question would be: what rules for evidence should a small group of often-not-very-influential people agree among themselves? In doing so, we recognise that very few policy actors will follow these rules.

What happens when we don’t divorce a discussion of (a) standards of evidence from (b) the use of evidence for policy impact?

The latter depends on far more than evidence, such as the usual factors we discuss in these workshops, including trust in the messenger, and providing a ‘timely’ message.  Perhaps a high-standard evidence base helps the former (providing a Kite Mark for evidence) and one aspect of the latter (the evidence is there when you demand it). However, policy studies-inspired messages go much further, such as in Three habits of successful entrepreneurs which describes the strategies people use for impact:

  1. They tell simple and persuasive stories to generate demand for their evidence
  2. They have a technically and politically feasible (evidence-based) policy solution ready to chase policy problems
  3. They adapt their strategies to the scale of their policy environments, akin to surfers in large and competitive political systems, but more like Poseidon in less competitive ‘policy communities’ or subnational venues.

In such cases, the availability of evidence becomes secondary to:

  1. the way you use evidence to frame a policy problem, which is often more about the way you connect information to policymaker demand than the quality of the evidence.

Table 1

  1. your skills in being able to spot the right time to present evidence-based solutions, which is not about a mythical policy cycle, and not really about the availability of evidence or speed of delivery.

Table 2

So, when we talk about any guidance for evidence advocates, such as pursued by INGSA, I think you will always find these tensions between evidence quality and scientific integrity on the one hand, and ‘timeliness’ or impact on the other. You don’t address the need for timely evidence simply by making sure that the evidence exists in a database.

I discuss these tensions further on the INGSA website: Principles of science advice to government: key problems and feasible solutions

.

.

*Perhaps you’d like to point out that when Ray Kinsella built it (the baseball field in his cornfield), he did come (the ghost of Shoeless Joe Jackson appeared to play baseball there). I’m sorry to have to tell you this, but actually that was Ray Liotta pretending to be Jackson.

 

2 Comments

Filed under Evidence Based Policymaking (EBPM), Uncategorized

Professor Grant Jordan

Grant died on Friday. He was my friend and mentor for a long time, and I’m glad that I knew him for so long.

Let me share four short stories with you, to give you a sense of Grant.

The first is of an internationally respected scholar. When Grant and I visited Hokkaido University in 2004, his talk was preceded by a warm and glowing reference by our hosts. One host held aloft Governing Under Pressure and described it as one of the most influential books of his time. It was certainly one of the biggest influences on me and many of my colleagues (as I describe below), and his work was valued by as many colleagues in the US as the UK.

governing under pressure

The second is of his continuous help to students and colleagues. I associate Grant’s working life with one image: his open office door. He kept an office directly across from the departmental office, ensuring that if any student came to us with a problem, he’d be the first to try to solve it. It might not sound like much to say that he knew Aberdeen University’s regulations inside-out, but it symbolised his continuous efforts to make sure that students benefited from them. He offered the same continuous help to many of our colleagues.

The third is of a quietly influential mentor. Some of Grant’s advice was cryptic, but all of it was useful. At key points of my career and intellectual development, he was there to offer pointers and challenge incomplete thought. For example, often his favourite approach was to quote Mandy Rice-Davies (‘he would say that’) to challenge any claims I made from elite interviews, and you can see the effect of such caution on much of my published work.

The fourth is of a funny person. There are odd-sounding times that I’ll remember, such as when Darren Halpin and I met up with Grant and his partner Andrea in Toronto (during an APSA conference), and they were already quite merry when we arrived. Or, when at the football together (Grant was a big Aberdeen FC fan), he would often offer me and my children some suspiciously warm toffees from his pocket. Maybe one of his funniest lines is now one of the most poignant: when I tried to do a decent speech on his career at his retirement dinner, he described it as a speech well suited to his funeral (I guess you had to be there to appreciate the humour!).

Jordan and Richardson’s intellectual legacy

Grant will leave an intellectual legacy. His work with Jeremy Richardson is still at the heart of my understanding of politics and policy. In my first undergraduate year at Strathclyde, Jeremy lectured on British politics and public policy. He presented an image of politics that drew me in (partly via Yes Minster) and an argument – made in partnership with Grant – that I still use most frequently to this day:

  • The size and scope of the state is so large that it is in danger of becoming unmanageable. The same can be said of the crowded environment in which huge numbers of actors seek policy influence. Consequently, the state’s component parts are broken down into policy sectors and sub-sectors, with power spread across government.
  • Elected policymakers can only pay attention to a tiny proportion of issues for which they are responsible. So, they pay attention to a small number and ignore the rest. In effect, they delegate policymaking responsibility to other actors such as bureaucrats, often at low levels of government.
  • At this level of government and specialisation, bureaucrats rely on specialist organisations for information and advice.
  • Those organisations trade that information/advice and other resources for access to, and influence within, the government (other resources may relate to who groups represent – such as a large, paying membership, an important profession, or a high status donor or corporation).
  • Therefore, most public policy is conducted primarily through small and specialist policy communities that process issues at a level of government not particularly visible to the public, and with minimal senior policymaker involvement.

They initially made this argument in the late 1970s and early 1980s, before the rise of ‘Thatcherism’ in the UK. Then, they used it to challenge the idea that Thatcherism marked a radical departure in policymaking. Of course, this new phase of policymaking was important and distinctive, but the same basic argument outlined above still applies, and Jordan went on to do further empirical studies, with colleagues such as William Maloney, to highlight a surprising amount of policymaking stability and continuity. In other words, Jordan and Richardson have shown, and continue to show that the UK generally does not live up to its ‘majoritarian’ image. It’s an argument that I use to this day.

Overall, I was very lucky to have Grant in my life for so long, and I hope he knew how many people shared this combination of admiration for his work and fondness of him.

 

 

5 Comments

Filed under Uncategorized

What’s That?

I apologise in advance.

In the olden days, my former lecturer would describe the wackily short amount of time you would have to explain a problem and possible solution to an elected policymaker: 1 or 2 pages of A4 (scroll to the end).

More recently, I heard a ramped-up version of this pressure: the time it takes to walk them to their car (a variant on the ‘elevator pitch’ idea).

It’s not, enough, though, is it? We need more of a sense of the pressure we face to explain complicated concepts quickly and concisely to different audiences.

So, I suggest this example, borne out of my recent experience as granddad to a 13-month old baby who has reached the stage of pointing randomly at things and saying ‘what’s that?’.

pull along mouse xylophone

I tried and failed to explain that (a) it’s a wooden mouse playing a xylophone, on wheels, which moves when you pull the string along the floor, and (b) the funny thing is, you don’t see many mice playing the xylophone these days.

For this type of audience, there’s only time to reply ‘a toy’ (or, perhaps if you are thinking laterally, ‘would you like a biscuit’?).

So, the new test of conciseness is: what can you explain in the length of time between a child pointing to one thing and saying ‘what’s that’ and the same child pointing to another thing and saying ‘what’s that?’.

Leave a comment

Filed under Uncategorized

My thoughts on how the all-women Wonder Woman screening affects me

Leave a comment

Filed under Uncategorized