Monthly Archives: September 2017

A 5-step strategy to make evidence count

5 stepsLet’s imagine a heroic researcher, producing the best evidence and fearlessly ‘speaking truth to power’. Then, let’s place this person in four scenarios, each of which combines a discussion of evidence, policy, and politics in different ways.

  1. Imagine your hero presents to HM Treasury an evidence-based report concluding that a unitary UK state would be far more efficient than a union state guaranteeing Scottish devolution. The evidence is top quality and the reasoning is sound, but the research question is ridiculous. The result of political deliberation and electoral choice suggests that your hero is asking a research question that does not deserve to be funded in the current political climate. Your hero is a clown.
  2. Imagine your hero presents to the Department of Health a report based on the systematic review of multiple randomised control trials. It recommends that you roll out an almost-identical early years or public health intervention across the whole country. We need high ‘fidelity’ to the model to ensure the correct ‘dosage’ and to measure its effect scientifically. The evidence is of the highest quality, but the research question is not quite right. The government has decided to devolve this responsibility to local public bodies and/ or encourage the co-production of public service design by local public bodies, communities, and service users. So, to focus narrowly on fidelity would be to ignore political choices (perhaps backed by different evidence) about how best to govern. If you don’t know the politics involved, you will ask the wrong questions or provide evidence with unclear relevance. Your hero is either a fool, naïve to the dynamics of governance, or a villain willing to ignore governance principles.        
  3. Imagine two fundamentally different – but equally heroic – professions with their own ideas about evidence. One favours a hierarchy of evidence in which RCTs and their systematic review is at the top, and service user and practitioner feedback is near the bottom. The other rejects this hierarchy completely, identifying the unique, complex relationship between practitioner and service user which requires high discretion to make choices in situations that will differ each time. Trying to resolve a debate between them with reference to ‘the evidence’ makes no sense. This is about a conflict between two heroes with opposing beliefs and preferences that can only be resolved through compromise or political choice. This is, oh I don’t know, Batman v Superman, saved by Wonder Woman.
  4. Imagine you want the evidence on hydraulic fracturing for shale oil and gas. We know that ‘the evidence’ follows the question: how much can we extract? How much revenue will it produce? Is it safe, from an engineering point of view? Is it safe, from a public health point of view? What will be its impact on climate change? What proportion of the public supports it? What proportion of the electorate supports it? Who will win and lose from the decision? It would be naïve to think that there is some kind of neutral way to produce an evidence-based analysis of such issues. The commissioning and integration of evidence has to be political. To pretend otherwise is a political strategy. Your hero may be another person’s villain.

Now, let’s use these scenarios to produce a 5-step way to ‘make evidence count’.

Step 1. Respect the positive role of politics

A narrow focus on making the supply of evidence count, via ‘evidence-based policymaking’, will always be dispiriting because it ignores politics or treats political choice as an inconvenience. If we:

  • begin with a focus on why we need political systems to make authoritative choices between conflicting preferences, and take governance principles seriously, we can
  • identify the demand for evidence in that context, then be more strategic and pragmatic about making evidence count, and
  • be less dispirited about the outcome.

In other words, think about the positive and necessary role of democratic politics before bemoaning post-truth politics and policy-based-evidence-making.

Step 2. Reject simple models of evidence-based policymaking

Policy is not made in a cycle containing a linear series of separate stages and we won’t ‘make evidence count’ by using it to inform our practices.


You might not want to give up the cycle image because it presents a simple account of how you should make policy. It suggests that we elect policymakers then: identify their aims, identify policies to achieve those aims, select a policy measure, ensure that the selection is legitimised by the population or its legislature, identify the necessary resources, implement and then evaluate. Or, policymakers aided by expert policy analysts make and legitimise choices, skilful public servants carry them out, and, policy analysts assess the results using evidence.

One compromise is to keep the cycle then show how messy it is in practice:

However, there comes a point when there is too much mess, and the image no longer helps you explain (a) to the public what you are doing, or (b) to providers of evidence how they should engage in political systems. By this point, simple messages from more complicated policy theories may be more useful.

Or, we may no longer want a cycle to symbolise a single source of policymaking authority. In a multi-level system, with many ‘centres’ possessing their own sources of legitimate authority, a single and simple policy cycle seems too artificial to be useful.

Step 3. Tell a simple story about your evidence

People are ‘cognitive misers’ seeking ‘rational’ and ‘irrational’ shortcuts to gather information for action, so you won’t get far if you bombard them with too much evidence. Policymakers already have too much evidence and they seek ways to reduce their cognitive load, relying on: (a) trusted sources of concise evidence relevant to their aims, and (b) their own experience, gut instinct, beliefs, and emotions.

The implication of both shortcuts is that we need to tell simple and persuasive stories about the substance and implications of the evidence we present. To say that ‘the evidence does not speak for itself’ may seem trite, but I’ve met too many people who assume naively that it will somehow ‘win the day’. In contrast, civil servants know that the evidence-informed advice they give to ministers needs to relate to the story that government ministers tell to the public.


Step 4.  Tailor your story to many audiences

In a complex or multi-level environment, one story to one audience (such as a minister) is not enough. If there are many key sources of policymaking authority – including public bodies with high autonomy, organisations and practitioners with the discretion to deliver services, and service users involved in designing services – there are many stories being told about what we should be doing and why. We may convince one audience and alienate (or fail to inspire) another with the same story.

Step 5. Clarify and address key dilemmas with political choice, not evidence

Let me give you one example of the dilemmas that must arise when you combine evidence and politics to produce policy: how do you produce a model of ‘evidence based best practice’ which combines evidence and governance principles in a consistent way? Here are 3 ideal-type models which answer the question in very different ways

Table 1 Three ideal types EBBP

The table helps us think through the tensions between models, built on very different principles of good evidence and governance.

In practice, you may want to combine different elements, perhaps while arguing that the loss of consistency is lower than the gain from flexibility. Or, the dynamics of political systems limit such choice or prompt ad hoc and inconsistent choices.

I built a lot of this analysis on the experiences of the Scottish Government, which juggles all three models, including a key focus on improvement method in its Early Years Collaborative.

However, Kathryn Oliver and I show that the UK government faces the same basic dilemma and addresses it in similar ways.

The example freshest in my mind is Sure Start. Its rationale was built on RCT evidence and systematic review. However, its roll-out was built more on local flexibility and service design than insistence on fidelity to a model. More recently, the Troubled Families programme initially set the policy agenda and criteria for inclusion, but increasingly invites local public bodies to select the most appropriate interventions, aided by the Early Intervention Foundation which reviews the evidence but does not insist on one-best-way. Emily St Denny and I explore these issues further in our forthcoming book on prevention policy, an exemplar case study of a field in which it is difficult to know how to ‘make evidence count’.

If you prefer a 3-step take home message:

  1. I think we use phrases like ‘impact’ and ‘make evidence count’ to reflect a vague and general worry about a decline in respect for evidence and experts. Certainly, when I go to large conferences of scientists, they usually tell a story about ‘post-truth’ politics.
  2. Usually, these stories do not acknowledge the difference between two different explanations for an evidence-policy gap: (a) pathological policymaking and corrupt politicians, versus (b) complex policymaking and politicians having to make choices despite uncertainty.
  3. To produce evidence with ‘impact’, and know how to ‘make evidence count’, we need to understand the policy process and the demand for evidence within it.

*Background. This is a post for my talk at the Government Economic Service and Government Social Research Service Annual Training Conference (15th September 2017). This year’s theme is ‘Impact and Future-Proofing: Making Evidence Count’. My brief is to discuss evidence use in the Scottish Government, but it faces the same basic question as the UK Government: how do you combine principles of evidence quality and governance principles? In other words, if you were in a position to design an (a) evidence-gathering system and (b) a political system, you’d soon find major points of tension between them. Resolving those tensions involves political choice, not more evidence. Of course, you are not in a position to design both systems, so the more complicated question is: how do you satisfy principles of evidence and governance in a complex policy process, often driven by policymaker psychology, over which you have little control?  Here are 7 different ‘answers’.

Powerpoint Paul Cairney @ GES GSRS 2017

Leave a comment

Filed under Evidence Based Policymaking (EBPM), public policy, Scottish politics, UK politics and policy

Policy Concepts in 500 Words: Social Construction and Policy Design

Why would a democratic political system produce ‘degenerative’ policy that undermines democracy? Social Construction and Policy Design (SCPD) describes two main ways in which policymaking alienates many citizens:

1. The Social Construction of Target Populations

High profile politics and electoral competition can cause alienation:

  1. Political actors compete to tell ‘stories’ to assign praise or blame to groups of people. For example, politicians describe value judgements about who should be rewarded or punished by government. They base them on stereotypes of ‘target populations’, by (a) exploiting the ways in which many people think about groups, or (b) making emotional and superficial judgements, backed up with selective use of facts.
  2. These judgements have a ‘feed-forward’ effect: they are reproduced in policies, practices, and institutions. Such ‘policy designs’ can endure for years or decades. The distribution of rewards and sanctions is cumulative and difficult to overcome.
  3. Policy design has an impact on citizens, who participate in politics according to how they are characterised by government. Many know they will be treated badly; their engagement will be dispiriting.

Some groups have the power to challenge the way they are described by policymakers (and the media and public), and receive benefits behind the scenes despite their poor image. However, many people feel powerless, become disenchanted with politics, and do not engage in the democratic process.

SCTP depicts this dynamic with a 2-by-2 table in which target populations are described positively/ negatively and more or less able to respond:

SCPD 500 words 2 by 2

2. Bureaucratic and expert politics

Most policy issues are not salient and politicised in this way. Yet, low salience can exacerbate problems of citizen exclusion. Policies dominated by bureaucratic interests often alienate citizens receiving services. Or a small elite dominates policymaking when there is high acceptance that (a) the best policy is ‘evidence based’, and (b) the evidence should come from experts.

Overall, SCPD describes a political system with major potential to diminish democracy, containing key actors (a) politicising issues to reward or punish populations or (b) depoliticising issues with reference to science and objectivity. In both cases, policy design is not informed by routine citizen participation.

Take home message for students: SCPD began as Schneider and Ingram’s description of the US political system’s failure to solve major problems including inequality, poverty, crime, racism, sexism, and effective universal healthcare and education. Think about how its key drivers apply elsewhere: (1) some people make and exploit quick and emotional judgements for political gain, and others refer to expertise to limit debate; (2) these judgements inform policy design; and, (3) policy design sends signals to citizens which can diminish or boost their incentive to engage in politics.

For more, see the 1000-word and 5000-word versions. The latter has a detailed guide to further reading.





Leave a comment

Filed under 500 words, Evidence Based Policymaking (EBPM), Psychology Based Policy Studies, public policy, Storytelling

How to design ‘maps’ for policymakers relying on their ‘internal compass’

This is a guest post by Dr Richard Simmons (below, in between Profs Alex Marsh and Catherine Farrell), discussing how to use insights from ‘cultural theory’ to think about how to design institutions to help policymakers think and make good decisions. The full paper has been submitted to the series for Policy and Politics called Practical Lessons from Policy Theories.

My reading of Richard’s argument is through the lens of debates on ‘evidence-based policymaking’ and policymaker psychology. Policymakers can only pay attention to a tiny proportion of the world, and a small proportion of their responsibilities. They combine ‘rational’ and ‘irrational’ informational shortcuts to act quickly and make ‘good enough’ decisions despite high uncertainty. You can choose to criticize their reliance on ‘cognitive frailties’ (and perhaps design institutions to limit their autonomous powers) or praise their remarkable ability to develop ‘fast and frugal’ heuristics (and perhaps work with them to refine such techniques). I think Richard takes a relatively sympathetic approach to describing ‘thinking, fast and slow’:

  1. His focus on an ‘internal compass’ describes aspects of fast thinking (using gut or instinct, emotion, habit, familiarity) but without necessarily equating a compass with negative cognitive ‘biases’ that get in the way of ‘rationality’.
  2. Instead, an internal compass can be remarkably useful, particularly if combined with a ‘map’ of the ‘policymaking terrain’. Terrain can describe the organisations, rules, and other sources of sources of direction and learning in a policymaking system.
  3. Both compass and map are necessary; they reinforce the value of each other.
  4. However, perhaps unlike a literal map, we cannot simply design one-size-fits-all advice for policymakers. We need to speak with them in some depth, to help them work out what they think the policy problem is and probe how they would like to solve it.
  5. In that context, the role of policy analysis is to help policymakers think about and ask the right questions as it is to provide tailor-made answers.

It is a paradox that in a world where there are often more questions than answers, policymakers more often seek to establish and then narrow the range of possible answers than to establish and then narrow the range of possible questions. There are different explanations for this:

  • One is that policymakers occupy a ‘rational’, ‘technical’ space, in which everything from real-time data to scientific evidence can be balanced in ‘problem-solving’. This means doing the background work to support authoritative choice between policy alternatives, perhaps via ‘structured interactions’, as a way to bring order to the weight of evidence and expertise.
  • Another is that policymakers occupy a ‘formally structured’, ‘political’ space, in which the contest to have ‘agenda-setting’ power has already been decided. For policy actors, this means learning not to ‘question why’ – accepting the legitimacy, if not the substantive nature, of their political masters’ concerns and (outwardly, at least) directing their attention accordingly.
  • A third explanation, however, is that policymakers occupy a ‘complex’ and ‘uncertain’ space, in which ‘What is a good question?’ is itself a good question. Yet often we lack good ways to ask questions about questions – at least, without encountering accusations of either ‘avoiding the problem’ or ‘re-politicising technical concerns’.

Given that questions are logically prior to a technical search for the ‘best answer’, it seems sensible that the search for the ‘best question’ should start away from the realm of ‘the technical’ (cf. Explanation 1). As a result, two possible options remain in response to ‘What is a good question?’:

  1. That it is a subjectively-normative question that depends on the eyes of the beholder, best aggregated and understood in the realm of ‘the political’ (which returns us to Explanation 2).
  2. That it is an objectively-normative question that depends on ‘social construction’ in policy work, best aggregated and understood in the realm of ‘the institutional’ (which returns us to Explanation 3).

Option 1 is the stuff of basic politics; it will not be explored further here. This leaves the ‘objectively-normative’ Option2 , which is less often explored. This option is ‘normative’ in the sense that it gives space to framing a problem in ways that acknowledges different sets of values and beliefs, that may be socially constructed in different ways. It is ‘objective’ in the sense that it seeks to resolve tensions between these different sets of values and beliefs in without re-opening the kind of explicit competition normally reserved for the realm of politics. Yet in its basis in the realm of institutions, some might ask: how is ‘objective’ analysis even possible?

Step Forward, Cultural Theory (CT)?

There are still, perhaps, a few ‘flat-earth’ policy actors who doubt the importance of institutions. Yet even for those who do not deny their influence, the prospect of ‘objective’ institutional analysis seems remote. By their very nature hard to define and intangible to the eye, institutions can seem esoteric, ephemeral, and resistant to meaningful measurement. However, new developments in Cultural Theory (CT) can help policymakers get a grip. Now well-established in policy circles, CT constitutes institutions along two dimensions into four (and only four) rival ‘cultural biases’ – hierarchy, individualism, egalitarianism and fatalism:

Simmons 1

Importantly, biases combine in different institutional patterns – and the mix matters. Dominant patterns tend to structure policy problems and guide policymakers’ response in different ways. Through exposure and experience, institutional patterns can become internalised in their ‘thought styles’; as an ‘internal compass’ that directs ‘fast-thinking’. No bad thing, perhaps – unless and until this sends them off course. Faulty compass readings arise when narrow thought-styles become ‘cultural blinkers’. As ‘practical wisdom’ may be present in more than one location, navigation risks arise if a course ahead is plotted that blocks out other constructions of the problem.

How would policymakers realise when they have led themselves astray? One way might be ‘slower thinking’ – reflection on their actions to question their constructions and promote dynamic learning. CT provides a parsimonious way of framing such reflection. Simplifying complex criteria into just four cultural categories, skilled ‘reflective policymakers’ are facilitated more quickly to ask ‘good’ questions. However, space for such ‘slow thinking’ is often limited in practical policy work. When this is closed-out by constraints of time and attention, what more has Cultural Theory to offer?

Recent work operationalises CT to both map institutions and chart ‘internal compass’ bearings. Using stakeholder surveys to ‘materialise the intangible’, institutions are mapped by visually overlaying policy actors’ perceptions of how policy problems ‘actually are’ governed, with those of how they ‘should be’ governed:

Simmons 2

Meanwhile, as points of congruence and dissonance emerge in this institutional environment, policymakers internal compass bearings show the likelihood that they might actually see them. Together, these tools up-the-odds of asking ‘good’ questions even further than reflection. Actors learn to navigate both change and the obstacles to change.

But is this not still too slow? This process may indeed seem slow, but intelligent investment in institutional analysis potentially has payoffs that can make it worth the wait. The ‘map’ immediately provides a provocation for more valid and reliable policy practice – definitively directing policy attention, no matter where the compass is pointing. Speed and accuracy increases. Not only this; over time, such purposive action serves to maintain, create and disrupt institutions. As new patterns emerge that subconsciously subvert existing thought-styles, the compass itself is recalibrated. There are fewer faulty readings to direct ‘fast thinking’. Speed and accuracy increases again…

For some, the tools provided by CT may seem blunt; for others, as esoteric and ephemeral as the institutions this theory purports to portray. The recent work reported here certainly requires further refinement to reinforce its validity and reliability. But the effort of doing so may be a small price to pay. The practical potential of CT’s meaningful measurement makes further progress a beguiling prospect.



1 Comment

Filed under Uncategorized

Three ways to encourage policy learning

Claire Claudio

This is a guest post by  Claire A. Dunlop (left) and Claudio M. Radaelli (right), discussing how to use insights from the Policy Learning literature to think about how to learn effectively or adapt to the processes of ‘learning’ in policymaking that are more about politics than education. The full paper has been submitted to the series for Policy and Politics called Practical Lessons from Policy Theories.

We often hear that university researchers are ‘all brains but no common sense’. There is often some truth to this stereotype. The literature on policy learning is an archetypal example of being high in IQ but low on street smarts. Researchers have generated a huge amount of ‘policy learning’ taxonomies, concepts and methods without showing what learning can offer policy-makers, citizens and societies.

This is odd because there is a substantive demand and need for practical insights on how to learn. Issues include economic growth, the control of corruption, and improvement in schools and health. Learning organisations range from ‘street level bureaucracies’ to international regulators like the European Union and the World Trade Organization.

To help develop a more practical agenda, we distil three major lessons from the policy learning literature.

1. Learning is often the by-product of politics, not the primary goal of policymakers

There is usually no clear incentive for political actors to learn how to improve public policy. Learning is often the by-product of bargaining, the effort to secure compliance with the law and rules, social participation, or problem-solving when there is radical uncertainty. This means that in politics we should not assume that politicians, bureaucrats, civil society organizations, experts interact to improve public policy. Consensus, participation, formal procedures, and social certification are more important.

Therefore, we have to learn how to design incentives so that the by-product of learning is actually generated. Otherwise, few actors will play the game of the policy-making process with learning as their first goal. Learning is all around us, but it appears in different forms, depending on whether the context is (a) bargaining, (b) compliance, (c) participation or (d) problem-solving under conditions of high uncertainty.

2. Each mode of learning has its triggers or hindrances

(a) Bargaining requires repeated interaction, low barriers to contract and mechanisms of preference aggregation.

(b) Compliance without trust in institutions is stymied.

(c) Participation needs its own deliberative spaces and a type of participant willing to go beyond the ‘dialogue of the deaf’. Without these two triggers, participation is chaotic, highly conflictual and inefficient.

(d) Expertise is key to problem-solving, but governments should design their advisory committees and special commissions of inquiry by recruiting a broad range of experts. The risk of excluding the next Galileo Galilei in a Ptolemaic committee is always there.

At the same time, there are specific hindrances:

(a) Bargaining stops when the winners are always the same (if you are thinking of Germany and Greece in the European Union you are spot-on).

(b) Hierarchy does not produce efficient compliance unless those at the top know exactly the solution to enforce.

(c) Incommensurable beliefs spoil participatory policy processes. If so, it’s better to switch to open democratic conflict, by counting votes in elections and referenda for example.

(d) Scientific scepticism and low policy capacity mar the work of experts in governmental bodies.

These triggers and hindrances have important lessons for design, perhaps prompting authorities (governments, regulators, public bodies) to switch from one context to another. For example, one can re-design the work of expert committees by including producers and consumers organizations or by allowing bargaining on the implementation of budgetary rules.

3. Beware the limitations of learning

We may get this precious by-product and avoid hindrances and traps, but still… learn the wrong lessons.

Latin America and Africa offer too many examples of diligent pupils who did exactly what they were supposed to do, but in the end implemented wrong policies. Perfect compliance does not provide breathing spaces to a policy and impairs the quality of innovation. We have to balance lay and professional knowledge. Bargaining does not allow us to learn about radical innovations; in some cases only a new participant can really change the nature of the game being played by the usual suspects.

So, whether the problem is learning how to fight organized crime and corruption, or to re-launch growth in Europe and development in Africa, the design of the policy process is crucial. For social actors, our analysis shows when and how they should try to change the nature of the game, or lobby for a re-design of the process. This lesson is often forgotten because social actors fight for a given policy objective, not for the parameters that define who does what and how in the policy process.

1 Comment

Filed under Evidence Based Policymaking (EBPM), public policy