Almost. I have sent a full draft following external feedback and review (next stage: copy-editing). All going well, it will be out in November 2019.
Here is the ACF story.
People engage in politics to turn their beliefs into policy. They form advocacy coalitions with people who share their beliefs, and compete with other coalitions. The action takes place within a subsystem devoted to a policy issue, and a wider policymaking process that provides constraints and opportunities to coalitions.
The policy process contains multiple actors and levels of government. It displays a mixture of intensely politicized disputes and routine activity. There is much uncertainty about the nature and severity of policy problems. The full effects of policy may be unclear for over a decade. The ACF sums it up in the following diagram:
Policy actors use their beliefs to understand, and seek influence in, this world. Beliefs about how to interpret the cause of and solution to policy problems, and the role of government in solving them, act as a glue to bind actors together within coalitions.
The outcome is often long-term policymaking stability and policy continuity because the ‘core’ beliefs of coalitions are unlikely to shift and one coalition may dominate the subsystem for long periods.
There are two main sources of change.
The ACF began as the study of US policymaking, focusing largely on environmental issues. It has changed markedly to reflect the widening of ACF scholarship to new policy areas, political systems, and methods.
For example, the flow diagram’s reference to the political system’s long term coalition opportunity structures is largely the response to insights from comparative international studies:
As such, like all theories in this series, the ACF discusses elements that it would treat as (a) universally applicable, such as the use of beliefs to address bounded rationality, and (b) context-specific, such as the motive and opportunity of specific people to organize collectively to translate their beliefs into policy.
Image source: Weible, Heikkila, Ingold, and Fischer (2016: 6)
I thank James Georgalakis for inviting me to speak at the inaugural event of IDS’ new Evidence into Policy and Practice Series, and the audience for giving extra meaning to my story about the politics of ‘evidence-based based policymaking’. The talk (using powerpoint) and Q&A is here:
James invited me to respond to some of the challenges raised to my talk – in his summary of the event – so here it is.
I’m working on a ‘show, don’t tell’ approach, leaving some of the story open to interpretation. As a result, much of the meaning of this story – and, in particular, the focus on limiting participation – depends on the audience.
For example, consider the impact of the same story on audiences primarily focused on (a) scientific evidence and policy, or (b) participation and power.
Normally, when I talk about evidence and policy, my audience is mostly people with scientific or public health backgrounds asking why do policymakers ignore scientific evidence? I am usually invited to ruffle feathers, mostly by challenging a – remarkably prevalent – narrative that goes like this:
In that context, I suggest that there are many claims to policy-relevant knowledge, policymakers have to ignore most information before making choices, and they are not in control of the policy process for which they are ostensibly in charge.
Limiting participation as a strategic aim
Then, I say to my audience that – if they are truly committed to maximising the use of scientific evidence in policy – they will need to consider how far they will go to get what they want. I use the metaphor of an ethical ladder in which each rung offers more influence in exchange for dirtier hands: tell stories and wait for opportunities, or demonise your opponents, limit participation, and humour politicians when they cherry-pick to reinforce emotional choices.
It’s ‘show don’t tell’ but I hope that the take-home point for most of the audience is that they shouldn’t focus so much on one aim – maximising the use of scientific evidence – to the detriment of other important aims, such as wider participation in politics beyond a reliance on a small number of experts. I say ‘keep your eyes on the prize’ but invite the audience to reflect on which prizes they should seek, and the trade-offs between them.
Limited participation – and ‘windows of opportunity’ – as an empirical finding
I did suggest that most policymaking happens away from the sphere of ‘exciting’ and ‘unruly’ politics. Put simply, people have to ignore almost every issue almost all of the time. Each time they focus their attention on one major issue, they must – by necessity – ignore almost all of the others.
For me, the political science story is largely about the pervasiveness of policy communities and policymaking out of the public spotlight.
The logic is as follows. Elected policymakers can only pay attention to a tiny proportion of their responsibilities. They delegate the rest to bureaucrats at lower levels of government. Bureaucrats lack specialist knowledge, and rely on other actors for information and advice. Those actors trade information for access. In many cases, they develop effective relationships based on trust and a shared understanding of the policy problem.
Trust often comes from a sense that everyone has proven to be reliable. For example, they follow norms or the ‘rules of the game’. One classic rule is to contain disputes within the policy community when actors don’t get what they want: if you complain in public, you draw external attention and internal disapproval; if not, you are more likely to get what you want next time.
For me, this is key context in which to describe common strategic concerns:
Where is the power analysis in all of this?
I rarely use the word power directly, partly because – like ‘politics’ or ‘democracy’ – it is an ambiguous term with many interpretations (see Box 3.1). People often use it without agreeing its meaning and, if it means everything, maybe it means nothing.
However, you can find many aspects of power within our discussion. For example, insider and outsider strategies relate closely to Schattschneider’s classic discussion in which powerful groups try to ‘privatise’ issues and less powerful groups try to ‘socialise’ them. Agenda setting is about using resources to make sure issues do, or do not, reach the top of the policy agenda, and most do not.
These aspects of power sometimes play out in public, when:
However, they are no less important when they play out routinely:
In other words, the word ‘power’ is often hidden because the most profound forms of power often seem to be hidden.
In the context of our discussion, power comes from the ability to define some evidence as essential and other evidence as low quality or irrelevant, and therefore define some people as essential or irrelevant. It comes from defining some issues as exciting and worthy of our attention, or humdrum, specialist and only relevant to experts. It is about the subtle, unseen, and sometimes thoughtless ways in which we exercise power to harness people’s existing beliefs and dominate their attention as much as the transparent ways in which we mobilise resources to publicise issues. Therefore, to ‘maximise the use of evidence’ sounds like an innocuous collective endeavour, but it is a highly political and often hidden use of power.
This post provides (a generous amount of) background for my ANZSOG talk Teaching evidence based policy to fly: transferring sound policies across the world.
The event’s description sums up key conclusions in the literature on policy learning and policy transfer:
In this post, I connect these conclusions to broader themes in policy studies, which suggest that:
As usual, I suggest that we focus less on how we think we’d like to do it, and more on how people actually do it.
Policy transfer describes the use of evidence about policy in one political system to help develop policy in another. Taken at face value, it sounds like a great idea: why would a government try to reinvent the wheel when another government has shown how to do it?
Therefore, wouldn’t it be nice if I turned up to the lecture, equipped with a ‘blueprint’ for ‘evidence based’ policy transfer, and declared how to do it in a series of realistic and straightforward steps? Unfortunately, there are three main obstacles:
Instead, policy learning is a political process – involving the exercise of power to determine what and how to learn – and it is difficult to separate policy transfer from the wider use of evidence and ideas in policy processes.
Let’s take each of these points in turn, before reflecting on their implications for any X-step guide:
3 reasons why ‘evidence based’ does not describe policymaking
Therefore, it is unclear how one government can, or should, generate evidence of another government’s policy success.
The generation of policy transfer lessons is a highly political process in which actors adapt to this need to prioritise information while competing with each other. They exercise power to: prioritise some information and downplay the rest, define the nature of the policy problem, and evaluate the success of another government’s solutions. There is a strong possibility that policymakers will import policy solutions without knowing if, and why, they were successful.
We should not treat ‘policy transfer’ as separate from the policy process in which policymakers and influencers engage. Rather, the evidence of international experience competes with many other sources of ideas and evidence within a complex policymaking system.
The literature on ‘policy learning’ tells a similar story
Studies of the use of evaluation evidence (perhaps to answer the question: was this policy successful?) have long described policymakers using the research process for many different purposes, from short term problem-solving and long-term enlightenment, to putting off decisions or using evidence cynically to support an existing policy.
We should therefore reject the temptation to (a) equate ‘policy learning’ with a simplistic process that we might associate with teachers transmitting facts to children, or (b) assume that adults simply change their beliefs when faced with new evidence. Rather, Dunlop and Radaelli describe policy learning as a political process in the following ways:
1.It is collective and rule-bound
Individuals combine cognition and emotion to process information, in organisations with rules that influence their motive and ability to learn, and in wider systems, in which many actors cooperate and compete to establish the rules of evidence gathering and analysis, or policymaking environments that constrain or facilitate their action.
2.’Evidence based’ is one of several types of policy learning
3.The process can be ‘dysfunctional’: driven by groupthink, limited analysis, and learning how to dominate policymaking, not improve policy.
Their analysis can produce relevant take-home points such as:
What does the literature on transfer tell us?
‘Policy transfer’ can describe a spectrum of activity:
In that context, some of the literature focuses on warning against unsuccessful policy transfer caused by factors such as:
However, other studies highlight some major qualifications:
The use of evidence to spread policy innovation requires a combination of profound political and governance choices
When encouraging policy diffusion within a political system, choices about: (a) what counts as ‘good’ evidence of policy success have a major connection to (b) what counts as good governance.
For example, consider these ideal-types or models in table 1:
In one scenario, we begin by relying primarily on RCT evidence (multiple international trials) and import a relatively fixed model, to ensure ‘fidelity’ to a proven intervention and allow us to measure its effect in a new context. This choice of good evidence limits the ability of subnational policymakers to adapt policy to local contexts.
In another scenario, we begin by relying primary on governance principles, such as to respect local discretion as well as incorporate practitioner and user experience as important knowledge claims. The choice of governance model relates closely to a less narrow sense of what counts as good evidence, but also a more limited ability to evaluate policy success scientifically.
In other words, the political choice to privilege some forms of evidence is difficult to separate from another political choice to privilege the role of one form of government.
Telling a policy transfer story: 11 questions to encourage successful evidence based policy transfer
In that context, these steps to evidence-informed policy transfer serve more to encourage reflection than provide a blueprint for action. I accept that 11 is less catchy than 10.
Points 1-3 represent the classic and necessary questions from policy studies: (1) ‘what is policy?’ (2) ‘how much did policy change?’ and (3) why? Until we have a good answer, we do not know how to draw comparable lessons. Learning from another government’s policy choices is no substitute for learning from more meaningful policy change.
4. Was the project introduced in a country or region which is sufficiently comparable? Comparability can relate to the size and type of country, the nature of the problem, the aims of the borrowing/ lending government and their measures of success.
5. Was it introduced nationwide, or in a region which is sufficiently representative of the national experience (it is not an outlier)?
6. How do we account for the role of scale, and the different cultures and expectations in each policy field?
Points 4-6 inform initial background discussions of case study reports. We need to focus on comparability when describing the context in which the original policy developed. It is not enough to state that two political systems are different. We need to identify the relevance and implications of the differences, from another government’s definition of the problem to the logistics of their task.
7. Has the project been evaluated independently, subject to peer review and/ or using measures deemed acceptable to the government?
8. Has the evaluation been of a sufficient period in proportion to the expected outcomes?
9. Are we confident that this project has been evaluated the most favourably – i.e. that our search for relevant lessons has been systematic, based on recognisable criteria (rather than reputations)?
10. Are we identifying ‘Good practice’ based on positive experience, ‘Promising approaches’ based on positive but unsystematic findings, ‘Research–based’ or based on ‘sound theory informed by a growing body of empirical research’, or ‘Evidence–based’, when ‘the programme or practice has been rigorously evaluated and has consistently been shown to work’?
Points 7-10 raise issues about the relationships between (a) what evidence we should use to evaluate success or potential, and (b) how long we should wait to declare success.
11. What will be the relationship between evidence and governance?
Should we identify the same basic model and transfer it uniformly, tell a qualitative story about the model and invite people to adapt it, or focus pragmatically on an eclectic range of evidential sources and focus on the training of the actors who will implement policy?
Information technology has allowed us to gather a huge amount of policy-relevant information across the globe. However, it has not solved the limitations we face in defining policy problems clearly, gathering evidence on policy solutions systematically, and generating international lessons that we can use to inform domestic policy processes.
This rise in available evidence is not a substitute for policy analysis and political choice. These choices range from how to adjudicate between competing policy preference, to how to define good evidence and good government. A lack of attention to these wider questions helps explain why – at least from some perspectives – policy transfer seems to fail.
Here is a guide to writing theory-driven policy analysis. Your aim is to identify a policy problem and solution, know your audience, and account for the complexity of policymaking.
At first, it may seem like daunting task to put together policy analysis and policy theory. On its own, policy analysis seems difficult but relatively straightforward: use evidence to identify and measure a policy problem, compare the merits of one or more solution, and make a recommendation on the steps to take us from policy to action.
However, policy process research tells us that people will engage emotionally with that evidence, and that policymakers operate in a complex system of which they have very limited knowledge and control.
So, how can we produce a policy analysis paper to which people will pay attention, and respond positively and effectively, under such circumstances? I focus on developing the critical analysis that will help you produce effective and feasible analysis. To do so, I show how policy analysis forms part of a collection of exercises to foster analysis informed by theory and reflection.
Aims of this document:
I have written this document as if part of a book to be called Teaching Public Policy and co-authored with Dr Emily St Denny.
For that audience, I have two aims: (1) to persuade policy scholars-as-teachers to adopt this kind of coursework in their curriculum; and, (2) to show students how to complete it effectively.
If you prefer shorter advice, see Writing a policy paper and blog post and Writing an essay on politics, policymaking, and policy change.
If you are interested in more background reading, see: The New Policy Sciences (by Paul Cairney and Chris Weible) which describes the need to combine policy theory-driven research with policy analysis; and, Practical Lessons from Policy Theories which describes eight attempts by scholars to translate policy theory into lessons that can be used for policy analysis.
The theories make more sense if you have read the corresponding 1000 Words posts (based on Cairney, 2012). Some of the forthcoming text will look familiar if you read my blog because I am consolidating several individual posts into an overall discussion.
I’m not quite there yet (the chapter is a first draft, a bit scrappy at times, and longer than a chapter should be), so all comments welcome (in the comments bit).
I tend to set this simple-looking question for coursework in policy modules: what is policy, how much has it changed, and why? Students get to choose the policy issue, timeframe (and sometimes the political system), and relevant explanatory concepts.
On the face of it, it looks super-simple: A+ for everyone!
Give it a few more seconds, and you can see the difficulties:
I try to encourage some creativity when solving this problem, but also advise students to keep their discussion as simple and jargon-free as possible (often by stretching an analogy with diving, in which a well-executed simple essay can score higher than a belly-flopped hard essay).
Choosing a format: the initial advice
For example, a question might be: What is tobacco policy in the UK, how much has it changed since the 1980s, and why? I use this example because I try to answer that – UK and global – question myself, even though my 2007 article on the UK is too theory-packed to be a good model for an undergraduate essay.
Choosing a format: the cautionary advice
You may be surprised about how difficult it is to answer a simple question like ‘what is policy?’ and I will give you considerable credit for considering how to define and measure it, by identifying, for example, the use of legislation/ regulation, funding, staff, and ‘nodality’ and/ or by considering the difference between, say, policy as a statement of intent or a long term outcome. In turn, a good description and explanation of policy change is difficult. If you are feeling ambitious, you can go further, to compare, say, two issues (such as tobacco and alcohol) or places (such UK Government policy and the policy of another country), but sometimes a simple and narrow discussion can be as, or more, effective. Similarly, you can use many theories or concepts to aid explanation, but often one theory will do. Note that (a) your description of your research question, and your essay structure, is more important than (b) your decision on what topic to focus or concepts to use.
Choosing a topic: the ‘joined up’ advice
The wider aim is to encourage students to think about the relationship between different perspectives on policy theory and analysis. For example, in a blog and policy analysis paper they try to generate attention to a policy problem and advocate a solution. Then, they draw on policy theories and concepts to reflect on their papers, highlighting (say): the need to identify the most important audience; the importance of framing issues with a mixture of evidence and emotional appeals; and, the need to present ‘feasible’ solutions.
The reflection can provide a useful segue to the essay, since we’re already identifying important policy problems, advocating change, reflecting on how best to encourage it – such as by presenting modest objectives – and then, in the essay, trying to explain (say) why governments have not taken that advice in the past. Their interest in the policy issue can prompt interest in researching the issue further; their knowledge of the issue and the policy process can help them develop politically-aware policy analysis. All going well, it produces a virtuous circle.
Some examples from my pet subject
Let me outline how I would begin to answer the three questions with reference to UK tobacco policy. I’m offering a brief summary of each section rather than presenting a full essay with more detail (partly to hold on to that idea of creativity – I don’t want students to use this description as a blueprint).
What is modern UK tobacco policy?
Tobacco policy in the UK is now one of the most restrictive in the world. The UK government has introduced a large number of policy instruments to encourage a major reduction of smoking in the population. They include: legislation to ban smoking in public places; legislation to limit tobacco advertising, promotion, and sponsorship; high taxes on tobacco products; unequivocal health education; regulations on tobacco ingredients; significant spending on customs and enforcement measures; and, plain packaging measures.
[Note that I selected only a few key measures to define policy. A fuller analysis might expand on why I chose them and why they are so important].
How much has policy changed since the 1980s?
Policy has changed radically since the post-war period, and most policy change began from the 1980s, but it was not until the 2000s onwards that the UK cemented its place as one of the most restrictive countries. The shift from the 1980s relates strongly to the replacement of voluntary agreements and limited measures with limited enforcement with legislative measures and stronger enforcement. The legislation to ban tobacco advertising, passed in 2002, replaced limited bans combined with voluntary agreements to (for example) keep billboards a certain distance from schools. The legislation to ban smoking in public places, passed in 2006 (2005 in Scotland), replaced voluntary measures which allowed smoking in most pubs and restaurants. Plain packaging measures, combined with large and graphic health warnings, replace branded packets which once had no warnings. Health education warnings have gone from stating the facts and inviting smokers to decide, and the promotion of harm reduction (smoke ‘low tar’), to an unequivocal message on the harms of smoking and passive smoking.
[Note that I describe these changes in broad terms. Other articles might ‘zoom’ in on specific instruments to show how exactly they changed]
Why has it changed?
This is the section of the essay in which we have to make a judgement about the type of explanation: should you choose one or many concepts; if many, do you focus on their competing or complementary insights; should you provide an extensive discussion of your chosen theory?
I normally recommend a very small number of concepts or simple discussion, largely because there is only so much you can say in an essay of 2-3000 words.
For example, a simple ‘hook’ is to ask if the main driver was the scientific evidence: did policy change as the evidence on smoking (and then passive smoking) related harm became more apparent? Is it a good case of ‘evidence based policymaking’? The answer may then note that policy change seemed to be 20-30 years behind the evidence [although I’d have to explain that statement in more depth] and set out the conditions in which this driver would have an effect.
In short, one might identify the need for a ‘policy environment’, shaped by policymakers, and conducive to a strong policy response based on the evidence of harm and a political choice to restrict tobacco use. It would relate to decisions by policymakers to: frame tobacco as a public health epidemic requiring a major government response (rather than primarily as an economic good or issue of civil liberties); place health departments or organisations at the heart of policy development; form networks with medical and public health groups at the expense of tobacco companies; and respond to greater public support for control, reduced smoking prevalence, and the diminishing economic value of tobacco.
This discussion can proceed conceptually, in a relatively straightforward way, or with the further aid of policy theories which ask further questions and help structure the answers.
For example, one might draw on punctuated equilibrium theory to help describe and explain shifts of public/media/ policymaker attention to tobacco, from low and positive in the 1950s to high and negative from the 1980s.
Or, one might draw on the ACF to explain how pro-tobacco coalitions helped slow down policy change by interpreting new scientific evidence though the ‘lens’ of well-established beliefs or approaches (examples from the 1950s include filter tips, low tar brands, and ventilation as alternatives to greater restrictions on smoking).
Any of these approaches will do, as long as you describe and justify your choice well. One cannot explain everything, so it may be better to try to explain one thing well.
You should get the impression from 1000 words that most policy changes are small or not radically different from the past: Lindblom identifies incrementalism; punctuated equilibrium highlights a huge number of small changes and small number of huge changes; the ACF compares routine learning by a dominant coalition to a ‘shock’ which prompts new subsystem and policy dynamics; and multiple streams identifies the conditions (rarely met) for major change.
Yet, I just gave you the impression that we don’t know how to define policy. If we can’t define it well, how can we measure it well enough to come to this conclusion so consistently?
Why is the measurement of policy change important?
We miss a lot if we equate policy with statements rather than outputs/ outcomes. We also miss a lot if we equate policy change with the most visible outputs such as legislation. I list 16 different policy instruments, although they tend to be grouped into smaller categories: focusing on regulation (including legislation) and resources (money and staffing) to accentuate the power at policymaker’s disposal; or regulatory/ distributive/ redistributive to suggest that some policy measures are more difficult to ‘sell’ than others.
We also give a limited picture if we equate change with outputs rather than outcomes, since a key insight from policy studies is that there is generally a gap between policymaker expectations and the actual result.
What are the key issues in measurement?
So, as in defining policy change, we need to make choices about what counts as policy in this instance to measure how much it has changed. For example, I have (a) written on one output as a key exemplar of policy change – legislation to ban smoking in public places for Scotland, England/ Wales, the UK, and (almost) EU – to show that a government is signalling major changes to come, but also (b) situated that policy instrument within a much broader discussion – of many tobacco policies in the UK and across the globe – to examine the extent to which it is already consistent with a well-established direction of travel.
To make such choices we need to consider:
How do we solve the problem?
The problem is that we can produce very different accounts of policy change from the same pool of evidence, by accentuating some measures and ignoring others, or putting more faith in some data more than others (e.g. during interviews).
Sometimes, my preferred solution is to compare more than one narrative of policy change. Another is simply to ‘show your work’.
Take home message for students: ‘show your work’ means explaining your logical process and step-by-step choices. Don’t just write that it is difficult to define policy and measure change. Instead, explain how you assess policy change in one important way, why you chose this way, and shine a light on the payoffs to your approach. Read up on how other scholars do it, to learn good practice and how to make your results comparable to theirs. Indeed, part of the benefit of using an established theory, to guide our analysis, is that we can engage in research systematically as a group.