Policy Analysis in 750 Words: what you need as an analyst versus policymaking reality

This post forms one part of the Policy Analysis in 750 words series overview. Note for the eagle eyed: you are not about to experience déjà vu. I’m just using the same introduction.

When describing ‘the policy sciences’, Lasswell distinguishes between:

1. ‘knowledge of the policy process’, to foster policy studies (the analysis of policy)
2. ‘knowledge in the process’, to foster policy analysis (analysis for policy)

The lines between each approach are blurry, and each element makes less sense without the other. However, the distinction is crucial to help us overcome the major confusion associated with this question:

Does policymaking proceed through a series of stages?

The short answer is no.

The longer answer is that you can find about 40 blog posts (of 500 and 1000 words) which compare (a) a stage-based model called the policy cycle, and (b) the many, many policy concepts and theories that describe a far messier collection of policy processes.

In a nutshell, most policy theorists reject this image because it oversimplifies a complex policymaking system. The image provides a great way to introduce policy studies, and serves a political purpose, but it does more harm than good:

1. Descriptively, it is profoundly inaccurate (unless you imagine thousands of policy cycles interacting with each other to produce less orderly behaviour and less predictable outputs).
2. Prescriptively, it gives you rotten advice about the nature of your policymaking task (for more on these points, see this chapter, article, article, and series).

Why does the stages/ policy cycle image persist? Two relevant explanations

 

1. It arose from a misunderstanding in policy studies

In another nutshell, Chris Weible and I argue (in a secret paper) that the stages approach represents a good idea gone wrong:

• If you trace it back to its origins, you will find Lasswell’s description of decision functions: intelligence, recommendation, prescription, invocation, application, appraisal and termination.
• These functions correspond reasonably well to a policy cycle’s stages: agenda setting, formulation, legitimation, implementation, evaluation, and maintenance, succession or termination.
• However, Lasswell was imagining functional requirements, while the cycle seems to describe actual stages.

In other words, if you take Lasswell’s list of what policy analysts/ policymakers need to do, multiple it by the number of actors (spread across many organisations or venues) trying to do it, then you get the multi-centric policy processes described by modern theories. If, instead, you strip all that activity down into a single cycle, you get the wrong idea.

2. It is a functional requirement of policy analysis

This description should seem familiar, because the classic policy analysis texts appear to describe a similar series of required steps, such as:

1. define the problem
2. identify potential solutions
3. choose the criteria to compare them
4. evaluate them in relation to their predicted outcomes
5. recommend a solution
6. monitor its effects
7. evaluate past policy to inform current policy.

However, these texts also provide a heavy dose of caution about your ability to perform these steps (compare Bardach, Dunn, Meltzer and Schwartz, Mintrom, Thissen and Walker, Weimer and Vining)

In addition, studies of policy analysis in action suggest that:

• an individual analyst’s need for simple steps, to turn policymaking complexity into useful heuristics and pragmatic strategies,

should not be confused with

• what actually happens when many policy analysts, influencers, and policymakers interact in policy processes (see Radin, and Brans, Geva-May, and Howlett).

What you need versus what you can expect

Overall, this discussion of policy studies and policy analysis reminds us of a major difference between:

1. Functional requirements. What you need from policymaking systems, to (a) manage your task (the 5-8 step policy analysis) and (b) understand and engage in policy processes (the simple policy cycle).
2. Actual processes and outcomes. What policy concepts and theories tell us about bounded rationality (which limit the comprehensiveness of your analysis) and policymaking complexity (which undermines your understanding and engagement in policy processes).

Of course, I am not about to provide you with a solution to these problems.

Still, this discussion should help you worry a little bit less about the circular arguments you will find in key texts: here are some simple policy analysis steps, but policymaking is not as ‘rational’ as the steps suggest, but (unless you can think of an alternative) there is still value in the steps, and so on.

See also:

The New Policy Sciences

Policy Analysis in 750 words: William Dunn (2017) Public Policy Analysis

Please see the Policy Analysis in 750 words series overview before reading the summary. This book is a whopper, with almost 500 pages and 101 (excellent) discussions of methods, so 800 words over budget seems OK to me. If you disagree, just read every second word.  By the time you reach the cat hanging in there baby you are about 300 (150) words away from the end.

William Dunn (2017) Public Policy Analysis 6^th Ed. (Routledge)

‘Policy analysis is a process of multidisciplinary inquiry aiming at the creation, critical assessment, and communication of policy-relevant knowledge … to solve practical problems … Its practitioners are free to choose among a range of scientific methods, qualitative as well as quantitative, and philosophies of science, so long as these yield reliable knowledge’ (Dunn, 2017: 2-3).

Dunn (2017: 4) describes policy analysis as pragmatic and eclectic. It involves synthesising policy relevant (‘usable’) knowledge, and combining it with experience and ‘practical wisdom’, to help solve problems with analysis that people can trust.

This exercise is ‘descriptive’, to define problems, and ‘normative’, to decide how the world should be and how solutions get us there (as opposed to policy studies/ research seeking primarily to explain what happens).

Dunn contrasts the ‘art and craft’ of policy analysts with other practices, including:

1. The idea of ‘best practice’ characterised by 5-step plans.

• In practice, analysis is influenced by: the cognitive shortcuts that analysts use to gather information; the role they perform in an organisation; the time constraints and incentive structures in organisations and political systems; the expectations and standards of their profession; and, the need to work with teams consisting of many professions/ disciplines (2017: 15-6)
• The cost (in terms of time and resources) of conducting multiple research and analytical methods is high, and highly constrained in political environments (2017: 17-8; compare with Lindblom)

2. The too-narrow idea of evidence-based policymaking

• The naïve attachment to ‘facts speak for themselves’ or ‘knowledge for its own sake’ undermines a researcher’s ability to adapt well to the evidence-demands of policymakers (2017: 68; 4 compare with Why don’t policymakers listen to your evidence?).

To produce ‘policy-relevant knowledge’ requires us to ask five questions before (Qs1-3) and after (Qs4-5) policy intervention (2017: 5-7; 54-6):

1. What is the policy problem to be solved?

• For example, identify its severity, urgency, cause, and our ability to solve it.
• Don’t define the wrong problem, such as by oversimplifying or defining it with insufficient knowledge.
• Key aspects of problems including ‘interdependency’ (each problem is inseparable from a host of others, and all problems may be greater than the sum of their parts), ‘subjectivity’ and ‘artificiality’ (people define problems), ‘instability’ (problems change rather than being solved), and ‘hierarchy’ (which level or type of government is responsible) (2017: 70; 75).
• Problems vary in terms of how many relevant policymakers are involved, how many solutions are on the agenda, the level of value conflict, and the unpredictability of outcomes (high levels suggest ‘wicked’ problems, and low levels ‘tame’) (2017: 75)
• ‘Problem-structuring methods’ are crucial, to: compare ways to define or interpret a problem, and ward against making too many assumptions about its nature and cause; produce models of cause-and-effect; and make a problem seem solve-able, such as by placing boundaries on its coverage. These methods foster creativity, which is useful when issues seem new and ambiguous, or new solutions are in demand (2017: 54; 69; 77; 81-107).
• Problem definition draws on evidence, but is primarily the exercise of power to reduce ambiguity through argumentation, such as when defining poverty as the fault of the poor, the elite, the government, or social structures (2017: 79; see Stone).

2. What effect will each potential policy solution have?

• Many ‘forecasting’ methods can help provide ‘plausible’ predictions about the future effects of current/ alternative policies (Chapter 4 contains a huge number of methods).
• ‘Creativity, insight, and the use of tacit knowledge’ may also be helpful (2017: 55).
• However, even the most-effective expert/ theory-based methods to extrapolate from the past are flawed, and it is important to communicate levels of uncertainty (2017: 118-23; see Spiegelhalter).

3. Which solutions should we choose, and why?

• ‘Prescription’ methods help provide a consistent way to compare each potential solution, in terms of its feasibility and predicted outcome, rather than decide too quickly that one is superior (2017: 55; 190-2; 220-42).
• They help to combine (a) an estimate of each policy alternative’s outcome with (b) a normative assessment.
• Normative assessments are based on values such as ‘equality, efficiency, security, democracy, enlightenment’ and beliefs about the preferable balance between state, communal, and market/ individual solutions (2017: 6; 205 see Weimer & Vining, Meltzer & Schwartz, and Stone on the meaning of these values).
• For example, cost benefit analysis (CBA) is an established – but problematic – economics method based on finding one metric – such as a $ value – to predict and compare outcomes (2017: 209-17; compare Weimer & Vining, Meltzer & Schwartz, and Stone)
• Cost effectiveness analysis uses a $ value for costs, but compared with other units of measurement for benefits (such as outputs per $) (2017: 217-9)
• Although such methods help us combine information and values to compare choices, note the inescapable role of power to decide whose values (and which outcomes, affecting whom) matter (2017: 204)

4. What were the policy outcomes?

• ‘Monitoring’ methods help identify (say): levels of compliance with regulations, if resources and services reach ‘target groups’, if money is spent correctly (such as on clearly defined ‘inputs’ such as public sector wages), and if we can make a causal link between the policy inputs/ activities/ outputs and outcomes (2017: 56; 251-5)
• Monitoring is crucial because it is so difficult to predict policy success, and unintended consequences are almost inevitable (2017: 250).
• However, the data gathered are usually no more than proxy indicators of outcomes. Further, the choice of indicators reflect what is available, ‘particular social values’, and ‘the political biases of analysts’ (2017: 262)
• The idea of ‘evidence based policy’ is linked strongly to the use of experiments and systematic review to identify causality (2017: 273-6; compare with trial-and-error learning in Gigerenzer, complexity theory, and Lindblom).

5. Did the policy solution work as intended? Did it improve policy outcomes?

• Although we frame policy interventions as ‘solutions’, few problems are ‘solved’. Instead, try to measure the outcomes and the contribution of your solution, and note that evaluations of success and ‘improvement’ are contested (2017: 57; 332-41).  
• Policy evaluation is not an objective process in which we can separate facts from values.
• Rather, values and beliefs are part of the criteria we use to gauge success (and even their meaning is contested – 2017: 322-32).
• We can gather facts about the policy process, and the impacts of policy on people, but this information has little meaning until we decide whose experiences matter.

Overall, the idea of ‘ex ante’ (forecasting) policy analysis is a little misleading, since policymaking is continuous, and evaluations of past choices inform current choices.

Policy analysis methods are ‘interdependent’, and ‘knowledge transformations’ describes the impact of knowledge regarding one question on the other four (2017: 7-13; contrast with Meltzer & Schwartz, Thissen & Walker).

Developing arguments and communicating effectively

Dunn (2017: 19-21; 348-54; 392) argues that ‘policy argumentation’ and the ‘communication of policy-relevant knowledge’ are central to policymaking’ (See Chapter 9 and Appendices 1-4 for advice on how to write briefs, memos, and executive summaries and prepare oral testimony).

He identifies seven elements of a ‘policy argument’ (2017: 19-21; 348-54), including:

• The claim itself, such as a description (size, cause) or evaluation (importance, urgency) of a problem, and prescription of a solution
• The things that support it (including reasoning, knowledge, authority)
• Incorporating the things that could undermine it (including any ‘qualifier’, the communication of uncertainty about current knowledge, and counter-arguments).

The key stages of communication (2017: 392-7; 405; 432) include:

1. ‘Analysis’, focusing on ‘technical quality’ (of the information and methods used to gather it), meeting client expectations, challenging the ‘status quo’, albeit while dealing with ‘political and organizational constraints’ and suggesting something that can actually be done.
2. ‘Documentation’, focusing on synthesising information from many sources, organising it into a coherent argument, translating from jargon or a technical language, simplifying, summarising, and producing user-friendly visuals.
3. ‘Utilization’, by making sure that (a) communications are tailored to the audience (its size, existing knowledge of policy and methods, attitude to analysts, and openness to challenge), and (b) the process is ‘interactive’ to help analysts and their audiences learn from each other.

 

 

Policy analysis and policy theory: systems thinking, evidence based policymaking, and policy cycles

Dunn (2017: 31-40) situates this discussion within a brief history of policy analysis, which culminated in new ways to express old ambitions, such as to:

1. Use ‘systems thinking’, to understand the interdependence between many elements in complex policymaking systems (see also socio-technical and socio-ecological systems).

• Note the huge difference between (a) policy analysis discussions of ‘systems thinking’ built on the hope that if we can understand them we can direct them, and (b) policy theory discussions that emphasise ‘emergence’ in the absence of central control (and presence of multi-centric policymaking).
• Also note that Dunn (2017: 73) describes policy problems – rather than policymaking – as complex systems. I’ll write another post (short, I promise) on the many different (and confusing) ways to use the language of complexity.

2. Promote ‘evidence based policy’, as the new way to describe an old desire for ‘technocratic’ policymaking that accentuates scientific evidence and downplays politics and values (see also 2017: 60-4).

In that context, see Dunn’s (47-52) discussion of comprehensive versus bounded rationality:

• Note the idea of ‘erotetic rationality’ in which people deal with their lack of knowledge of a complex world by giving up on the idea of certainty (accepting their ‘ignorance’), in favour of a continuous process of ‘questioning and answering’.
• This approach is a pragmatic response to the lack of order and predictability of policymaking systems, which limits the effectiveness of a rigid attachment to ‘rational’ 5 step policy analyses (compare with Meltzer & Schwartz).

Dunn (2017: 41-7) also provides an unusually useful discussion of the policy cycle. Rather than seeing it as a mythical series of orderly stages, Dunn highlights:

1. Lasswell’s original discussion of policymaking functions (or functional requirements of policy analysis, not actual stages to observe), including: ‘intelligence’ (gathering knowledge), ‘promotion’ (persuasion and argumentation while defining problems), ‘prescription’, ‘invocation’ and ‘application’ (to use authority to make sure that policy is made and carried out), and ‘appraisal’ (2017: 42-3).
2. The constant interaction between all notional ‘stages’ rather than a linear process: attention to a policy problem fluctuates, actors propose and adopt solutions continuously, actors are making policy (and feeding back on its success) as they implement, evaluation (of policy success) is not a single-shot document, and previous policies set the agenda for new policy (2017: 44-5).

In that context, it is no surprise that the impact of a single policy analyst is usually minimal (2017: 57). Sorry to break it to you. Hang in there, baby.

 

Policy Concepts in 1000 Words: The Policy Process

We talk a lot about ‘the policy process’ without really saying what it is. If you are new to policy studies, maybe you think that you’ll learn what it is eventually if you read enough material. This would be a mistake! Instead, when you seek a definition of the policy process, you’ll find two common responses:

1. Many will seek to define policy or public policy instead of ‘the policy process’.
2. Some will describe the policy process as a policy cycle with stages.

Both responses seem inadequate: one avoids giving an answer, and another gives the wrong answer!

However, we can combine elements of each approach to give you just enough of a sense of ‘the policy process’ to continue reading the full ‘1000 words’ series:

1. The beauty of the ‘what is policy?’ question …

… is that we don’t give you an answer. It may seem frustrating at first to fail to find a definitive answer, but eventually you’ll accept this problem! The more important outcome is to use the ‘what is policy?’ question to develop analytical skills, to allow you to define policy in more specific circumstances (such as, what are the key elements of policy in this case study?), and ask more useful and specific questions about policy and policymaking. So, look at the questions we need to ask if we begin with the definition, ‘the sum total of government action, from signals of intent to the final outcomes’: does action include statements of intent? Do we include unintended policy outcomes? Are all policymakers in government? What about the things policymakers choose not to do? And so on.

2. The beauty of the policy cycle approach …

… is that it provides a simple way to imagine policy ‘dynamics’, or events and choices producing a never-ending sequence of other events and choices. Look at the stages model to identify many different tasks within one ‘process’, and to get the sense that policymaking is continuous and often ‘its own cause’. It’s not a good description of what actually happens, but it describes what some might like to happen, and used by many governments to describe what they do. Consequently, we can’t simply ignore it, at least without providing a better description, a better plan, and a better way for governments to justify what they do.

There are more complicated but better ways of describing policymaking dynamics

This picture is the ‘policy process’ equivalent of my definition of public policy. It captures the main elements of the policy process described – albeit in different ways – by most policy theories in this series. I present it here to give you enough of an answer – to ‘what is the policy process?’ – to help you ask more questions.

In the middle is ‘policy choice’

At the heart of most policy theory is ‘bounded rationality’, which describes (a) the cognitive limits of all people, and (b) how policymakers overcome such limits to make decisions (in the absence of NZT). In short, they use ‘rational’ and ‘irrational’ shortcuts to action, but these are provocative terms to prompt further reading (on, for example, ‘evidence-based policymaking’).

‘Rational’ describes goal-oriented activity: people may have limits to their attention and ‘information processing’, but they find systematic ways to respond, by setting goals and producing criteria to find the best information. ‘Irrational’ describes aspects of psychology: people draw on habit, emotions, their ‘gut’ or intuition, well-established beliefs, and their familiarity with information to make often-almost-instant decisions.

Surrounding choice is what we’ll call the ‘policy environment’

Environment is a metaphor we’ll use to describe the combination of key elements of the policy process which (a) I describe separately in further 1000 words posts, and (b) policy theories bring together to produce an overall picture of policy dynamics.

There are 5 or 6 key elements. In the picture are 6, reflecting the way Tanya Heikkila and I describe it (and the fact that I had 7 boxes to fill). In real life, I describe 5 because I have 5 digits on each hand. If you are Count Tyrone Rugen you have more choice.

Policy environments are made up of:

1. A wide range of actors (which can be individuals and organisations with the ability to deliberate and act) making or influencing policy at many levels and types of government.
2. Institutions, defined as the rules followed by actors. Some are formal, written down, and easy to identify. Others are informal, reproduced via processes like socialisation, and difficult to spot and describe.
3. Networks, or the relationships between policymakers and influencers. Some are wide open, competitive, and contain many actors. Others are relatively closed, insulated from external attention, and contain few actors.
4. Ideas, or the beliefs held and shared by actors. There is often a tendency for certain beliefs or ‘paradigms’ to dominate discussion, constraining or facilitating the progress of new ‘ideas’ as policy solutions.
5. Context and events. Context describes the policy conditions – including economic, social, demographic, and technological factors – that provide the context for policy choice, and are often outside of the control of policymakers. Events can be routine and predictable, or unpredictable ‘focusing’ events that prompt policymaker attention to lurch at short notice.

This picture is only the beginning of analysis, raising further questions that will make more sense when you read further, including: should policymaker choice be at the centre of this picture? Why are there arrows in the cycle but not in my picture? Should we describe complex policymaking ‘systems’ rather than ‘environments’? How exactly does each element in the ‘policy environment’ or ‘system’ relate to the other?

The answer to the final question can only be found in each theory of the policy process, and each theory describes this relationship in a different way. Let’s not worry about that just now! We’ll return to this issue at the end, when thinking about how to combine the insights of many theories.

 

 

 

5 images of the policy process

A picture tells a thousand words but, in policy studies, those words are often misleading or unclear. The most useful images can present the least useful advice, or capture a misleading metaphor. Images from the most useful theories are useful when you already know the theory, but far more difficult to grasp initially.

So, I present two examples from each, then describe what a compromise image might look like, to combine something that is easy to pick up and use but also not misleading or merely metaphorical.

Why do we need it? It is common practice at workshops and conferences for some to present policy process images on powerpoint and for others to tweet photos of them, generally with little critical discussion of what they say and how useful they are. I’d like to see as-simple but more-useful images spread this way.

1. The policy cycle

The policy cycle is perhaps the most used and known image. It divides the policy process into a series of stages (described in 1000 words and 500 words). It oversimplifies, and does not explain, a complex policymaking system. We are better to imagine, for example, thousands of policy cycles interacting with each other to produce less orderly behaviour and less predictable outputs.

For students, we have dozens of concepts and theories which serve as better ways to understand policymaking.

Policymakers have more use for the cycle, to tell a story of what they’d like to do: identify aims, identify policies to achieve those aims, select a policy measure, ensure that the selection is legitimised by the population or its legislature, identify the necessary resources, implement, then evaluate the policy.

As such, this story still pops up from time to time:

Really helpful slide from SG Chief Social Researcher on the Gov. 'Policy Cycle'

Useful for anyone working in the Policy Sphere. @SPRExchange pic.twitter.com/jtLEGhD0oD

— TILLY ROBINSON-MILES FRGS (she/her) 🌻 (@TRobinsonMiles) December 9, 2020

Yet, most presentations from policymakers, advisers, and practitioners modify the cycle image to show how messy life really is:

The messy policy process and the role of evidence #evidence4gov @effectiveserv pic.twitter.com/g8kLOhxet5

— Majella McCloskey (@MajellaMcCl) June 27, 2017

Discussing the challenges of evidence based policy including complexity of the landscape @OECD #evidence4gov pic.twitter.com/1yu3UIKKjO

— Rob Davies (@r0bdavies) June 26, 2017

Update 18.2.20: I knocked this up one in my garage for the book The Politics of Policy Analysis. See in particular what you need as an analyst versus policymaking reality, which argues that the cycle (or 5-step policy analysis) describes what policy analysts would like to do (not what happens).

2. The multiple streams metaphor

The ‘multiple streams’ approach uses metaphor to describe this messier world (described in 1000 words and 500 words). Instead of a linear cycle – in which policymakers define problems, then ask for potential solutions, then select one – we describe these ‘stages’ as independent ‘streams’. Each stream – heightened attention to a problem (problem stream), an available and feasible solution (policy stream), and the motive to select it (politics stream) – must come together during a ‘window of opportunity’ or the opportunity is lost.

Many people like MSA because it contains a flexible metaphor which is simple to pick up and use. However, it’s so flexible that I’ve seen many different ways to visualise – and make sense of – the metaphor, including literal watery streams, which suggest that when they come together they are hard to separate.  There is the Ghostbusters metaphor which shows that key actors (‘entrepreneurs’) help couple the streams. There is also Howlett et al’s attempt to combine the streams and cycles metaphors (reproduced here, and criticised here).

However, I’d encourage Kingdon’s space launch metaphor in which policymakers will abort the mission unless every factor is just right.

3. The punctuated equilibrium graph

Punctuated equilibrium theory is one of the most important approaches to policy dynamics, now backed up with a wealth of data from the Comparative Agendas Project. The image (in True et al, 2007) describes the result of the policy process rather than the process itself. It describes government budgets in the US, although we can find very similar images from studies of budgets in many other countries and in many measures of policy change.

It sums up a profoundly important message about policy change: we find a huge number of very small changes, and a very small number of huge changes. Compare the distribution of values in this image with the ‘normal distribution’ (the dotted line). It shows a ‘leptokurtic’ distribution, with most values deviating minimally from the mean (and the mean change in each budget item is small), but with a high number of ‘outliers’.

The image helps sum up a key aim of PET, to measure and try to explain long periods of policymaking stability, and policy continuity, disrupted by short but intense periods of instability and change. One explanation relates to ‘bounded rationality’: policymakers have to ignore almost all issues while paying attention to some. The lack of ‘macropolitical’ attention to most issues helps explain stability and continuity, while lurches of attention can help explain instability (although attention can fade before ‘institutions’ feel the need to respond).

Here I am, pointing at this graph:

@CairneyPaul providing much food for thought on multi-level decision making in energy policy at @UKERCHQ assembly '17 pic.twitter.com/zKoImOhGNZ

— Prof. Christian Brand (@_chris_brand_) July 13, 2017

4. The advocacy coalition framework ‘flow diagram’

The ACF presents an ambitious image of the policy process, in which we zoom out to consider how key elements fit together in a process containing many actors and levels of government. Like many policy theories, it situates most of the ‘action’ in policy networks or subsystems, showing that some issues involve intensely politicized disputes containing many actors while others are treated as technical and processed routinely, largely by policy specialists, out of the public spotlight.

The ACF suggests that people get into politics to turn their beliefs into policy, form coalitions with people who share their beliefs, and compete with coalitions of actors who share different beliefs. This competition takes place in a policy subsystem, in which coalitions understand new evidence through the lens of their beliefs, and exercise power to make sure that their interpretation is accepted. The other boxes describe the factors – the ‘parameters’ likely to be stable during the 10-year period of study, the partial sources of potential ‘shocks’ to the subsystem, and the need and ability of key actors to form consensus for policy change (particularly in political systems with PR elections) – which constrain and facilitate coalition action.

5. What do we need from a new image?

I recommend an image that consolidates or synthesises existing knowledge and insights. It is tempting to produce something that purports to be ‘new’ but, as with ‘new’ concepts or ‘new’ policy theories, how could we accumulate insights if everyone simply declared novelty and rejected the science of the past?

For me, the novelty should be in the presentation of the image, to help people pick up and use a wealth of policy studies which try to capture two key dynamics:

1. Policy choice despite uncertainty and ambiguity.

Policymakers can only pay attention to a tiny proportion of issues. They use ‘rational’ and ‘irrational’ cognitive shortcuts to make decisions quickly, despite their limited knowledge of the world, and the possibility to understand policy problems from many perspectives.

2. A policy environment which constrains and facilitates choice.

Such environments are made up of:

1. Actors (individuals and organisations) influencing policy at many levels and types of government
2. Institutions: a proliferation of rules and norms followed by different levels or types of government
3. Networks: relationships between policymakers and influencers
4. Ideas: a tendency for certain beliefs or ‘paradigms’ to dominate discussion
5. Context and events: economic, social, demographic, and technological conditions provide the context for policy choice, and routine/ unpredictable events can prompt policymaker attention to lurch at short notice.

The implications of both dynamics are fairly easy to describe in tables (for example, while describing MSA) and to cobble together quickly in a SmartArt picture:

However, note at least three issues with such a visual presentation:

1. Do we put policymakers and choice at the centre? If so, it could suggest (a bit like the policy cycle) that a small number of key actors are at the ‘centre’ of the process, when we might prefer to show that their environment, or the interaction between many actors, is more important.
2. Do we show only the policy process or relate it to the ‘outside world’?
3. There are many overlaps between concepts. For example, we seek to describe the use and reproduction of rules in ‘institutions’ and ‘networks’, while those rules relate strongly to ‘ideas’. Further, ‘networks’ could sum up ‘actors interacting in many levels/ types of government’. So, ideally, we’d have overlapping shapes to denote overlapping relationships and understandings, but it would really mess up the simplicity of the image.

Of course, the bigger issue is that the image I provide is really just a vehicle to put text on a screen (in the hope that it will be shared). At best it says ‘note these concepts’. It does not show causal relationships. It does not describe any substantial interaction between the concepts to show cause and effect (such as, event A prompted policy choice B).

However, if we tried to bring in that level of detail, I think we would quickly end up with the messy process already described in relation to the policy cycle. Or, we would need to provide a more specific and less generally applicable model of policymaking.

So, right now, this image is a statement of intent. I want to produce something better, but don’t yet know what ‘better’ looks like. There is no ‘general theory’ of policymaking, so can we have a general image? Or, like ‘what is policy?’ discussions, do we produce an answer largely to raise more questions?

Update: please compare with the turtle diagram, below, and explored in more depth here.

___

Here I am, looking remarkably pleased with my SmartArt skills

😀😀😀Diagram on policy environment by @CairneyPaul #UKERCAA17 pic.twitter.com/hADP9gS5ds

— Sam Hampton (@samhampton) July 13, 2017

Policy in 500 Words: if the policy cycle does not exist, what do we do?

It is easy to reject the empirical value of the policy cycle, but difficult to replace it as a practical tool. I identify the implications for students, policymakers, and the actors seeking influence in the policy process.

A policy cycle divides the policy process into a series of stages:

• Agenda setting. Identifying problems that require government attention, deciding which issues deserve the most attention and defining the nature of the problem.
• Policy formulation. Setting objectives, identifying the cost and estimating the effect of solutions, choosing from a list of solutions and selecting policy instruments.
• Legitimation. Ensuring that the chosen policy instruments have support. It can involve one or a combination of: legislative approval, executive approval, seeking consent through consultation with interest groups, and referenda.
• Implementation. Establishing or employing an organization to take responsibility for implementation, ensuring that the organization has the resources (such as staffing, money and legal authority) to do so, and making sure that policy decisions are carried out as planned.
• Evaluation. Assessing the extent to which the policy was successful or the policy decision was the correct one; if it was implemented correctly and, if so, had the desired effect.
• Policy maintenance, succession or termination. Considering if the policy should be continued, modified or discontinued.

Most academics (and many practitioners) reject it because it oversimplifies, and does not explain, a complex policymaking system in which: these stages may not occur (or occur in this order), or we are better to imagine thousands of policy cycles interacting with each other to produce less orderly behaviour and predictable outputs.

But what do we do about it?

The implications for students are relatively simple: we have dozens of concepts and theories which serve as better ways to understand policymaking. In the 1000 Words series, I give you 25 to get you started.

The implications for policymakers are less simple because they cycle may be unrealistic and useful. Stages can be used to organise policymaking in a simple way: identify policymaker aims, identify policies to achieve those aims, select a policy measure, ensure that the selection is legitimised by the population or its legislature, identify the necessary resources, implement and then evaluate the policy. The idea is simple and the consequent advice to policy practitioners is straightforward.  A major unresolved challenge for scholars and practitioners is to describe a more meaningful, more realistic, analytical model to policymakers and give advice on how to act and justify action in the same straightforward way. So, in this article, I discuss how to reconcile policy advice based on complexity and pragmatism with public and policymaker expectations.

The implications for actors trying to influence policymaking can be dispiriting: how can we engage effectively in the policy process if we struggle to understand it? So, in this page (scroll down – it’s long!), I discuss how to present evidence in complex policymaking systems.

Take home message for students. It is easy to describe then assess the policy cycle as an empirical tool, but don’t stop there. Consider how to turn this insight into action. First, examine the many ways in which we use concepts to provide better descriptions and explanations. Then, think about the practical implications. What useful advice could you give an elected policymaker, trying to juggle pragmatism with accountability? What strategies would you recommend to actors trying to influence the policy process?

The Politics of Evidence Based Policymaking:3 messages

Really, it’s three different ways to make the same argument in the number of words that suits you:

1. Guardian post (700 words): ‘When presenting evidence to policymakers, scientists and other experts need to engage with the policy process that exists, not the one we wish existed’
2. Public Administration Review article (3000 words) To Bridge the Divide between Evidence and Policy: Reduce Ambiguity as Much as Uncertainty (free version)
3. Book (40,000 words)The Politics of Evidence Based Policymaking (free version)

For even more words, see my EBPM page

What is Policy?

Compare with What is the Policy Process? and What is public policy and why does it matter?

The first thing we do when studying public policy is to try to define it – as, for example, the sum total of government action, from signals of intent to the final outcomes. This sort of definition produces more questions:

• Does ‘government action’ include what policymakers say they will do as well as what they actually do? An unfulfilled promise may not always seem like policy.
• Does it include the effects of a decision as well as the decision itself? A policy outcome may not resemble the initial policy aims.
• What is ‘the government’ and does it include elected and unelected policymakers? Many individuals, groups and organisations influence policy and help carry it out.
• Does public policy include what policymakers do not do. Policy is about power, which is often exercised to keep important issues off the public, media and government agenda.

The second thing we do is point to the vast scale of government, which is too big to be understood without some simplifying concepts and theories. It is also too big to be managed. We soon learn that the vast majority of policymaking takes place in the absence of meaningful public attention. The ‘public’ simply does not have the time to pay attention to government. Even when it pays attention to some issues, the debate is simplified and does not give a good account of the complicated nature of policy problems.

We also learn that government is too big to be managed by elected policymakers. Instead, they divide government into manageable units and devolve almost all decisions to bureaucrats and organisations (including ‘street level’).  They are responsible for government, but they simply do not have the time to pay attention to anything but a tiny proportion.

So, a big part of public policy is about what happens when neither the public nor elected policymakers have the ability to pay attention to what goes on in their name. That’s what makes it seem so messed up and so interesting at the same time.

It’s also what makes policy studies look so weird. We often reject a focus on high-profile elected policymakers, because we know that the action takes place elsewhere. We often focus on the day-to-day practices of organisations far removed from the ‘top’ or the ‘centre’. We ‘zoom in’ and ‘zoom out’ to gain several perspectives on the same thing. We spend a lot of time gnashing our teeth about how you can identify and measure policy change (still, no-one has cracked this one) and compare it with the past and the experience of other countries. We try to come up with ways to demonstrate that inaction is often more significant than action. When you ask us a question, your eyes will glaze over while we try to explain, ‘well, that’s really 12 questions’. We come up with wacky names to describe policymaking and bristle if you call it ‘jargon’. It’s because policymaking is complicated and it takes skill, and some useful concepts, to make it look simple.

To read more, see: Policy Concepts in 1000 words

(to store the podcast, right click and save this link)

‘Evidence-based Policymaking’ and the Study of Public Policy

This post accompanies a 40 minute lecture (download) which considers ‘evidence-based policymaking’ (EBPM) through the lens of policy theory. The theory is important, to give us a language with which to understand EBPM as part of a wider discussion of the policy process, while the lens of EBPM allows us to think through the ‘real world’ application of concepts and theories.

To that end, I’ll make three key points:

1. Definitions and clarity are important. ‘Evidence-based policymaking’, ‘evidence-based policy’ and related phrases such as ‘policy based evidence’ are used incredibly loosely in public debates. A focus on basic questions in policy studies – what is policy, and how can we measure policy change? – helps us clarify the issues, reject superficial debates on ‘evidence-based policy versus policy-based evidence’, and in some cases identify the very different assumptions people make about how policymaking works and should work.
2. Realistic models are important. Discussing EBPM helps us identify the major flaws in simple models of policymaking such as the ‘policy cycle’. I’ll discuss the insights we gain by considering how policy scholars describe the implications of policymaker ‘bounded rationality’ and policymaking complexity.
3. Realistic strategies are important. There is a lot of academic discussion of the need to overcome ‘barriers’ between evidence and policy. It is often atheoretical, producing naïve recommendations about improving the supply of evidence and training policymakers to understand it. I identify two more useful (but potentially controversial) strategies: be manipulative and learn where the ‘action’ is.

Definitions and clarity are important, so what is ‘evidence-based policymaking’?

What is Policy? It is incredibly difficult to say what policy is and measure how much it has changed. I use the working definition, ‘the sum total of government action, from signals of intent to the final outcomes’ to raise important qualifications: (a) it is problematic to conflate what people say they will do and what they actually do; (b) a policy outcome can be very different from the intention; (c) policy is made routinely through cooperation between elected and unelected policymakers and actors with no formal role in the process; (d) policymaking is also about the power not to do something. It is also important to identify the many components or policy instruments that make up policies, including: the level of spending; the use of economic incentives/ penalties; regulations and laws; the use of voluntary agreements and codes of conduct; the provision of public services; education campaigns; funding for scientific studies or advocacy; organisational change; and, the levels of resources/ methods dedicated to policy implementation (2012a: 26).

In that context, we are trying to capture a process in which actors make and deliver ‘policy’ continuously, not identify a set-piece event which provides a single opportunity to use a piece of scientific evidence to prompt a policymaker response.

Who are the policymakers? The intuitive definition is ‘people who make policy’, but there are two important distinctions: (1) between elected and unelected participants, since people such as civil servants also make important decisions; (2) between people and organisations, with the latter used as a shorthand to refer to a group of people making decisions collectively. There are blurry dividing lines between the people who make and influence policy. Terms such as ‘policy community’ suggest that policy decisions are made by a collection of people with formal responsibility and informal influence. Consequently, we need to make clear what we mean by ‘policymakers’ when we identify how they use evidence.

What is evidence? We can define evidence as an argument backed by information. Scientific evidence describes information produced in a particular way. Some describe ‘scientific’ broadly, to refer to information gathered systematically using recognised methods, while others refer to a specific hierarchy of scientific methods, with randomized control trials (RCTs) and the systematic review of RCTs at the top. This is a crucial point:

policymakers will seek many kinds of information that many scientists would not consider to be ‘the evidence’.

This discussion helps identify two key points of potential confusion when people discuss EBPM:

1. When you describe ‘evidence-based policy’ and EBPM you need to clarify what the policy is and who is making it. This is not just about some elected politicians making announcements.
2. When you describe ‘evidence’ you need to clarify what counts as evidence and what an ‘evidence-based’ policy response would look like. This point is at the heart of often fruitless discussions about ‘policy based evidence’, which seems to describe almost a dozen alleged mistakes by policymakers (relating to ignoring evidence, using the wrong kinds, and/ or producing a disproportionate response).

Realistic models are important, so what is wrong with the policy cycle?

One traditional way to understand policymaking in the ‘real world’ is to compare it to an ideal-type: what happens when the conditions of the ideal-type are not met? We do this in particular with the ‘policy cycle and ‘comprehensive rationality’.

So, consider this modified ideal-type of EBPM:

• There is a core group of policymakers at the ‘centre’, making policy from the ‘top down’, breaking down their task into clearly defined and well-ordered stages;
• Scientists are in a privileged position to help those policymakers make good decisions by getting them as close as possible to the ideal of ‘comprehensive rationality’ in which they have the best information available to inform all options and consequences.

So far, so good (although you might stop to consider who is best placed to provide evidence, and who – or which methods of evidence gathering – should be privileged or excluded), but what happens when we move away from the ideal-type? Here are two insights from a forthcoming paper (Cairney Oliver Wellstead 26.1.16).

Lessons from policy theory: 1. Identify multi-level policymaking environments

First, policymaking takes place in less ordered and predictable policy environment, exhibiting:

• a wide range of actors (individuals and organisations) influencing policy at many levels of government
• a proliferation of rules and norms followed by different levels or types of government
• close relationships (‘networks’) between policymakers and powerful actors
• a tendency for certain beliefs or ‘paradigms’ to dominate discussion
• shifting policy conditions and events that can prompt policymaker attention to lurch at short notice.

A focus on this bigger picture shifts our attention from the use of scientific evidence by an elite group of elected policymakers at the ‘top’ to its use by a wide range of influential actors in a multi-level policy process. It shows scientists and practitioners that they are competing with many actors to present evidence in a particular way to secure a policymaker audience. Support for particular solutions varies according to which organisation takes the lead and how it understands the problem. Some networks are close-knit and difficult to access because bureaucracies have operating procedures that favour particular sources of evidence and some participants over others, and there is a language – indicating what ways of thinking are in good ‘currency’ (such as ‘value for money’) – that takes time to learn. Well-established beliefs provide the context for policymaking: new evidence on the effectiveness of a policy solution has to be accompanied by a shift of attention and successful persuasion. In some cases, social or economic ‘crises’ can prompt lurches of attention from one issue to another, and some forms of evidence can be used to encourage that shift. In this context, too many practitioner studies analyse, for example, a singular point of central government decision rather than the longer term process. Overcoming barriers to influence in that small part of the process will not provide an overall solution.

Lessons from policy theory: 2. Policymakers use two ‘shortcuts’ to make decisions

How do policymakers deal with their ‘bounded rationality’? They employ two kinds of shortcut: ‘rational’, by pursuing clear goals and prioritizing certain kinds and sources of information, and ‘irrational’, by drawing on emotions, gut feelings, deeply held beliefs, habits, and the familiar to make decisions quickly. Consequently, the focus of policy theories is on the links between evidence, persuasion, and framing (in the wider context of a tendency for certain beliefs to dominate discussion).

Framing refers to the ways in which we understand, portray, and categorise issues. Problems are multi-faceted, but bounded rationality limits the attention of policymakers, and actors compete to highlight one image at the expense of others. The outcome of this process determines who is involved (for example, portraying an issue as technical limits involvement to experts), who is responsible for policy, how much attention they pay, and what kind of solution they favour. For example, tobacco control is more likely when policymakers view it primarily as a public health epidemic rather than an economic good, while ‘fracking’ policy depends on its primary image as a new oil boom or environmental disaster (I discuss both examples in depth here).

Scientific evidence plays a part in this process, but we should not exaggerate the ability of scientists to win the day with reference to evidence. Rather, policy theories signal the strategies that practitioners may have to adopt to increase demand for their evidence:

• to combine facts with emotional appeals, to prompt lurches of policymaker attention from one policy image to another (punctuated equilibrium theory)
• to tell simple stories which are easy to understand, help manipulate people’s biases, apportion praise and blame, and highlight the moral and political value of solutions (narrative policy framework)
• to interpret new evidence through the lens of the pre-existing beliefs of actors within coalitions, some of which dominate policy networks (advocacy coalition framework)
• to produce a policy solution that is feasible and exploit a time when policymakers have the opportunity to adopt it (multiple streams analysis).

Further, the impact of a framing strategy may not be immediate, even if it appears to be successful. Scientific evidence may prompt a lurch of attention to a policy problem, prompting a shift of views in one venue or the new involvement of actors from other venues. However, for example, it can take years to produce support for an ‘evidence-based’ policy solution, built on its technical and political feasibility (will it work as intended, and do policymakers have the motive and opportunity to select it?).

This discussion helps identify two key points of potential confusion when people discuss the policy cycle and comprehensive rationality:

1. These concepts are there to help us understand what doesn’t happen. What are the real world implications of the limits to these models?
2. They do not help you give good advice to people trying to influence the policy process. A focus on going through policymaking ‘stages’ and improving ‘rationality’ is always relevant when you give advice to policymakers. However unrealistic these models are, you would still want to gather the maximum information and go through a process of stages. This is very different from (a) giving advice on how to influence the process, or (b) evaluating the pros and cons of a political system with reference to ideal-types.

Realistic strategies are important, so how far should you go to overcome ‘barriers’ between evidence and policy?

You can’t take the politics out of EBPM. Even the selection of ‘the evidence’ is political (should evidence be scientific, and what counts as scientific evidence?).

Further, providers of scientific evidence face major dilemmas when they seek to maximise the ‘impact’ of their research. Armed with this knowledge of the policy process, how should you seek to engage and influence decisions made within it?

If you are interested in this final discussion, please see the short video here and the follow up blog post: Political science improves our understanding of evidence-based policymaking, but does it produce better advice?

See also:

This post is one of many on EBPM. The full list is here: https://paulcairney.wordpress.com/ebpm/

To bridge the divide between evidence and policy: reduce ambiguity as much as uncertainty

 

Can anyone use the ‘tools’ of policy research?

We often use the metaphor of ‘tools’ to describe the ways in which people use public policy analysis. For some, the suggestion is that – like a hammer and chisel – anyone could pick them up and use them. Yet, let’s think about that metaphor some more:

• Anyone can use a hammer and chisel, but many people lop of their fingers or break things while doing so.
• You use a hammer and chisel for a particular purpose, such as to create a sculpture. We can all whack a bit of stone with a blunt object, but it takes skill to turn it into something worth your attention.
• Many tools require training before you should use them: power tools, lasers, MRIs, the screws to insert frames into broken bones, and so on.

My argument is this: you wouldn’t trust a political scientist to make you a piece of art, fix your window, do your laser eyesight correction, or scan your brain, because it is generally not a good idea to pick up and use these tools without training. So, have a quick think about who you would trust with policy research tools if they just picked them out of the shed and used them for the first time.

What sort of training do you need to use policy research tools effectively?

Let me give you three examples, bearing in mind that the tools metaphor will get annoying soon:

1. You like the look of the policy cycle.

It offers a simple way to turn evidence into policy: you use evidence to identify a problem, provide a range of feasible evidence-based solutions, choose the best solution, then legitimise, implement, and evaluate the solution. However, you soon find that the cycle is the equivalent of, say, a manual carpet sweeper. The technology has moved on, and we now have a much improved understanding of the policy process. In empirical policy analysis, the cycle remains as a way to begin our discussion before identifying more useful concepts and theories.

Using the tools metaphor, you need regular training to know about the state of the art of the technology we use.

2. You like the look of multiple streams analysis.

It too offers a simple way into the subject. Further, it remains a well-respected and much-used tool for analysis. Let’s say it is like the X-ray. It has been used for decades and it remains a key tool in medicine (and security). You need some training to operate it and, crucially, your training would not stop at ‘here is how we used it in the 1980s’.

In other words, many people pick up Kingdon’s classic book and apply its simple insights without much reference to 3 decades of conceptual advance (much contemporary MSA was developed by other people) and hundreds of other empirical applications.

Using the tools metaphor, you need regular training to keep up to date with the ways in which people use the technology.

3. You want to pick and choose insights from several theories.

This is a fairly common exercise: people pick and choose concepts, adapt them to produce their models, and apply different concepts in different ways. If you are optimistic, you will think of something like a Dremel which has the same starting point/ base unit and dozens of compatible attachments. If, like me, you are not so optimistic, you imagine a frying pan radio or an X-ray machine glued onto the side of an MRI. It can be fruitful to combine the insights of concepts and theories, but not without thinking about the trade-offs and the compatibility between concepts (which prompts some scholars to identify one kind of tool to replace another).

Using the tools metaphor, you need regular training to know how compatible each tool is with the other, and if one is used to replace another.

This last point is crucial if you want to go beyond using a tool for a one-off project, to compare your insights and lessons with other people. Many people will want to know how you fared when you used the same tools/ approach/ language as them, and you can learn from each other’s experiences. Indeed, the aim of theory is to produce comparable and, if possible, generalisable insights, Relatively few people will want to learn from someone who glues an X-ray to an MRI, and it will be difficult to generalise from the experience.

The upshot is that you can indeed pick up some policy research tools and use them to improve your understanding of the world. Indeed, I encourage you to do so in this series of posts which outlines the concepts you are most likely to see stocked in Home Depot.

However, I also suggest that you use them as the first step of your project (or engage the help of more qualified people), since most of these concepts come with a training manual that can take years to read.

Policy Concepts in 1000 Words: The Policy Cycle and its Stages

See also What is Policy? and the Policy concepts in 1000 words series

(podcast download)

The classic way to study policymaking is to break it down into stages. The stages have changed over the years, and vary by country, but the basic ideas remain the same:

1. Descriptive. Let’s simplify a complex world by identifying its key elements.
2. Prescriptive. Let’s work out how to make policy, to translate public demands into government action (or at least to carry out government policy).

A cycle divides the policy process into a series of stages, from a notional starting point at which policymakers begin to think about a policy problem to a notional end point at which a policy has been implemented and policymakers think about how successful it has been before deciding what to do next. The image is of a continuous process rather than a single event. The evaluation stage of policy 1 represents the first stage of policy 2, as lessons learned in the past set the agenda for choices to be made in the future:

• Agenda setting. Identifying problems that require government attention, deciding which issues deserve the most attention and defining the nature of the problem.
• Policy formulation. Setting objectives, identifying the cost and estimating the effect of solutions, choosing from a list of solutions and selecting policy instruments.
• Legitimation. Ensuring that the chosen policy instruments have support. It can involve one or a combination of: legislative approval, executive approval, seeking consent through consultation with interest groups, and referenda.
• Implementation. Establishing or employing an organization to take responsibility for implementation, ensuring that the organization has the resources (such as staffing, money and legal authority) to do so, and making sure that policy decisions are carried out as planned.
• Evaluation. Assessing the extent to which the policy was successful or the policy decision was the correct one; if it was implemented correctly and, if so, had the desired effect.
• Policy maintenance, succession or termination. Considering if the policy should be continued, modified or discontinued.

The cycle is useful in many ways. It is simple and understandable. It can be applied to all political systems. The emphasis on cycles highlights fluid policymaking.  There is also a wide range of important studies (and key debates) based on the analysis of particular stages – such as the top-down versus bottom-up approaches to the study of policymaking.

However, the stages approach is no longer central to policy studies, partly because it does not help explain what it describes, and partly because it oversimplifies a complex world (does it also seem to take the politics out of policymaking? In other words, note the often-fraught politics of seemingly-innocuous stages such as evaluation). The policymaking system may be seen more as a collection of thousands of policy cycles, which interact with each other to produce much less predictable outcomes.  Indeed, many of the theories or concepts outlined in this series serve as replacements for a focus on cycles (see the The Advocacy Coalition Framework and Multiple Streams Analysis in particular).

The prescriptive side of cycles and stages is a bit more interesting, because it may be both unrealistic and useful at the same time. Stages can be used to organise policymaking in a simple way: identify policymaker aims, identify policies to achieve those aims, select a policy measure, ensure that the selection is legitimised by the population or its legislature, identify the necessary resources, implement and then evaluate the policy.  The academic idea is simple and the consequent advice to policy practitioners is straightforward.  It is difficult – but not impossible – to describe a more meaningful, more realistic, analytical model to policymakers (and give advice on how to act) in the same straightforward way.

Related articles

• https://paulcairney.wordpress.com/1000-words/
• For more on the prescriptive side, see How can policy theory have an impact on policymaking? and chapter 2 of The Politics of Evidence-based Policymaking