Policy Analysis in 750 words: Classic 5-step advice

‘Policy analysis’ describes the identification of a policy problem and possible solutions.

Classic models of policy analysis are client-oriented. Most texts identify the steps that a policy analysis should follow, from identifying a problem and potential solutions, to finding ways to predict and evaluate the impact of each solution. Each text describes this process in different ways, as outlined in Boxes 1-5. However, for the most part, they follow the same five steps:

1. Define a policy problem identified by your client.
2. Identify technically and politically feasible solutions.
3. Use value-based criteria and political goals to compare solutions.
4. Predict the outcome of each feasible solution.
5. Make a recommendation to your client.

Further, they share the sense that analysts need to adapt pragmatically to a political environment. Assume that your audience is not an experienced policy analyst. Assume a political environment in which there is limited attention or time to consider problems, and some policy solutions will be politically infeasible. Describe the policy problem for your audience: to help them see it as something worthy of their energy. Discuss a small number of possible solutions, the differences between them, and their respective costs and benefits. Keep it short with the aid of visual techniques that sum up the issue concisely, to minimise cognitive load and make the problem seem solvable.

Box 1. Bardach (2012) A Practical Guide for Policy Analysis

1. ‘Define the problem’. Provide a diagnosis of a policy problem, using rhetoric and eye-catching data to generate attention.
2. ‘Assemble some evidence’. Gather relevant data efficiently.
3. ‘Construct the alternatives’. Identify the relevant and feasible policy solutions that your audience might consider.
4. ‘Select the criteria’. Typical value judgements relate to efficiency, equity and fairness, the trade-off between individual freedom and collective action, and the extent to which a policy process involves citizens in deliberation.
5. ‘Project the outcomes’. Focus on the outcomes that key actors care about (such as value for money), and quantify and visualise your predictions if possible.
6. ‘Confront the trade-offs’. Compare the pros and cons of each solution, such as how much of a bad service policymakers will accept to cut costs.
7. ‘Decide’. Examine your case through the eyes of a policymaker.
8. ‘Tell your story’. Identify your target audience and tailor your case. Weigh up the benefits of oral versus written presentation. Provide an executive summary. Focus on coherence and clarity. Keep it simple and concise. Avoid jargon.

Box 2. Dunn (2017) Public Policy Analysis

1. What is the policy problem to be solved? Identify its severity, urgency, cause, and our ability to solve it. Don’t define the wrong problem, such as by oversimplifying.
2. What effect will each potential policy solution have? ‘Forecasting’ methods can help provide ‘plausible’ predictions about the future effects of current/ alternative policies.
3. Which solutions should we choose, and why? Normative assessments are based on values such as ‘equality, efficiency, security, democracy, enlightenment’ and beliefs about the preferable balance between state, communal, and market/ individual solutions (2017: 6; 205).
4. What were the policy outcomes? ‘Monitoring is crucial because it is difficult to predict policy success, and unintended consequences are inevitable (2017: 250).
5. Did the policy solution work as intended? Did it improve policy outcomes? Try to measure the outcomes your solution, while noting that evaluations are contested (2017: 332-41).

Box 3. Meltzer and Schwartz (2019) Policy Analysis as Problem Solving

1. ‘Define the problem’. Problem definition is a political act of framing, as part of a narrative to evaluate the nature, cause, size, and urgency of an issue.
2. ‘Identify potential policy options (alternatives) to address the problem’. Identify many possible solutions, then select the ‘most promising’ for further analysis (2019: 65).
3. ‘Specify the objectives to be attained in addressing the problem and the criteria  to  evaluate  the  attainment  of  these  objectives  as  well as  the  satisfaction  of  other  key  considerations  (e.g.,  equity,  cost, equity, feasibility)’.
4. ‘Assess the outcomes of the policy options in light of the criteria and weigh trade-offs between the advantages and disadvantages of the options’.
5. ‘Arrive at a recommendation’. Make a preliminary recommendation to inform an iterative process, drawing feedback from clients and stakeholder groups (2019: 212).

Box 4. Mintrom (2012) Contemporary Policy Analysis

1. ‘Engage in problem definition’. Define the nature of a policy problem, and the role of government in solving it, while engaging with many stakeholders (2012: 3; 58-60).
2. ‘Propose alternative responses to the problem’. Identify how governments have addressed comparable problems, and a previous policy’s impact (2012: 21).
3. ‘Choose criteria for evaluating each alternative policy response’. ‘Effectiveness, efficiency, fairness, and administrative efficiency’ are common (2012: 21).
4. ‘Project the outcomes of pursuing each policy alternative’. Estimate the cost of a new policy, in comparison with current policy, and in relation to factors such as savings to society or benefits to certain populations.
5. ‘Identify and analyse trade-offs among alternatives’. Use your criteria and projections to compare each alternative in relation to their likely costs and benefits.
6. ‘Report findings and make an argument for the most appropriate response’. Client-oriented advisors identify the beliefs of policymakers and tailor accordingly (2012: 22).

Box 5 Weimer and Vining (2017) Policy Analysis: Concepts and Practice

1. ‘Write to Your Client’. Having a client such as an elected policymaker requires you to address the question they ask, by their deadline, in a clear and concise way that they can understand (and communicate to others) quickly (2017: 23; 370-4).
2. ‘Understand the Policy Problem’. First, ‘diagnose the undesirable condition’. Second, frame it as ‘a market or government failure (or maybe both)’.
3. ‘Be Explicit About Values’ (and goals). Identify (a) the values to prioritise, such as ‘efficiency’, ‘equity’, and ‘human dignity’, and (b) ‘instrumental goals’, such as ‘sustainable public finance or political feasibility’, to generate support for solutions.
4. ‘Specify Concrete Policy Alternatives’. Explain potential solutions in sufficient detail to predict the costs and benefits of each ‘alternative’ (including current policy).
5. ‘Predict and Value Impacts’. Short deadlines dictate that you use ‘logic and theory, rather than systematic empirical evidence’ to make predictions efficiently (2017: 27)
6. ‘Consider the Trade-Offs’. Each alternatives will fulfil certain goals more than others. Produce a summary table to make value-based choices about trade-offs (2017: 356-8).
7. ‘Make a Recommendation’. ‘Unless your client asks you not to do so, you should explicitly recommend one policy’ (2017: 28).

This is an excerpt from The Politics of Policy Analysis, found here: https://paulcairney.wordpress.com/policy-analysis-in-750-words/

Policy Analysis in 750 Words: how much impact can you expect from your analysis?

This post forms one part of the Policy Analysis in 750 words series overview.

Throughout this series you may notice three different conceptions about the scope of policy analysis:

1. ‘Ex ante’ (before the event) policy analysis. Focused primarily on defining a problem, and predicting the effect of solutions, to inform current choice (as described by Meltzer and Schwartz and Thissen and Walker).
2. ‘Ex post’ (after the event) policy analysis. Focused primarily on monitoring and evaluating that choice, perhaps to inform future choice (as described famously by Weiss).
3. Some combination of both, to treat policy analysis as a continuous (never-ending) process (as described by Dunn).

As usual, these are not hard-and-fast distinctions, but they help us clarify expectations in relation to different scenarios.

1. The impact of old-school ex ante policy analysis

Radin provides a valuable historical discussion of policymaking with the following elements:

• a small number of analysts, generally inside government (such as senior bureaucrats, scientific experts, and – in particular- economists),
• giving technical or factual advice,
• about policy formulation,
• to policymakers at the heart of government,
• on the assumption that policy problems would be solved via analysis and action.

This kind of image signals an expectation for high impact: policy analysts face low competition, enjoy a clearly defined and powerful audience, and their analysis is expected to feed directly into choice.

Radin goes on to describe a much different, modern policy environment: more competition, more analysts spread across and outside government, with a less obvious audience, and – even if there is a client – high uncertainty about where the analysis fits into the bigger picture.

Yet, the impetus to seek high and direct impact remains.

This combination of shifting conditions but unshifting hopes/ expectations helps explain a lot of the pragmatic forms of policy analysis you will see in this series, including:

• Keep it catchy, gather data efficiently, tailor your solutions to your audience, and tell a good story (Bardach)
• Speak with an audience in mind, highlight a well-defined problem and purpose, project authority, use the right form of communication, and focus on clarity, precision, conciseness, and credibility ( Smith)
• Address your client’s question, by their chosen deadline, in a clear and concise way that they can understand (and communicate to others) quickly (Weimer and Vining)
• Client-oriented advisors identify the beliefs of policymakers and anticipate the options worth researching (Mintrom)
• Identify your client’s resources and motivation, such as how they seek to use your analysis, the format of analysis they favour (make it ‘concise’ and ‘digestible’), their deadline, and their ability to make or influence the policies you might suggest (Meltzer and Schwartz).
• ‘Advise strategically’, to help a policymaker choose an effective solution within their political context (Thissen and Walker).
• Focus on producing ‘policy-relevant knowledge’ by adapting to the evidence-demands of policymakers and rejecting a naïve attachment to ‘facts speaking for themselves’ or ‘knowledge for its own sake’ (Dunn).

2. The impact of research and policy evaluation

Many of these recommendations are familiar to scientists and researchers, but generally in the context of far lower expectations about their likely impact, particularly if those expectations are informed by policy studies (compare Oliver & Cairney with Cairney & Oliver).

In that context, Weiss’ work is a key reference point. It gives us a menu of ways in which policymakers might use policy evaluation (and research evidence more widely):

• to inform solutions to a problem identified by policymakers
• as one of many sources of information used by policymakers, alongside ‘stakeholder’ advice and professional and service user experience
• as a resource used selectively by politicians, with entrenched positions, to bolster their case
• as a tool of government, to show it is acting (by setting up a scientific study), or to measure how well policy is working
• as a source of ‘enlightenment’, shaping how people think over the long term (compare with this discussion of ‘evidence based policy’ versus ‘policy based evidence’).

In other words, researchers may have a role, but they struggle (a) to navigate the politics of policy analysis, (b) find the right time to act, and (c) to secure attention, in competition with many other policy actors.

3. The potential for a form of continuous impact

Dunn suggests that the idea of ‘ex ante’ policy analysis is misleading, since policymaking is continuous, and evaluations of past choices inform current choices. Think of each policy analysis steps as ‘interdependent’, in which new knowledge to inform one step also informs the other four. For example, routine monitoring helps identify compliance with regulations, if resources and services reach ‘target groups’, if money is spent correctly, and if we can make a causal link between the policy solutions and outcomes. Its impact is often better seen as background information with intermittent impact.

Key conclusions to bear in mind

1. The demand for information from policy analysts may be disproportionately high when policymakers pay attention to a problem, and disproportionately low when they feel that they have addressed it.
2. Common advice for policy analysts and researchers often looks very similar: keep it concise, tailor it to your audience, make evidence ‘policy relevant’, and give advice (don’t sit on the fence). However, unless researchers are prepared to act quickly, to gather data efficiently (not comprehensively), to meet a tight brief for a client, they are not really in the impact business described by most policy analysis texts.
3. A lot of routine, continuous, impact tends to occur out of the public spotlight, based on rules and expectations that most policy actors take for granted.

Further reading

See the Policy Analysis in 750 words series overview to continue reading on policy analysis.

See the ‘evidence-based policymaking’ page to continue reading on research impact.

Bristol powerpoint: Paul Cairney Bristol EBPM January 2020

Policy Analysis in 750 Words: what you need as an analyst versus policymaking reality

This post forms one part of the Policy Analysis in 750 words series overview. Note for the eagle eyed: you are not about to experience déjà vu. I’m just using the same introduction.

When describing ‘the policy sciences’, Lasswell distinguishes between:

1. ‘knowledge of the policy process’, to foster policy studies (the analysis of policy)
2. ‘knowledge in the process’, to foster policy analysis (analysis for policy)

The lines between each approach are blurry, and each element makes less sense without the other. However, the distinction is crucial to help us overcome the major confusion associated with this question:

Does policymaking proceed through a series of stages?

The short answer is no.

The longer answer is that you can find about 40 blog posts (of 500 and 1000 words) which compare (a) a stage-based model called the policy cycle, and (b) the many, many policy concepts and theories that describe a far messier collection of policy processes.

In a nutshell, most policy theorists reject this image because it oversimplifies a complex policymaking system. The image provides a great way to introduce policy studies, and serves a political purpose, but it does more harm than good:

1. Descriptively, it is profoundly inaccurate (unless you imagine thousands of policy cycles interacting with each other to produce less orderly behaviour and less predictable outputs).
2. Prescriptively, it gives you rotten advice about the nature of your policymaking task (for more on these points, see this chapter, article, article, and series).

Why does the stages/ policy cycle image persist? Two relevant explanations

 

1. It arose from a misunderstanding in policy studies

In another nutshell, Chris Weible and I argue (in a secret paper) that the stages approach represents a good idea gone wrong:

• If you trace it back to its origins, you will find Lasswell’s description of decision functions: intelligence, recommendation, prescription, invocation, application, appraisal and termination.
• These functions correspond reasonably well to a policy cycle’s stages: agenda setting, formulation, legitimation, implementation, evaluation, and maintenance, succession or termination.
• However, Lasswell was imagining functional requirements, while the cycle seems to describe actual stages.

In other words, if you take Lasswell’s list of what policy analysts/ policymakers need to do, multiple it by the number of actors (spread across many organisations or venues) trying to do it, then you get the multi-centric policy processes described by modern theories. If, instead, you strip all that activity down into a single cycle, you get the wrong idea.

2. It is a functional requirement of policy analysis

This description should seem familiar, because the classic policy analysis texts appear to describe a similar series of required steps, such as:

1. define the problem
2. identify potential solutions
3. choose the criteria to compare them
4. evaluate them in relation to their predicted outcomes
5. recommend a solution
6. monitor its effects
7. evaluate past policy to inform current policy.

However, these texts also provide a heavy dose of caution about your ability to perform these steps (compare Bardach, Dunn, Meltzer and Schwartz, Mintrom, Thissen and Walker, Weimer and Vining)

In addition, studies of policy analysis in action suggest that:

• an individual analyst’s need for simple steps, to turn policymaking complexity into useful heuristics and pragmatic strategies,

should not be confused with

• what actually happens when many policy analysts, influencers, and policymakers interact in policy processes (see Radin, and Brans, Geva-May, and Howlett).

What you need versus what you can expect

Overall, this discussion of policy studies and policy analysis reminds us of a major difference between:

1. Functional requirements. What you need from policymaking systems, to (a) manage your task (the 5-8 step policy analysis) and (b) understand and engage in policy processes (the simple policy cycle).
2. Actual processes and outcomes. What policy concepts and theories tell us about bounded rationality (which limit the comprehensiveness of your analysis) and policymaking complexity (which undermines your understanding and engagement in policy processes).

Of course, I am not about to provide you with a solution to these problems.

Still, this discussion should help you worry a little bit less about the circular arguments you will find in key texts: here are some simple policy analysis steps, but policymaking is not as ‘rational’ as the steps suggest, but (unless you can think of an alternative) there is still value in the steps, and so on.

See also:

The New Policy Sciences

Policy Analysis in 750 Words: Defining policy problems and choosing solutions

This post forms one part of the Policy Analysis in 750 words series overview.

When describing ‘the policy sciences’, Lasswell distinguishes between:

1. ‘knowledge of the policy process’, to foster policy studies (the analysis of policy)
2. ‘knowledge in the process’, to foster policy analysis (analysis for policy)

The idea is that both elements are analytically separable but mutually informative: policy analysis is crucial to solving real policy problems, policy studies inform the feasibility of analysis, the study of policy analysts informs policy studies, and so on.

Both elements focus on similar questions – such as What is policy? – and explore their descriptive (what do policy actors do?) and prescriptive (what should they do?) implications.

1. What is the policy problem?

Policy studies tend to describe problem definition in relation to framing, narrative, social construction, power, and agenda setting.

Actors exercise power to generate attention for their preferred interpretation, and minimise attention to alternative frames (to help foster or undermine policy change, or translate their beliefs into policy).

Policy studies incorporate insights from psychology to understand (a) how policymakers might combine cognition and emotion to understand problems, and therefore (b) how to communicate effectively when presenting policy analysis.

Policy studies focus on the power to reduce ambiguity rather than simply the provision of information to reduce uncertainty. In other words, the power to decide whose interpretation of policy problems counts, and therefore to decide what information is policy-relevant.

This (unequal) competition takes place within a policy process over which no actor has full knowledge or control.

The classic 5-8 step policy analysis texts focus on how to define policy problems well, but they vary somewhat in their definition of doing it well (see also C.Smith):

• Bardach recommends using rhetoric and eye-catching data to generate attention
• Weimer and Vining and Mintrom recommend beginning with your client’s ‘diagnosis’, placing it in a wider perspective to help analyse it critically, and asking yourself how else you might define it (see also Bacchi, Stone)
• Meltzer and Schwartz and Dunn identify additional ways to contextualise your client’s definition, such as by generating a timeline to help ‘map’ causation or using ‘problem-structuring methods’ to compare definitions and avoid making too many assumptions on a problem’s cause.
• Thissen and Walker compare ‘rational’ and ‘argumentative’ approaches, treating problem definition as something to be measured scientifically or established rhetorically (see also Riker).

These approaches compare with more critical accounts that emphasise the role of power and politics to determine whose knowledge is relevant (L.T.Smith) and whose problem definition counts (Bacchi, Stone). Indeed, Bacchi and Stone provide a crucial bridge between policy analysis and policy studies by reflecting on what policy analysts do and why.

2. What is the policy solution?

In policy studies, it is common to identify counterintuitive or confusing aspects of policy processes, including:

• Few studies suggest that policy responses actually solve problems (and many highlight their potential to exacerbate them). Rather, ‘policy solutions’ is shorthand for proposed or alleged solutions.
• Problem definition often sets the agenda for the production of ‘solutions’, but note the phrase solutions chasing problems (when actors have their ‘pet’ solutions ready, and they seek opportunities to promote them).

Policy studies: problem definition informs the feasibility and success of solutions

Generally speaking, to define the problem is to influence assessments of the feasibility of solutions:

• Technical feasibility. Will they work as intended, given the alleged severity and cause of the problem?
• Political feasibility. Will they receive sufficient support, given the ways in which key policy actors weigh up the costs and benefits of action?

Policy studies highlight the inextricable connection between technical and political feasibility. Put simply, (a) a ‘technocratic’ choice about the ‘optimality’ of a solution is useless without considering who will support its adoption, and (b) some types of solution will always be a hard sell, no matter their alleged effectiveness (Box 2.3 below).

In that context, policy studies ask: what types of policy tools or instruments are actually used, and how does their use contribute to policy change? Measures include the size, substance, speed, and direction of policy change.

In turn, problem definition informs: the ways in which actors will frame any evaluation of policy success, and the policy-relevance of the evidence to evaluate solutions. Simple examples include:

• If you define tobacco in relation to: (a) its economic benefits, or (b) a global public health epidemic, evaluations relate to (a) export and taxation revenues, or (b) reductions in smoking in the population.
• If you define ‘fracking’ in relation to: (a) seeking more benefits than costs, or (b) minimising environmental damage and climate change, evaluations relate to (a) factors such as revenue and effective regulation, or simply (b) how little it takes place.

Policy analysis: recognising and pushing boundaries

Policy analysis texts tend to accommodate these insights when giving advice:

• Bardach recommends identifying solutions that your audience might consider, perhaps providing a range of options on a notional spectrum of acceptability.
• Smith highlights the value of ‘precedent’, or relating potential solutions to previous strategies.
• Weimer and Vining identify the importance of ‘a professional mind-set’ that may be more important than perfecting ‘technical skills’
• Mintrom notes that some solutions are easier to sell than others
• Meltzer and Schwartz describe the benefits of making a preliminary recommendation to inform an iterative process, drawing feedback from clients and stakeholder groups
• Dunn warns against too-narrow forms of ‘evidence based’ analysis which undermine a researcher’s ability to adapt well to the evidence-demands of policymakers
• Thissen and Walker relate solution feasibility to a wide range of policy analysis ‘styles’

Still, note the difference in emphasis.

Policy analysis education/ training may be about developing the technical skills to widen definitions and apply many criteria to compare solutions.

Policy studies suggest that problem definition and a search for solutions takes place in an environment where many actors apply a much narrower lens and are not interested in debates on many possibilities (particularly if they begin with a solution).

I have exaggerated this distinction between each element, but it is worth considering the repeated interaction between them in practice: politics and policymaking provide boundaries for policy analysis, analysis could change those boundaries, and policy studies help us reflect on the impact of analysts.

I’ll take a quick break, then discuss how this conclusion relates to the idea of ‘entrepreneurial’ policy analysis.

Further reading

Understanding Public Policy (2020: 28) describes the difference between governments paying for and actually using the ‘tools of policy formulation’. To explore this point, see ‘The use and non-use of policy appraisal tools in public policy making‘ and The Tools of Policy Formulation.

Policy Analysis in 750 Words: Reflecting on your role as a policy analyst

This post forms one part of the Policy Analysis in 750 words series overview.

One aim of this series is to combine insights from policy research (1000, 500) and policy analysis texts.

If we take key insights from policy theories seriously, we can use them to identify (a) the constraints to policy analytical capacity, and (b) the ways in which analysts might address them. I use the idea of policy analyst archetypes to compare a variety of possible responses.

Key constraints to policy analytical capacity

Terms like ‘bounded rationality’ highlight major limits on the ability of humans and organisations to process information.

• Humans use heuristics or cognitive shortcuts to process enough information to make choices, and institutions are the rules used by organisations to limit information processing.

Terms like policymaking ‘context’, ‘environments’, and multi-centric policymaking suggest that the policy process is beyond the limits of policymaker understanding and control.

• Policy actors need to find ways to act, with incomplete information about the problem they seek to solve and the likely impact of their ‘solution’.
• They gather information to help reduce uncertainty, but problem definition is really about exercising power to reduce ambiguity: select one way to interpret a problem (at the expense of most others), and limit therefore limit the relevance and feasibility of solutions.
• This context informs how actors might use the tools of policy analysis. Key texts in this series highlight the use of tools to establish technical feasibility (will it work as intended?), but policymakers also select tools for their political feasibility (who will support or oppose this measure?).

How might policy analysts address these constraints ethically?

Most policy analysis texts (in this series) consider the role of professional ethics and values during the production of policy analysis. However, they also point out that there is not a clearly defined profession and associated code of conduct (e.g. see Adachi). In that context, let us begin with some questions about the purpose of policy analysis and your potential role:

1. Is your primary role to serve individual clients or some notion of the ‘public good’?
2. Should you maximise your role as an individual or play your part in a wider profession?
3. What is the balance between the potential benefits of individual ‘entrepreneurship’ and collective ‘co-productive’ processes?
4. Which policy analysis techniques should you prioritise?
5. What forms of knowledge and evidence count in policy analysis?
6. What does it mean to communicate policy analysis responsibly?
7. Should you provide a clear recommendation or encourage reflection?

 

Policy analysis archetypes: pragmatists, entrepreneurs, manipulators, storytellers, and decolonisers

In that context, I have created a story of policy analysis archetypes to identify the elements that each text emphasises.

The pragmatic policy analyst

• Bardach provides the classic simple, workable, 8-step system to present policy analysis to policymakers while subject to time and resource-pressed political conditions.
• Dunn also uses Wildavsky’s famous phrase ‘art and craft’ to suggest that scientific and ‘rational’ methods can only take us so far.

The professional, client–oriented policy analyst

• Weimer and Vining provide a similar 7-step client-focused system, but incorporating a greater focus on professional development and economic techniques (such as cost-benefit-analysis) to emphasise a particular form of professional analyst.
• Meltzer and Schwartz also focus on advice to clients, but with a greater emphasis on a wide variety of methods or techniques (including service design) to encourage the co-design of policy analysis with clients.

The communicative policy analyst

•  C. Smith focuses on how to write and communicate policy analysis to clients in a political context.
• Compare with Spiegelhalter and Gigerenzer on how to communicate responsibly when describing uncertainty, probability, and risk.

The manipulative policy analyst.

• Riker helps us understand the relationship between two aspects of agenda setting: the rules/ procedures to make choice, and the framing of policy problems and solutions.

The entrepreneurial policy analyst

• Mintrom shows how to combine insights from studies of policy entrepreneurship and policy analysis, to emphasise the benefits of collaboration and creativity.

The questioning policy analyst

• Bacchi  analyses the wider context in which people give and use such advice, to identify the emancipatory role of analysis and encourage policy analysts to challenge dominant social constructions of problems and populations.

The storytelling policy analyst

• Stone identifies the ways in which people use storytelling and argumentation techniques to define problems and justify solutions. This process is about politics and power, not objectivity and optimal solutions.

The decolonizing policy analyst.

• L.T. Smith does not describe policy analysis directly, but shows how the ‘decolonization of research methods’ can inform the generation and use of knowledge.
• Compare with Hindess on the ways in which knowledge-based hierarchies rely on an untenable, circular logic.
• Compare with Michener’s thread, discussing Doucet’s new essay on (a) the role of power and knowledge in limiting (b) the ways in which we gather evidence to analyse policy problems.

Incisive essay from @bailabomba on studying the use of research evidence through critical perspectives that center the marginalized. There is so too much good stuff in here to summarize via twitter (you should just read it). But let me point out a few things that resonated (1/n) https://t.co/nIahyIjwBo

— Jamila Michener (@povertyscholar) December 17, 2019

Using archetypes to define the problem of policy analysis

Studies of the field (e.g. Radin plus Brans, Geva-May, and Howlett) suggest that there are many ways to do policy analysis. Further, as Thissen and Walker describe, such roles are not mutually exclusive, your views on their relative value could change throughout the process of analysis, and you could perform many of these roles.

Further, each text describes multiple roles, and some seem clustered together:

• pragmatic, client-orientated, and communicative could sum-up the traditional 5-8 step approaches, while
• questioning, storytelling, and decolonizing could sum up an important (‘critical’) challenge to narrow ways of thinking about policy analysis and the use of information.

Still, the emphasis matters.

Each text is setting an agenda or defining the problem of policy analysis more-or-less in relation to these roles. Put simply, the more you are reading about economic theory and method, the less you are reading about dominance and manipulation.

How can you read further?

Michener’s ‘Policy Feedback in a Racialized Polity’ connects to studies of historical institutionalism, and reminds us to use insights from policy theories to identify the context for policy analysis.

I have co-authored a lot about uncertainty/ ambiguity in relation to ‘evidence based policymaking’, including:

• Fostering Evidence-informed Policy Making: Uncertainty Versus Ambiguity(PDF)
• To Bridge the Divide between Evidence and Policy: Reduce Ambiguity as Much as Uncertainty
• How to communicate effectively with policymakers: combine insights from psychology and policy studies

See also The new policy sciences for a discussion of how these issues inform Lasswell’s original vision for the policy sciences (combining the analysis of and for policy).

Policy Analysis in 750 words: Wil Thissen and Warren Walker (2013) Public Policy Analysis

Please see the Policy Analysis in 750 words series overview before reading the summary. Please note that this is an edited book and the full list of authors (PDF) is here. I’m using the previous sentence as today’s excuse for not sticking to 750 words.

Wil Thissen and Warren Walker (editors) (2013) Public Policy Analysis: New Developments (Springer)

Our premise is that there is no single, let alone ‘one best’, way of conducting policy analyses (Thissen and Walker, 2013: 2)

Thissen and Walker (2013: 2) begin by identifying the proliferation of (a) policy analysts inside and outside government, (b) the many approaches and methods that could count as policy analysis (see Radin), and therefore (c) a proliferation of concepts to describe it.

Like Vining and Weimar, they distinguish between:

1. Policy analysis, as the advice given to clients before they make a choice. Thissen and Walker (2013: 4) describe analysts working with a potential range of clients, when employed directly by governments or organisations, or acting more as entrepreneurs with multiple audiences in mind (compare with Bardach, Weimer & Vining, Mintrom).
2. Policy process research, as the study of such actors within policymaking systems (see 500 and 1000).

Policy theory: implications for policy analysis

Policy process research informs our understanding of policy analysis, identifying what analysts and their clients (a) can and cannot do, which informs (b) what they should do.

As Enserink et al (2012: 12-3) describe, policy analysis (analysis for policy) will differ profoundly if the policy process is ‘chaotic and messy’ rather than ‘neat and rational’.

The range of policy concepts and theories (analysis of policy) at our disposal helps add meaning to policy analysis as a practice. Like Radin, Enserink et al trace historic attempts to seek ‘rational’ policy analysis then conclude that modern theories – describing policymaking complexity – are ‘more in line with political reality’ (2012: 13-6).

As such, policy analysis shifts from:

(a) A centralised process with few actors inside government, to (b) a messy process including many policymakers and influencers, inside and outside government

(a) Translating science into policy, to (b) a competition to frame issues and assess policy-relevant knowledge

(a) An ‘optimal’ solution from one perspective, to (b) a negotiated solution based on many perspectives (in which optimality is contested)

(a) Analysing a policy problem/ solution with a common metric (such as cost benefit analysis), to (b) developing skills relating to: stakeholder analysis, network management, collaboration, mediation or conflict resolution based on sensitivity to the role of different beliefs, and the analysis of policymaking institutions to help resolve fragmentation (2013: 17-34).

Their Table 2.1 (2012: 35) outlines these potential differences (pop your reading glasses on …. now!):

In many cases, the role of an analyst remains uncertain. If we follow the ACF story, does an analyst appeal to one coalition or seek to mediate between them? If we follow MSA, do they wait for a ‘window of opportunity’ or seek to influence problem definition and motivation to adopt certain solutions?

Policy Analysis: implications for policy theory

In that context, rather than identify a 5-step plan for policy analysis, Mayer et al (2013: 43-50) suggest that policy analysts tend to perform one or more of six activities:

1. ‘Research and analyze’, to collect information relevant to policy problems.
2. ‘Design and recommend’, to produce a range of potential solutions.
3. ‘Clarify values and arguments’, to identify potential conflicts and facilitate high quality debate.
4. ‘Advise strategically’, to help a policymaker choose an effective solution within their political context.
5. ‘Democratize’, to pursue a ‘normative and ethical objective: it should further equal access to, and influence on, the policy process for all stakeholders’ (2013: 47)
6. ‘Mediate’, to foster many forms of cooperation between governments, stakeholders (including business), researchers, and/ or citizens.

Styles of policy analysis

Policy analysts do not perform these functions sequentially or with equal weight.

Rather, Mayer et al (2013: 50-5) describe ‘six styles of policy analysis’ that vary according to the analyst’s ‘assumptions about science (epistemology), democracy, learning, and change’ (and these assumptions may change during the process):

1. Rational, based on the idea that we can conduct research in a straightforward way within a well-ordered policy process (or modify the analysis to reflect limits to research and order).
2. Argumentative, based on a competition to define policy problems and solutions (see Stone).
3. Client advice, based on the assumption that analysis is part of a ‘political game’, and analysts bring knowledge of political strategy and policymaking complexity.
4. Participatory, to facilitate a more equal access to information and debate among citizens.
5. Process, based on the idea that the faithful adherence to good procedures aids high quality analysis (and perhaps mitigates an ‘erratic and volatile’ policy process)
6. Interactive, based on the idea that the rehearsal of many competing perspectives is useful to policymaker deliberations (compare with reflexive learning).

In turn, these styles prompt different questions to evaluate the activities associated with analysis (2013: 56):

In relation to the six policy analysis activities,

• the criteria for good policy analysis include: the quality of knowledge, usefulness of advice to clients and stakeholders, quality of argumentation, pragmatism of advice, transparency of processes, and ability to secure a mediated settlement (2013: 58).
• The positive role for analysts includes ‘independent scientist’ or expert, ‘ethicist’, ‘narrator’, ‘counsellor’, ‘entrepreneur’,’ democratic advocate’, or ‘facilitator’ (2013: 59).

Further, their – rather complicated – visualisations of these roles (e.g. p60; compare with the Appendix) project the (useful) sense that (a) individuals face a trade-off between roles (even if they seek to combine some), and (b) many people making many trade-offs adds up to a complex picture of activity.

Therefore, we should bear in mind that

(a) there exist some useful 5-step guides for budding analyst, but

(b) even if they adopt a simple strategy, analysts will also need to find ways to understand and engage with a complex policymaking systems containing a huge number of analysts, policymakers, and influencers.

Policy Analysis styles: implications for problem definition and policy design

Thissen (2013: 66-9) extends the focus on policymaking context and policy analysis styles to problem definition, including:

1. A rational approach relies on research knowledge to diagnose problems (the world is knowable, use the best scientific methods to produce knowledge, and subject the results to scientific scrutiny).
2. A ‘political game model’ emphasises key actors and their perspectives, value conflicts, trust, and interdependence (assess the potential to make deals and use skills of mediation and persuasion to secure them).

These different starting points influence they ways in which analysts might take steps to identify: how people perceive policy problems, if other definitions are more useful, how to identify a problem’s cause and effect, and the likely effect of a proposed solution, communicate uncertainty, and relate the results to a ‘policy arena’ with its own rules on resolving conflict and producing policy instruments (2013: 70-84; 93-4).

Similarly, Bots (2013: 114) suggests that these styles inform a process of policy design, constructed to change people’s minds during repeated interactions with clients (such as by appealing to scientific evidence or argumentation).

Bruijn et al (2013: 134-5) situate such activities in modern discussions of policy analysis:

1. In multi-centric systems, with analysts focused less on ‘unilateral decisions using command and control’ and more on ‘consultation and negotiation among stakeholders’ in networks.

• The latter are necessary because there will always be contestation about what the available information tells us about the problem, often without a simple way to negotiate choices on solutions.

2. In relation to categories of policy problems, including

• ‘tamed’ (high knowledge/ technically solvable, with no political conflict)
• ‘untamed ethical/political’ (technically solvable, with high moral and political conflict)
• ‘untamed scientific’ (high consensus but low scientific knowledge)
• ‘untamed’ problems (low consensus, low knowledge).

Put simply, ‘rational’ approaches may help address low knowledge, while other skills are required to manage processes such as conflict resolution and stakeholder engagement (2013: 136-40)

Policy Analysis styles: implications for models

Part 2 of the book relates such styles (and assumptions about how ‘rational’ and comprehensive our analyses can be) to models of policy analysis. For example,

1. Walker and van Daalen (2013: 157-84) explore models designed to compare the status quo with a future state, often based on the (shaky) assumption that the world is knowable and we can predict with sufficient accuracy the impact of policy solutions.
2. Hermans and Cunningham (2013: 185-213) describe models to trace agent behaviour in networks and systems, and create multiple possible scenarios, which could help explore the ‘implementability’ of policies.
3. Walker et al (2013: 215-61) relate policy analysis styles to how analysts might deal with uncertainty.

• Some models may serve primarily to reduce ‘epistemic’ uncertainty associated with insufficient knowledge about the future (perhaps with a focus on methods and statistical analysis).
• Others may focus on resolving ambiguity, in which many actors may define/ interpret problems and feasible solutions in different ways.

Overall, this book contains one of the most extensive discussions of 101 different technical models for policy analysis, but the authors emphasise their lack of value without initial clarity about (a) our beliefs regarding the nature of policymaking and (b) the styles of analysis we should use to resolve policy problems. Few of these initial choices can be resolved simply with reference to scientific analysis or evidence.

Policy Analysis in 750 words: Beryl Radin, B (2019) Policy Analysis in the Twenty-First Century

Please see the Policy Analysis in 750 words series overview before reading the summary. As usual, the 750-word description is more for branding than accuracy.

Beryl Radin (2019) Policy Analysis in the Twenty-First Century (Routledge)

‘The basic relationship between a decision-maker (the client) and an analyst has moved from a two-person encounter to an extremely complex and diverse set of interactions’ (Radin, 2019: 2).

Many texts in this series continue to highlight the client-oriented nature of policy analysis (Weimer and Vining), but within a changing policy process that has altered the nature of that relationship profoundly.

This new policymaking environment requires new policy analysis skills and training (see Mintrom), and limits the applicability of classic 8-step (or 5-step) policy analysis techniques (2019: 82).

We can use Radin’s work to present two main stories of policy analysis:

1. The old ways of making policy resembled a club, or reflected a clear government hierarchy, involving:

• a small number of analysts, generally inside government (such as senior bureaucrats, scientific experts, and – in particular- economists),
• giving technical or factual advice,
• about policy formulation,
• to policymakers at the heart of government,
• on the assumption that policy problems would be solved via analysis and action.

2. Modern policy analysis is characterised by a more open and politicised process in which:

• many analysts, inside and outside government,
• compete to interpret facts, and give advice,
• about setting the agenda, and making, delivering, and evaluating policy,
• across many policymaking venues,
• often on the assumption that governments have a limited ability to understand and solve complex policy problems.

As a result, the client-analyst relationship is increasingly fluid:

In previous eras, the analyst’s client was a senior policymaker, the main focus was on the analyst-client relationship, and ‘both analysts and clients did not spend much time or energy thinking about the dimensions of the policy environment in which they worked’ (2019: 59). Now, in a multi-centric policymaking environment:

1. It is tricky to identify the client.

• We could imagine the client to be someone paying for the analysis, someone affected by its recommendations, or all policy actors with the ability to act on the advice (2019: 10).
• If there is ‘shared authority’ for policymaking within one political system, a ‘client’ (or audience) may be a collection of policymakers and influencers spread across a network containing multiple types of government, non-governmental actors, and actors responsible for policy delivery (2019: 33).
• The growth in international cooperation also complicates the idea of a single client for policy advice (2019: 33-4)
• This shift may limit the ‘face-to-face encounters’ that would otherwise provide information for – and perhaps trust in – the analyst (2019: 2-3).

2. It is tricky to identify the analyst

• Radin (2019: 9-25) traces, from the post-war period in the US, a major expansion of policy analysts, from the notional centre of policymaking in federal government towards analysts spread across many venues, inside government (across multiple levels, ‘policy units’, and government agencies) and congressional committees, and outside government (such as in influential think tanks).
• Policy analysts can also be specialist external companies contracted by organisations to provide advice (2019: 37-8).
• This expansion shifted the image of many analysts, from a small number of trusted insiders towards many being treated as akin to interest groups selling their pet policies (2019: 25-6).
• The nature – and impact – of policy analysis has always been a little vague, but now it seems more common to suggest that ‘policy analysts’ may really be ‘policy advocates’ (2019: 44-6).
• As such, they may now have to work harder to demonstrate their usefulness (2019: 80-1) and accept that their analysis will have a limited impact (2019: 82, drawing on Weiss’ discussion of ‘enlightenment’).

Consequently, the necessary skills of policy analysis have changed:

Although many people value systematic policy analysis (and many rely on economists), an effective analyst does not simply apply economic or scientific techniques to analyse a problem or solution, or rely on one source of expertise or method, as if it were possible to provide ‘neutral information’ (2019: 26).

Indeed, Radin (2019: 31; 48) compares the old ‘acceptance that analysts would be governed by the norms of neutrality and objectivity’ with

(a) increasing calls to acknowledge that policy analysis is part of a political project to foster some notion of public good or ‘public interest’, and

(b)  Stone’s suggestion that the projection of reason and neutrality is a political strategy.

In other words, the fictional divide between political policymakers and neutral analysts is difficult to maintain.

Rather, think of analysts as developing wider skills to operate in a highly political environment in which the nature of the policy issue is contested, responsibility for a policy problem is unclear, and it is not clear how to resolve major debates on values and priorities:

• Some analysts will be expected to see the problem from the perspective of a specific client with a particular agenda.
• Other analysts may be valued for their flexibility and pragmatism, such as when they acknowledge the role of their own values, maintain or operate within networks, communicate by many means, and supplement ‘quantitative data’ with ‘hunches’ when required (2019: 2-3; 28-9).

Radin (2019: 21) emphasises a shift in skills and status

The idea of (a) producing new and relatively abstract ideas, based on high control over available information, at the top of a hierarchical organisation, makes way for (b) developing the ability to:

• generate a wider understanding of organisational and policy processes, reflecting the diffusion of power across multiple policymaking venues
• identify a map of stakeholders,
• manage networks of policymakers and influencers,
• incorporate ‘multiple and often conflicting perspectives’,
• make and deliver more concrete proposals (2019: 59-74), while recognising
• the contested nature of information, and the practices sued to gather it, even during multiple attempts to establish the superiority of scientific evidence (2019: 89-103),
• the limits to a government’s ability to understand and solve problems (2019: 95-6),
• the inescapable conflict over trade-offs between values and goals, which are difficult to resolve simply by weighting each goal (2019: 105-8; see Stone), and
• do so flexibly, to recognise major variations in problem definition, attention and networks across different policy sectors and notional ‘stages’ of policymaking (2019: 75-9; 84).

Radin’s (2019: 48) overall list of relevant skills include:

1. ‘Case study methods, Cost- benefit analysis, Ethical analysis, Evaluation, Futures analysis, Historical analysis, Implementation analysis, Interviewing, Legal analysis, Microeconomics, Negotiation, mediation, Operations research, Organizational analysis, Political feasibility analysis, Public speaking, Small- group facilitation, Specific program knowledge, Statistics, Survey research methods, Systems analysis’

They develop alongside analytical experience and status, from the early career analyst trying to secure or keep a job, to the experienced operator looking forward to retirement (2019: 54-5)

A checklist for policy analysts

Based on these skills requirements, the contested nature of evidence, and the complexity of the policymaking environment, Radin (2019: 128-31) produces a 4-page checklist of – 91! – questions for policy analysts.

For me, it serves two main functions:

1. It is a major contrast to the idea that we can break policy analysis into a mere 5-8 steps (rather, think of these small numbers as marketing for policy analysis students, akin to 7-minute abs)
2. It presents policy analysis as an overwhelming task with absolutely no guarantee of policy impact.

To me, this cautious, eyes-wide-open, approach is preferable to the sense that policy analysts can change the world if they just get the evidence and the steps right.

Further Reading:

1. Iris Geva-May (2005) ‘Thinking Like a Policy Analyst. Policy Analysis as a Clinical Profession’, in Geva-May (ed) Thinking Like a Policy Analyst. Policy Analysis as a Clinical Profession (Basingstoke: Palgrave)

Although the idea of policy analysis may be changing, Geva-May (2005: 15) argues that it remains a profession with its own set of practices and ways of thinking. As with other professions (like medicine), it would be unwise to practice policy analysis without education and training or otherwise learning the ‘craft’ shared by a policy analysis community (2005: 16-17). For example, while not engaging in clinical diagnosis, policy analysts can draw on 5-step process to diagnose a policy problem and potential solutions (2005: 18-21). Analysts may also combine these steps with heuristics to determine the technical and political feasibility of their proposals (2005: 22-5), as they address inevitable uncertainty and their own bounded rationality (2005: 26-34; see Gigerenzer on heuristics). As with medicine, some aspects of the role – such as research methods – can be taught in graduate programmes, while others may be better suited to on the job learning (2005: 36-40). If so, it opens up the possibility that there are many policy analysis professions to reflect different cultures in each political system (and perhaps the venues within each system).

2. Vining and Weimar’s take on the distinction between policy analysis and policy process research