Policy Analysis in 750 words: Classic 5-step advice

‘Policy analysis’ describes the identification of a policy problem and possible solutions.

Classic models of policy analysis are client-oriented. Most texts identify the steps that a policy analysis should follow, from identifying a problem and potential solutions, to finding ways to predict and evaluate the impact of each solution. Each text describes this process in different ways, as outlined in Boxes 1-5. However, for the most part, they follow the same five steps:

1. Define a policy problem identified by your client.
2. Identify technically and politically feasible solutions.
3. Use value-based criteria and political goals to compare solutions.
4. Predict the outcome of each feasible solution.
5. Make a recommendation to your client.

Further, they share the sense that analysts need to adapt pragmatically to a political environment. Assume that your audience is not an experienced policy analyst. Assume a political environment in which there is limited attention or time to consider problems, and some policy solutions will be politically infeasible. Describe the policy problem for your audience: to help them see it as something worthy of their energy. Discuss a small number of possible solutions, the differences between them, and their respective costs and benefits. Keep it short with the aid of visual techniques that sum up the issue concisely, to minimise cognitive load and make the problem seem solvable.

Box 1. Bardach (2012) A Practical Guide for Policy Analysis

1. ‘Define the problem’. Provide a diagnosis of a policy problem, using rhetoric and eye-catching data to generate attention.
2. ‘Assemble some evidence’. Gather relevant data efficiently.
3. ‘Construct the alternatives’. Identify the relevant and feasible policy solutions that your audience might consider.
4. ‘Select the criteria’. Typical value judgements relate to efficiency, equity and fairness, the trade-off between individual freedom and collective action, and the extent to which a policy process involves citizens in deliberation.
5. ‘Project the outcomes’. Focus on the outcomes that key actors care about (such as value for money), and quantify and visualise your predictions if possible.
6. ‘Confront the trade-offs’. Compare the pros and cons of each solution, such as how much of a bad service policymakers will accept to cut costs.
7. ‘Decide’. Examine your case through the eyes of a policymaker.
8. ‘Tell your story’. Identify your target audience and tailor your case. Weigh up the benefits of oral versus written presentation. Provide an executive summary. Focus on coherence and clarity. Keep it simple and concise. Avoid jargon.

Box 2. Dunn (2017) Public Policy Analysis

1. What is the policy problem to be solved? Identify its severity, urgency, cause, and our ability to solve it. Don’t define the wrong problem, such as by oversimplifying.
2. What effect will each potential policy solution have? ‘Forecasting’ methods can help provide ‘plausible’ predictions about the future effects of current/ alternative policies.
3. Which solutions should we choose, and why? Normative assessments are based on values such as ‘equality, efficiency, security, democracy, enlightenment’ and beliefs about the preferable balance between state, communal, and market/ individual solutions (2017: 6; 205).
4. What were the policy outcomes? ‘Monitoring is crucial because it is difficult to predict policy success, and unintended consequences are inevitable (2017: 250).
5. Did the policy solution work as intended? Did it improve policy outcomes? Try to measure the outcomes your solution, while noting that evaluations are contested (2017: 332-41).

Box 3. Meltzer and Schwartz (2019) Policy Analysis as Problem Solving

1. ‘Define the problem’. Problem definition is a political act of framing, as part of a narrative to evaluate the nature, cause, size, and urgency of an issue.
2. ‘Identify potential policy options (alternatives) to address the problem’. Identify many possible solutions, then select the ‘most promising’ for further analysis (2019: 65).
3. ‘Specify the objectives to be attained in addressing the problem and the criteria  to  evaluate  the  attainment  of  these  objectives  as  well as  the  satisfaction  of  other  key  considerations  (e.g.,  equity,  cost, equity, feasibility)’.
4. ‘Assess the outcomes of the policy options in light of the criteria and weigh trade-offs between the advantages and disadvantages of the options’.
5. ‘Arrive at a recommendation’. Make a preliminary recommendation to inform an iterative process, drawing feedback from clients and stakeholder groups (2019: 212).

Box 4. Mintrom (2012) Contemporary Policy Analysis

1. ‘Engage in problem definition’. Define the nature of a policy problem, and the role of government in solving it, while engaging with many stakeholders (2012: 3; 58-60).
2. ‘Propose alternative responses to the problem’. Identify how governments have addressed comparable problems, and a previous policy’s impact (2012: 21).
3. ‘Choose criteria for evaluating each alternative policy response’. ‘Effectiveness, efficiency, fairness, and administrative efficiency’ are common (2012: 21).
4. ‘Project the outcomes of pursuing each policy alternative’. Estimate the cost of a new policy, in comparison with current policy, and in relation to factors such as savings to society or benefits to certain populations.
5. ‘Identify and analyse trade-offs among alternatives’. Use your criteria and projections to compare each alternative in relation to their likely costs and benefits.
6. ‘Report findings and make an argument for the most appropriate response’. Client-oriented advisors identify the beliefs of policymakers and tailor accordingly (2012: 22).

Box 5 Weimer and Vining (2017) Policy Analysis: Concepts and Practice

1. ‘Write to Your Client’. Having a client such as an elected policymaker requires you to address the question they ask, by their deadline, in a clear and concise way that they can understand (and communicate to others) quickly (2017: 23; 370-4).
2. ‘Understand the Policy Problem’. First, ‘diagnose the undesirable condition’. Second, frame it as ‘a market or government failure (or maybe both)’.
3. ‘Be Explicit About Values’ (and goals). Identify (a) the values to prioritise, such as ‘efficiency’, ‘equity’, and ‘human dignity’, and (b) ‘instrumental goals’, such as ‘sustainable public finance or political feasibility’, to generate support for solutions.
4. ‘Specify Concrete Policy Alternatives’. Explain potential solutions in sufficient detail to predict the costs and benefits of each ‘alternative’ (including current policy).
5. ‘Predict and Value Impacts’. Short deadlines dictate that you use ‘logic and theory, rather than systematic empirical evidence’ to make predictions efficiently (2017: 27)
6. ‘Consider the Trade-Offs’. Each alternatives will fulfil certain goals more than others. Produce a summary table to make value-based choices about trade-offs (2017: 356-8).
7. ‘Make a Recommendation’. ‘Unless your client asks you not to do so, you should explicitly recommend one policy’ (2017: 28).

This is an excerpt from The Politics of Policy Analysis, found here: https://paulcairney.wordpress.com/policy-analysis-in-750-words/

Policy Analysis in 750 Words: how much impact can you expect from your analysis?

This post forms one part of the Policy Analysis in 750 words series overview.

Throughout this series you may notice three different conceptions about the scope of policy analysis:

1. ‘Ex ante’ (before the event) policy analysis. Focused primarily on defining a problem, and predicting the effect of solutions, to inform current choice (as described by Meltzer and Schwartz and Thissen and Walker).
2. ‘Ex post’ (after the event) policy analysis. Focused primarily on monitoring and evaluating that choice, perhaps to inform future choice (as described famously by Weiss).
3. Some combination of both, to treat policy analysis as a continuous (never-ending) process (as described by Dunn).

As usual, these are not hard-and-fast distinctions, but they help us clarify expectations in relation to different scenarios.

1. The impact of old-school ex ante policy analysis

Radin provides a valuable historical discussion of policymaking with the following elements:

• a small number of analysts, generally inside government (such as senior bureaucrats, scientific experts, and – in particular- economists),
• giving technical or factual advice,
• about policy formulation,
• to policymakers at the heart of government,
• on the assumption that policy problems would be solved via analysis and action.

This kind of image signals an expectation for high impact: policy analysts face low competition, enjoy a clearly defined and powerful audience, and their analysis is expected to feed directly into choice.

Radin goes on to describe a much different, modern policy environment: more competition, more analysts spread across and outside government, with a less obvious audience, and – even if there is a client – high uncertainty about where the analysis fits into the bigger picture.

Yet, the impetus to seek high and direct impact remains.

This combination of shifting conditions but unshifting hopes/ expectations helps explain a lot of the pragmatic forms of policy analysis you will see in this series, including:

• Keep it catchy, gather data efficiently, tailor your solutions to your audience, and tell a good story (Bardach)
• Speak with an audience in mind, highlight a well-defined problem and purpose, project authority, use the right form of communication, and focus on clarity, precision, conciseness, and credibility ( Smith)
• Address your client’s question, by their chosen deadline, in a clear and concise way that they can understand (and communicate to others) quickly (Weimer and Vining)
• Client-oriented advisors identify the beliefs of policymakers and anticipate the options worth researching (Mintrom)
• Identify your client’s resources and motivation, such as how they seek to use your analysis, the format of analysis they favour (make it ‘concise’ and ‘digestible’), their deadline, and their ability to make or influence the policies you might suggest (Meltzer and Schwartz).
• ‘Advise strategically’, to help a policymaker choose an effective solution within their political context (Thissen and Walker).
• Focus on producing ‘policy-relevant knowledge’ by adapting to the evidence-demands of policymakers and rejecting a naïve attachment to ‘facts speaking for themselves’ or ‘knowledge for its own sake’ (Dunn).

2. The impact of research and policy evaluation

Many of these recommendations are familiar to scientists and researchers, but generally in the context of far lower expectations about their likely impact, particularly if those expectations are informed by policy studies (compare Oliver & Cairney with Cairney & Oliver).

In that context, Weiss’ work is a key reference point. It gives us a menu of ways in which policymakers might use policy evaluation (and research evidence more widely):

• to inform solutions to a problem identified by policymakers
• as one of many sources of information used by policymakers, alongside ‘stakeholder’ advice and professional and service user experience
• as a resource used selectively by politicians, with entrenched positions, to bolster their case
• as a tool of government, to show it is acting (by setting up a scientific study), or to measure how well policy is working
• as a source of ‘enlightenment’, shaping how people think over the long term (compare with this discussion of ‘evidence based policy’ versus ‘policy based evidence’).

In other words, researchers may have a role, but they struggle (a) to navigate the politics of policy analysis, (b) find the right time to act, and (c) to secure attention, in competition with many other policy actors.

3. The potential for a form of continuous impact

Dunn suggests that the idea of ‘ex ante’ policy analysis is misleading, since policymaking is continuous, and evaluations of past choices inform current choices. Think of each policy analysis steps as ‘interdependent’, in which new knowledge to inform one step also informs the other four. For example, routine monitoring helps identify compliance with regulations, if resources and services reach ‘target groups’, if money is spent correctly, and if we can make a causal link between the policy solutions and outcomes. Its impact is often better seen as background information with intermittent impact.

Key conclusions to bear in mind

1. The demand for information from policy analysts may be disproportionately high when policymakers pay attention to a problem, and disproportionately low when they feel that they have addressed it.
2. Common advice for policy analysts and researchers often looks very similar: keep it concise, tailor it to your audience, make evidence ‘policy relevant’, and give advice (don’t sit on the fence). However, unless researchers are prepared to act quickly, to gather data efficiently (not comprehensively), to meet a tight brief for a client, they are not really in the impact business described by most policy analysis texts.
3. A lot of routine, continuous, impact tends to occur out of the public spotlight, based on rules and expectations that most policy actors take for granted.

Further reading

See the Policy Analysis in 750 words series overview to continue reading on policy analysis.

See the ‘evidence-based policymaking’ page to continue reading on research impact.

Bristol powerpoint: Paul Cairney Bristol EBPM January 2020

Policy Analysis in 750 Words: what you need as an analyst versus policymaking reality

This post forms one part of the Policy Analysis in 750 words series overview. Note for the eagle eyed: you are not about to experience déjà vu. I’m just using the same introduction.

When describing ‘the policy sciences’, Lasswell distinguishes between:

1. ‘knowledge of the policy process’, to foster policy studies (the analysis of policy)
2. ‘knowledge in the process’, to foster policy analysis (analysis for policy)

The lines between each approach are blurry, and each element makes less sense without the other. However, the distinction is crucial to help us overcome the major confusion associated with this question:

Does policymaking proceed through a series of stages?

The short answer is no.

The longer answer is that you can find about 40 blog posts (of 500 and 1000 words) which compare (a) a stage-based model called the policy cycle, and (b) the many, many policy concepts and theories that describe a far messier collection of policy processes.

In a nutshell, most policy theorists reject this image because it oversimplifies a complex policymaking system. The image provides a great way to introduce policy studies, and serves a political purpose, but it does more harm than good:

1. Descriptively, it is profoundly inaccurate (unless you imagine thousands of policy cycles interacting with each other to produce less orderly behaviour and less predictable outputs).
2. Prescriptively, it gives you rotten advice about the nature of your policymaking task (for more on these points, see this chapter, article, article, and series).

Why does the stages/ policy cycle image persist? Two relevant explanations

 

1. It arose from a misunderstanding in policy studies

In another nutshell, Chris Weible and I argue (in a secret paper) that the stages approach represents a good idea gone wrong:

• If you trace it back to its origins, you will find Lasswell’s description of decision functions: intelligence, recommendation, prescription, invocation, application, appraisal and termination.
• These functions correspond reasonably well to a policy cycle’s stages: agenda setting, formulation, legitimation, implementation, evaluation, and maintenance, succession or termination.
• However, Lasswell was imagining functional requirements, while the cycle seems to describe actual stages.

In other words, if you take Lasswell’s list of what policy analysts/ policymakers need to do, multiple it by the number of actors (spread across many organisations or venues) trying to do it, then you get the multi-centric policy processes described by modern theories. If, instead, you strip all that activity down into a single cycle, you get the wrong idea.

2. It is a functional requirement of policy analysis

This description should seem familiar, because the classic policy analysis texts appear to describe a similar series of required steps, such as:

1. define the problem
2. identify potential solutions
3. choose the criteria to compare them
4. evaluate them in relation to their predicted outcomes
5. recommend a solution
6. monitor its effects
7. evaluate past policy to inform current policy.

However, these texts also provide a heavy dose of caution about your ability to perform these steps (compare Bardach, Dunn, Meltzer and Schwartz, Mintrom, Thissen and Walker, Weimer and Vining)

In addition, studies of policy analysis in action suggest that:

• an individual analyst’s need for simple steps, to turn policymaking complexity into useful heuristics and pragmatic strategies,

should not be confused with

• what actually happens when many policy analysts, influencers, and policymakers interact in policy processes (see Radin, and Brans, Geva-May, and Howlett).

What you need versus what you can expect

Overall, this discussion of policy studies and policy analysis reminds us of a major difference between:

1. Functional requirements. What you need from policymaking systems, to (a) manage your task (the 5-8 step policy analysis) and (b) understand and engage in policy processes (the simple policy cycle).
2. Actual processes and outcomes. What policy concepts and theories tell us about bounded rationality (which limit the comprehensiveness of your analysis) and policymaking complexity (which undermines your understanding and engagement in policy processes).

Of course, I am not about to provide you with a solution to these problems.

Still, this discussion should help you worry a little bit less about the circular arguments you will find in key texts: here are some simple policy analysis steps, but policymaking is not as ‘rational’ as the steps suggest, but (unless you can think of an alternative) there is still value in the steps, and so on.

See also:

The New Policy Sciences

Policy Analysis in 750 Words: Defining policy problems and choosing solutions

This post forms one part of the Policy Analysis in 750 words series overview.

When describing ‘the policy sciences’, Lasswell distinguishes between:

1. ‘knowledge of the policy process’, to foster policy studies (the analysis of policy)
2. ‘knowledge in the process’, to foster policy analysis (analysis for policy)

The idea is that both elements are analytically separable but mutually informative: policy analysis is crucial to solving real policy problems, policy studies inform the feasibility of analysis, the study of policy analysts informs policy studies, and so on.

Both elements focus on similar questions – such as What is policy? – and explore their descriptive (what do policy actors do?) and prescriptive (what should they do?) implications.

1. What is the policy problem?

Policy studies tend to describe problem definition in relation to framing, narrative, social construction, power, and agenda setting.

Actors exercise power to generate attention for their preferred interpretation, and minimise attention to alternative frames (to help foster or undermine policy change, or translate their beliefs into policy).

Policy studies incorporate insights from psychology to understand (a) how policymakers might combine cognition and emotion to understand problems, and therefore (b) how to communicate effectively when presenting policy analysis.

Policy studies focus on the power to reduce ambiguity rather than simply the provision of information to reduce uncertainty. In other words, the power to decide whose interpretation of policy problems counts, and therefore to decide what information is policy-relevant.

This (unequal) competition takes place within a policy process over which no actor has full knowledge or control.

The classic 5-8 step policy analysis texts focus on how to define policy problems well, but they vary somewhat in their definition of doing it well (see also C.Smith):

• Bardach recommends using rhetoric and eye-catching data to generate attention
• Weimer and Vining and Mintrom recommend beginning with your client’s ‘diagnosis’, placing it in a wider perspective to help analyse it critically, and asking yourself how else you might define it (see also Bacchi, Stone)
• Meltzer and Schwartz and Dunn identify additional ways to contextualise your client’s definition, such as by generating a timeline to help ‘map’ causation or using ‘problem-structuring methods’ to compare definitions and avoid making too many assumptions on a problem’s cause.
• Thissen and Walker compare ‘rational’ and ‘argumentative’ approaches, treating problem definition as something to be measured scientifically or established rhetorically (see also Riker).

These approaches compare with more critical accounts that emphasise the role of power and politics to determine whose knowledge is relevant (L.T.Smith) and whose problem definition counts (Bacchi, Stone). Indeed, Bacchi and Stone provide a crucial bridge between policy analysis and policy studies by reflecting on what policy analysts do and why.

2. What is the policy solution?

In policy studies, it is common to identify counterintuitive or confusing aspects of policy processes, including:

• Few studies suggest that policy responses actually solve problems (and many highlight their potential to exacerbate them). Rather, ‘policy solutions’ is shorthand for proposed or alleged solutions.
• Problem definition often sets the agenda for the production of ‘solutions’, but note the phrase solutions chasing problems (when actors have their ‘pet’ solutions ready, and they seek opportunities to promote them).

Policy studies: problem definition informs the feasibility and success of solutions

Generally speaking, to define the problem is to influence assessments of the feasibility of solutions:

• Technical feasibility. Will they work as intended, given the alleged severity and cause of the problem?
• Political feasibility. Will they receive sufficient support, given the ways in which key policy actors weigh up the costs and benefits of action?

Policy studies highlight the inextricable connection between technical and political feasibility. Put simply, (a) a ‘technocratic’ choice about the ‘optimality’ of a solution is useless without considering who will support its adoption, and (b) some types of solution will always be a hard sell, no matter their alleged effectiveness (Box 2.3 below).

In that context, policy studies ask: what types of policy tools or instruments are actually used, and how does their use contribute to policy change? Measures include the size, substance, speed, and direction of policy change.

In turn, problem definition informs: the ways in which actors will frame any evaluation of policy success, and the policy-relevance of the evidence to evaluate solutions. Simple examples include:

• If you define tobacco in relation to: (a) its economic benefits, or (b) a global public health epidemic, evaluations relate to (a) export and taxation revenues, or (b) reductions in smoking in the population.
• If you define ‘fracking’ in relation to: (a) seeking more benefits than costs, or (b) minimising environmental damage and climate change, evaluations relate to (a) factors such as revenue and effective regulation, or simply (b) how little it takes place.

Policy analysis: recognising and pushing boundaries

Policy analysis texts tend to accommodate these insights when giving advice:

• Bardach recommends identifying solutions that your audience might consider, perhaps providing a range of options on a notional spectrum of acceptability.
• Smith highlights the value of ‘precedent’, or relating potential solutions to previous strategies.
• Weimer and Vining identify the importance of ‘a professional mind-set’ that may be more important than perfecting ‘technical skills’
• Mintrom notes that some solutions are easier to sell than others
• Meltzer and Schwartz describe the benefits of making a preliminary recommendation to inform an iterative process, drawing feedback from clients and stakeholder groups
• Dunn warns against too-narrow forms of ‘evidence based’ analysis which undermine a researcher’s ability to adapt well to the evidence-demands of policymakers
• Thissen and Walker relate solution feasibility to a wide range of policy analysis ‘styles’

Still, note the difference in emphasis.

Policy analysis education/ training may be about developing the technical skills to widen definitions and apply many criteria to compare solutions.

Policy studies suggest that problem definition and a search for solutions takes place in an environment where many actors apply a much narrower lens and are not interested in debates on many possibilities (particularly if they begin with a solution).

I have exaggerated this distinction between each element, but it is worth considering the repeated interaction between them in practice: politics and policymaking provide boundaries for policy analysis, analysis could change those boundaries, and policy studies help us reflect on the impact of analysts.

I’ll take a quick break, then discuss how this conclusion relates to the idea of ‘entrepreneurial’ policy analysis.

Further reading

Understanding Public Policy (2020: 28) describes the difference between governments paying for and actually using the ‘tools of policy formulation’. To explore this point, see ‘The use and non-use of policy appraisal tools in public policy making‘ and The Tools of Policy Formulation.

Policy Analysis in 750 Words: Reflecting on your role as a policy analyst

This post forms one part of the Policy Analysis in 750 words series overview.

One aim of this series is to combine insights from policy research (1000, 500) and policy analysis texts.

If we take key insights from policy theories seriously, we can use them to identify (a) the constraints to policy analytical capacity, and (b) the ways in which analysts might address them. I use the idea of policy analyst archetypes to compare a variety of possible responses.

Key constraints to policy analytical capacity

Terms like ‘bounded rationality’ highlight major limits on the ability of humans and organisations to process information.

• Humans use heuristics or cognitive shortcuts to process enough information to make choices, and institutions are the rules used by organisations to limit information processing.

Terms like policymaking ‘context’, ‘environments’, and multi-centric policymaking suggest that the policy process is beyond the limits of policymaker understanding and control.

• Policy actors need to find ways to act, with incomplete information about the problem they seek to solve and the likely impact of their ‘solution’.
• They gather information to help reduce uncertainty, but problem definition is really about exercising power to reduce ambiguity: select one way to interpret a problem (at the expense of most others), and limit therefore limit the relevance and feasibility of solutions.
• This context informs how actors might use the tools of policy analysis. Key texts in this series highlight the use of tools to establish technical feasibility (will it work as intended?), but policymakers also select tools for their political feasibility (who will support or oppose this measure?).

How might policy analysts address these constraints ethically?

Most policy analysis texts (in this series) consider the role of professional ethics and values during the production of policy analysis. However, they also point out that there is not a clearly defined profession and associated code of conduct (e.g. see Adachi). In that context, let us begin with some questions about the purpose of policy analysis and your potential role:

1. Is your primary role to serve individual clients or some notion of the ‘public good’?
2. Should you maximise your role as an individual or play your part in a wider profession?
3. What is the balance between the potential benefits of individual ‘entrepreneurship’ and collective ‘co-productive’ processes?
4. Which policy analysis techniques should you prioritise?
5. What forms of knowledge and evidence count in policy analysis?
6. What does it mean to communicate policy analysis responsibly?
7. Should you provide a clear recommendation or encourage reflection?

 

Policy analysis archetypes: pragmatists, entrepreneurs, manipulators, storytellers, and decolonisers

In that context, I have created a story of policy analysis archetypes to identify the elements that each text emphasises.

The pragmatic policy analyst

• Bardach provides the classic simple, workable, 8-step system to present policy analysis to policymakers while subject to time and resource-pressed political conditions.
• Dunn also uses Wildavsky’s famous phrase ‘art and craft’ to suggest that scientific and ‘rational’ methods can only take us so far.

The professional, client–oriented policy analyst

• Weimer and Vining provide a similar 7-step client-focused system, but incorporating a greater focus on professional development and economic techniques (such as cost-benefit-analysis) to emphasise a particular form of professional analyst.
• Meltzer and Schwartz also focus on advice to clients, but with a greater emphasis on a wide variety of methods or techniques (including service design) to encourage the co-design of policy analysis with clients.

The communicative policy analyst

•  C. Smith focuses on how to write and communicate policy analysis to clients in a political context.
• Compare with Spiegelhalter and Gigerenzer on how to communicate responsibly when describing uncertainty, probability, and risk.

The manipulative policy analyst.

• Riker helps us understand the relationship between two aspects of agenda setting: the rules/ procedures to make choice, and the framing of policy problems and solutions.

The entrepreneurial policy analyst

• Mintrom shows how to combine insights from studies of policy entrepreneurship and policy analysis, to emphasise the benefits of collaboration and creativity.

The questioning policy analyst

• Bacchi  analyses the wider context in which people give and use such advice, to identify the emancipatory role of analysis and encourage policy analysts to challenge dominant social constructions of problems and populations.

The storytelling policy analyst

• Stone identifies the ways in which people use storytelling and argumentation techniques to define problems and justify solutions. This process is about politics and power, not objectivity and optimal solutions.

The decolonizing policy analyst.

• L.T. Smith does not describe policy analysis directly, but shows how the ‘decolonization of research methods’ can inform the generation and use of knowledge.
• Compare with Hindess on the ways in which knowledge-based hierarchies rely on an untenable, circular logic.
• Compare with Michener’s thread, discussing Doucet’s new essay on (a) the role of power and knowledge in limiting (b) the ways in which we gather evidence to analyse policy problems.

Incisive essay from @bailabomba on studying the use of research evidence through critical perspectives that center the marginalized. There is so too much good stuff in here to summarize via twitter (you should just read it). But let me point out a few things that resonated (1/n) https://t.co/nIahyIjwBo

— Jamila Michener (@povertyscholar) December 17, 2019

Using archetypes to define the problem of policy analysis

Studies of the field (e.g. Radin plus Brans, Geva-May, and Howlett) suggest that there are many ways to do policy analysis. Further, as Thissen and Walker describe, such roles are not mutually exclusive, your views on their relative value could change throughout the process of analysis, and you could perform many of these roles.

Further, each text describes multiple roles, and some seem clustered together:

• pragmatic, client-orientated, and communicative could sum-up the traditional 5-8 step approaches, while
• questioning, storytelling, and decolonizing could sum up an important (‘critical’) challenge to narrow ways of thinking about policy analysis and the use of information.

Still, the emphasis matters.

Each text is setting an agenda or defining the problem of policy analysis more-or-less in relation to these roles. Put simply, the more you are reading about economic theory and method, the less you are reading about dominance and manipulation.

How can you read further?

Michener’s ‘Policy Feedback in a Racialized Polity’ connects to studies of historical institutionalism, and reminds us to use insights from policy theories to identify the context for policy analysis.

I have co-authored a lot about uncertainty/ ambiguity in relation to ‘evidence based policymaking’, including:

• Fostering Evidence-informed Policy Making: Uncertainty Versus Ambiguity(PDF)
• To Bridge the Divide between Evidence and Policy: Reduce Ambiguity as Much as Uncertainty
• How to communicate effectively with policymakers: combine insights from psychology and policy studies

See also The new policy sciences for a discussion of how these issues inform Lasswell’s original vision for the policy sciences (combining the analysis of and for policy).

Policy Analysis in 750 words: Marleen Brans, Iris Geva-May, and Michael Howlett (2017) Routledge Handbook of Comparative Policy Analysis

Please see the Policy Analysis in 750 words series overview before reading the summary (and click here for the full list of authors). This post is a mere 500 words over budget (not including these words describing the number of words).

Marleen Brans, Iris Geva-May, and Michael Howlett (editors) (2017) Routledge Handbook of Comparative Policy Analysis (London: Routledge)

‘The Handbook … covers … the state of the art knowledge about the science, art and craft of policy analysis in different countries, at different levels of government and by all relevant actors in and outside government who contribute to the analysis of problems and the search for policy solutions’ (Brans et al, 2017: 1).

This book focuses on the interaction between (in Lasswell’s terms) ‘analysis for policy’ (policy analysis) and ‘analysis of policy’ (policy process research). In other words,

• what can the study of policy analysis tell us about policymaking, and
• what can studies of policymaking tell budding policy analysts about the nature of their task in relation to their policymaking environment?

Brans et al’s (2017: 1-6) opening discussion suggests that this task is rather unclear and complicated. They highlight the wide range of activity described by the term ‘policy analysis’:

1. The scope of policy analysis is wide, and its meaning unclear

Analysts can be found in many levels and types of government, in bodies holding governments to account, and outside of government, including interest groups, think tanks, and specialist firms (such as global accountancy or management consultancy firms – Saint-Martin, 2017).

Further, ‘what counts’ as policy analysis can relate to the people that do it, the rules they follow, the processes in which they engage, the form of outputs, and the expectations of clients (Veselý, 2017: 103; Vining and Boardman, 2017: 264).

2. The role of a policy analyst varies remarkably in relation to context

It varies over time, policy area, type of government (such as central, subnational, local), country, type of political system (e.g. majoritarian and consensus democracies), and ‘policy style’.

3. Analysis involves ‘science, art and craft’ and the rules are written and unwritten

The process of policy analysis – such as to gather and analyse information, define problems, design and compare solutions, and give policy advice – includes ‘applied social and scientific research as well as more implicit forms of practical knowledge’, and ‘both formal and informal professional practices’ (see also studies of institutions and networks).

4. The policy process is complex.

It is difficult to identify a straightforward process in which analysts are clearly engaged in multiple, well-defined ‘stages’ of policymaking.

5. Key principles and practices can be institutionalised, contested, or non-existent.

The idea of policy analysis principles – ‘of transparency, effectiveness, efficiency and accountability through systematic and evidence-based analysis’ – may be entrenched in places like the US but not globally.

In some political systems (particularly in the ‘Anglo-Saxon family of nations’), the most-described forms of policy analysis (in the 750 words series) may be taken for granted (2017: 4):

• Scholars encourage or criticise 5-step policy analysis techniques.
• Some methods (such as cost-benefit-analysis) are institutionalised and difficult to replace (or even supplement with more inclusive approaches).
• There is a high commitment to the (super-problematic) idea of ‘evidence based policymaking’ (Brans et al, 2017: 5).

Even so, the status of science and expertise is often contested, particularly in relation to salient and polarised issues, or more generally:

• During ‘attempts by elected politicians to restore the primacy of political judgement in the policymaking process, at the expense of technical or scientific evidence’ (2017: 5).
• When the ‘blending of expert policy analysis with public consultation and participation’ makes ‘advice more competitive and contested’ (2017: 5).
• When evidence based really means evidence informed, given that there are many legitimate claims to knowledge, and evidence forms one part of a larger process of policy design (van Nispen and de Jong, 2017: 153).

In many political systems, there may be less criticism of the idea of ‘systematic and evidence-based analysis’ because there less capacity to process information. It is difficult to worry about excessively technocratic approaches if they do not exist (a point that CW made to me just before I read this book).

Implications for policy analysis

1. It is difficult to think of policy analysis as a ‘profession’.

We may wonder if ‘policy analysis’ can ever be based on common skills and methods (such as described by Scott, 2017, and in Weimer and Vining), connected to ‘formal education and training’, a ‘a code of professional conduct’, and the ability of organisations to control membership (Adachi, 2017: 28; compare with Radin and Geva-May).

2. Policy analysis is a loosely-defined collection of practices that vary according to context.

Policy analysis may, instead, be considered a collection of ‘styles’ (Hassenteufel and Zittoun, 2017), influenced by:

• competing analytical approaches in different political systems (2017: 65)
• bureaucratic capacity for analysis (Mendez and Dussauge-Laguna, 2017: 82)
• a relative tendency to contract out analysis (Veselý, 2017: 113)
• the types and remits of advisory bodies (e.g. are they tasked simply with offering expert advice, or also to encourage wider participation to generate knowledge?) (Crowley and Head, 2017)
• the level of government in which analysts work, such as ‘subnational’ (Newman, 2017) or ‘local’ (Lundin and Öberg, 2017)
• the type of activity, such as when (‘performance’) budgeting analysis is influenced heavily by economic methods and ‘new public management’ reforms (albeit with limited success, followed by attempts at reform) (van Nispen and de Jong, 2017: 143-52)

Policy analysis can also describe a remarkably wide range of activity, including:

• Public inquiries (Marier, 2017)
• Advice to MPs, parliaments, and their committees (Wolfs and De Winter, 2017)
• The strategic analysis of public opinion or social media data (Rothmayr Allison, 2017; Kuo and Cheng, 2017)
• A diverse set of activities associated with ‘think tanks’ (Stone and Ladi, 2017) and ‘political party think tanks’ (Pattyn et al, 2017)
• Analysis for and by ‘business associations’ (Vining and Boardman, 2017), unions (Schulze and Schroeder, 2017), and voluntary/ non-profit organisations (Evans et al, 2017), all of whom juggle policy advice to government with keeping members on board.
• The more-or-less policy relevant work of academic researchers (Blum and Brans, 2017; compare with Dunn and see the EBPM page).

3. The analysis of and for policy is not so easy to separate in practice.

When defining policy analysis largely as a collection of highly-variable practices, in complex policymaking systems, we can see the symbiotic relationship between policy analysis and policy research. Studying policy analysis allows us to generate knowledge of policy processes. Policy process research demonstrates that the policymaking context influences how we think about policy analysis.

4. Policy analysis education and training is incomplete without policy process research

Put simply, we should not assume that graduates in ‘policy analysis’ will enter a central government with high capacity, coherent expectations, and a clear demand for the same basic skills. Yet, Fukuyama argues that US University programmes largely teach students:

‘a battery of quantitative methods … applied econometrics, cost-benefit analysis, decision analysis, and, most recently, use of randomized experiments for program evaluation’ that ‘will tell you what the optimal policy should be’, but not ‘how to achieve that outcome. The world is littered with optimal policies that don’t have a snowball’s chance in hell of being adopted’.

In that context, additional necessary skills include: stakeholder mapping, to identify who is crucial to policy success, defining policy problems in a way that stakeholders and policymakers can support, and including those actors continuously during a process of policy design and delivery. These skills are described at more length by Radin and Geva May, while Botha et al (2017) suggest that the policy analysis programmes (across North American and European Universities) offer a more diverse range of skills (and support for experiential learning) than Fukuyama describes.