Daily Archives: December 18, 2019

Policy Analysis in 750 Words: Who should be involved in the process of policy analysis?

This post forms one part of the Policy Analysis in 750 words series overview.

Think of two visions for policy analysis. It should be primarily:

These choices are not mutually exclusive, but there are key tensions between them that should not be ignored, such as when we ask:

  • how many people should be involved in policy analysis?
  • whose knowledge counts?
  • who should control policy design?

Perhaps we can only produce a sensible combination of the two if we clarify their often very different implications for policy analysis. Let’s begin with one story for each and see where they take us.

A story of ‘evidence-based policymaking’

One story of ‘evidence based’ policy analysis is that it should be based on the best available evidence of ‘what works’.

Often, the description of the ‘best’ evidence relates to the idea that there is a notional hierarchy of evidence according to the research methods used.

At the top would be the systematic review of randomised control trials, and nearer the bottom would be expertise, practitioner knowledge, and stakeholder feedback.

This kind of hierarchy has major implications for policy learning and transfer, such as when importing policy interventions from abroad or ‘scaling up’ domestic projects.

Put simply, the experimental method is designed to identify the causal effect of a very narrowly defined policy intervention. Its importation or scaling up would be akin to the description of medicine, in which the evidence suggests the causal effect of a specific active ingredient to be administered with the correct dosage. A very strong commitment to a uniform model precludes the processes we might associate with co-production, in which many voices contribute to a policy design to suit a specific context (see also: the intersection between evidence and policy transfer).

A story of co-production in policymaking

One story of ‘co-produced’ policy analysis is that it should be ‘reflexive’ and based on respectful conversations between a wide range of policymakers and citizens.

Often, the description is of the diversity of valuable policy relevant information, with scientific evidence considered alongside community voices and normative values.

This rejection of a hierarchy of evidence also has major implications for policy learning and transfer. Put simply, a co-production method is designed to identify the positive effect – widespread ‘ownership’ of the problem and commitment to a commonly-agreed solution – of a well-discussed intervention, often in the absence of central government control.

Its use would be akin to a collaborative governance mechanism, in which the causal mechanism is perhaps the process used to foster agreement (including to produce the rules of collective action and the evaluation of success) rather than the intervention itself. A very strong commitment to this process precludes the adoption of a uniform model that we might associate with narrowly-defined stories of evidence based policymaking.

Where can you find these stories in the 750-words series?

  1. Texts focusing on policy analysis as evidence-based/ informed practice (albeit subject to limits) include: Weimer and Vining, Meltzer and Schwartz, Brans, Geva-May, and Howlett (compare with Mintrom, Dunn)
  2. Texts on being careful while gathering and analysing evidence include: Spiegelhalter
  3. Texts that challenge the ‘evidence based’ story include: Bacchi, T. Smith, Hindess, Stone

 

How can you read further?

See the EBPM page and special series ‘The politics of evidence-based policymaking: maximising the use of evidence in policy

There are 101 approaches to co-production, but let’s see if we can get away with two categories:

  1. Co-producing policy (policymakers, analysts, stakeholders). Some key principles can be found in Ostrom’s work and studies of collaborative governance.
  2. Co-producing research to help make it more policy-relevant (academics, stakeholders). See the Social Policy and Administration special issue ‘Inside Co-production’ and Oliver et al’s ‘The dark side of coproduction’ to get started.

To compare ‘epistemic’ and ‘reflexive’ forms of learning, see Dunlop and Radaelli’s ‘The lessons of policy learning: types, triggers, hindrances and pathologies

My interest has been to understand how governments juggle competing demands, such as to (a) centralise and localise policymaking, (b) encourage uniform and tailored solutions, and (c) embrace and reject a hierarchy of evidence. What could possibly go wrong when they entertain contradictory objectives? For example:

  • Paul Cairney (2019) “The myth of ‘evidence based policymaking’ in a decentred state”, forthcoming in Public Policy and Administration(Special Issue, The Decentred State) (accepted version)
  • Paul Cairney (2019) ‘The UK government’s imaginative use of evidence to make policy’, British Politics, 14, 1, 1-22 Open AccessPDF
  • Paul Cairney and Kathryn Oliver (2017) ‘Evidence-based policymaking is not like evidence-based medicine, so how far should you go to bridge the divide between evidence and policy?’ Health Research Policy and Systems (HARPS), DOI: 10.1186/s12961-017-0192-x PDF
  • Paul Cairney (2017) “Evidence-based best practice is more political than it looks: a case study of the ‘Scottish Approach’”, Evidence and Policy, 13, 3, 499-515 PDF

 

4 Comments

Filed under 750 word policy analysis, Evidence Based Policymaking (EBPM), public policy

Policy Analysis in 750 words: the old page

I am redesigning the 750 words page. This is the old version.

The posts in this new series summarise key texts in policy analysis. They present the most common advice about how to ‘do’ policy analysis (to identify a policy problem and possible solutions) and situate this advice within the study of politics, power, and public policy.

This combination of ‘how to’ advice and ‘what actually happens’ research allows you to produce policy analyses and reflect on the political and pragmatic choices you need to make. Policy analysis is not a ‘rational’ or ‘technocratic’ process and we should not pretend otherwise. Rather, our aim in this series is to understand policy analysis through the lens of the policy theories that highlight:

  • a competition to frame problems and identify the technical and political feasibility of solutions; in
  • a policymaking environment over which no one has full understanding or control (even if elected policymakers need to project their control to boost their image of governing competence), during which
  • governments add new policy solutions to an existing, complex, mix of solutions (rather than working from a blank canvas).

In that context, you may find that the summaries make more sense with reference to four main themes from the 1000 words and 500 word policy theories series (and this draft document):

  1. How do you define a policy problem, and what types of solutions are available?
  2. How does your chosen solution relate to existing policies?
  3. Who is your audience, and what can you realistically expect them to do?
  4. Who should be involved in the process of policy analysis?

I then add theme (5): Reflecting on your role as a policy analyst.

In each case, I prompt you to reflect on how (a) your knowledge of politics and policy processes, informs (b) the strategies you adopt when constructing policy analysis. The following description is a long read, but I think it provides essential context that could make the difference between effective and ineffective analysis. My MPP students can also note that this page is the same number of words as the policy analysis/ reflection exercise.

  1. How do you define a policy problem, and what types of solutions are available?

The classic introduction to policy analysis is to ask what is policy? and explore the types of policy tools or instruments that may be used to produce policy change.

Modern discussions also incorporate insights from psychology to understand (a) how policymakers might combine cognition and emotion to reduce uncertainty and ambiguity, to understand problems, and therefore (b) how to communicate effectively when presenting policy analysis. This process is about the power to reduce ambiguity rather than simply the provision of information to reduce uncertainty.

In turn, problem definition influences assessments of the – technical and political -feasibility of solutions, and the ways in which actors will frame any evaluation of policy success.

      2. How does your chosen solution relate to existing policies?

Although you may propose the adoption of one (or more) policy instrument, it will likely add to many others. Governments already combine a large number of instruments to make policy, including legislation, expenditure, economic incentives and penalties, education, and various forms of service delivery. Those instruments combine to represent a complex policy mix whose overall effects are not simple to predict. This interaction provides essential context, particularly if you are asked to provide, say, a simple logic model or ‘theory of change’ to describe the likely impact of your new solution.

      3. Who is your audience, and what can you realistically expect them to do?

For example, imagine writing policy analysis in the ideal-type world of a single powerful ‘comprehensively rational’ policymaker at the heart of an orderly policy cycle. Your analysis would be relatively simple, and you would not need to worry about what happens after you make a recommendation for policy change. You could focus on widely-used tools such as cost-benefit analysis and know where the results would feed into the policy process. I have perhaps over-egged this ideal-type pudding, but I think a lot of traditional policy analysis buys into this basic idea and focuses primarily on the science of analysis rather than the political policymaking context in which it takes place.

Then imagine a far messier and less predictable world in which the nature of the policy issue is highly contestedresponsibility for policy is unclear, and no single ‘centre’ has the power to turn a recommendation into an outcome. This image is a key feature of policy process theories, which describe:

  • Many policymakers and influencers spread across many levels and types of government (as the venues in which authoritative choice takes place). Consequently, it is not a straightforward task to identify and know your audience, particularly if the problem you seek to solve requires a combination of policy instruments controlled by different actors.
  • Each venue resembles an institution driven by formal and informal rules. Formal rules are written-down or widely-known. Informal rules are unwritten, difficult to understand, and may not even be understood in the same way by participants. Consequently, it is difficult to know if your solution will be a good fit with the standard operating procedures of organisations (and therefore if it is politically feasible or too challenging).
  • Policymakers and influencers operate in ‘subsystems’, forming networks built on resources such as trust or coalitions based on shared beliefs. Effective policy analysis may require you to engage with – or become part of – such networks, to allow you to understand the unwritten rules of the game and encourage your audience to trust the messenger. In some cases, the rules relate to your willingness to accept current losses for future gains, to accept the limited impact of your analysis now in the hope of acceptance at the next opportunity.
  • Actors relate their analysis to shared understandings of the world – how it is, and how it should be – which are often so well-established as to be taken for granted. Common terms include paradigms, hegemons, core beliefs, and monopolies of understandings. These dominant frames of reference give meaning to your policy solution. They prompt you to couch your solutions in terms of, for example, a strong attachment to evidence-based cases in public health, value for money in treasury departments, or with regard to core principles such as liberalism or socialism in different political systems.
  • Your solutions relate to socioeconomic context and the events that seem (a) impossible to ignore and (b) out of the control of policymakers. Such factors range from a political system’s geography, demography, social attitudes, economy, while events can be routine elections or unexpected crises. To some extent, you could see yourself as a policy entrepreneur and treat events opportunistically, as ‘windows of opportunity’ to encourage new solutions. Alternatively, a general awareness of the unpredictability of events can prompt you to be modest in your claims, since the policymaking environment may be more important to outcomes than your favoured policy.

What would you recommend under these conditions?  The terms of your cost-benefit analysis would be contested (at least until there is agreement on what the problem is, and how you would measure the success of a solution). Further, even the most sophisticated ‘evidence based’ analysis of a policy problem will fall flat if uninformed by good analysis of the policy process.

Note that these problems amplify the limitations to policy analysis that are more likely to be described in this series. For example,  Weimer and Vining invest about 200 pages in economic analyses of markets and government, often highlighting a gap between (a) our ability to model and predict economic and social behaviour, and (b) what actually happens when governments intervene. To this, we should add the gap between policymakers’ formal responsibilities versus actual control of policy processes and outcomes.

      4. Who should be involved in the process of policy analysis?

Think of two visions for policy analysis. It should be primarily (1) ‘evidence based’ or (2) ‘coproduced’. While these choices are not mutually exclusive, there are key tensions between them that should not be ignored, such as when we ask:

  • how many people should be involved in policy analysis?
  • whose knowledge counts?
  • who should control policy design?

Perhaps we can only produce a sensible combination of the two if we clarify their often very different implications for policy analysis. Let’s begin with one story for each (see also Approach 1 versus Approach 2) and see where they take us.

One story of ‘evidence based’ policy analysis is that it should be based on the best available evidence of ‘what works’. Often, the description of the ‘best’ evidence relates to the idea that there is a notional hierarchy of evidence (according to the research methods used). At the top would be the systematic review of randomised control trials, and nearer the bottom would be expertise, practitioner knowledge, and stakeholder feedback. This kind of hierarchy has major implications for policy learning and transfer, such as when importing policy interventions from abroad or ‘scaling up’ domestic projects. Put simply, the experimental method is designed to identify the causal effect of a very narrowly defined policy intervention. Its importation or scaling up would be akin to the description of medicine, in which the evidence suggests the causal effect of a specific active ingredient to be administered with the correct dosage. A very strong commitment to a uniform model precludes the processes we might associate with co-production, in which many voices contribute to a policy design to suit a specific context.

One story of ‘co-produced’ policy analysis is that it should be ‘reflexive’ and based on respectful conversations between a wide range of policymakers and citizens. Often, the description is of the diversity of valuable policy relevant information, with scientific evidence considered alongside community voices and normative values. This rejection of a hierarchy of evidence also has major implications for policy learning and transfer. Put simply, a co-production method is designed to identify the positive effect – widespread ‘ownership’ of the problem and commitment to a commonly-agreed solution – of a well-discussed intervention, often in the absence of central government control. Its use would be akin to a collaborative governance mechanism, in which the causal mechanism is perhaps the process used to foster agreement (including to produce the rules of collective action and the evaluation of success) rather than the intervention itself. A very strong commitment to this process precludes the adoption of a uniform model that we might associate with narrowly-defined stories of evidence based policymaking.

      5. Reflecting on your role as a policy analyst

If we take insights from policy theories seriously, we need to incorporate them into policy analysis, to consider policymaker psychology and policymaking context alongside the tools of policy analysis. We also need to consider the role of societal and professional values during the production of policy analysis. In other words, consider the limits to your influence and the ethics of your task.

In that context, I have begun to create a story of policy analysis archetypes to help explain this point in context:

  • The pragmatic policy analystBardach provides a simple, workable 8-step system to present policy analysis to policymakers while subject to time and resource-pressed political conditions (and Dunnis pragmatic for other reasons).
  • The professional, clientoriented policy analystWeimer and Vining provide a similar 7-step client-focused system, but incorporating a greater focus on professional development and economic techniques (such as cost-benefit-analysis) to foster efficiency.
  • The communicative policy analystCatherine Smith focuses on how to write and communicate policy analysis to clients in a political context.
  • The entrepreneurial policy analystMintrom shows how to incorporate insights from studies of policy entrepreneurship.
  • The questioning policy analystBacchi’s aim is to analyse the process in which people give and use such advice and to encourage policy analysts to question their role.
  • The storytelling policy analyst. Stone’s aim is to identify the ways in which people use storytelling and argumentation techniques to define problems and justify solutions.
  • The decolonizing policy analystLinda Tuhiwai Smith does not describe policy analysis directly, but shows how the ‘decolonization of research methods’ informs new approaches to policymaking.
  • The manipulative policy analyst. A discussion of Riker helps us understand the relationship between two aspects of agenda setting: the rules/ procedures to make choice, and the framing of policy problems and solutions.

These descriptions allow you to reflect on your role, as part of a wider political or policymaking system:

  1. Is your primary role to serve and communicate to clients? Is it to get what you want?
  2. Are you primarily seeking to maximise your role as an individual or your part in a wider profession?
  3. What is the balance between the potential benefits of individual ‘entrepreneurship’ and collective ‘co-productive’ processes?
  4. Which policy analysis techniques and forms of knowledge do you prioritise?

The initial list of texts

My aim is to summarise the texts below (based initially on a module guide by Dr Raul Pacheco-Vega and any further suggestions you may have) and incorporate this analysis into the draft document How to write theory-driven policy analysis. See also Writing a Policy Paper. The reference has a weblink when a summary is available.

Eugene Bardach (2012) A Practical Guide for Policy Analysis 5th ed. (CQ Press) (see also A step-by-step policy analysis using Bardach’s Eight Step Model)

Carol Bacchi (2009) Analysing Policy: What’s the problem represented to be? (NSW: Pearson Australia)

Eugene Bardach and Erik Patashnik (2015) A Practical Guide for Policy Analysis 5th ed. (International Edition) (CQ Press)

Catherine Smith (2015) Writing Public Policy: A Practical Guide to Communicating in the Policy Making Process (Oxford: Oxford University Press)

David Weimer and Aidan Vining (2017) Policy Analysis: Concepts and Practice, 6th Edition (London: Routledge)

Michael Mintrom (2012) Contemporary Policy Analysis (Oxford: Oxford University Press) (see also Contemporary Policy Analysis (Mintrom 2012))

Dunn, W. (2017) Public Policy Analysis 6th Ed. (London: Routledge)

Geva-May, I. (2005) ‘Thinking Like a Policy Analyst. Policy Analysis as a Clinical Profession’, in Geva-May, I. (ed) Thinking Like a Policy Analyst. Policy Analysis as a Clinical Profession (Basingstoke: Palgrave) (scroll down after Radin) (see also Policy analysis as a clinical profession)

Meltzer, R. and Schwartz, A. (2019) Policy Analysis as Problem Solving (London: Routledge)

Radin, B. (2019) Policy Analysis in the Twenty-First Century (London: Routledge)

Marleen Brans, Iris Geva-May, and Michael Howlett (2017) Routledge Handbook of Comparative Policy Analysis

Thissen, W. and Walker, W. (Eds.). (2013) Public Policy Analysis (London: Springer)

Please let me know if you see some weird omissions from the list. They should give advice about policy analysis rather than describe the policy process (the latter are covered in the 1000 words and 500 word policy theories series). That said, these are some of the books that I use to widen-out the definition of policy analysis books:

William H. Riker (1986) The Art of Political Manipulation (New Haven: Yale University Press)

Linda Tuhiwai Smith (2012) Decolonizing Methodologies (London: Zed Books) (also discusses Lorde, Santos)

Barry Hindess (1977) Philosophy and Methodology in the Social Sciences

Policy Analysis in 750 words: Using Statistics and Explaining Risk (Sincerely)

Policy Analysis in 750 words: Deborah Stone (2012) Policy Paradox

Leave a comment

Filed under 750 word policy analysis, Uncategorized

Policy Analysis in 750 words: Marleen Brans, Iris Geva-May, and Michael Howlett (2017) Routledge Handbook of Comparative Policy Analysis

Please see the Policy Analysis in 750 words series overview before reading the summary (and click here for the full list of authors). This post is a mere 500 words over budget (not including these words describing the number of words).

Brans et al 2017 cover

Marleen Brans, Iris Geva-May, and Michael Howlett (editors) (2017) Routledge Handbook of Comparative Policy Analysis (London: Routledge)

The Handbook … covers … the state of the art knowledge about the science, art and craft of policy analysis in different countries, at different levels of government and by all relevant actors in and outside government who contribute to the analysis of problems and the search for policy solutions’ (Brans et al, 2017: 1).

This book focuses on the interaction between (in Lasswell’s terms) ‘analysis for policy’ (policy analysis) and ‘analysis of policy’ (policy process research). In other words,

  • what can the study of policy analysis tell us about policymaking, and
  • what can studies of policymaking tell budding policy analysts about the nature of their task in relation to their policymaking environment?

Brans et al’s (2017: 1-6) opening discussion suggests that this task is rather unclear and complicated. They highlight the wide range of activity described by the term ‘policy analysis’:

  1. The scope of policy analysis is wide, and its meaning unclear

Analysts can be found in many levels and types of government, in bodies holding governments to account, and outside of government, including interest groups, think tanks, and specialist firms (such as global accountancy or management consultancy firms – Saint-Martin, 2017).

Further, ‘what counts’ as policy analysis can relate to the people that do it, the rules they follow, the processes in which they engage, the form of outputs, and the expectations of clients (Veselý, 2017: 103; Vining and Boardman, 2017: 264).

  1. The role of a policy analyst varies remarkably in relation to context

It varies over time, policy area, type of government (such as central, subnational, local), country, type of political system (e.g. majoritarian and consensus democracies), and ‘policy style’.

  1. Analysis involves ‘science, art and craft’ and the rules are written and unwritten

The process of policy analysis – such as to gather and analyse information, define problems, design and compare solutions, and give policy advice – includes ‘applied social and scientific research as well as more implicit forms of practical knowledge’, and ‘both formal and informal professional practices’ (see also studies of institutions and networks).

  1. The policy process is complex.

It is difficult to identify a straightforward process in which analysts are clearly engaged in multiple, well-defined ‘stages’ of policymaking.

  1. Key principles and practices can be institutionalised, contested, or non-existent.

The idea of policy analysis principles – ‘of transparency, effectiveness, efficiency and accountability through systematic and evidence-based analysis’ – may be entrenched in places like the US but not globally.

In some political systems (particularly in the ‘Anglo-Saxon family of nations’), the most-described forms of policy analysis (in the 750 words series) may be taken for granted (2017: 4):

Even so, the status of science and expertise is often contested, particularly in relation to salient and polarised issues, or more generally:

  • During ‘attempts by elected politicians to restore the primacy of political judgement in the policymaking process, at the expense of technical or scientific evidence’ (2017: 5).
  • When the ‘blending of expert policy analysis with public consultation and participation’ makes ‘advice more competitive and contested’ (2017: 5).
  • When evidence based really means evidence informed, given that there are many legitimate claims to knowledge, and evidence forms one part of a larger process of policy design (van Nispen and de Jong, 2017: 153).

In many political systems, there may be less criticism of the idea of ‘systematic and evidence-based analysis’ because there less capacity to process information. It is difficult to worry about excessively technocratic approaches if they do not exist (a point that CW made to me just before I read this book).

Implications for policy analysis

  1. It is difficult to think of policy analysis as a ‘profession’.

We may wonder if ‘policy analysis’ can ever be based on common skills and methods (such as described by Scott, 2017, and in Weimer and Vining), connected to ‘formal education and training’, a ‘a code of professional conduct’, and the ability of organisations to control membership (Adachi, 2017: 28; compare with Radin and Geva-May).

  1. Policy analysis is a loosely-defined collection of practices that vary according to context.

Policy analysis may, instead, be considered a collection of ‘styles’ (Hassenteufel and Zittoun, 2017), influenced by:

  • competing analytical approaches in different political systems (2017: 65)
  • bureaucratic capacity for analysis (Mendez and Dussauge-Laguna, 2017: 82)
  • a relative tendency to contract out analysis (Veselý, 2017: 113)
  • the types and remits of advisory bodies (e.g. are they tasked simply with offering expert advice, or also to encourage wider participation to generate knowledge?) (Crowley and Head, 2017)
  • the level of government in which analysts work, such as ‘subnational’ (Newman, 2017) or ‘local’ (Lundin and Öberg, 2017)
  • the type of activity, such as when (‘performance’) budgeting analysis is influenced heavily by economic methods and ‘new public management’ reforms (albeit with limited success, followed by attempts at reform) (van Nispen and de Jong, 2017: 143-52)

Policy analysis can also describe a remarkably wide range of activity, including:

  • Public inquiries (Marier, 2017)
  • Advice to MPs, parliaments, and their committees (Wolfs and De Winter, 2017)
  • The strategic analysis of public opinion or social media data (Rothmayr Allison, 2017; Kuo and Cheng, 2017)
  • A diverse set of activities associated with ‘think tanks’ (Stone and Ladi, 2017) and ‘political party think tanks’ (Pattyn et al, 2017)
  • Analysis for and by ‘business associations’ (Vining and Boardman, 2017), unions (Schulze and Schroeder, 2017), and voluntary/ non-profit organisations (Evans et al, 2017), all of whom juggle policy advice to government with keeping members on board.
  • The more-or-less policy relevant work of academic researchers (Blum and Brans, 2017; compare with Dunn and see the EBPM page).
  1. The analysis of and for policy is not so easy to separate in practice.

When defining policy analysis largely as a collection of highly-variable practices, in complex policymaking systems, we can see the symbiotic relationship between policy analysis and policy research. Studying policy analysis allows us to generate knowledge of policy processes. Policy process research demonstrates that the policymaking context influences how we think about policy analysis.

  1. Policy analysis education and training is incomplete without policy process research

Put simply, we should not assume that graduates in ‘policy analysis’ will enter a central government with high capacity, coherent expectations, and a clear demand for the same basic skills. Yet, Fukuyama argues that US University programmes largely teach students:

a battery of quantitative methods … applied econometrics, cost-benefit analysis, decision analysis, and, most recently, use of randomized experiments for program evaluation’ that ‘will tell you what the optimal policy should be’, but not ‘how to achieve that outcome. The world is littered with optimal policies that don’t have a snowball’s chance in hell of being adopted’.

In that context, additional necessary skills include: stakeholder mapping, to identify who is crucial to policy success, defining policy problems in a way that stakeholders and policymakers can support, and including those actors continuously during a process of policy design and delivery. These skills are described at more length by Radin and Geva May, while Botha et al (2017) suggest that the policy analysis programmes (across North American and European Universities) offer a more diverse range of skills (and support for experiential learning) than Fukuyama describes.

6 Comments

Filed under 750 word policy analysis, Evidence Based Policymaking (EBPM), public policy