Tag Archives: facts don’t speak for themselves

Policy Analysis in 750 words: William Dunn (2017) Public Policy Analysis

Please see the Policy Analysis in 750 words series overview before reading the summary. This book is a whopper, with almost 500 pages and 101 (excellent) discussions of methods, so 800 words over budget seems OK to me. If you disagree, just read every second word.  By the time you reach the cat hanging in there baby you are about 300 (150) words away from the end.

Dunn 2017 cover

William Dunn (2017) Public Policy Analysis 6th Ed. (Routledge)

Policy analysis is a process of multidisciplinary inquiry aiming at the creation, critical assessment, and communication of policy-relevant knowledge … to solve practical problemsIts practitioners are free to choose among a range of scientific methods, qualitative as well as quantitative, and philosophies of science, so long as these yield reliable knowledge’ (Dunn, 2017: 2-3).

Dunn (2017: 4) describes policy analysis as pragmatic and eclectic. It involves synthesising policy relevant (‘usable’) knowledge, and combining it with experience and ‘practical wisdom’, to help solve problems with analysis that people can trust.

This exercise is ‘descriptive’, to define problems, and ‘normative’, to decide how the world should be and how solutions get us there (as opposed to policy studies/ research seeking primarily to explain what happens).

Dunn contrasts the ‘art and craft’ of policy analysts with other practices, including:

  1. The idea of ‘best practice’ characterised by 5-step plans.
  • In practice, analysis is influenced by: the cognitive shortcuts that analysts use to gather information; the role they perform in an organisation; the time constraints and incentive structures in organisations and political systems; the expectations and standards of their profession; and, the need to work with teams consisting of many professions/ disciplines (2017: 15-6)
  • The cost (in terms of time and resources) of conducting multiple research and analytical methods is high, and highly constrained in political environments (2017: 17-8; compare with Lindblom)
  1. The too-narrow idea of evidence-based policymaking
  • The naïve attachment to ‘facts speak for themselves’ or ‘knowledge for its own sake’ undermines a researcher’s ability to adapt well to the evidence-demands of policymakers (2017: 68; 4 compare with Why don’t policymakers listen to your evidence?).

To produce ‘policy-relevant knowledge’ requires us to ask five questions before (Qs1-3) and after (Qs4-5) policy intervention (2017: 5-7; 54-6):

  1. What is the policy problem to be solved?
  • For example, identify its severity, urgency, cause, and our ability to solve it.
  • Don’t define the wrong problem, such as by oversimplifying or defining it with insufficient knowledge.
  • Key aspects of problems including ‘interdependency’ (each problem is inseparable from a host of others, and all problems may be greater than the sum of their parts), ‘subjectivity’ and ‘artificiality’ (people define problems), ‘instability’ (problems change rather than being solved), and ‘hierarchy’ (which level or type of government is responsible) (2017: 70; 75).
  • Problems vary in terms of how many relevant policymakers are involved, how many solutions are on the agenda, the level of value conflict, and the unpredictability of outcomes (high levels suggest ‘wicked’ problems, and low levels ‘tame’) (2017: 75)
  • ‘Problem-structuring methods’ are crucial, to: compare ways to define or interpret a problem, and ward against making too many assumptions about its nature and cause; produce models of cause-and-effect; and make a problem seem solve-able, such as by placing boundaries on its coverage. These methods foster creativity, which is useful when issues seem new and ambiguous, or new solutions are in demand (2017: 54; 69; 77; 81-107).
  • Problem definition draws on evidence, but is primarily the exercise of power to reduce ambiguity through argumentation, such as when defining poverty as the fault of the poor, the elite, the government, or social structures (2017: 79; see Stone).
  1. What effect will each potential policy solution have?
  • Many ‘forecasting’ methods can help provide ‘plausible’ predictions about the future effects of current/ alternative policies (Chapter 4 contains a huge number of methods).
  • ‘Creativity, insight, and the use of tacit knowledge’ may also be helpful (2017: 55).
  • However, even the most-effective expert/ theory-based methods to extrapolate from the past are flawed, and it is important to communicate levels of uncertainty (2017: 118-23; see Spiegelhalter).
  1. Which solutions should we choose, and why?
  • ‘Prescription’ methods help provide a consistent way to compare each potential solution, in terms of its feasibility and predicted outcome, rather than decide too quickly that one is superior (2017: 55; 190-2; 220-42).
  • They help to combine (a) an estimate of each policy alternative’s outcome with (b) a normative assessment.
  • Normative assessments are based on values such as ‘equality, efficiency, security, democracy, enlightenment’ and beliefs about the preferable balance between state, communal, and market/ individual solutions (2017: 6; 205 see Weimer & Vining, Meltzer & Schwartz, and Stone on the meaning of these values).
  • For example, cost benefit analysis (CBA) is an established – but problematic – economics method based on finding one metric – such as a $ value – to predict and compare outcomes (2017: 209-17; compare Weimer & Vining, Meltzer & Schwartz, and Stone)
  • Cost effectiveness analysis uses a $ value for costs, but compared with other units of measurement for benefits (such as outputs per $) (2017: 217-9)
  • Although such methods help us combine information and values to compare choices, note the inescapable role of power to decide whose values (and which outcomes, affecting whom) matter (2017: 204)
  1. What were the policy outcomes?
  • ‘Monitoring’ methods help identify (say): levels of compliance with regulations, if resources and services reach ‘target groups’, if money is spent correctly (such as on clearly defined ‘inputs’ such as public sector wages), and if we can make a causal link between the policy inputs/ activities/ outputs and outcomes (2017: 56; 251-5)
  • Monitoring is crucial because it is so difficult to predict policy success, and unintended consequences are almost inevitable (2017: 250).
  • However, the data gathered are usually no more than proxy indicators of outcomes. Further, the choice of indicators reflect what is available, ‘particular social values’, and ‘the political biases of analysts’ (2017: 262)
  • The idea of ‘evidence based policy’ is linked strongly to the use of experiments and systematic review to identify causality (2017: 273-6; compare with trial-and-error learning in Gigerenzer, complexity theory, and Lindblom).
  1. Did the policy solution work as intended? Did it improve policy outcomes?
  • Although we frame policy interventions as ‘solutions’, few problems are ‘solved’. Instead, try to measure the outcomes and the contribution of your solution, and note that evaluations of success and ‘improvement’ are contested (2017: 57; 332-41).  
  • Policy evaluation is not an objective process in which we can separate facts from values.
  • Rather, values and beliefs are part of the criteria we use to gauge success (and even their meaning is contested – 2017: 322-32).
  • We can gather facts about the policy process, and the impacts of policy on people, but this information has little meaning until we decide whose experiences matter.

Overall, the idea of ‘ex ante’ (forecasting) policy analysis is a little misleading, since policymaking is continuous, and evaluations of past choices inform current choices.

Policy analysis methods are ‘interdependent’, and ‘knowledge transformations’ describes the impact of knowledge regarding one question on the other four (2017: 7-13; contrast with Meltzer & Schwartz, Thissen & Walker).

Developing arguments and communicating effectively

Dunn (2017: 19-21; 348-54; 392) argues that ‘policy argumentation’ and the ‘communication of policy-relevant knowledge’ are central to policymaking’ (See Chapter 9 and Appendices 1-4 for advice on how to write briefs, memos, and executive summaries and prepare oral testimony).

He identifies seven elements of a ‘policy argument’ (2017: 19-21; 348-54), including:

  • The claim itself, such as a description (size, cause) or evaluation (importance, urgency) of a problem, and prescription of a solution
  • The things that support it (including reasoning, knowledge, authority)
  • Incorporating the things that could undermine it (including any ‘qualifier’, the communication of uncertainty about current knowledge, and counter-arguments).

The key stages of communication (2017: 392-7; 405; 432) include:

  1. ‘Analysis’, focusing on ‘technical quality’ (of the information and methods used to gather it), meeting client expectations, challenging the ‘status quo’, albeit while dealing with ‘political and organizational constraints’ and suggesting something that can actually be done.
  2. ‘Documentation’, focusing on synthesising information from many sources, organising it into a coherent argument, translating from jargon or a technical language, simplifying, summarising, and producing user-friendly visuals.
  3. ‘Utilization’, by making sure that (a) communications are tailored to the audience (its size, existing knowledge of policy and methods, attitude to analysts, and openness to challenge), and (b) the process is ‘interactive’ to help analysts and their audiences learn from each other.

 

hang-in-there-baby

 

Policy analysis and policy theory: systems thinking, evidence based policymaking, and policy cycles

Dunn (2017: 31-40) situates this discussion within a brief history of policy analysis, which culminated in new ways to express old ambitions, such as to:

  1. Use ‘systems thinking’, to understand the interdependence between many elements in complex policymaking systems (see also socio-technical and socio-ecological systems).
  • Note the huge difference between (a) policy analysis discussions of ‘systems thinking’ built on the hope that if we can understand them we can direct them, and (b) policy theory discussions that emphasise ‘emergence’ in the absence of central control (and presence of multi-centric policymaking).
  • Also note that Dunn (2017: 73) describes policy problems – rather than policymaking – as complex systems. I’ll write another post (short, I promise) on the many different (and confusing) ways to use the language of complexity.
  1. Promote ‘evidence based policy, as the new way to describe an old desire for ‘technocratic’ policymaking that accentuates scientific evidence and downplays politics and values (see also 2017: 60-4).

In that context, see Dunn’s (47-52) discussion of comprehensive versus bounded rationality:

  • Note the idea of ‘erotetic rationality’ in which people deal with their lack of knowledge of a complex world by giving up on the idea of certainty (accepting their ‘ignorance’), in favour of a continuous process of ‘questioning and answering’.
  • This approach is a pragmatic response to the lack of order and predictability of policymaking systems, which limits the effectiveness of a rigid attachment to ‘rational’ 5 step policy analyses (compare with Meltzer & Schwartz).

Dunn (2017: 41-7) also provides an unusually useful discussion of the policy cycle. Rather than seeing it as a mythical series of orderly stages, Dunn highlights:

  1. Lasswell’s original discussion of policymaking functions (or functional requirements of policy analysis, not actual stages to observe), including: ‘intelligence’ (gathering knowledge), ‘promotion’ (persuasion and argumentation while defining problems), ‘prescription’, ‘invocation’ and ‘application’ (to use authority to make sure that policy is made and carried out), and ‘appraisal’ (2017: 42-3).
  2. The constant interaction between all notional ‘stages’ rather than a linear process: attention to a policy problem fluctuates, actors propose and adopt solutions continuously, actors are making policy (and feeding back on its success) as they implement, evaluation (of policy success) is not a single-shot document, and previous policies set the agenda for new policy (2017: 44-5).

In that context, it is no surprise that the impact of a single policy analyst is usually minimal (2017: 57). Sorry to break it to you. Hang in there, baby.

hang-in-there-baby

 

11 Comments

Filed under 750 word policy analysis, public policy

Policy Analysis in 750 words: Using Statistics and Explaining Risk (Spiegelhalter and Gigerenzer)

Please see the Policy Analysis in 750 words series overview before reading the summary. This post is close to 750 words if you divide it by 2.

David Spiegelhalter (2018) The Art of Statistics: Learning from Data (Pelican, hardback)

Gerd Gigerenzer (2015) Risk Savvy (Penguin)

Spiegelhalter cover

Policy analysis: the skilful consumption and communication of information

Some use the phrase ‘lies, damned lies, and statistics’ to suggest that people can manipulate the presentation of information to reinforce whatever case they want to make. Common examples include the highly selective sharing of data, and the use of misleading images to distort the size of an effect or strength of a relationship between ‘variables’ (when we try to find out if a change in one thing causes a change in another).

In that context, your first aim is to become a skilled consumer of information.

Or, you may be asked to gather and present data as part of your policy analysis, and don’t seek to mislead people (see Mintrom and compare with Riker).

Your second aim is to become an ethical and skilled communicator of information.

In each case, a good rule of thumb is to assume that the analysts who help policymakers learn how to consume and interpret evidence are more influential than the researchers who produce it.

Research and policy analysis are not so different

Although research is not identical to policy analysis, it highlights similar ambitions and issues. Indeed, Spiegelhalter’s (2018: 6-7) description of ‘using data to help us understand the world and make better judgements’ sounds like Mintrom, and the PPDAC approach – identify a problem, plan how to study it, collect and manage data, analyse, and draw/ communicate conclusions (2018: 14) – is not so different from the ‘steps’ to policy analysis that you will find in Bardach or Weimer and Vining.

PPDAC requires us to understand what people need to do to ‘turn the world into data’, such as to produce precise definitions of things and use observation and modelling to estimate their number or the likelihood of their occurrence (2018: 6-7).

More importantly, consider our inability to define things precisely – e.g. economic activity and unemployment, or wellbeing and happiness – and need to accept that our estimates (a) come with often high levels of uncertainty, and (b) are ‘only the starting point to real understanding of the world’ (2018: 8-9).

In that context, the technical skill to gather and analyse information is necessary for research, while the skill to communicate findings is necessary to avoid misleading your audience.

The pitfalls of information communication

Speigelhalter’s initial discussion highlights the great potential to mislead, via:

  1. deliberate manipulation,
  2. a poor grasp of statistics, and/ or
  3. insufficient appreciation of (a) your non-specialist audience’s potential reaction to (b) different ways to frame the same information (2018: 354-62), perhaps based on
  4. the unscientific belief that scientists are objective and can communicate the truth in a neutral way, rather than storytellers with imperfect data (2018: 68-9; 307; 338; 342-53).

Potentially influential communications include (2018: 19-38):

  1. The type of visual, with bar or line-based charts often more useful than pie charts (and dynamic often better than static – 2018: 71)
  2. The point at which you cut off the chart’s axis to downplay or accentuate the difference between results
  3. Framing the results positively (e.g. survival rate) versus negatively (e.g. death rate)
  4. Describing a higher relative risk (e.g. 18%) or absolute risk (e.g. from 6 in 100 to 7 in 100 cases)
  5. Describing risk in relation to decimal places, percentages, or numbers out of 100
  6. Using the wrong way to describe an average (mode, median, or mean – 2018: 46)
  7. Using a language familiar to specialists but confusing to – and subject to misinterpretation by – non-specialists (e.g. odds ratios)
  8. Translating numbers into words (e.g. what does ‘very likely’ mean?) to describe probability (2018: 320).

These problems with the supply of information combine with the ways that citizens and policymakers consume it.

People use cognitive shortcuts, such as emotions and heuristics, to process information (see p60 of Understanding Public Policy, reproduced below).

It can make them vulnerable to framing and manipulation, and prompt them to change their behaviour after misinterpreting evidence in relation to risk: e.g. eating certain foods (2018: 33), anticipating the weather, taking medicines, or refusing to fly after a vivid event (Gigerenzer, 2015: 2-13).

p60 UPP 2nd ed heuristics

Dealing with scientific uncertainty

Communication is important, but the underlying problem may be actual scientific uncertainty about the ability of our data to give us accurate knowledge of the world, such as when:

  1. We use a survey of a sample population, in the hope that (a) respondents provide accurate answers, and (b) their responses provide a representative picture of the population we seek to understand. In such cases, professional standards and practices exist to minimise, but not remove biases associated with questions and sampling (2018: 74).
  2. Some people ignore (and other people underestimate) the ‘margin of error’ in surveys, even though they could be larger than the reported change in data (2018: 189-92; 247).
  3. Alternatives to surveys have major unintended consequences, such as when government statistics are collected unsystematically or otherwise misrepresent outcomes (2018: 84-5)
  4. ‘Correlation does not equal causation’ (see also The Book of Why).
  • The cause of an association between two things could be either of those things, or another thing (2018: 95-9; 110-15).
  • It is usually prohibitively expensive to conduct and analyse research – such as multiple ‘randomised control trials’ to establish cause and effect in the same ways as medicines trials (2018: 104) – to minimise doubt.
  • Further, our complex and uncontrolled world is not as conducive to the experimental trials of social and economic policies.
  1. The misleading appearance of a short term trend often relates to ‘chance variation’ rather than a long-term trend (e.g. in PISA education tables or murder rates – 2018: 131; 249).
  2. The algorithms used to process huge amounts of data may contain unhelpful rules and misplaced assumptions that bias the results, and this problem is worse if the rules are kept secret (2018: 178-87)
  3. Calculating the probability of events is difficult to do, agree how to do, and to understand (2018: 216-20; 226; 239; 304-7).
  4. The likelihood of identifying ‘false positive’ results in research is high (2018: 278-80). Note the comparison to finding someone guilty when innocent, or innocent when guilty (2018: 284 and compare with Gigerenzer, 2015: 33-7; 161-8). However, the professional incentive to minimise these outcomes or admit the research’s limitations is low (2018: 278; 287; 294-302)

Developing statistical and risk ‘literacy’

In that context, Spiegelhalter (2019: 369-71) summarises key ways to consume data effectively, asking: how rigorous is the study, how much uncertainty remains, if the measures are chosen and communicated well, if you can trust the source to do the work well and not spin the results, if the claim fits with other evidence and has a good explanation, and if the effect is important and relevant to key populations. However:

  1. Many such texts describe how they would like the world to work, and give advice to people to help foster that world. The logical conclusion is that this world does not exist, and most people do not have the training, or use these tips, to describe or consume statistics and their implications well.
  2. Policymaking is about making choices, often under immense time and political pressure, in the face of uncertainty versus ambiguity, and despite our inability to understand policy problems or the likely impact of solutions.

I’m not suggesting that, as a result, you should go full Riker. Rather, as with most of the posts in this series, reflect on how you would act – and expect others to act – during the (very long/ not very likely) transition from your world to this better one. What if your collective task is to make just enough sense of the available information, and your options, to make good enough choices?

Gigerenzer cover

In that context, Gigerenzer (2015) identifies the steps we can take to become ‘risk savvy’.

Begin by rejecting (a) the psychological drive to seek the ‘safety blanket’ of certainty (and avoid the ‘fear of doing something wrong and being blamed’), which causes people to (b) place too much faith in necessarily-flawed technologies or tests to reduce uncertainty, instead of (c) learning some basic tools to assess risk while accepting the inevitability of uncertainty and ambiguity (2015: 18-20; 43; 32-40).

Then, employ simple ‘mind tools’ to assess and communicate risk in each case:

  1. Communicate risk using appropriate visuals, categories, and descriptions of risk (e.g. with reference to absolute risk and ‘natural frequences’, expressed as a proportion of 100 (not a % or decimal) and be sceptical if others do not (2015: 25-7; 168)
  • E.g. do not confuse calculations of risk based (a) on known frequencies (such as the coin toss), and (b) unknown frequencies (such as outcomes of complex systems) (2015: 21-6).
  • E.g. be aware of the difference between (a) the accuracy of a test of a problem (its ability to minimise false positive/ negative results), (b) the likelihood that you have the problem it is testing, and (c) the extent to which you will benefit from an intervention for that problem (2015: 33-7; 161-8; 194-7)
  1. Use heuristics that are shown to be efficient and reliable in particular situations.
  • rely frequently on ‘gut feeling’ (‘a judgment 1. that appears quickly in consciousness, 2. whose underlying reasons we are not fully aware of, yet 3. it is strong enough to act upon’)
  • accept the counterintuitive sense that ‘ignoring information can lead to better, faster, and safer decisions’ (2018: 30-1)
  • equate intuition with ‘unconscious intelligence based on personal experience and smart rules of thumb. You need both intuition and reasoning to be rational’ (2018: 123-4)
  • find efficient ways to trust in other people and practices (2018: 99-103)
  • ‘satisfice’ (choose the first option that satisfies an adequate threshold, rather than consider every option) (2018: 148-9)
  1. Value ‘good errors’ that allow us to learn efficiently (via ‘trial and error’) (2015: 47-51)

Wait a minute

I like these Gigerenzer-style messages a lot and, on reflection, seem to make most of my choices using my gut and trial-and-error (and I only electrocuted myself that one time; I’m told that I barked).

Some of his examples – e.g. ask if a hospital uses airline-style checklists (2015: 53; see also Radin’s checklist), or ask your doctor how they would treat their relative, not yours (2015: 63) – are intuitively appealing. The explainers on risk are profoundly important.

However, note that Gigerenzer devotes a lot of his book to describing the defensive nature of sectors such as business, medicine, and government, linked strongly the absence of the right ‘culture’ to allow learning through error.

Trial and error is a big feature in complexity theory, and Lindblom’s incrementalism, but also be aware that you are surrounded by people whose heuristic may be ‘make sure you don’t get the blame’ (or ‘procedure over performance’ (2018: 65). To recommend trial-and-error policy analysis may be a hard sell.

Further reading

This post is part of the Policy Analysis in 750 words series

The 500 and 1000 words series describe how people act under conditions of bounded rationality and policymaking complexity

Winners and losers: communicating the potential impacts of policies (by Cameron Brick, Alexandra Freeman, Steven Wooding, William Skylark, Theresa Marteau & David Spiegelhalter)

See Policy in 500 Words: Social Construction and Policy Design and ask yourself if Gigerenzer’s (2015: 69) ‘fear whatever your social group fears’ is OK when you are running from a lion, but not if you are cooperating with many target populations.

The study of punctuated equilibrium theory is particularly relevant, since its results reject the sense that policy change follows a ‘normal distribution’. See the chart below (from Theories of the Policy Process 2; also found in 5 Images of the Policy Process) and visit the Comparative Agendas Project to see how they gather the data.

True et al figure 6.2

10 Comments

Filed under 750 word policy analysis, public policy