This post first appeared on LSE British Politics and Policy (27.11.20) and is based on this article in British Politics.
Paul Cairneyassesses government policy in the first half of 2020. He identifies the intense criticism of its response so far, encouraging more systematic assessments grounded in policy research.
In March 2020, COVID-19 prompted policy change in the UK at a speed and scale only seen during wartime. According to the UK government, policy was informed heavily by science advice. Prime Minister Boris Johnson argued that, ‘At all stages, we have been guided by the science, and we will do the right thing at the right time’. Further, key scientific advisers such as Sir Patrick Vallance emphasised the need to gather evidence continuously to model the epidemic and identify key points at which to intervene, to reduce the size of the peak of population illness initially, then manage the spread of the virus over the longer term.
Both ministers and advisors emphasised the need for individual behavioural change, supplemented by government action, in a liberal democracy in which direct imposition is unusual and unsustainable. However, for its critics, the government experience has quickly become an exemplar of policy failure.
Initial criticisms include that ministers did not take COVID-19 seriously enough in relation to existing evidence, when its devastating effect was apparent in China in January and Italy from February; act as quickly as other countries to test for infection to limit its spread; or introduce swift-enough measures to close schools, businesses, and major social events. Subsequent criticisms highlight problems in securing personal protective equipment (PPE), testing capacity, and an effective test-trace-and-isolate system. Some suggest that the UK government was responding to the ‘wrong pandemic’, assuming that COVID-19 could be treated like influenza. Others blame ministers for not pursuing an elimination strategy to minimise its spread until a vaccine could be developed. Some criticise their over-reliance on models which underestimated the R (rate of transmission) and ‘doubling time’ of cases and contributed to a 2-week delay of lockdown. Many describe these problems and delays as the contributors to the UK’s internationally high number of excess deaths.
How can we hold ministers to account in a meaningful way?
I argue that these debates are often fruitless and too narrow because they do not involve systematic policy analysis, take into account what policymakers can actually do, or widen debate to consider whose lives matter to policymakers. Drawing on three policy analysis perspectives, I explore the questions that we should ask to hold ministers to account in a way that encourages meaningful learning from early experience.
These questions include:
Was the government’s definition of the problem appropriate? Much analysis of UK government competence relates to specific deficiencies in preparation (such as shortages in PPE), immediate action (such as to discharge people from hospitals to care homes without testing them for COVID-19), and implementation (such as an imperfect test-trace-and-isolate system). The broader issue relates to its focus on intervening in late March to protect healthcare capacity during a peak of infection, rather than taking a quicker and more precautionary approach. This judgment relates largely to its definition of the policy problem which underpins every subsequent policy intervention.
Did the government select the right policy mix at the right time? Who benefits most from its choices?
Most debates focus on the ‘lock down or not?’ question without exploring fully the unequal impact of any action. The government initially relied on exhortation, based on voluntarism and an appeal to social responsibility. Initial policy inaction had unequal consequences on social groups, including people with underlying health conditions, black and ethnic minority populations more susceptible to mortality at work or discrimination by public services, care home residents, disabled people unable to receive services, non-UK citizens obliged to pay more to live and work while less able to access public funds, and populations (such as prisoners and drug users) that receive minimal public sympathy. Then, in March, its ‘stay at home’ requirement initiated a major new policy and different unequal impacts in relation to the income, employment, and wellbeing of different groups. These inequalities are list in more general discussions of impacts on the whole population.
Did the UK government make the right choices on the trade-offs between values, and what impacts could the government have reasonably predicted?
Initially, the most high-profile value judgment related to freedom from state coercion to reduce infection versus freedom from the harm of infection caused by others. Then, values underpinned choices on the equitable distribution of measures to mitigate the economic and wellbeing consequences of lockdown. A tendency for the UK government to project centralised and ‘guided by the science’ policymaking has undermined public deliberation on these trade-offs between policies. The latter will be crucial to ongoing debates on the trade-offs associated with national and regional lockdowns.
Did the UK government combine good policy with good policymaking?
A problem like COVID-19 requires trial-and-error policymaking on a scale that seems incomparable to previous experiences. It requires further reflection on how to foster transparent and adaptive policymaking and widespread public ownership for unprecedented policy measures, in a political system characterised by (a) accountability focused incorrectly on strong central government control and (b) adversarial politics that is not conducive to consensus seeking and cooperation.
These additional perspectives and questions show that too-narrow questions – such as was the UK government ‘following the science’ – do not help us understand the longer term development and wider consequences of UK COVID-19 policy. Indeed, such a narrow focus on science marginalises wider discussions of values and the populations that are most disadvantaged by government policy.
Paul Cairney (2020) ‘The UK Government’s COVID-19 policy: assessing evidence-informed policy analysis in real time’, British Politicshttps://rdcu.be/b9zAk (PDF)
The coronavirus feels like a new policy problem that requires new policy analysis. The analysis should be informed by (a) good evidence, translated into (b) good policy. However, don’t be fooled into thinking that either of those things are straightforward. There are simple-looking steps to go from defining a problem to making a recommendation, but this simplicity masks the profoundly political process that must take place. Each step in analysis involves political choices to prioritise some problems and solutions over others, and therefore prioritise some people’s lives at the expense of others.
My article in British Politics takes us through those steps in the UK, and situates them in a wider political and policymaking context. This post is shorter, and only scratches the surface of analysis.
5 steps to policy analysis
Define the problem.
Perhaps we can sum up the initial UK government approach as: (a) the impact of this virus and illness will be a level of death and illness that could overwhelm the population and exceed the capacity of public services, so (b) we need to contain the virus enough to make sure it spreads in the right way at the right time, so (c) we need to encourage and make people change their behaviour (primarily via hygiene and social distancing). However, there are many ways to frame this problem to emphasise the importance of some populations over others, and some impacts over others.
Identify technically and politically feasible solutions.
Solutions are not really solutions: they are policy instruments that address one aspect of the problem, including taxation and spending, delivering public services, funding research, giving advice to the population, and regulating or encouraging changes to social behaviour. Each new instrument contributes an existing mix, with unpredictable and unintended consequences. Some instruments seem technically feasible (they will work as intended if implemented), but will not be adopted unless politically feasible (enough people support their introduction). Or vice versa. From the UK government’s perspective, this dual requirement rules out a lot of responses.
Use values and goals to compare solutions.
Typical judgements combine: (a) broad descriptions of values such as efficiency, fairness, freedom, security, and human dignity, (b) instrumental goals, such as sustainable policymaking (can we do it, and for how long?), and political feasibility (will people agree to it, and will it make me more or less popular or trusted?), and (c) the process to make choices, such as the extent to which a policy process involves citizens or stakeholders (alongside experts) in deliberation. They combine to help policymakers come to high profile choices (such as the balance between individual freedom and state coercion), and low profile but profound choices (to influence the level of public service capacity, and level of state intervention, and therefore who and how many people will die).
Predict the outcome of each feasible solution.
It is difficult to envisage a way for the UK Government to publicise all of the thinking behind its choices (Step 3) and predictions (Step 4) in a way that would encourage effective public deliberation. People often call for the UK Government to publicise its expert advice and operational logic, but I am not sure how they would separate it from their normative logic about who should live or die, or provide a frank account without unintended consequences for public trust or anxiety. If so, one aspect of government policy is to keep some choices implicit and avoid a lot of debate on trade-offs. Another is to make choices continuously without knowing what their impact will be (the most likely scenario right now).
Make a choice, or recommendation to your client.
Your recommendation or choice would build on these four steps. Define the problem with one framing at the expense of the others. Romanticise some people and not others. Decide how to support some people, and coerce or punish others. Prioritise the lives of some people in the knowledge that others will suffer or die. Do it despite your lack of expertise and profoundly limited knowledge and information. Learn from experts, but don’t assume that only scientific experts have relevant knowledge (decolonise; coproduce). Recommend choices that, if damaging, could take decades to fix after you’ve gone. Consider if a policymaker is willing and able to act on your advice, and if your proposed action will work as intended. Consider if a government is willing and able to bear the economic and political costs. Protect your client’s popularity, and trust in your client, at the same time as protecting lives. Consider if your advice would change if the problem seemed to change. If you are writing your analysis, maybe keep it down to one sheet of paper (in other words, fewer words than in this post up to this point).
Policy analysis is not as simple as these steps suggest, and further analysis of the wider policymaking environment helps describe two profound limitations to simple analytical thought and action.
Policymakers must ignore almost all evidence
The amount of policy relevant information is infinite, and capacity is finite. So, individuals and governments need ways to filter out almost all of it. Individuals combine cognition and emotion to help them make choices efficiently, and governments have equivalent rules to prioritise only some information. They include: define a problem and a feasible response, seek information that is available, understandable, and actionable, and identify credible sources of information and advice. In that context, the vague idea of trusting or not trusting experts is nonsense, and the larger post highlights the many flawed ways in which all people decide whose expertise counts.
They do not control the policy process.
Policymakers engage in a messy and unpredictable world in which no single ‘centre’ has the power to turn a policy recommendation into an outcome.
There are many policymakers and influencers spread across a political system. For example, consider the extent to which each government department, devolved governments, and public and private organisations are making their own choices that help or hinder the UK government approach.
Most choices in government are made in ‘subsystems’, with their own rules and networks, over which ministers have limited knowledge and influence.
The social and economic context, and events, are largely out of their control.
The take home messages (if you accept this line of thinking)
The coronavirus is an extreme example of a general situation: policymakers will always have very limited knowledge of policy problems and control over their policymaking environment. They make choices to frame problems narrowly enough to seem solvable, rule out most solutions as not feasible, make value judgements to try help some more than others, try to predict the results, and respond when the results do not match their hopes or expectations.
This is not a message of doom and despair. Rather, it encourages us to think about how to influence government, and hold policymakers to account, in a thoughtful and systematic way that does not mislead the public or exacerbate the problem we are seeing. No one is helping their government solve the problem by saying stupid shit on the internet (OK, that last bit was a message of despair).
Further reading:
The article (PDF) sets out these arguments in much more detail, with some links to further thoughts and developments.
This series of ‘750 words’ posts summarises key texts in policy analysis and tries to situate policy analysis in a wider political and policymaking context. Note the focus on whose knowledge counts, which is not yet a big feature of this crisis.
These series of 500 words and 1000 words posts (with podcasts) summarise concepts and theories in policy studies.
This is the long version. It is long. Too long to call a blog post. Let’s call it a ‘living document’ that I update and amend as new developments arise (then start turning into a more organised paper). In most cases, I am adding tweets, so the date of the update is embedded. If I add a new section, I will add a date. If you seek specific topics (like ‘herd immunity’), it might be worth doing a search. The short version is shorter.
The coronavirus feels like a new policy problem. Governments already have policies for public health crises, but the level of uncertainty about the spread and impact of this virus seems to be taking it to a new level of policy, media, and public attention. The UK Government’s Prime Minister calls it ‘the worst public health crisis for a generation’.
As such, there is no shortage of opinions on what to do, but there is a shortage of well-considered opinions, producing little consensus. Many people are rushing to judgement and expressing remarkably firm opinions about the best solutions, but their contributions add up to contradictory evaluations, in which:
the government is doing precisely the right thing or the completely wrong thing,
we should listen to this expert saying one thing or another expert saying the opposite.
Lots of otherwise-sensible people are doing what they bemoan in politicians: rushing to judgement, largely accepting or sharing evidence only if it reinforces that judgement, and/or using their interpretation of any new development to settle scores with their opponents.
Yet, anyone who feels, without uncertainty, that they have the best definition of, and solution to, this problem is a fool. If people are also sharing bad information and advice, they are dangerous fools. Further, as Professor Madley puts it (in the video below), ‘anyone who tells you they know what’s going to happen over the next six months is lying’.
In that context, how can we make sense of public policy to address the coronavirus in a more systematic way?
Studies of policy analysis and policymaking do not solve a policy problem, but they at least give us a language to think it through.
In each step, note how quickly it is possible to be overwhelmed by uncertainty and ambiguity, even when the issue seems so simple at first.
Note how difficult it is to move from Step 1, and to separate Step 1 from the others. It is difficult to define the problem without relating it to the solution (or to the ways in which we will evaluate each solution).
Let’s relate that analysis to research on policymaking, to understand the wider context in which people pay attention to, and try to address, important problems that are largely out of their control.
Throughout, note that I am describing a thought process as simply as I can, not a full examination of relevant evidence. I am highlighting the problems that people face when ‘diagnosing’ policy problems, not trying to diagnose it myself. To do so, I draw initially on common advice from the key policy analysis texts (summaries of the texts that policy analysis students are most likely to read) that simplify the process a little too much. Still, the thought process that it encourages took me hours alone (spread over three days) to produce no real conclusion. Policymakers and advisers, in the thick of this problem, do not have that luxury of time or uncertainty.
In our latest guest blog, Jonny Pearson-Stuttard, RSPH Trustee and Public Health Doctor @imperialcollege sets out what we know about the spread of coronavirus to date, and why the Government has taken the measures it hashttps://t.co/XM7zKKjwtE
Provide a diagnosis of a policy problem, using rhetoric and eye-catching data to generate attention.
Identify its severity, urgency, cause, and our ability to solve it. Don’t define the wrong problem, such as by oversimplifying.
Problem definition is a political act of framing, as part of a narrative to evaluate the nature, cause, size, and urgency of an issue.
Define the nature of a policy problem, and the role of government in solving it, while engaging with many stakeholders.
‘Diagnose the undesirable condition’ and frame it as ‘a market or government failure (or maybe both)’.
Coronavirus as a physical problem is not the same as a coronavirus policy problem. To define the physical problem is to identify the nature, spread, and impact of a virus and illness on individuals and populations. To define a policy problem, we identify the physical problem and relate it (implicitly or explicitly) to what we think a government can, and should, do about it. Put more provocatively, it is only a policy problem if policymakers are willing and able to offer some kind of solution.
This point may seem semantic, but it raises a profound question about the capacity of any government to solve a problem like an epidemic, or for governments to cooperate to solve a pandemic. It is easy for an outsider to exhort a government to ‘do something!’ (or ‘ACT NOW!’) and express certainty about what would happen. However, policymakers inside government:
Do not enjoy the same confidence that they know what is happening, or that their actions will have their intended consequences, and
Will think twice about trying to regulate social behaviour under those circumstances, especially when they
Know that any action or inaction will benefit some and punish others.
For example, can a government make people wash their hands? Or, if it restricts gatherings at large events, can it stop people gathering somewhere else, with worse impact? If it closes a school, can it stop children from going to their grandparents to be looked after until it reopens? There are 101 similar questions and, in each case, I reckon the answer is no. Maybe government action has some of the desired impact; maybe not. If you agree, then the question might be: what would it really take to force people to change their behaviour?
The answer is: often too much for a government to consider (in a liberal democracy), particularly if policymakers are informed that it will not have the desired impact.
A couple of key takeaways from our analysis of early COVID-19 dynamics in Wuhan:
1. We estimated that the control measures introduced – unprecedented interventions that will have had a huge social and psychological toll – reduced transmission by around 55% in space of 2 weeks 1/
If so, the UK government’s definition of the policy problem will incorporate this implicit question: what can we do if we can influence, but not determine (or even predict well) how people behave?
Uncertainty about the coronavirus plus uncertainty about policy impact
Now, add that general uncertainty about the impact of government to this specific uncertainty about the likely nature and spread of the coronavirus:
The ideal spread involves all well people sharing the virus first, while all vulnerable people (e.g. older, and/or with existing health problems that affect their immune systems) protected in one isolated space, but it won’t happen like that; so, we are trying to minimise damage in the real world.
We mainly track the spread via deaths, with data showing a major spike appearing one month later, so the problem may only seem real to most people when it is too late to change behaviour
A lot of the spread will happen inside homes, where the role of government is minimal (compared to public places). So, for example, the impact of school closures could be good (isolation) or make things worse (children spreading the virus to vulnerable relatives) (see also ‘we don’t know [if the UKG decision not to close schools] was brilliant or catastrophic’). [Update 18.3.20: as it turned out, the First Minister’s argument for closing Scottish schools was that there were too few teachers available).
The choice in theory is between a rapid epidemic with a high peak, or a slowed-down epidemic over a longer period, but ‘anyone who tells you they know what’s going to happen over the next six months is lying’.
Maybe this epidemic will be so memorable as to shift social behaviour, but so much depends on trying to predict (badly) if individuals will actually change (see also Spiegelhalter on communicating risk).
None of this account tells policymakers what to do, but at least it helps them clarify three key aspects of their policy problem:
The impact of this virus and illness could overwhelm the population, to the extent that it causes mass deaths, causes a level of illness that exceeds the capacity of health services to treat, and contributes to an unpredictable amount of social and economic damage.
We need to contain the virus enough to make sure it (a) spreads at the right speed and/or (b) peaks at the right time. The right speed seems to be: a level that allows most people to recover alone, while the most vulnerable are treated well in healthcare settings that have enough capacity. The right time seems to be the part of the year with the lowest demand on health services (e.g. summer is better than winter). In other words, (a) reduce the size of the peak by ‘flattening the curve’, and/or (b) find the right time of year to address the peak, while (c) anticipating more than one peak.
My impression is that the most frequently-expressed aim is (a) …
Yesterday we entered the Delay phase of our #COVID_19uk Action Plan. @UKScienceChief explained why this is important.
It allows us to #FlattenTheCurve, which means reducing the impact in the short-term to ensure our health & care system can effectively protect vulnerable people pic.twitter.com/1I45C3v38V
— Department of Health and Social Care (@DHSCgovuk) March 13, 2020
… while the UK Government’s Deputy Chief Medical Officer also seems to be describing (b):
Dr Jenny Harries, Deputy Chief Medical Officer, came into Downing Street to answer some of the most commonly asked questions on coronavirus. pic.twitter.com/KCdeHsaz6a
We need to encourage or coerce people to change their behaviour, to look after themselves (e.g. by handwashing) and forsake their individual preferences for the sake of public health (e.g. by self-isolating or avoiding vulnerable people). Perhaps we can foster social trust and empathy to encourage responsible individual action. Perhaps people will only protect others if obliged to do so (compare Stone; Ostrom; game theory).
See also: From across the Ditch: How Australia has to decide on the least worst option for COVID-19 (Prof Tony Blakely on three bad options: (1) the likelihood of ‘elimination’ of the virus before vaccination is low; (2) an 18-month lock-down will help ‘flatten the curve’; (3) ‘to prepare meticulously for allowing the pandemic to wash through society over a period of six or so months. To tool up the production of masks and medical supplies. To learn as quickly as possible which treatments of people sick with COVID-19 saves lives. To work out our strategies for protection of the elderly and those with a chronic condition (for whom the mortality from COVID-19 is much higher’).
Why politicians fear being accused of over reaction. Which in turn might prevent them from reacting appropriately when a real crisis hits 👇🏽👇🏽 https://t.co/UrxHTAs2z5
If you are still with me, I reckon you would have worded those aims slightly differently, right? There is some ambiguity about these broad intentions, partly because there is some uncertainty, and partly because policymakers need to set rather vague intentions to generate the highest possible support for them. However, vagueness is not our friend during a crisis involving such high anxiety. Further, they are only delaying the inevitable choices that people need to make to turn a complex multi-faceted problem into something simple enough to describe and manage. The problem may be complex, but our attention focuses only on a small number of aspects, at the expense of the rest. Examples that have arisen, so far, include to accentuate:
The health of the whole population or people who would be affected disproportionately by the illness.
For example, the difference in emphasis affects the health advice for the relatively vulnerable (and the balance between exhortation and reassurance)
Inequalities in relation to health, socio-economic status (e.g. income, gender, race, ethnicity), or the wider economy.
For example, restrictive measures may reduce the risk of harm to some, but increase the burden on people with no savings or reliable sources of income.
For example, some people are hoarding large quantities of home and medical supplies that (a) other people cannot afford, and (b) some people cannot access, despite having higher need.
For example, social distancing will limit the spread of the virus (see the nascent evidence), but also produce highly unequal forms of social isolation that increase the risk of domestic abuse (possibly exacerbated by school closures) and undermine wellbeing. Or, there will be major policy changes, such as to the rules to detain people under mental health legislation, regarding abortion, or in relation to asylum (note: some of these tweets are from the US, partly because I’m seeing more attention to race – and the consequence of systematic racism on the socioeconomic inequalities so important to COVID-19 mortality – than in the UK).
COVID-19 has brought new focus to women’s continued inequality. Without a gendered response to both the health and economic crises, gender inequality will be further cemented. Read more on the blog: https://t.co/zYxSFpUTNE
“The epidemic has had a huge impact on domestic violence,” said Wan. “According to our statistics, 90% of the causes of violence are related to the COVID-19 epidemic.” https://t.co/xswemtf548
I just asked a DC cop what he’s noticed since the coronavirus sent people home. “More domestic violence,” he said, without missing a beat. https://t.co/kv9zH5VNj1
While black people make up about 12% of Michigan’s population, they make up about 40% of all COVID-19 deaths reported.
A social epidemiologist says the numbers don’t say everything, but there's something that can’t be ignored: inequality. @MichiganRadiohttps://t.co/bWsqFaCrUJ
Available evidence (though injuriously limited) shows that Black people are being infected & dying of #coronavirus at higher rates. Disproportionate Black suffering is what many of us have suspected and feared because it is consistent with the entirety of American history. https://t.co/qzmXvGCGvV
#Coronavirus is not the 'great equalizer'—race matters:
“I believe that the actions and omissions of world leaders in charge of fighting the #COVID19 pandemic will reveal historical and current impacts of colonial violence and continued health inequities” https://t.co/nUuBIKfrVL
— Dr. Malinda S. Smith (@MalindaSmith) April 6, 2020
BAME lives matter, so far they account for:
– 100% of Dr deaths – 50% nurse deaths – 35% of Patients in ICU
Yet account for only 14% of population and account for 44% of NHS staff. Who is asking the questions, why the disparity?https://t.co/VOL8FAmy45
BBC news reports on the disproportionate deaths of African Americans & minorities in the US from #COVID19, but silence on similar issues in the UK. Why? Where is the reporting? Where is the accountability? https://t.co/DkGPjfnWG1
What the coronavirus bill will do: https://t.co/qoBdKKr64H Mental Health Act – detention implemented using just one doctor’s opinion (not 2) & AMHP, & temporarily allow extension or removal of time limits to allow for greater flexibility where services are less able to respond
English obviously, but fascinating that have issued an explicitly ethical framework for COVID decisions re mental health and incapacity. Can Scotland do same? https://t.co/WccPntZOwf
WOW – government has relaxed restrictions on WHERE abortions can take place, temporary inclusion of 'the home' as a legal site for abortion: https://t.co/Vw714fWXEM
Abortion services for women from Northern Ireland remain available free of charge in England. This provision will continue until services are available to meet these needs in Northern Ireland. For more information, visit: https://t.co/YYjop5lSgUpic.twitter.com/M8k95aIisM
BREAKING NEWS!!!! The Home Office have confirmed that ALL evictions and terminations of asylum support have been paused for 3 months. Find out more and read the letter from Home Office Minister Chris Philp confirming this on our website at: https://t.co/KDlVr4PHyP
NEW Editorial: While responding to #COVID19, policy makers should consider the risk of deepening health inequalities. If vulnerable groups are not properly identified, the consequences of this pandemic will be even more devastating https://t.co/BrypuXH6vSpic.twitter.com/hka3nLzxdv
In relation to Prison Rule Changes – these would only ever be used as an absolute last resort, in order to protect staff & those in our care. I can confirm that emergency changes to showering have not been implemented in any establishment.
For example, governments cannot ignore the impact of their actions on the economy, however much they emphasise mortality, health, and wellbeing. Most high-profile emphasis was initially on the fate of large and small businesses, and people with mortgages, but a long period of crisis will a tip the balance from low income to unsustainable poverty (even prompting Iain Duncan Smith to propose policy change), and why favour people who can afford a mortgage over people scraping the money together for rent?
So…. Govt income protection package includes….. 1. 80% of wage costs up to £2500 2. Deferred VAT. 3. £7 billion uplift to Universal Credit and Woring Tax crdit. 4. £1 billion to cover 30% of house rental costs. 5. Self employed to get same as sickness benefit payments.
A need for more communication and exhortation, or for direct action to change behaviour.
The short term (do everything possible now) or long term (manage behaviour over many months).
The Imperial College COVID report is being discussed. But a major takeaway from it will likely survive discussion: the human cost of a pure mitigation strategy is inacceptable, whilst a pure suppression strategy is unsustainable over time (thread)
How to maintain trust in the UK government when (a) people are more or less inclined to trust a the current part of government and general trust may be quite low, and (b) so many other governments are acting differently from the UK.
For example, note the visible presence of the Prime Minister, but also his unusually high deference to unelected experts such as (a) UK Government senior scientists providing direct advice to ministers and the public, and (b) scientists drawing on limited information to model behaviour and produce realistic scenarios (we can return to the idea of ‘evidence-based policymaking’ later). This approach is not uncommon with epidemics/ pandemics (LD was then the UK Government’s Chief Medical Officer):
For example, note how often people are second guessing and criticising the UK Government position (and questioning the motives of Conservative ministers).
For example, people often try to lay blame for viruses on certain populations, based on their nationality, race, ethnicity, sexuality, or behaviour (e.g. with HIV).
For example, the (a) association between the coronavirus and China and Chinese people (e.g. restrict travel to/ from China; e.g. exacerbate racism), initially overshadowed (b) the general role of international travellers (e.g. place more general restrictions on behaviour), and (c) other ways to describe who might be responsible for exacerbating a crisis.
For social scientists wondering “what can I do now?” here’s a challenge:@cp_roth@LukasHenselEcon & others ran a survey with 2500 Italians yday & found that:
Under ‘normal’ policymaking circumstances, we would expect policymakers to resolve this ambiguity by exercising power to set the agenda and make choices that close off debate. Attention rises at first, a choice is made, and attention tends to move on to something else. With the coronavirus, attention to many different aspects of the problem has been lurching remarkably quickly. The definition of the policy problem often seems to be changing daily or hourly, and more quickly than the physical problem. It will also change many more times, particularly when attention to each personal story of illness or death prompts people to question government policy every hour. If the policy problem keeps changing in these ways, how could a government solve it?
@alexwickham doing fine work as a journalist again. Gets right into Government somehow and tells people what is going on.
10 Days That Changed Britain: "Heated" Debate Between Scientists Forced Boris Johnson To Act On Coronavirus https://t.co/hDLEAPT3Z0
Public expenditure (e.g. to boost spending for emergency care, crisis services, medical equipment)
Economic incentives and disincentives (e.g. to reduce the cost of business or borrowing, or tax unhealthy products)
Linking spending to entitlement or behaviour (e.g. social security benefits conditional on working or seeking work, perhaps with the rules modified during crises)
Formal regulations versus voluntary agreements (e.g. making organisations close, or encouraging them to close)
Public services: universal or targeted, free or with charges, delivered directly or via non-governmental organisations
As a result, what we call ‘policy’ is really a complex mix of instruments adopted by one or more governments. A truism in policy studies is that it is difficult to define or identify exactly what policy is because (a) each new instrument adds to a pile of existing measures (with often-unpredictable consequences), and (b) many instruments designed for individual sectors tend, in practice, to intersect in ways that we cannot always anticipate. When you think through any government response to the coronavirus, note how every measure is connected to many others.
Further, it is a truism in public policy that there is a gap between technical and political feasibility: the things that we think will be most likely to work as intended if implemented are often the things that would receive the least support or most opposition. For example:
Redistributing income and wealth to reduce socio-economic inequalities (e.g. to allay fears about the impact of current events on low-income and poverty) seems to be less politically feasible than distributing public services to deal with the consequences of health inequalities.
Providing information and exhortation seems more politically feasible than the direct regulation of behaviour. Indeed, compared to many other countries, the UK Government seems reluctant to introduce ‘quarantine’ style measures to restrict behaviour.
Under ‘normal’ circumstances, governments may be using these distinctions as simple heuristics to help them make modest policy changes while remaining sufficiently popular (or at least looking competent). If so, they are adding or modifying policy instruments during individual ‘windows of opportunity’ for specific action, or perhaps contributing to the sense of incremental change towards an ambitious goal.
Right now, we may be pushing the boundaries of what seems possible, since crises – and the need to address public anxiety – tend to change what seems politically feasible. However, many options that seem politically feasible may not be possible (e.g. to buy a lot of extra medical/ technology capacity quickly), or may not work as intended (e.g. to restrict the movement of people). Think of technical and political feasibility as necessary but insufficient on their own, which is a requirement that rules out a lot of responses.
Add in the UK legislation and we see that it is a major feat simply to account for all of the major moving parts (while noting that much policy change is not legislative)https://t.co/gKsIx7aHr2pic.twitter.com/Ms6fjaDbhF
A few 'somewhat overwritten' newspaper stories today using some of our quotes on PPE. Here is exactly what we are saying, in the tone in which we are saying it: https://t.co/j6PO420WSF
Typical value judgements relate to efficiency, equity and fairness, the trade-off between individual freedom and collective action, and the extent to which a policy process involves citizens in deliberation.
Normative assessments are based on values such as ‘equality, efficiency, security, democracy, enlightenment’ and beliefs about the preferable balance between state, communal, and market/ individual solutions
‘Specify the objectives to be attained in addressing the problem and the criteria to evaluate the attainment of these objectives as well as the satisfaction of other key considerations (e.g., equity, cost, equity, feasibility)’.
‘Effectiveness, efficiency, fairness, and administrative efficiency’ are common.
Identify (a) the values to prioritise, such as ‘efficiency’, ‘equity’, and ‘human dignity’, and (b) ‘instrumental goals’, such as ‘sustainable public finance or political feasibility’, to generate support for solutions.
Instrumental questions may include: Will this intervention produce the intended outcomes? Is it easy to get agreement and maintain support? Will it make me popular, or diminish trust in me even further?
How to weigh the many future health problems and deaths caused by the lockdown against those saved? How to account for the worse effects of the lockdown on the young and the poor? Near impossible ethical choices that government will have to make. https://t.co/DJgwE4b3rd
Step 3 is the most simple-looking but difficult task. Remember that it is a political, not technical, process. It is also a political process that most people would like to avoid doing (at least publicly) because it involves making explicit the ways in which we prioritise some people over others. Public policy is the choice to help some people and punish or refuse to help others (and includes the choice to do nothing).
Policy analysis texts describe a relatively simple procedure of identifying criteria and producing a table (with a solution in each row, and criteria in each column) to compare the trade-offs between each solution. However, these criteria are notoriously difficult to define, and people resolve that problem by exercising power to decide what each term means, and whose interests should be served when they resolve trade-offs. For example, see Stone on whose needs come first, who benefits from each definition of fairness, and how technical-looking processes such as ‘cost benefit analysis’ mask political choices.
Right now, the most obvious and visible trade-off, accentuated in the UK, is between individual freedom and collective action, or the balance between state, communal, and market/ individual solutions. In comparison with many countries (and China and Italy in particular), the UK Government seems to be favouring individual action over state quarantine measures. However, most trade-offs are difficult to categorise
What should be the balance between efforts to minimise the deaths of some (generally in older populations) and maximise the wellbeing of others? This is partly about human dignity during crisis, how we treat different people fairly, and the balance of freedom and coercion.
How much should a government spend to keep people alive using intensive case or expensive medicines, when the money could be spent improving the lives of far more people? This is partly about human dignity, the relative efficiency of policy measures, and fairness.
If you are like me, you don’t really want to answer such questions (indeed, even writing them looks callous). If so, one way to resolve them is to elect policymakers to make such choices on our behalf (perhaps aided by experts in moral philosophy, or with access to deliberative forums). To endure, this unusually high level of deference to elected ministers requires some kind of reciprocal act:
"I hope the UK government will be transparent about its decision-making; willing to listen to NHS staff concerns; humble in learning from other countries’ experiences; and pro-active in building relationships with them."https://t.co/CYUyvij2bK
I agree. There is a need to show that divergent opinons in the public health/virology expert sector have been heard, debates have been had and conclusions explained. This is what I need as a citizen. Also casting the public not a bog roll stowing mob is not helpful or kind. https://t.co/g61Nypcqlc
The Guardian calls this document a “secret” briefing from Public Health England. At a time of national crisis there is no place for secrecy from health experts. If you want public support, share your data, scenarios, and forecasts. Now. https://t.co/O8BpDlCJ7H
I am glad Johnson has listened, but we shouldn't have to drag him kicking and screaming to these decisions. A daily update is a basic step. Transparency, honesty, compassion are vital in this time of a global crisis! no more secret briefings PM.https://t.co/eMxZnMehUp
The CSA and CMO say they will publish the models underlying their strategy on Covid-19. Sharing the data and models is important for accountability, testing and learning. https://t.co/rOuJWwy93i
Dear Boris – Number 10 needs a professional communications operation, immediately. (Open letter to the Prime Minister. Britain has some great comms specialists. He needs to hire one of them urgently) https://t.co/8w6MBYHHbm
Still, I doubt that governments are making reportable daily choices with reference to a clear and explicit view of what the trade-offs and priorities should be, because their choices are about who will die, and their ability to predict outcomes is limited.
Focus on the outcomes that key actors care about (such as value for money), and quantify and visualise your predictions if possible. Compare the pros and cons of each solution, such as how much of a bad service policymakers will accept to cut costs.
‘Assess the outcomes of the policy options in light of the criteria and weigh trade-offs between the advantages and disadvantages of the options’.
Estimate the cost of a new policy, in comparison with current policy, and in relation to factors such as savings to society or benefits to certain populations. Use your criteria and projections to compare each alternative in relation to their likely costs and benefits.
Explain potential solutions in sufficient detail to predict the costs and benefits of each ‘alternative’ (including current policy).
Short deadlines dictate that you use ‘logic and theory, rather than systematic empirical evidence’ to make predictions efficiently.
Monitoring is crucial because it is difficult to predict policy success, and unintended consequences are inevitable. Try to measure the outcomes of your solution, while noting that evaluations are contested.
It is difficult to envisage a way for the UK Government to publicise the thinking behind its choices (Step 3) and predictions (Step 4) in a way that would encourage effective public deliberation, rather than a highly technical debate between a small number of academics:
Ferguson et al (link) simulate outbreak response. Positive: They show suppression (lockdown R0<1) is essential as mitigation (R0>1, “flattening the curve”) results in massive hospital overload and many dead. BUT 1/3 (review attached)https://t.co/srbBS7F1s5pic.twitter.com/qbEymBdOqm
I’m conscious that lots of people would like to see and run the pandemic simulation code we are using to model control measures against COVID-19. To explain the background – I wrote the code (thousands of lines of undocumented C) 13+ years ago to model flu pandemics…
Further, people often call for the UK Government to publicise its expert advice and operational logic, but I am not sure how they would separate it from their normative logic, or provide a frank account without unintended consequences for public trust or anxiety. If so, government policy involves (a) to keep some choices implicit to avoid a lot of debate on trade-offs, and (b) to make general statements about choices when they do not know what their impact will be.
Examine your case through the eyes of a policymaker. Keep it simple and concise.
Make a preliminary recommendation to inform an iterative process, drawing feedback from clients and stakeholder groups
Client-oriented advisors identify the beliefs of policymakers and tailor accordingly.
‘Unless your client asks you not to do so, you should explicitly recommend one policy’
I now invite you to make a recommendation (step 5) based on our discussion so far (steps 1-4). Define the problem with one framing at the expense of the others. Romanticise some people and not others. Decide how to support some people, and coerce or punish others. Prioritise the lives of some people in the knowledge that others will suffer or die. Do it despite your lack of expertise and profoundly limited knowledge and information. Learn from experts, but don’t assume that only scientific experts have relevant knowledge (decolonise; coproduce). Recommend choices that, if damaging, could take decades to fix after you’ve gone. Consider if a policymaker is willing and able to act on your advice, and if your proposed action will work as intended. Consider if a government is willing and able to bear the economic and political costs. Protect your client’s popularity, and trust in your client, at the same time as protecting lives. Consider if your advice would change if the problem would seem to change. If you are writing your analysis, maybe keep it down to one sheet of paper (and certainly far fewer words than in this post). Better you than me.
Please now watch this video before I suggest that things are not so simple.
Would that policy analysis were so simple
Imagine writing policy analysis in an imaginary world, in which there is a single powerful ‘rational’ policymaker at the heart of government, making policy via an orderly series of stages.
Your audience would be easy to identify at each stage, your analysis would be relatively simple, and you would not need to worry about what happens after you make a recommendation for policy change (since the selection of a solution would lead to implementation). You could adopt a simple 5 step policy analysis method, use widely-used tools such as cost-benefit analysis to compare solutions, and know where the results would feed into the policy process.
Studies of policy analysts describe how unrealistic this expectation tends to be (Radin, Brans, Thissen).
For example, there are many policymakers, analysts, influencers, and experts spread across political systems, and engaging with 101 policy problems simultaneously, which suggests that it is not even clear how everyone fits together and interacts in what we call (for the sake of simplicity) ‘the policy process’.
Instead, we can describe real world policymaking with reference to two factors.
The wider policymaking environment: 1. Limiting the use of evidence
First, policymakers face ‘bounded rationality’, in which they only have the ability to pay attention to a tiny proportion of available facts, are unable to separate those facts from their values (since we use our beliefs to evaluate the meaning of facts), struggle to make clear and consistent choices, and do not know what impact they will have. The consequences can include:
Limited attention, and lurches of attention. Policymakers can only pay attention to a tiny proportion of their responsibilities, and policymaking organizations struggle to process all policy-relevant information. They prioritize some issues and information and ignore the rest.
Power and ideas. Some ways of understanding and describing the world dominate policy debate, helping some actors and marginalizing others.
Beliefs and coalitions. Policymakers see the world through the lens of their beliefs. They engage in politics to turn their beliefs into policy, form coalitions with people who share them, and compete with coalitions who don’t.
Dealing with complexity. They engage in ‘trial-and-error strategies’ to deal with uncertain and dynamic environments (see the new section on trial-and-error- at the end).
Framing and narratives. Policy audiences are vulnerable to manipulation when they rely on other actors to help them understand the world. People tell simple stories to persuade their audience to see a policy problem and its solution in a particular way.
The social construction of populations. Policymakers draw on quick emotional judgements, and social stereotypes, to propose benefits to some target populations and punishments for others.
Rules and norms. Institutions are the formal rules and informal understandings that represent a way to narrow information searches efficiently to make choices quickly.
Learning. Policy learning is a political process in which actors engage selectively with information, not a rational search for truth.
Evidence-based or expert-informed policymaking
Don’t think science can or should make decisions Donna. In conditions of uncertainty, it must inform decision makers who must be transparent about the choices they make and be held to account for them https://t.co/Wj4s9IS6fO
Put simply, policymakers cannot oversee a simple process of ‘evidence-based policymaking’. Rather, to all intents and purposes:
They need to find ways to ignore most evidence so that they can focus disproportionately on some. Otherwise, they will be unable to focus well enough to make choices. The cognitive and organisational shortcuts, described above, help them do it almost instantly.
They also use their experience to help them decide – often very quickly – what evidence is policy-relevant under the circumstances. Relevance can include:
How it relates to the policy problem as they define it (Step 1).
If it relates to a feasible solution (Step 2).
If it is timely, available, understandable, and actionable.
If it seems credible, such as from groups representing wider populations, or from people they trust.
They use a specific shortcut: relying on expertise.
However, the vague idea of trusting or not trusting experts is a nonsense, largely because it is virtually impossible to set a clear boundary between relevant/irrelevant experts and find a huge consensus on (exactly) what is happening and what to do. Instead, in political systems, we define the policy problem or find other ways to identify the most relevant expertise and exclude other sources of knowledge.
In the UK Government’s case, it appears to be relying primarily on expertise from its own general scientific advisers, medical and public health advisers, and – perhaps more controversially – advisers on behavioural public policy.
Not a thread but an interesting exchange on #coronavirus & Behavioural Sciences including readings from @LiamDelaneyEcon https://t.co/7Yn89XwOk6
Here’s my article on why I wish my fellow psychologists and “behavioural scientists” would just stop talking about the coronavirus: https://t.co/ofjJWdIY9v
Right now, it is difficult to tell exactly how and why it relies on each expert (at least when the expert is not in a clearly defined role, in which case it would be irresponsible not to consider their advice). Further, there are regular calls on Twitter for ministers to be more open about their decisions.
Key point from @jameswilsdon 'It is problematic if political choices are being made and then the science advice system has to front them up. There needs to be a clearer sense of where science advice ends and political judgement begins.'https://t.co/TjLCJDZijO via @timeshighered
However, don’t underestimate the problems of identifying why we make choices, then justifying one expert or another (while avoiding pointless arguments), or prioritising one form of advice over another. Look, for example, at the kind of short-cuts that intelligent people use, which seem sensible enough, but would receive much more intense scrutiny if presented in this way by governments:
Sophisticated speculation by experts in a particular field, shared widely (look at the RTs), but questioned by other experts in another field:
2. This all assumes I'm correct in what I think the govt are doing and why. I could be wrong – and wouldn't be surprised. But it looks to me like. . .
— Professor Ian Donald 3.5% (@iandonald_psych) March 13, 2020
As many have said, it would be good to get an official version of this, with acknowledged uncertainties and sources of evidence https://t.co/jxgoysYb3L
But what happened is that they have as a group fallen into a logical error in their attempts to model what will bring this epidemic under control. They have not appreciated that the answer to this question is adaptive behavior change. 3/17
It would be really helpful to project risk of covid death with and without mitigation strategies? Possible to map with inside / outside projections (ie what we gain/ don’t gain with current measures ?)
Experts in one field trusting certain experts in another field based on personal or professional interaction:
Lots of concern about UK's approach to #COVID19. I'm not an epidemiologist or a virologist (=> can't judge the detail) but I knew Patrick Vallance before he was famous and I believe he is a man of integrity. Same for Chris Whitty. Science, not politics, is driving their thinking.
— Trisha Greenhalgh 😷 #BlackLivesMatter (@trishgreenhalgh) March 14, 2020
Experts in one field not trusting a government’s approach based on its use of one (of many) sources of advice:
Why is UK government listening to the ‘nudge unit’ on the pandemic, and not expert epidemiologists and the WHO? You would think the ‘anti-experts’ approach would have at least on this occasion, with so many lives at risk, given way to a scientific approach https://t.co/QZIicXYpsj
Experts representing a community of experts, criticising another expert (Prof John Ashton), for misrepresenting the amount of expert scepticism of government experts (yes, I am trying to confuse you):
The Chief Medical Officer @CMO_England and his team have the 100% support and backing of the Public Health community. Every DPH I know thinks he is doing an amazing job in difficult circumstances Sorry but JRA is just demonstrating he is out of touch on this https://t.co/ExmOjEgum0
Expert debate on how well policymakers are making policy based on expert advice
Disagree.
Not much audible consensus amongst scientists anywhere for UK approach. Science can only illuminate value judgements yet now used a shield for determining them. UK science advice has always been characterised by old boys, political operators. Blurring is concerning. https://t.co/iBt07QfvqH
Finding quite-sensible ways to trust certain experts over others, such as because they can be held to account in some way (and may be relatively worried about saying any old shit on the internet):
My current approach to making sense of conflicting expert opinion on #coronavirus: no expert is infallible, but some are accountable and others are not, and I will value the opinions of those who are accountable above the opinions of those who are not.
There are many more examples in which the shortcut to expertise is fine, but not particularly better than another shortcut (and likely to include a disproportionately high number of white men with STEM backgrounds).
Update: of course, they are better than the volume trumps expertise approach:
This meme is spreading (you could say, in a not very funny joke, that it has gone viral). The WHO Director-General did not say this (brief thread). https://t.co/3eMfy70tKZ
For what it’s worth, I tend to favour experts who:
(a) establish the boundaries of their knowledge, (b) admit to high uncertainty about the overall problem:
After having spent considerable time thinking how to mitigate and manage this pandemic, and analysing the available data. I failed to identify the best course of action. Even worse, I'm not sure there is such a thing as an acceptable solution to the problem we are facing. (2/12)
— Prof Francois Balloux (@BallouxFrancois) March 14, 2020
I would challenge anyone to provide an accurate estimate of prevalence. The difference between models & real life is that with models we can set the parameters as if they are known. In real life these parameters are as clear as mud. Extract 04/13/2020 https://t.co/Qg2OrCo8tR
(c) (in this case) make it clear that they are working on scenarios, not simple prediction
I am deeply uncomfortable with the message that UK is actively pursuing ‘herd immunity’ as the main COVID-19 strategy. Our group’s scenario modelling has focused on reducing two main things: peak healthcare demand and deaths… 1/
"Prediction models are just estimates of what might happen and a model is only as good as the data that goes into it." https://t.co/KXDILsbZgr via @ConversationUK
(d) examine critically the too-simple ideas that float around, such as the idea that the UK Government should emulate ‘what works’ somewhere else
It's easy to say 'let's just do what Wuhan did', but the measures there have involved a change to daily life that really has been unimaginable in scale and impact. And as we've seen, China cannot sustain them indefinitely. 3/
A lot of my colleagues in the @LSHTM modelling centre (@cmmid_lshtm) have been working extremely hard to help expand the COVID-19 evidence base over the past two months. I'd like to take a moment to highlight some of their work… 1/
8. There's no gotcha-ism. Updating your models and predictions in light of new evidence and new inferential methods and insightful counterpoints from colleagues isn't a sign of weakness, it's *doing science*.
I do not agree with this interpretation. Multiple papers that tested people at high risk found that asymptomatic infection is relatively uncommon, in the range of 6-32%. https://t.co/gv5e2upEwz
(e) situate their own position (in Prof Sridhar’s case, for mass testing) within a broader debate
Scientific community is well-intentioned but split in two camps: one argues why sacrifice short-term social/economic well-being if everyone will get virus regardless, & other which says we have to buy time in short-term & save lives now while figuring out exit plan.
How much effort does your govt want to put into suppressing this outbreak? There is no quick fix or easy solution. S.Korea & Germany show what huge govt effort, planning, strong leadership, & doing utmost to protect population look like. Do everything v. do minimum.
Been saying 3 objectives for weeks. Not to attack anyone, but to highlight what we have learned so far: 1. Testing, tracing, isolating 2. Protect health workers with PPE & testing 3. Buy time for NHS
Two weeks ago Boris Johnson said Britain was aiming to eventually test 250,000 people a day. The reality is still far off the aspiration https://t.co/2SHX40B9Ul
My new blog on whether Covid raises everyone’s relative risk of dying by a similar amount. https://t.co/76NSNuDJ3i Latest ONS data shows that, of recent death registrations, the proportion linked to Covid does not depend on age.
However, note that most of these experts are from a very narrow social background, and from very narrow scientific fields (first in modelling, then likely in testing), despite the policy problem being largely about (a) who, and how many people, a government should try to save, and (b) how far a government should go to change behaviour to do it (Update 2.4.20: I wrote that paragraph before adding so many people to the list). It is understandable to defer in this way during a crisis, but it also contributes to a form of ‘depoliticisation’ that masks profound choices that benefit some people and leave others vulnerable to harm.
— Louis M M Coiffait (@LouisMMCoiffait) April 6, 2020
See also: ‘What’s important is social distancing’ coronavirus testing ‘is a side issue’, says Deputy Chief Medical Officer [Professor Jonathan Van-Tam talks about the important distinction between a currently available test to see if someone has contracted the virus (an antigen test) and a forthcoming test to see if someone has had and recovered from COVID-19 (an antibody test)]. The full interview is here (please feel free to ignore the editorialising of the uploader):
We might need to change our criteria to decide on capacity and resources. COVID-19 showed that the standard CEO approach of doing more with less is no good. German planners have apparently safely ignored this holy managerial mantra. @Breconomicshttps://t.co/MKi3f1Pueq
Cross country comparisons of the efficacy of anti covid19 policies are going to be hard. There are so many likely inputs; and data on them is scarce and noisy.
The UK Govts chief medical officer has conceded that Germany “got ahead” in testing people for Covid-19 and said the UK needed to learn from that. Ministers have been challenged repeatedly during the pandemic over their failure to increase testing. https://t.co/V0bgcMR7l0
He says there is not as much scrutiny as we might normally wish and says concerns raised about human rights, the length of powers and need for safeguards should be heeded in Westminster. He also commits to legislate for reporting requirements for use of powers by SG 4/5
Glad Scottish Government recognise need for ethical guidance on Covid 19, and hope they can say more on human rights in next version https://t.co/GiyTd2Xksu
This is an excellent initiative from @policescotland – commissioning @johndscott to provide independent scrutiny of new Coronavirus Emergency Powers. Policing is by consent of the people, this step hopefully gives further public reassurance on the application of powers https://t.co/6MtrqdTqIm
Unprecedented restrictions are in force in order to limit social contact and slow the spread of the coronavirus. But the govt and police must make clear what is enforceable and what is guidance if they are to retain the trust and confidence of the public https://t.co/ieLcg2qVE5pic.twitter.com/mBOK2fppH2
— Institute for Gov (@instituteforgov) April 5, 2020
The wider policymaking environment: 2. Limited control
Second, policymakers engage in a messy and unpredictable world in which no single ‘centre’ has the power to turn a policy recommendation into an outcome. I normally use the following figure to think through the nature of a complex and unwieldy policymaking environment of which no ‘centre’ of government has full knowledge or control.
It helps us identify (further) the ways in which we can reject the idea that the UK Prime Minister and colleagues can fully understand and solve policy problems:
Actors. The environment contains many policymakers and influencers spread across many levels and types of government (‘venues’).
For example, consider how many key decisions that (a) have been made by organisations not in the UK central government, and (b) are more or less consistent with its advice, including:
Devolved governments announcing their own healthcare and public health responses (although the level of UK coordination seems more significant than the level of autonomy).
Public sector employers initiating or encouraging at-home working (and many Universities moving quickly from in-person to online teaching)
Private organisations cancelling cultural and sporting events.
There’s some coverage today suggesting Scotland proposing different policy to rest of UK on over 70s. This isn’t so. The policy of social distancing, not isolation, set out here by @jasonleitch is the policy all 4 nations have been discussing at COBR – and will do so again today. https://t.co/D89nwUDZTb
This is interesting, particularly the contrast with the approach to Brexit. The key difference is that Brexit blurred the boundaries between reserved and devolved competences in a way that health does not. https://t.co/4kSIcQFmJf
Context and events. Policy solutions relate to socioeconomic context and events which can be impossible to ignore and out of the control of policymakers. The coronavirus, and its impact on so many aspects on population health and wellbeing, is an extreme example of this problem.
Networks, Institutions, and Ideas. Policymakers and influencers operate in subsystems (specialist parts of political systems). They form networks or coalitions built on the exchange of resources or facilitated by trust underpinned by shared beliefs or previous cooperation. Many different parts of government have practices driven by their own formal and informal rules. Formal rules are often written down or known widely. Informal rules are the unwritten rules, norms and practices that are difficult to understand, and may not even be understood in the same way by participants. Political actors relate their analysis to shared understandings of the world – how it is, and how it should be – which are often so established as to be taken for granted. These dominant frames of reference establish the boundaries of the political feasibility of policy solutions. These kinds of insights suggest that most policy decisions are considered, made, and delivered in the name of – but not in the full knowledge of – government ministers.
Trial and error policymaking in complex policymaking systems (17.3.20)
One way of viewing the UK's COVID-19 policy is that it changed to reflect changing evidence. That is fair; it's both how science-guided policy *should* work, and how I think the govt's advisors are seeing it, as per the Imperial paper. But… 1/
There are many ways to conceptualise this policymaking environment, but few theories provide specific advice on what to do, or how to engage effectively in it. One notable exception is the general advice that comes from complexity theory, including:
Law-like behaviour is difficult to identify – so a policy that was successful in one context may not have the same effect in another.
Policymaking systems are difficult to control; policy makers should not be surprised when their policy interventions do not have the desired effect.
Policy makers in the UK have been too driven by the idea of order, maintaining rigid hierarchies and producing top-down, centrally driven policy strategies. An attachment to performance indicators, to monitor and control local actors, may simply result in policy failure and demoralised policymakers.
Policymaking systems or their environments change quickly. Therefore, organisations must adapt quickly and not rely on a single policy strategy.
On this basis, there is a tendency in the literature to encourage the delegation of decision-making to local actors:
Rely less on central government driven targets, in favour of giving local organisations more freedom to learn from their experience and adapt to their rapidly-changing environment.
To deal with uncertainty and change, encourage trial-and-error projects, or pilots, that can provide lessons, or be adopted or rejected, relatively quickly.
Encourage better ways to deal with alleged failure by treating ‘errors’ as sources of learning (rather than a means to punish organisations) or setting more realistic parameters for success/ failure (although see this example and this comment).
Encourage a greater understanding, within the public sector, of the implications of complex systems and terms such as ‘emergence’ or ‘feedback loops’.
In other words, this literature, when applied to policymaking, tends to encourage a movement from centrally driven targets and performance indicators towards a more flexible understanding of rules and targets by local actors who are more able to understand and adapt to rapidly-changing local circumstances.
Now, just imagine the UK Government taking that advice right now. I think it is fair to say that it would be condemned continuously (even more so than right now). Maybe that is because it is the wrong way to make policy in times of crisis. Maybe it is because too few people are willing and able to accept that the role of a small group of people at the centre of government is necessarily limited, and that effective policymaking requires trial-and-error rather than a single, fixed, grand strategy to be communicated to the public. The former highlights policy that changes with new information and perspective. The latter highlights errors of judgement, incompetence, and U-turns. In either case, the advice is changing as estimates of the coronavirus’ impact change:
I think this tension, in the way that we understand UK government, helps explain some of the criticism that it faces when changing its advice to reflect changes in its data or advice. This criticism becomes intense when people also question the competence or motives of ministers (and even people reporting the news) more generally, leading to criticism that ranges from mild to outrageous:
Incredible detail in this FT story: up until last week, the UK was basing its coronavirus control policy on a model based on hospitalisation rates for 😲a different disease😲 with lower rates of intensive care need than coronavirus pic.twitter.com/7rJYh9sqg2
Laura Kuenssberg says (BBC) that, “The science has changed.” This is not true. The science has been the same since January. What has changed is that govt advisors have at last understood what really took place in China and what is now taking place in Italy. It was there to see.
We can’t keep changing our #COVID19 control policies whenever the results of the “mathematical modelling” change. We need to implement standard WHO-approved epidemic control policies hard and fast, as well as providing more support to frontline NHS staff. https://t.co/HAM9OqbmqW
There may be perfectly valid or at least debatable reasons for each but obfuscation does not help public to understand uncertainty around decisions. In other words, not communicating rationale = incompetence (as in incompetent in terms of state craft, not nec individual decision)
One wonders if Brit leaders have decided that the ultimate way to cut national budgets is to cull the herd of the weak, those who require costly NHS care, and pray for "herd immunity" among the rest. Cruel, cost effective #COVID19 strategy?@richardhorton1
For me, this casual reference to a government policy to ‘cull the heard of the weak’ is outrageous, but you can find much worse on Twitter. It reflects wider debate on whether ‘herd immunity’ is or is not government policy. Much of it relates to interpretation of government statements, based on levels of trust/distrust in the UK Government, its Prime Minister and Secretaries of State, and the Prime Minister’s special adviser
I have enormous respect for the SAGE team and scientific advisors trying to understand the situation & inform the UK's response. If this article is accurate & partisan hacks were deliberately sacrificing lives based on their information, its scandalous. A week ago I was saying… https://t.co/WYsHbj6o0a
If you read the whole article you will see that Dominic Cummings has been, for the last 10 days, the most zealous advocate of a tough lockdown. Which is what his critics seem to want. The world is not black and white
1. Wilful misinterpretation (particularly on Twitter). For example, in the early development and communication of policy, Boris Johnson was accused (in an irresponsibly misleading way) of advocating for herd immunity rather than restrictive measures.
Below is one of the most misleading videos of its type. Look at how it cuts each segment into a narrative not provided by ministers or their advisors (see also this stinker):
The herd immunity strategy would’ve likely caused hundreds of thousands of deaths. They even told us so.
2. The accentuation of a message not being emphasised by government spokespeople.
See for example this interview, described by Sky News (13.3.20) as: The government’s chief scientific adviser Sir Patrick Vallance has told Sky News that about 60% of people will need to become infected with coronavirus in order for the UK to enjoy “herd immunity”. You might be forgiven for thinking that he was on Sky extolling the virtues of a strategy to that end (and expressing sincere concerns on that basis). This was certainly the write-up in respected papers like the FT (UK’s chief scientific adviser defends ‘herd immunity’ strategy for coronavirus). Yet, he was saying nothing of the sort. Rather, when prompted, he discussed herd immunity in relation to the belief that COVID-19 will endure long enough to become as common as seasonal flu.
The same goes for Vallance’s interview on the same day (13.3.20) during Radio 4’s Today programme (transcribed by the Spectator, which calls Vallance the author, and gives ittheheadline “How ‘herd immunity’ can help fight coronavirus” as if it is his main message). The Today Programme also tweeted only 30 seconds to single out that brief exchange:
Sir Patrick Vallance, the govt chief scientific adviser, says the thinking behind current approach to #coronavirus is to try and "reduce the peak" and to build up a "degree of herd immunity so that more people are immune to the disease". #R4Today
Yet, clearly his overall message – in this and other interviews – was that some interventions (e.g. staying at home; self-isolating with symptoms) would have bigger effects than others (e.g. school closures; prohibiting mass gatherings) during the ‘flattening of the peak’ strategy (‘What we don’t want is everybody to end up getting it in a short period of time so that we swamp and overwhelm NHS services’). Rather than describing ‘herd immunity’ as a strategy, he is really describing how to deal with its inevitability (‘Well, I think that we will end up with a number of people getting it’).
For anyone who thinks it was all obvious in January and February reading these minutes is a sobering experience. What comes over is the real uncertainty about what could be foretold from the Chinese experience and the ease with which the disease could be transmitted.4/n
Toby Young 'expert'. Nobody, including the Oxford team, believes this is true. Shame on The Sun for publishing this irresponsible rubbish. Shame on Toby Young for cynical misrepresentation of the science. pic.twitter.com/17hrOPW9b8
[OK, that proved to be a big departure from the trial-and-error discussion. Here we are, back again]
In some cases, maybe people are making the argument that trial-and-error is the best way to respond quickly, and adapt quickly, in a crisis but that the UK Government version is not what, say, the WHO thinks of as good kind of adaptive response. It is not possible to tell, at least from the general ways in which they justify acting quickly.
Dr Michael J Ryan, Executive Director at WHO. An off the cuff answer to a question at today's virtual press conference. Inspiring stuff! pic.twitter.com/Q4EUs8V1dG
The coronavirus is an extreme example of a general situation: policymakers will always have very limited knowledge of policy problems and control over their policymaking environment. They make choices to frame problems narrowly enough to seem solvable, rule out most solutions as not feasible, make value judgements to try help some more than others, try to predict the results, and respond when the results to not match their hopes or expectations.
This is not a message of doom and despair. Rather, it encourages us to think about how to influence government, and hold policymakers to account, in a thoughtful and systematic way that does not mislead the public or exacerbate the problem we are seeing.
Further reading, until I can think of a better conclusion:
This series of ‘750 words’ posts summarises key texts in policy analysis and tries to situate policy analysis in a wider political and policymaking context. Note the focus on whose knowledge counts, which is not yet a big feature of this crisis.
These series of 500 words and 1000 words posts (with podcasts) summarise concepts and theories in policy studies.
The scientific response to COVID-19 demands speed. But changing incentives and norms in academic science may be pushing the enterprise toward fast science at the expense of good science. Read Dan Sarewitz's editor's journal in the Spring 2020 ISSUES: https://t.co/JSSS45eTze
— Issues in Science and Technology (@ISSUESinST) April 7, 2020
#politvirus Public Health has always been #political because it’s actions impact on politics, economics, commercial interests, personal freedoms – this becomes most obvious in crisis – it will be key to analyse the political responses to #Covid_19 if we want to be better prepared https://t.co/JkUZrVeAxv
An assessment of the Government's response to date – written by Chair of Global Health at Edinburgh University..Prof Devi Sridhar https://t.co/N31QtFmQ2p
This is a really important paper. Partisanship is a huge influence on timing of state public health measures- Republican governors and Trump majorities slow adoption of measures. This might have big mortality effects in a few weeks. https://t.co/BEOAM69aSw
One reason Germany has so many ventilators (and intensive care beds) given in The Times: Not just more money in the system but design of hospital payment rates through the insurance system has driven up ICU investment be hospital managers pic.twitter.com/7R062IJI2k
This is worrying. Singapore was held up as one of the models for how to control #COVID19 through a sophisticated programme of testing and tracing without having to resort to the kinds of lockdowns many other countries are going through. https://t.co/6R0LY4IhuO
Today’s reflection- A number of Swedes are pretty shit at social distancing and probably need at least a modicum of discipline- the notion that we should be so very different here is ludicrous
WATCH: "Some countries initially talked about herd immunity as a strategy. In New Zealand we never, ever considered that. It would have meant tens of thousands of New Zealanders dying" — New Zealand Prime Minister @jacindaardernpic.twitter.com/W1ei6OUUyr
An online form to report lockdown breaches undermines the trust we have in each other – unhelpful in even the most benign of situations, and downright dangerous right now, writes Michael Macaulay. https://t.co/XCrnpfEVJt
Speechless every time someone says that this was totally unexpected & nobody saw this coming. See chapter 3: 'Preparing for the Worst: A Rapidly Spreading, Lethal Respiratory Pathogen' published by the @WHO Sept 2019. https://t.co/23qTrz7dN9
People are facing uncertainty for days, weeks & months. We need a manageable way forward to keep the health, social & economic costs at a minimum. My analysis on where COVID-19 response is heading & how it could end: https://t.co/qLDm8tv8a9
I wish the late great Mick Moran were still around – it feels like the next chapter of his analysis of the modern British state urgently needs to be written. https://t.co/ffxegGKVCu
I’m writing a book about @ExtinctionR. Here are some thoughts about today’s controversy. 1. This may or may not be a legit XR group. 2. That may matter because it may be done in order to smear XR & climate activism generally 1/n https://t.co/NyQhbv53a3
Cautionary words for anyone tempted to say "this must be good for the climate" or, worse, "this shows we can tackle climate change".
COVID19 is a re-framing of the climate issues – a dramatically changed context for the response – but those climate issues haven't gone away. https://t.co/gixVwnk6gq
We are concerned about regulation rollbacks which impact the food system slipping under the radar at the moment – we are going to be keeping an eye on things and use hashtag #Covid19Watchdoghttps://t.co/niinfSWv6f#TuesdayThoughts
A study in politics – when leadership fails. Would those that were ready to bash the @WHO take the time to read this? The critical issue for all countries is: what did they do after the PHEIC was declared? Why did USA and China not work together to fight #COVID19https://t.co/zK7hcEbU80
Not a single voice from the Global South – that’s not good enough if you are reporting on a global organisation – @who has 194 member states – it’s not the donors who should be running it #COVID19#geopoliticshttps://t.co/xqTaFEYLap
— Professor Paul Cairney (@CairneyPaul) April 9, 2020
The Australian #COVID19 modelling was published today. My thanks to James McCaw (@j_mccaw) for checking this thread. I’ll do two threads – one explaining the results and how we might interpret them; and another to try to explain how these models work. https://t.co/O6sGwggY9W
This was so predictable. Ireland was already closing pubs and restaurants. #COVIDー19 . Cheltenham Festival ‘spread coronavirus across country’ | News | The Times https://t.co/QVQnJblJiH
— Andrea Catherwood (@acatherwoodnews) April 3, 2020
expert comments about comparison between the COVID-19 situation in Ireland and the UKhttps://t.co/y4OBOhdbtT
‘Policy analysis’ describes the identification of a policy problem and possible solutions.
Classic models of policy analysis are client-oriented. Most texts identify the steps that a policy analysis should follow, from identifying a problem and potential solutions, to finding ways to predict and evaluate the impact of each solution. Each text describes this process in different ways, as outlined in Boxes 1-5. However, for the most part, they follow the same five steps:
Define a policy problem identified by your client.
Identify technically and politically feasible solutions.
Use value-based criteria and political goals to compare solutions.
Predict the outcome of each feasible solution.
Make a recommendation to your client.
Further, they share the sense that analysts need to adapt pragmatically to a political environment. Assume that your audience is not an experienced policy analyst. Assume a political environment in which there is limited attention or time to consider problems, and some policy solutions will be politically infeasible. Describe the policy problem for your audience: to help them see it as something worthy of their energy. Discuss a small number of possible solutions, the differences between them, and their respective costs and benefits. Keep it short with the aid of visual techniques that sum up the issue concisely, to minimise cognitive load and make the problem seem solvable.
Box 1. Bardach (2012) A Practical Guide for Policy Analysis
‘Define the problem’. Provide a diagnosis of a policy problem, using rhetoric and eye-catching data to generate attention.
‘Assemble some evidence’. Gather relevant data efficiently.
‘Construct the alternatives’. Identify the relevant and feasible policy solutions that your audience might consider.
‘Select the criteria’. Typical value judgements relate to efficiency, equity and fairness, the trade-off between individual freedom and collective action, and the extent to which a policy process involves citizens in deliberation.
‘Project the outcomes’. Focus on the outcomes that key actors care about (such as value for money), and quantify and visualise your predictions if possible.
‘Confront the trade-offs’. Compare the pros and cons of each solution, such as how much of a bad service policymakers will accept to cut costs.
‘Decide’. Examine your case through the eyes of a policymaker.
‘Tell your story’. Identify your target audience and tailor your case. Weigh up the benefits of oral versus written presentation. Provide an executive summary. Focus on coherence and clarity. Keep it simple and concise. Avoid jargon.
Box 2. Dunn (2017) Public Policy Analysis
What is the policy problem to be solved? Identify its severity, urgency, cause, and our ability to solve it. Don’t define the wrong problem, such as by oversimplifying.
What effect will each potential policy solution have? ‘Forecasting’ methods can help provide ‘plausible’ predictions about the future effects of current/ alternative policies.
Which solutions should we choose, and why? Normative assessments are based on values such as ‘equality, efficiency, security, democracy, enlightenment’ and beliefs about the preferable balance between state, communal, and market/ individual solutions (2017: 6; 205).
What were the policy outcomes? ‘Monitoring is crucial because it is difficult to predict policy success, and unintended consequences are inevitable (2017: 250).
Did the policy solution work as intended? Did it improve policy outcomes? Try to measure the outcomes your solution, while noting that evaluations are contested (2017: 332-41).
Box 3. Meltzer and Schwartz (2019) Policy Analysis as Problem Solving
‘Define the problem’. Problem definition is a political act of framing, as part of a narrative to evaluate the nature, cause, size, and urgency of an issue.
‘Identify potential policy options (alternatives) to address the problem’. Identify many possible solutions, then select the ‘most promising’ for further analysis (2019: 65).
‘Specify the objectives to be attained in addressing the problem and the criteria to evaluate the attainment of these objectives as well as the satisfaction of other key considerations (e.g., equity, cost, equity, feasibility)’.
‘Assess the outcomes of the policy options in light of the criteria and weigh trade-offs between the advantages and disadvantages of the options’.
‘Arrive at a recommendation’. Make a preliminary recommendation to inform an iterative process, drawing feedback from clients and stakeholder groups (2019: 212).
‘Engage in problem definition’. Define the nature of a policy problem, and the role of government in solving it, while engaging with many stakeholders (2012: 3; 58-60).
‘Propose alternative responses to the problem’. Identify how governments have addressed comparable problems, and a previous policy’s impact (2012: 21).
‘Choose criteria for evaluating each alternative policy response’. ‘Effectiveness, efficiency, fairness, and administrative efficiency’ are common (2012: 21).
‘Project the outcomes of pursuing each policy alternative’. Estimate the cost of a new policy, in comparison with current policy, and in relation to factors such as savings to society or benefits to certain populations.
‘Identify and analyse trade-offs among alternatives’. Use your criteria and projections to compare each alternative in relation to their likely costs and benefits.
‘Report findings and make an argument for the most appropriate response’. Client-oriented advisors identify the beliefs of policymakers and tailor accordingly (2012: 22).
Box 5 Weimer and Vining (2017) Policy Analysis: Concepts and Practice
‘Write to Your Client’. Having a client such as an elected policymaker requires you to address the question they ask, by their deadline, in a clear and concise way that they can understand (and communicate to others) quickly (2017: 23; 370-4).
‘Understand the Policy Problem’. First, ‘diagnose the undesirable condition’. Second, frame it as ‘a market or government failure (or maybe both)’.
‘Be Explicit About Values’ (and goals). Identify (a) the values to prioritise, such as ‘efficiency’, ‘equity’, and ‘human dignity’, and (b) ‘instrumental goals’, such as ‘sustainable public finance or political feasibility’, to generate support for solutions.
‘Specify Concrete Policy Alternatives’. Explain potential solutions in sufficient detail to predict the costs and benefits of each ‘alternative’ (including current policy).
‘Predict and Value Impacts’. Short deadlines dictate that you use ‘logic and theory, rather than systematic empirical evidence’ to make predictions efficiently (2017: 27)
‘Consider the Trade-Offs’. Each alternatives will fulfil certain goals more than others. Produce a summary table to make value-based choices about trade-offs (2017: 356-8).
‘Make a Recommendation’. ‘Unless your client asks you not to do so, you should explicitly recommend one policy’ (2017: 28).
Incisive essay from @bailabomba on studying the use of research evidence through critical perspectives that center the marginalized. There is so too much good stuff in here to summarize via twitter (you should just read it). But let me point out a few things that resonated (1/n) https://t.co/nIahyIjwBo
Research and policy analysis for marginalized groups
For Doucet (2019: 1), it begins by describing the William T. Grant Foundation’s focus on improving the ‘use of research evidence’ (URE), and the key questions that we should ask when improving URE:
For what purposes do policymakers find evidence useful?
For example, usefulness could be defined by the researchers providing evidence, the policymakers using it, the stakeholders involved in coproduction, or the people affected by research and policy (compare with Bacchi, Stone and Who should be involved in the process of policy analysis?).
How do critical theories inform these questions? (compare with T. Smith)
First, they remind us that so-called ‘rational’ policy processes have incorporated research evidence to help:
‘maintain power hierarchies and accept social inequity as a given. Indeed, research has been historically and contemporaneously (mis)used to justify a range of social harms from enslavement, colonial conquest, and genocide, to high-stakes testing, disproportionality in child welfare services, and “broken windows” policing’ (Doucet, 2019: 2)
Second, they help us redefine usefulness in relation to:
‘how well research evidence communicates the lived experiences of marginalized groups so that the understanding of the problem and its response is more likely to be impactful to the community in the ways the community itself would want’ (Doucet, 2019: 3)
In that context, potential responses include to:
Recognise the ways in which research and policy combine to reproduce the subordination of social groups.
Specific mechanism include: judging marginalised groups harshly according to ‘Western, educated, industrialized, rich and democratic’ norms (‘WEIRD’)
Reject the idea that scientific research can be seen as objective or neutral (and that researchers are beyond reproach for their role in subordination).
Give proper recognition to ‘experiential knowledge’ and ‘transdiciplinary approaches’ to knowledge production, rather than privileging scientific knowledge.
Commit to social justice, to help ‘eliminate oppressions and to emancipate and empower marginalized groups’, such as by disrupting ‘the policies and practices that disproportionately harm marginalized groups’ (2019: 5-7)
Develop strategies to ‘center race’, ‘democratize’ research production, and ‘leverage’ transdisciplinary methods (including poetry, oral history and narrative, art, and discourse analysis – compare with Lorde) (2019: 10-22)
Policy analysis in a ‘racialized polity’
A key way to understand these processes is to use, and improve, policy theories to explain the dynamics and impacts of a racialized political system. For example, ‘policy feedback theory’ (PFT) draws on elements from historical institutionalism and SCPD to identify the rules, norms, and practices that reinforce subordination.
In particular, Michener’s (2019: 424) ‘Policy Feedback in a Racialized Polity’ develops a ‘racialized feedback framework (RFF)’ to help explain the ‘unrelenting force with which racism and White supremacy have pervaded social, economic, and political institutions in the United States’. Key mechanisms include (2019: 424-6):
‘Channelling resources’, in which the rules, to distribute government resources, benefit some social groups and punish others.
Examples include: privileging White populations in social security schemes and the design/ provision of education, and punishing Black populations disproportionately in prisons (2019: 428-32).
‘Generating interests’, in which ‘racial stratification’ is a key factor in the power of interest groups (and balance of power in them).
‘Shaping interpretive schema’, in which race is a lens through which actors understand, interpret, and seek to solve policy problems.
The ways in which centralization (making policy at the federal level) or decentralization influence policy design.
For example, the ‘historical record’ suggests that decentralization is more likely to ‘be a force of inequality than an incubator of power for people of color’ (2019: 433).
Insufficient attention to race and racism: what are the implications for policy analysis?
One potential consequence of this lack of attention to race, and the inequalities caused by racism in policy, is that we place too much faith in the vague idea of ‘pragmatic’ policy analysis.
Throughout the 750 words series, you will see me refer generally to the benefits of pragmatism:
In that context, pragmatism relates to the idea that policy analysis consists of ‘art and craft’, in which analysts assess what is politically feasible if taking a low-risk client-oriented approach.
In this context, pragmatism may be read as a euphemism for conservatism and status quo protection.
In other words, other posts in the series warn against too-high expectations for entrepreneurial and systems thinking approaches to major policy change, but they should not be read as an excuse to reject ambitious plans for much-needed changes to policy and policy analysis (compare with Meltzer and Schwartz, who engage with this dilemma in client-oriented advice).
This post forms one part of the Policy Analysis in 750 words series overview and connects to previous posts on complexity.The first 750 words tick along nicely, then there is a picture of a cat hanging in there baby to signal where it can all go wrong. I updated it (22.6.20) to add category 11 then again (30.9.20) when I realised that the former category 11 was a lot like 6.
There are a million-and-one ways to describe systems and systems thinking. These terms are incredibly useful, but also at risk of meaning everything and therefore nothing (compare with planning and consultation).
We need to acknowledge these limitations properly, to accept our limitations, and avoid the mechanistic language of ‘policy levers’ which exaggerate human or government control.
Complex systems thinking could be the future of policymaking.
Six meanings of complex systems in policy and policymaking
Let’s begin by trying to clarify many meanings of complex system and relate them to systems thinking storylines.
For example, you will encounter three different meanings of complex system in this series alone, and each meaning presents different implications for systems thinking:
Policy outcomes seem to ‘emerge’ from policymaking systems in the absence of central government control. As such, we should rely less on central government driven targets (in favour of local discretion to adapt to environments), encourage trial-and-error learning, and rethink the ways in which we think about government ‘failure’ (see, for example, Hallsworth on ‘system stewardship’, the OECD on ‘Systemic Thinking for Policy Making‘, and this thread)
Systems thinking is about learning and adapting to the limits to policymaker control.
Dunn (2017: 73) describes the interdependent nature of problems:
“Subjectively experienced problems – crime, poverty, unemployment, inflation, energy, pollution, health, security – cannot be decomposed into independent subsets without running the risk of producing an approximately right solution to the wrong problem. A key characteristic of systems of problems is that the whole is greater – that is, qualitatively different – than the simple sum of its parts” (contrast with Meltzer and Schwartz on creating a ‘boundary’ to make problems seem solveable).
Systems thinking is about addressing policy problems holistically.
Used to explain the transition from unsustainable to sustainable energy systems.
Systems thinking is about identifying the role of new technologies, protected initially in a ‘niche’, and fostered by a supportive ‘social and political environment’.
Used to explain how and why policy actors might cooperate to manage finite resources.
Systems thinking is about identifying the conditions under which actors develop layers of rules to foster trust and cooperation.
Performing the metaphor of systems
Governments often use the language of complex systems – rather loosely – to indicate an awareness of the interconnectedness of things. They often perform systems thinking to give the impression that they are thinking and acting differently, but without backing up their words with tangible changes to policy instruments.
Systems thinking is about projecting the sense that (a) policy and policymaking is complicated, but (b) governments can still look like they are in control.
Four more meanings of systems thinking
Now, let’s compare these storylines with a small sample of wider conceptions of systems thinking:
Systems thinking was about the human ability to turn potential chaos into well-managed systems (such as ‘large technical systems’ to distribute energy)
The new way of accepting complexity but seeking to make an impact
Based on the idea that we can identify ‘leverage points’, or the places that help us ‘intervene in a system’ (see Meadows then compare with Arnold and Wade).
Systems thinking is about the human ability to use a small shift in a system to produce profound changes in that system.
A way to rethink cause-and-effect
Based on the idea that current research methods are too narrowly focused on linearity rather than the emergent properties of systems of behaviour (for example, Rutter et al on how to analyse the cumulative effect of public health interventions, and Greenhalgh on responding more effectively to pandemics).
Systems thinking is about rethinking the ways in which governments, funders, or professions conduct policy-relevant research on social behaviour.
How can we clarify systems thinking and use it effectively in policy analysis?
Now, imagine you are in a room of self-styled systems thinkers, and that no-one has yet suggested a brief conversation to establish what you all mean by systems thinking. I reckon you can make a quick visual distinction by seeing who looks optimistic.
I’ll be the morose-looking guy sitting in the corner, waiting to complain about ambiguity, so you would probably be better off sitting next to Luke Craven who still ‘believes in the power of systems thinking’.
If you can imagine some amalgam of these pessimistic/ optimistic positions, perhaps the conversation would go like this:
Reasons to expect some useful collaboration.
Some of these 10 discussions seem to complement each other. For example:
We can use 3 and 9 to reject one narrow idea of ‘evidence-based policymaking’, in which the focus is on (a) using experimental methods to establish cause and effect in relation to one policy instrument, without showing (b) the overall impact on policy and outcomes (e.g. compare FNP with more general ‘families’ policy).
1-3 and 10 might be about the need for policy analysts to show humility when seeking to understand and influence complex policy problems, solutions, and policymaking systems.
In other words, you could define systems thinking in relation to the need to rethink the ways in which we understand – and try to address – policy problems. If so, you can stop here and move on to the next post. There is no benefit to completing this post.
Reasons to expect the same old frustrating discussions based on no-one defining terms well enough (collectively) to collaborate effectively (beyond using the same buzzwords).
Although all of these approaches use the language of complex systems and systems thinking, note some profound differences:
Holding on versus letting go.
Some are about intervening to take control of systems or, at least, make a disproportionate difference from a small change.
Some are about accepting our inability to understand, far less manage, these systems.
Talking about different systems.
Some are about managing policymaking systems, and others about social systems (or systems of policy problems), without making a clear connection between both endeavours.
For example, if you use approach 9 to rethink societal cause-and-effect, are you then going to pretend that you can use approach 7 to do something about it? Or, will our group have a difficult discussion about the greater likelihood of 6 (metaphorical policymaking) in the context of 1 (the inability of governments to control the policymaking systems we need to solve the problems raised by 9).
In that context, the reason that I am sitting in the corner, looking so morose, is that too much collective effort goes into (a) restating, over and over and over again, the potential benefits of systems thinking, leaving almost no time for (b) clarifying systems thinking well enough to move on to these profound differences in thinking. Systems thinking has not even helped us solve these problems with systems thinking.
Throughout this series you may notice three different conceptions about the scope of policy analysis:
‘Ex ante’ (before the event) policy analysis. Focused primarily on defining a problem, and predicting the effect of solutions, to inform current choice (as described by Meltzer and Schwartz and Thissen and Walker).
‘Ex post’ (after the event) policy analysis. Focused primarily on monitoring and evaluating that choice, perhaps to inform future choice (as described famously by Weiss).
Some combination of both, to treat policy analysis as a continuous (never-ending) process (as described by Dunn).
As usual, these are not hard-and-fast distinctions, but they help us clarify expectations in relation to different scenarios.
The impact of old-school ex ante policy analysis
Radin provides a valuable historical discussion of policymaking with the following elements:
a small number of analysts, generally inside government (such as senior bureaucrats, scientific experts, and – in particular- economists),
giving technical or factual advice,
about policy formulation,
to policymakers at the heart of government,
on the assumption that policy problems would be solved via analysis and action.
This kind of image signals an expectation for high impact: policy analysts face low competition, enjoy a clearly defined and powerful audience, and their analysis is expected to feed directly into choice.
Radin goes on to describe a much different, modern policy environment: more competition, more analysts spread across and outside government, with a less obvious audience, and – even if there is a client – high uncertainty about where the analysis fits into the bigger picture.
Yet, the impetus to seek high and direct impact remains.
This combination of shifting conditions but unshifting hopes/ expectations helps explain a lot of the pragmatic forms of policy analysis you will see in this series, including:
Keep it catchy, gather data efficiently, tailor your solutions to your audience, and tell a good story (Bardach)
Speak with an audience in mind, highlight a well-defined problem and purpose, project authority, use the right form of communication, and focus on clarity, precision, conciseness, and credibility ( Smith)
Address your client’s question, by their chosen deadline, in a clear and concise way that they can understand (and communicate to others) quickly (Weimer and Vining)
Client-oriented advisors identify the beliefs of policymakers and anticipate the options worth researching (Mintrom)
Identify your client’s resources and motivation, such as how they seek to use your analysis, the format of analysis they favour (make it ‘concise’ and ‘digestible’), their deadline, and their ability to make or influence the policies you might suggest (Meltzer and Schwartz).
‘Advise strategically’, to help a policymaker choose an effective solution within their political context (Thissen and Walker).
Focus on producing ‘policy-relevant knowledge’ by adapting to the evidence-demands of policymakers and rejecting a naïve attachment to ‘facts speaking for themselves’ or ‘knowledge for its own sake’ (Dunn).
The impact of research and policy evaluation
Many of these recommendations are familiar to scientists and researchers, but generally in the context of far lower expectations about their likely impact, particularly if those expectations are informed by policy studies (compare Oliver & Cairney with Cairney & Oliver).
In that context, Weiss’ work is a key reference point. It gives us a menu of ways in which policymakers might use policy evaluation (and research evidence more widely):
to inform solutions to a problem identified by policymakers
as one of many sources of information used by policymakers, alongside ‘stakeholder’ advice and professional and service user experience
as a resource used selectively by politicians, with entrenched positions, to bolster their case
as a tool of government, to show it is acting (by setting up a scientific study), or to measure how well policy is working
as a source of ‘enlightenment’, shaping how people think over the long term (compare with this discussion of ‘evidence based policy’ versus ‘policy based evidence’).
In other words, researchers may have a role, but they struggle (a) to navigate the politics of policy analysis, (b) find the right time to act, and (c) to secure attention, in competition with many other policy actors.
The potential for a form of continuous impact
Dunn suggests that the idea of ‘ex ante’ policy analysis is misleading, since policymaking is continuous, and evaluations of past choices inform current choices. Think of each policy analysis steps as ‘interdependent’, in which new knowledge to inform one step also informs the other four. For example, routine monitoring helps identify compliance with regulations, if resources and services reach ‘target groups’, if money is spent correctly, and if we can make a causal link between the policy solutions and outcomes. Its impact is often better seen as background information with intermittent impact.
Key conclusions to bear in mind
The demand for information from policy analysts may be disproportionately high when policymakers pay attention to a problem, and disproportionately low when they feel that they have addressed it.
Common advice for policy analysts and researchers often looks very similar: keep it concise, tailor it to your audience, make evidence ‘policy relevant’, and give advice (don’t sit on the fence). However, unless researchers are prepared to act quickly, to gather data efficiently (not comprehensively), to meet a tight brief for a client, they are not really in the impact business described by most policy analysis texts.
A lot of routine, continuous, impact tends to occur out of the public spotlight, based on rules and expectations that most policy actors take for granted.
When describing ‘the policy sciences’, Lasswell distinguishes between:
‘knowledge of the policy process’, to foster policy studies (the analysis of policy)
‘knowledge in the process’, to foster policy analysis (analysis for policy)
The lines between each approach are blurry, and each element makes less sense without the other. However, the distinction is crucial to help us overcome the major confusion associated with this question:
Does policymaking proceed through a series of stages?
The short answer is no.
The longer answer is that you can find about 40 blog posts (of 500 and 1000 words) which compare (a) a stage-based model called the policy cycle, and (b) the many, many policy concepts and theories that describe a far messier collection of policy processes.
In a nutshell, most policy theorists reject this image because it oversimplifies a complex policymaking system. The image provides a great way to introduce policy studies, and serves a political purpose, but it does more harm than good:
Prescriptively, it gives you rotten advice about the nature of your policymaking task (for more on these points, see this chapter, article, article, and series).
Why does the stages/ policy cycle image persist? Two relevant explanations
It arose from a misunderstanding in policy studies
In another nutshell, Chris Weible and I argue (in a secret paper) that the stages approach represents a good idea gone wrong:
If you trace it back to its origins, you will find Lasswell’s description of decision functions: intelligence, recommendation, prescription, invocation, application, appraisal and termination.
These functions correspond reasonably well to a policy cycle’s stages: agenda setting, formulation, legitimation, implementation, evaluation, and maintenance, succession or termination.
However, Lasswell was imagining functional requirements, while the cycle seems to describe actual stages.
In other words, if you take Lasswell’s list of what policy analysts/ policymakers need to do, multiple it by the number of actors (spread across many organisations or venues) trying to do it, then you get the multi-centric policy processes described by modern theories. If, instead, you strip all that activity down into a single cycle, you get the wrong idea.
It is a functional requirement of policy analysis
This description should seem familiar, because the classic policy analysis texts appear to describe a similar series of required steps, such as:
define the problem
identify potential solutions
choose the criteria to compare them
evaluate them in relation to their predicted outcomes
In addition, studies of policy analysis in action suggest that:
an individual analyst’sneed for simple steps, to turn policymaking complexity into useful heuristics and pragmatic strategies,
should not be confused with
what actually happens when many policy analysts, influencers, and policymakers interact in policy processes (see Radin, and Brans, Geva-May, and Howlett).
What you need versus what you can expect
Overall, this discussion of policy studies and policy analysis reminds us of a major difference between:
Functional requirements. What you need from policymaking systems, to (a) manage your task (the 5-8 step policy analysis) and (b) understand and engage in policy processes (the simple policy cycle).
Actual processes and outcomes. What policy concepts and theories tell us about bounded rationality (which limit the comprehensiveness of your analysis) and policymaking complexity (which undermines your understanding and engagement in policy processes).
Of course, I am not about to provide you with a solution to these problems.
When describing ‘the policy sciences’, Lasswell distinguishes between:
‘knowledge of the policy process’, to foster policy studies (the analysis of policy)
‘knowledge in the process’, to foster policy analysis (analysis for policy)
The idea is that both elements are analytically separable but mutually informative: policy analysis is crucial to solving real policy problems, policy studies inform the feasibility of analysis, the study of policy analysts informs policy studies, and so on.
Both elements focus on similar questions – such as What is policy? – and explore their descriptive (what do policy actors do?) and prescriptive (what should they do?) implications.
Policy studies focus on the power to reduce ambiguity rather than simply the provision of information to reduce uncertainty. In other words, the power to decide whose interpretation of policy problems counts, and therefore to decide what information is policy-relevant.
This (unequal) competition takes place within a policy process over which no actor has full knowledge or control.
The classic 5-8 step policy analysis texts focus on how to define policy problems well, but they vary somewhat in their definition of doing it well (see also C.Smith):
Bardach recommends using rhetoric and eye-catching data to generate attention
Weimer and Vining and Mintrom recommend beginning with your client’s ‘diagnosis’, placing it in a wider perspective to help analyse it critically, and asking yourself how else you might define it (see also Bacchi, Stone)
Meltzer and Schwartz and Dunn identify additional ways to contextualise your client’s definition, such as by generating a timeline to help ‘map’ causation or using ‘problem-structuring methods’ to compare definitions and avoid making too many assumptions on a problem’s cause.
Thissen and Walker compare ‘rational’ and ‘argumentative’ approaches, treating problem definition as something to be measured scientifically or established rhetorically (see also Riker).
These approaches compare with more critical accounts that emphasise the role of power and politics to determine whose knowledge is relevant (L.T.Smith) and whose problem definition counts (Bacchi, Stone). Indeed, Bacchi andStone provide a crucial bridge between policy analysis and policy studies by reflecting on what policy analysts do and why.
What is the policy solution?
In policy studies, it is common to identify counterintuitive or confusing aspects of policy processes, including:
Few studies suggest that policy responses actually solve problems (and many highlight their potential to exacerbate them). Rather, ‘policy solutions’ is shorthand for proposed or alleged solutions.
Problem definition often sets the agenda for the production of ‘solutions’, but note the phrase solutions chasing problems (when actors have their ‘pet’ solutions ready, and they seek opportunities to promote them).
Policy studies: problem definition informs the feasibility and success of solutions
Generally speaking, to define the problem is to influence assessments of the feasibility of solutions:
Technical feasibility. Will they work as intended, given the alleged severity and cause of the problem?
Political feasibility. Will they receive sufficient support, given the ways in which key policy actors weigh up the costs and benefits of action?
Policy studies highlight the inextricable connection between technical and political feasibility. Put simply, (a) a ‘technocratic’ choice about the ‘optimality’ of a solution is useless without considering who will support its adoption, and (b) some types of solution will always be a hard sell, no matter their alleged effectiveness (Box 2.3 below).
In turn, problem definition informs: the ways in which actors will frame any evaluation of policy success, and the policy-relevance of the evidence to evaluate solutions. Simple examples include:
If you define tobacco in relation to: (a) its economic benefits, or (b) a global public health epidemic, evaluations relate to (a) export and taxation revenues, or (b) reductions in smoking in the population.
If you define ‘fracking’ in relation to: (a) seeking more benefits than costs, or (b) minimising environmental damage and climate change, evaluations relate to (a) factors such as revenue and effective regulation, or simply (b) how little it takes place.
Policyanalysis: recognising and pushing boundaries
Policy analysis texts tend to accommodate these insights when giving advice:
Bardach recommends identifying solutions that your audience might consider, perhaps providing a range of options on a notional spectrum of acceptability.
Smith highlights the value of ‘precedent’, or relating potential solutions to previous strategies.
Weimer and Vining identify the importance of ‘a professional mind-set’ that may be more important than perfecting ‘technical skills’
Mintrom notes that some solutions are easier to sell than others
Meltzer and Schwartz describe the benefits of making a preliminary recommendation to inform an iterative process, drawing feedback from clients and stakeholder groups
Dunn warns against too-narrow forms of ‘evidence based’ analysis which undermine a researcher’s ability to adapt well to the evidence-demands of policymakers
Thissen and Walker relate solution feasibility to a wide range of policy analysis ‘styles’
Still, note the difference in emphasis.
Policy analysis education/ training may be about developing the technical skills to widen definitions and apply many criteria to compare solutions.
Policy studies suggest that problem definition and a search for solutions takes place in an environment where many actors apply a much narrower lens and are not interested in debates on many possibilities (particularly if they begin with a solution).
I have exaggerated this distinction between each element, but it is worth considering the repeated interaction between them in practice: politics and policymaking provide boundaries for policy analysis, analysis could change those boundaries, and policy studies help us reflect on the impact of analysts.
I’ll take a quick break, then discuss how this conclusion relates to the idea of ‘entrepreneurial’ policy analysis.
One aim of this series is to combine insights from policy research (1000, 500) and policy analysis texts.
In this case, modern theories of the policy process help you identify your audience and their capacity to follow your advice. This simple insight may have a profound impact on the advice you give.
Policy analysis for an ideal-type world
For our purposes, an ideal-type is an abstract idea, which highlights hypothetical features of the world, to compare with ‘real world’ descriptions. It need not be an ideal to which we aspire. For example, comprehensive rationality describes the ideal type, and bounded rationality describes the ‘real world’ limitations to the ways in which humans and organisations process information.
Imagine writing policy analysis in the ideal-type world of a single powerful ‘comprehensively rational’ policymaker at the heart of government, making policy via an orderly policy cycle.
Your audience would be easy to identify, your analysis would be relatively simple, and you would not need to worry about what happens after you make a recommendation for policy change.
You could adopt a simple 5-8 step policy analysis method, use widely-used tools such as cost-benefit analysis to compare solutions, and know where the results would feed into the policy process.
I have perhaps over-egged this ideal-type pudding, but I think a lot of traditional policy analyses tapped into this basic idea and focused more on the science of analysis than the political and policymaking context in which it takes place (see Radin and Brans, Geva-May, and Howlett).
This image is a key feature of policy process theories, which describe:
Many policymakers and influencers spread across many levels and types of government (as the venues in which authoritative choice takes place). Consequently, it is not a straightforward task to identify and know your audience, particularly if the problem you seek to solve requires a combination of policy instruments controlled by different actors.
Each venue resembles an institution driven by formal and informal rules. Formal rules are written-down or widely-known. Informal rules are unwritten, difficult to understand, and may not even be understood in the same way by participants. Consequently, it is difficult to know if your solution will be a good fit with the standard operating procedures of organisations (and therefore if it is politically feasible or too challenging).
Policymakers and influencers operate in ‘subsystems’, forming networks built on resources such as trust or coalitions based on shared beliefs. Effective policy analysis may require you to engage with – or become part of – such networks, to allow you to understand the unwritten rules of the game and encourage your audience to trust the messenger. In some cases, the rules relate to your willingness to accept current losses for future gains, to accept the limited impact of your analysis now in the hope of acceptance at the next opportunity.
Actors relate their analysis to shared understandings of the world – how it is, and how it should be – which are often so well-established as to be taken for granted. Common terms include paradigms, hegemons, core beliefs, and monopolies of understandings. These dominant frames of reference give meaning to your policy solution. They prompt you to couch your solutions in terms of, for example, a strong attachment to evidence-based cases in public health, value for money in treasury departments, or with regard to core principles such as liberalism or socialism in different political systems.
Your solutions relate to socioeconomic context and the events that seem (a) impossible to ignore and (b) out of the control of policymakers. Such factors range from a political system’s geography, demography, social attitudes, and economy, while events can be routine elections or unexpected crises.
What would you recommend under these conditions? Rethinking 5-step analysis
There is a large gap between policymakers’ (a) formal responsibilities versus (b) actual control of policy processes and outcomes. Even the most sophisticated ‘evidence based’ analysis of a policy problem will fall flat if uninformed by such analyses of the policy process. Further, the terms of your cost-benefit analysis will be highly contested (at least until there is agreement on what the problem is, and how you would measure the success of a solution).
Modern policy analysis texts try to incorporate such insights from policy theories while maintaining a focus on 5-8 steps. For example:
Meltzer and Schwartz contrast their ‘flexible’ and ‘iterative’ approach with a too- rigid ‘rationalistic approach’.
Bardachand Dunn emphasise the value of political pragmatism and the ‘art and craft’ of policy analysis.
Weimer and Vininginvest 200 pages in economic analyses of markets and government, often highlighting a gap between (a) our ability to model and predict economic and social behaviour, and (b) what actually happens when governments intervene.
Mintrom invites you to see yourself as a policy entrepreneur, to highlight the value of of ‘positive thinking’, creativity, deliberation, and leadership, and perhaps seek ‘windows of opportunity’ to encourage new solutions. Alternatively, a general awareness of the unpredictability of events can prompt you to be modest in your claims, since the policymaking environment may be more important (than your solution) to outcomes.
Thissen and Walker focus more on a range of possible roles than a rigid 5-step process.
Without this wider perspective, we are focusing on policy analysis as a process rather than considering the political context in which analysts use it.
Additional posts on entrepreneurs and ‘systems thinking’ [to be added] encourage us to reflect on the limits to policy analysis in multi-centric policymaking systems.
One aim of this series is to combine insights from policy research (1000, 500) and policy analysis texts.
If we take key insights from policy theories seriously, we can use them to identify (a) the constraints to policy analytical capacity, and (b) the ways in which analysts might address them. I use the idea of policy analyst archetypes to compare a variety of possible responses.
Key constraints to policy analytical capacity
Terms like ‘bounded rationality’ highlight major limits on the ability of humans and organisations to process information.
Humans use heuristics or cognitive shortcuts to process enough information to make choices, and institutions are the rules used by organisations to limit information processing.
Policy actors need to find ways to act, with incomplete information about the problem they seek to solve and the likely impact of their ‘solution’.
They gather information to help reduce uncertainty, but problem definition is really about exercising power to reduce ambiguity: select one way to interpret a problem (at the expense of most others), and limit therefore limit the relevance and feasibility of solutions.
This context informs how actors might use the tools of policy analysis. Key texts in this series highlight the use of tools to establish technical feasibility (will it work as intended?), but policymakers also select tools for their political feasibility (who will support or oppose this measure?).
How might policy analysts address these constraints ethically?
Most policy analysis texts (in this series) consider the role of professional ethics and values during the production of policy analysis. However, they also point out that there is not a clearly defined profession and associated code of conduct (e.g. see Adachi). In that context, let us begin with some questions about the purpose of policy analysis and your potential role:
Is your primary role to serve individual clients or some notion of the ‘public good’?
Should you maximise your role as an individual or play your part in a wider profession?
What is the balance between the potential benefits of individual ‘entrepreneurship’ and collective ‘co-productive’ processes?
Which policy analysis techniques should you prioritise?
What forms of knowledge and evidence count in policy analysis?
What does it mean to communicate policy analysis responsibly?
Should you provide a clear recommendation or encourage reflection?
Policy analysis archetypes: pragmatists, entrepreneurs, manipulators, storytellers, and decolonisers
In that context, I have created a story of policy analysis archetypes to identify the elements that each text emphasises.
The pragmatic policy analyst
Bardach provides the classic simple, workable, 8-step system to present policy analysis to policymakers while subject to time and resource-pressed political conditions.
Dunn also uses Wildavsky’s famous phrase ‘art and craft’ to suggest that scientific and ‘rational’ methods can only take us so far.
The professional, client–oriented policy analyst
Weimer and Vining provide a similar 7-step client-focused system, but incorporating a greater focus on professional development and economic techniques (such as cost-benefit-analysis) to emphasise a particular form of professional analyst.
Meltzer and Schwartz also focus on advice to clients, but with a greater emphasis on a wide variety of methods or techniques (including service design) to encourage the co-design of policy analysis with clients.
The communicative policy analyst
C. Smith focuses on how to write and communicate policy analysis to clients in a political context.
Compare with Spiegelhalter and Gigerenzer on how to communicate responsibly when describing uncertainty, probability, and risk.
The manipulative policy analyst.
Riker helps us understand the relationship between two aspects of agenda setting: the rules/ procedures to make choice, and the framing of policy problems and solutions.
The entrepreneurial policy analyst
Mintrom shows how to combine insights from studies of policy entrepreneurship and policy analysis, to emphasise the benefits of collaboration and creativity.
The questioning policy analyst
Bacchi analyses the wider context in which people give and use such advice, to identify the emancipatory role of analysis and encourage policy analysts to challenge dominant social constructions of problems and populations.
The storytelling policy analyst
Stone identifies the ways in which people use storytelling and argumentation techniques to define problems and justify solutions. This process is about politics and power, not objectivity and optimal solutions.
The decolonizing policy analyst.
L.T. Smith does not describe policy analysis directly, but shows how the ‘decolonization of research methods’ can inform the generation and use of knowledge.
Compare with Hindess on the ways in which knowledge-based hierarchies rely on an untenable, circular logic.
Compare with Michener’s thread, discussing Doucet’s new essay on (a) the role of power and knowledge in limiting (b) the ways in which we gather evidence to analyse policy problems.
Incisive essay from @bailabomba on studying the use of research evidence through critical perspectives that center the marginalized. There is so too much good stuff in here to summarize via twitter (you should just read it). But let me point out a few things that resonated (1/n) https://t.co/nIahyIjwBo
Using archetypes to define the problem of policy analysis
Studies of the field (e.g. Radin plus Brans, Geva-May, and Howlett) suggest that there are many ways to do policy analysis. Further, as Thissen and Walker describe, such roles are notmutually exclusive, your views on their relative value could change throughout the process of analysis, and you could perform many of these roles.
Further, each text describes multiple roles, and some seem clustered together:
pragmatic, client-orientated, and communicative could sum-up the traditional 5-8 step approaches, while
questioning, storytelling, and decolonizing could sum up an important (‘critical’) challenge to narrow ways of thinking about policy analysis and the use of information.
Still, the emphasis matters.
Each text is setting an agenda or defining the problem of policy analysis more-or-less in relation to these roles. Put simply, the more you are reading about economic theory and method, the less you are reading about dominance and manipulation.
See also The new policy sciences for a discussion of how these issues inform Lasswell’s original vision for the policy sciences (combining the analysis of and for policy).
These choices are not mutually exclusive, but there are key tensions between them that should not be ignored, such as when we ask:
how many people should be involved in policy analysis?
whose knowledge counts?
who should control policy design?
Perhaps we can only produce a sensible combination of the two if we clarify their often very different implications for policy analysis. Let’s begin with one story for each and see where they take us.
A story of ‘evidence-based policymaking’
One story of ‘evidence based’ policy analysis is that it should be based on the best available evidence of ‘what works’.
Often, the description of the ‘best’ evidence relates to the idea that there is a notional hierarchy of evidence according to the research methods used.
At the top would be the systematic review of randomised control trials, and nearer the bottom would be expertise, practitioner knowledge, and stakeholder feedback.
This kind of hierarchy has major implications for policy learning and transfer, such as when importing policy interventions from abroad or ‘scaling up’ domestic projects.
Put simply, the experimental method is designed to identify the causal effect of a very narrowly defined policy intervention. Its importation or scaling up would be akin to the description of medicine, in which the evidence suggests the causal effect of a specific active ingredient to be administered with the correct dosage. A very strong commitment to a uniform model precludes the processes we might associate with co-production, in which many voices contribute to a policy design to suit a specific context (see also: the intersection between evidence and policy transfer).
A story of co-production in policymaking
One story of ‘co-produced’ policy analysis is that it should be ‘reflexive’ and based on respectful conversations between a wide range of policymakers and citizens.
Often, the description is of the diversity of valuable policy relevant information, with scientific evidence considered alongside community voices and normative values.
This rejection of a hierarchy of evidence also has major implications for policy learning and transfer. Put simply, a co-production method is designed to identify the positive effect – widespread ‘ownership’ of the problem and commitment to a commonly-agreed solution – of a well-discussed intervention, often in the absence of central government control.
Its use would be akin to a collaborative governance mechanism, in which the causal mechanism is perhaps the process used to foster agreement (including to produce the rules of collective action and the evaluation of success) rather than the intervention itself. A very strong commitment to this process precludes the adoption of a uniform model that we might associate with narrowly-defined stories of evidence based policymaking.
Where can you find these stories in the 750-words series?
There are 101 approaches to co-production, but let’s see if we can get away with two categories:
Co-producing policy (policymakers, analysts, stakeholders). Some key principles can be found in Ostrom’s work and studies of collaborative governance.
Co-producing research to help make it more policy-relevant (academics, stakeholders). See the Social Policy and Administration special issue ‘Inside Co-production’ and Oliver et al’s ‘The dark side of coproduction’ to get started.
My interest has been to understand how governments juggle competing demands, such as to (a) centralise and localise policymaking, (b) encourage uniform and tailored solutions, and (c) embrace and reject a hierarchy of evidence. What could possibly go wrong when they entertain contradictory objectives? For example:
Paul Cairney (2019) “The myth of ‘evidence based policymaking’ in a decentred state”, forthcoming in Public Policy and Administration(Special Issue, The Decentred State) (accepted version)
Paul Cairney (2019) ‘The UK government’s imaginative use of evidence to make policy’, British Politics, 14, 1, 1-22 Open AccessPDF
Paul Cairney and Kathryn Oliver (2017) ‘Evidence-based policymaking is not like evidence-based medicine, so how far should you go to bridge the divide between evidence and policy?’ Health Research Policy and Systems (HARPS), DOI: 10.1186/s12961-017-0192-xPDF
Paul Cairney (2017) “Evidence-based best practice is more political than it looks: a case study of the ‘Scottish Approach’”, Evidence and Policy, 13, 3, 499-515 PDF
‘The Handbook … covers … the state of the art knowledge about the science, art and craft of policy analysis in different countries, at different levels of government and by all relevant actors in and outside government who contribute to the analysis of problems and the search for policy solutions’ (Brans et al, 2017: 1).
This book focuses on the interaction between (in Lasswell’s terms) ‘analysis for policy’ (policy analysis) and ‘analysis of policy’ (policy process research). In other words,
what can the study of policy analysis tell us about policymaking, and
what can studies of policymaking tell budding policy analysts about the nature of their task in relation to their policymaking environment?
Brans et al’s (2017: 1-6) opening discussion suggests that this task is rather unclear and complicated. They highlight the wide range of activity described by the term ‘policy analysis’:
The scope of policy analysis is wide, and its meaning unclear
Analysts can be found in many levels and types of government, in bodies holding governments to account, and outside of government, including interest groups, think tanks, and specialist firms (such as global accountancy or management consultancy firms – Saint-Martin, 2017).
Further, ‘what counts’ as policy analysis can relate to the people that do it, the rules they follow, the processes in which they engage, the form of outputs, and the expectations of clients (Veselý, 2017: 103; Vining and Boardman, 2017: 264).
The role of a policy analyst varies remarkably in relation to context
It varies over time, policy area, type of government (such as central, subnational, local), country, type of political system (e.g. majoritarian and consensus democracies), and ‘policy style’.
Analysis involves ‘science, art and craft’ and the rules are written and unwritten
The process of policy analysis – such as to gather and analyse information, define problems, design and compare solutions, and give policy advice – includes ‘applied social and scientific research as well as more implicit forms of practical knowledge’, and ‘both formal and informal professional practices’ (see also studies of institutions and networks).
The policy process is complex.
It is difficult to identify a straightforward process in which analysts are clearly engaged in multiple, well-defined ‘stages’ of policymaking.
Key principles and practices can be institutionalised, contested, or non-existent.
The idea of policy analysis principles – ‘of transparency, effectiveness, efficiency and accountability through systematic and evidence-based analysis’ – may be entrenched in places like the US but not globally.
In some political systems (particularly in the ‘Anglo-Saxon family of nations’), the most-described forms of policy analysis (in the 750 words series) may be taken for granted (2017: 4):
Even so, the status of science and expertise is often contested, particularly in relation to salient and polarised issues, or more generally:
During ‘attempts by elected politicians to restore the primacy of political judgement in the policymaking process, at the expense of technical or scientific evidence’ (2017: 5).
When the ‘blending of expert policy analysis with public consultation and participation’ makes ‘advice more competitive and contested’ (2017: 5).
When evidence based really means evidence informed, given that there are many legitimate claims to knowledge, and evidence forms one part of a larger process of policy design (van Nispen and de Jong, 2017: 153).
In many political systems, there may be less criticism of the idea of ‘systematic and evidence-based analysis’ because there less capacity to process information. It is difficult to worry about excessively technocratic approaches if they do not exist (a point that CW made to me just before I read this book).
Implications for policy analysis
It is difficult to think of policy analysis as a ‘profession’.
We may wonder if ‘policy analysis’ can ever be based on common skills and methods (such as described by Scott, 2017, and in Weimer and Vining), connected to ‘formal education and training’, a ‘a code of professional conduct’, and the ability of organisations to control membership (Adachi, 2017: 28; compare with Radin and Geva-May).
Policy analysis is a loosely-defined collection of practices that vary according to context.
Policy analysis may, instead, be considered a collection of ‘styles’ (Hassenteufel and Zittoun, 2017), influenced by:
competing analytical approaches in different political systems (2017: 65)
bureaucratic capacity for analysis (Mendez and Dussauge-Laguna, 2017: 82)
a relative tendency to contract out analysis (Veselý, 2017: 113)
the types and remits of advisory bodies (e.g. are they tasked simply with offering expert advice, or also to encourage wider participation to generate knowledge?) (Crowley and Head, 2017)
the level of government in which analysts work, such as ‘subnational’ (Newman, 2017) or ‘local’ (Lundin and Öberg, 2017)
the type of activity, such as when (‘performance’) budgeting analysis is influenced heavily by economic methods and ‘new public management’ reforms (albeit with limited success, followed by attempts at reform) (van Nispen and de Jong, 2017: 143-52)
Policy analysis can also describe a remarkably wide range of activity, including:
Public inquiries (Marier, 2017)
Advice to MPs, parliaments, and their committees (Wolfs and De Winter, 2017)
The strategic analysis of public opinion or social media data (Rothmayr Allison, 2017; Kuo and Cheng, 2017)
A diverse set of activities associated with ‘think tanks’ (Stone and Ladi, 2017) and ‘political party think tanks’ (Pattyn et al, 2017)
Analysis for and by ‘business associations’ (Vining and Boardman, 2017), unions (Schulze and Schroeder, 2017), and voluntary/ non-profit organisations (Evans et al, 2017), all of whom juggle policy advice to government with keeping members on board.
The more-or-less policy relevant work of academic researchers (Blum and Brans, 2017; compare with Dunn and see the EBPM page).
The analysis of and for policy is not so easy to separate in practice.
When defining policy analysis largely as a collection of highly-variable practices, in complex policymaking systems, we can see the symbiotic relationship between policy analysis and policy research. Studying policy analysis allows us to generate knowledge of policy processes. Policy process research demonstrates that the policymaking context influences how we think about policy analysis.
Policy analysis education and training is incomplete without policy process research
Put simply, we should not assume that graduates in ‘policy analysis’ will enter a central government with high capacity, coherent expectations, and a clear demand for the same basic skills. Yet, Fukuyama argues that US University programmes largely teach students:
‘a battery of quantitative methods … applied econometrics, cost-benefit analysis, decision analysis, and, most recently, use of randomized experiments for program evaluation’ that ‘will tell you what the optimal policy should be’, but not ‘how to achieve that outcome. The world is littered with optimal policies that don’t have a snowball’s chance in hell of being adopted’.
In that context, additional necessary skills include: stakeholder mapping, to identify who is crucial to policy success, defining policy problems in a way that stakeholders and policymakers can support, and including those actors continuously during a process of policy design and delivery. These skills are described at more length by Radin and Geva May, while Botha et al (2017) suggest that the policy analysis programmes (across North American and European Universities) offer a more diverse range of skills (and support for experiential learning) than Fukuyama describes.
Please see the Policy Analysis in 750 words series overview before reading the summary. This book is a whopper, with almost 500 pages and 101 (excellent) discussions of methods, so 800 words over budget seems OK to me. If you disagree, just read every second word. By the time you reach the cat hanging in there baby you are about 300 (150) words away from the end.
‘Policy analysis is a process of multidisciplinary inquiry aiming at the creation, critical assessment, and communication of policy-relevant knowledge … to solve practical problems … Its practitioners are free to choose among a range of scientific methods, qualitative as well as quantitative, and philosophies of science, so long as these yield reliable knowledge’ (Dunn, 2017: 2-3).
Dunn (2017: 4) describes policy analysis as pragmatic and eclectic. It involves synthesising policy relevant (‘usable’) knowledge, and combining it with experience and ‘practical wisdom’, to help solve problems with analysis that people can trust.
This exercise is ‘descriptive’, to define problems, and ‘normative’, to decide how the world should be and how solutions get us there (as opposed to policy studies/ research seeking primarily to explain what happens).
Dunn contrasts the ‘art and craft’ of policy analysts with other practices, including:
The idea of ‘best practice’ characterised by 5-step plans.
In practice, analysis is influenced by: the cognitive shortcuts that analysts use to gather information; the role they perform in an organisation; the time constraints and incentive structures in organisations and political systems; the expectations and standards of their profession; and, the need to work with teams consisting of many professions/ disciplines (2017: 15-6)
The cost (in terms of time and resources) of conducting multiple research and analytical methods is high, and highly constrained in political environments (2017: 17-8; compare with Lindblom)
The naïve attachment to ‘facts speak for themselves’ or ‘knowledge for its own sake’ undermines a researcher’s ability to adapt well to the evidence-demands of policymakers (2017: 68; 4 compare with Why don’t policymakers listen to your evidence?).
To produce ‘policy-relevant knowledge’ requires us to ask five questions before (Qs1-3) and after (Qs4-5) policy intervention (2017: 5-7; 54-6):
What is the policy problem to be solved?
For example, identify its severity, urgency, cause, and our ability to solve it.
Don’t define the wrong problem, such as by oversimplifying or defining it with insufficient knowledge.
Key aspects of problems including ‘interdependency’ (each problem is inseparable from a host of others, and all problems may be greater than the sum of their parts), ‘subjectivity’ and ‘artificiality’ (people define problems), ‘instability’ (problems change rather than being solved), and ‘hierarchy’ (which level or type of government is responsible) (2017: 70; 75).
Problems vary in terms of how many relevant policymakers are involved, how many solutions are on the agenda, the level of value conflict, and the unpredictability of outcomes (high levels suggest ‘wicked’ problems, and low levels ‘tame’) (2017: 75)
‘Problem-structuring methods’ are crucial, to: compare ways to define or interpret a problem, and ward against making too many assumptions about its nature and cause; produce models of cause-and-effect; and make a problem seem solve-able, such as by placing boundaries on its coverage. These methods foster creativity, which is useful when issues seem new and ambiguous, or new solutions are in demand (2017: 54; 69; 77; 81-107).
Problem definition draws on evidence, but is primarily the exercise of power to reduce ambiguity through argumentation, such as when defining poverty as the fault of the poor, the elite, the government, or social structures (2017: 79; see Stone).
What effect will each potential policy solution have?
Many ‘forecasting’ methods can help provide ‘plausible’ predictions about the future effects of current/ alternative policies (Chapter 4 contains a huge number of methods).
‘Creativity, insight, and the use of tacit knowledge’ may also be helpful (2017: 55).
However, even the most-effective expert/ theory-based methods to extrapolate from the past are flawed, and it is important to communicate levels of uncertainty (2017: 118-23; see Spiegelhalter).
Which solutions should we choose, and why?
‘Prescription’ methods help provide a consistent way to compare each potential solution, in terms of its feasibility and predicted outcome, rather than decide too quickly that one is superior (2017: 55; 190-2; 220-42).
They help to combine (a) an estimate of each policy alternative’s outcome with (b) a normative assessment.
Normative assessments are based on values such as ‘equality, efficiency, security, democracy, enlightenment’ and beliefs about the preferable balance between state, communal, and market/ individual solutions (2017: 6; 205 see Weimer & Vining, Meltzer & Schwartz, and Stone on the meaning of these values).
For example, cost benefit analysis (CBA) is an established – but problematic – economics method based on finding one metric – such as a $ value – to predict and compare outcomes (2017: 209-17; compare Weimer & Vining, Meltzer & Schwartz, and Stone)
Cost effectiveness analysis uses a $ value for costs, but compared with other units of measurement for benefits (such as outputs per $) (2017: 217-9)
Although such methods help us combine information and values to compare choices, note the inescapable role of power to decide whose values (and which outcomes, affecting whom) matter (2017: 204)
What were the policy outcomes?
‘Monitoring’ methods help identify (say): levels of compliance with regulations, if resources and services reach ‘target groups’, if money is spent correctly (such as on clearly defined ‘inputs’ such as public sector wages), and if we can make a causal link between the policy inputs/ activities/ outputs and outcomes (2017: 56; 251-5)
Monitoring is crucial because it is so difficult to predict policy success, and unintended consequences are almost inevitable (2017: 250).
However, the data gathered are usually no more than proxy indicators of outcomes. Further, the choice of indicators reflect what is available, ‘particular social values’, and ‘the political biases of analysts’ (2017: 262)
The idea of ‘evidence based policy’ is linked strongly to the use of experiments and systematic review to identify causality (2017: 273-6; compare with trial-and-error learning in Gigerenzer, complexity theory, and Lindblom).
Did the policy solution work as intended? Did it improve policy outcomes?
Although we frame policy interventions as ‘solutions’, few problems are ‘solved’. Instead, try to measure the outcomes and the contribution of your solution, and note that evaluations of success and ‘improvement’ are contested (2017: 57; 332-41).
Policy evaluation is not an objective process in which we can separate facts from values.
Rather, values and beliefs are part of the criteria we use to gauge success (and even their meaning is contested – 2017: 322-32).
We can gather facts about the policy process, and the impacts of policy on people, but this information has little meaning until we decide whose experiences matter.
Overall, the idea of ‘ex ante’ (forecasting) policy analysis is a little misleading, since policymaking is continuous, and evaluations of past choices inform current choices.
Policy analysis methods are ‘interdependent’, and ‘knowledge transformations’ describes the impact of knowledge regarding one question on the other four (2017: 7-13; contrast with Meltzer & Schwartz, Thissen & Walker).
Developing arguments and communicating effectively
Dunn (2017: 19-21; 348-54; 392) argues that ‘policy argumentation’ and the ‘communication of policy-relevant knowledge’ are central to policymaking’ (See Chapter 9 and Appendices 1-4 for advice on how to write briefs, memos, and executive summaries and prepare oral testimony).
He identifies seven elements of a ‘policy argument’ (2017: 19-21; 348-54), including:
The claim itself, such as a description (size, cause) or evaluation (importance, urgency) of a problem, and prescription of a solution
The things that support it (including reasoning, knowledge, authority)
Incorporating the things that could undermine it (including any ‘qualifier’, the communication of uncertainty about current knowledge, and counter-arguments).
The key stages of communication (2017: 392-7; 405; 432) include:
‘Analysis’, focusing on ‘technical quality’ (of the information and methods used to gather it), meeting client expectations, challenging the ‘status quo’, albeit while dealing with ‘political and organizational constraints’ and suggesting something that can actually be done.
‘Documentation’, focusing on synthesising information from many sources, organising it into a coherent argument, translating from jargon or a technical language, simplifying, summarising, and producing user-friendly visuals.
‘Utilization’, by making sure that (a) communications are tailored to the audience (its size, existing knowledge of policy and methods, attitude to analysts, and openness to challenge), and (b) the process is ‘interactive’ to help analysts and their audiences learn from each other.
Policy analysis and policy theory: systems thinking, evidence based policymaking, and policy cycles
Dunn (2017: 31-40) situates this discussion within a brief history of policy analysis, which culminated in new ways to express old ambitions, such as to:
Note the huge difference between (a) policy analysis discussions of ‘systems thinking’ built on the hope that if we can understand them we can direct them, and (b) policy theory discussions that emphasise ‘emergence’ in the absence of central control (and presence of multi-centric policymaking).
Also note that Dunn (2017: 73) describes policy problems – rather than policymaking – as complex systems. I’ll write another post (short, I promise) on the many different (and confusing) ways to use the language of complexity.
Promote ‘evidence based policy’, as the new way to describe an old desire for ‘technocratic’ policymaking that accentuates scientific evidence and downplays politics and values (see also 2017: 60-4).
Note the idea of ‘erotetic rationality’ in which people deal with their lack of knowledge of a complex world by giving up on the idea of certainty (accepting their ‘ignorance’), in favour of a continuous process of ‘questioning and answering’.
This approach is a pragmatic response to the lack of order and predictability of policymaking systems, which limits the effectiveness of a rigid attachment to ‘rational’ 5 step policy analyses (compare with Meltzer & Schwartz).
Dunn (2017: 41-7) also provides an unusually useful discussion of the policy cycle. Rather than seeing it as a mythical series of orderly stages, Dunn highlights:
Lasswell’s original discussion of policymaking functions (or functional requirements of policy analysis, not actual stages to observe), including: ‘intelligence’ (gathering knowledge), ‘promotion’ (persuasion and argumentation while defining problems), ‘prescription’, ‘invocation’ and ‘application’ (to use authority to make sure that policy is made and carried out), and ‘appraisal’ (2017: 42-3).
The constant interaction between all notional ‘stages’ rather than a linear process: attention to a policy problem fluctuates, actors propose and adopt solutions continuously, actors are making policy (and feeding back on its success) as they implement, evaluation (of policy success) is not a single-shot document, and previous policies set the agenda for new policy (2017: 44-5).
In that context, it is no surprise that the impact of a single policy analyst is usually minimal (2017: 57). Sorry to break it to you. Hang in there, baby.
Please see the Policy Analysis in 750 words series overview before reading the summary. Please note that this is an edited book and the full list of authors (PDF) is here. I’m using the previous sentence as today’s excuse for not sticking to 750 words.
Our premise is that there is no single, let alone ‘one best’, way of conducting policy analyses (Thissen and Walker, 2013: 2)
Thissen and Walker (2013: 2) begin by identifying the proliferation of (a) policy analysts inside and outside government, (b) the many approaches and methods that could count as policy analysis (see Radin), and therefore (c) a proliferation of concepts to describe it.
Policy analysis, as the advice given to clients before they make a choice. Thissen and Walker (2013: 4) describe analysts working with a potential range of clients, when employed directly by governments or organisations, or acting more as entrepreneurs with multiple audiences in mind (compare with Bardach, Weimer & Vining, Mintrom).
Policy process research, as the study of such actors within policymaking systems (see 500 and 1000).
Policy theory: implications for policy analysis
Policy process research informs our understanding of policy analysis, identifying what analysts and their clients (a) can and cannot do, which informs (b) what they should do.
As Enserink et al (2012: 12-3) describe, policy analysis (analysis for policy) will differ profoundly if the policy process is ‘chaotic and messy’ rather than ‘neat and rational’.
The range of policy concepts and theories (analysis of policy) at our disposal helps add meaning to policy analysis as a practice. Like Radin, Enserink et al trace historic attempts to seek ‘rational’ policy analysis then conclude that modern theories – describing policymaking complexity – are ‘more in line with political reality’ (2012: 13-6).
As such, policy analysis shifts from:
(a) A centralised process with few actors inside government, to (b) a messy process including many policymakers and influencers, inside and outside government
(a) Translating science into policy, to (b) a competition to frame issues and assess policy-relevant knowledge
(a) An ‘optimal’ solution from one perspective, to (b) a negotiated solution based on many perspectives (in which optimality is contested)
(a) Analysing a policy problem/ solution with a common metric (such as cost benefit analysis), to (b) developing skills relating to: stakeholder analysis, network management, collaboration, mediation or conflict resolution based on sensitivity to the role of different beliefs, and the analysis of policymaking institutions to help resolve fragmentation (2013: 17-34).
Their Table 2.1 (2012: 35) outlines these potential differences (pop your reading glasses on …. now!):
In many cases, the role of an analyst remains uncertain. If we follow the ACF story, does an analyst appeal to one coalition or seek to mediate between them? If we follow MSA, do they wait for a ‘window of opportunity’ or seek to influence problem definition and motivation to adopt certain solutions?
Policy Analysis: implications for policy theory
In that context, rather than identify a 5-step plan for policy analysis, Mayer et al (2013: 43-50) suggest that policy analysts tend to perform one or more of six activities:
‘Research and analyze’, to collect information relevant to policy problems.
‘Design and recommend’, to produce a range of potential solutions.
‘Clarify values and arguments’, to identify potential conflicts and facilitate high quality debate.
‘Advise strategically’, to help a policymaker choose an effective solution within their political context.
‘Democratize’, to pursue a ‘normative and ethical objective: it should further equal access to, and influence on, the policy process for all stakeholders’ (2013: 47)
‘Mediate’, to foster many forms of cooperation between governments, stakeholders (including business), researchers, and/ or citizens.
Styles of policy analysis
Policy analysts do not perform these functions sequentially or with equal weight.
Rather, Mayer et al (2013: 50-5) describe ‘six styles of policy analysis’ that vary according to the analyst’s ‘assumptions about science (epistemology), democracy, learning, and change’ (and these assumptions may change during the process):
Rational, based on the idea that we can conduct research in a straightforward way within a well-ordered policy process (or modify the analysis to reflect limits to research and order).
Argumentative, based on a competition to define policy problems and solutions (see Stone).
Client advice, based on the assumption that analysis is part of a ‘political game’, and analysts bring knowledge of political strategy and policymaking complexity.
Participatory, to facilitate a more equal access to information and debate among citizens.
Process, based on the idea that the faithful adherence to good procedures aids high quality analysis (and perhaps mitigates an ‘erratic and volatile’ policy process)
Interactive, based on the idea that the rehearsal of many competing perspectives is useful to policymaker deliberations (compare with reflexive learning).
In turn, these styles prompt different questions to evaluate the activities associated with analysis (2013: 56):
In relation to the six policy analysis activities,
the criteria for good policy analysis include: the quality of knowledge, usefulness of advice to clients and stakeholders, quality of argumentation, pragmatism of advice, transparency of processes, and ability to secure a mediated settlement (2013: 58).
The positive role for analysts includes ‘independent scientist’ or expert, ‘ethicist’, ‘narrator’, ‘counsellor’, ‘entrepreneur’,’ democratic advocate’, or ‘facilitator’ (2013: 59).
Further, their – rather complicated – visualisations of these roles (e.g. p60; compare with the Appendix) project the (useful) sense that (a) individuals face a trade-off between roles (even if they seek to combine some), and (b) many people making many trade-offs adds up to a complex picture of activity.
Therefore, we should bear in mind that
(a) there exist some useful 5-step guides for budding analyst, but
(b) even if they adopt a simple strategy, analysts will also need to find ways to understand and engage with a complex policymaking systems containing a huge number of analysts, policymakers, and influencers.
Policy Analysis styles: implications for problem definition and policy design
Thissen (2013: 66-9) extends the focus on policymaking context and policy analysis styles to problem definition, including:
A rational approach relies on research knowledge to diagnose problems (the world is knowable, use the best scientific methods to produce knowledge, and subject the results to scientific scrutiny).
A ‘political game model’ emphasises key actors and their perspectives, value conflicts, trust, and interdependence (assess the potential to make deals and use skills of mediation and persuasion to secure them).
These different starting points influence they ways in which analysts might take steps to identify: how people perceive policy problems, if other definitions are more useful, how to identify a problem’s cause and effect, and the likely effect of a proposed solution, communicate uncertainty, and relate the results to a ‘policy arena’ with its own rules on resolving conflict and producing policy instruments (2013: 70-84; 93-4).
Similarly, Bots (2013: 114) suggests that these styles inform a process of policy design, constructed to change people’s minds during repeated interactions with clients (such as by appealing to scientific evidence or argumentation).
Bruijn et al (2013: 134-5) situate such activities in modern discussions of policy analysis:
In multi-centric systems, with analysts focused less on ‘unilateral decisions using command and control’ and more on ‘consultation and negotiation among stakeholders’ in networks.
The latter are necessary because there will always be contestation about what the available information tells us about the problem, often without a simple way to negotiate choices on solutions.
In relation to categories of policy problems, including
‘tamed’ (high knowledge/ technically solvable, with no political conflict)
‘untamed ethical/political’ (technically solvable, with high moral and political conflict)
‘untamed scientific’ (high consensus but low scientific knowledge)
Put simply, ‘rational’ approaches may help address low knowledge, while other skills are required to manage processes such as conflict resolution and stakeholder engagement (2013: 136-40)
Policy Analysis styles: implications for models
Part 2 of the book relates such styles (and assumptions about how ‘rational’ and comprehensive our analyses can be) to models of policy analysis. For example,
Walker and van Daalen (2013: 157-84) explore models designed to compare the status quo with a future state, often based on the (shaky) assumption that the world is knowable and we can predict with sufficient accuracy the impact of policy solutions.
Hermans and Cunningham (2013: 185-213) describe models to trace agent behaviour in networks and systems, and create multiple possible scenarios, which could help explore the ‘implementability’ of policies.
Walker et al (2013: 215-61) relate policy analysis styles to how analysts might deal with uncertainty.
Some models may serve primarily to reduce ‘epistemic’ uncertainty associated with insufficient knowledge about the future (perhaps with a focus on methods and statistical analysis).
Others may focus on resolving ambiguity, in which many actors may define/ interpret problems and feasible solutions in different ways.
Overall, this book contains one of the most extensive discussions of 101 different technical models for policy analysis, but the authors emphasise their lack of value without initial clarity about (a) our beliefs regarding the nature of policymaking and (b) the styles of analysis we should use to resolve policy problems. Few of these initial choices can be resolved simply with reference to scientific analysis or evidence.
Please see the Policy Analysis in 750 words series overview before reading the summary. This post might well represent the largest breach of the ‘750 words’ limit, so please get comfortable. I have inserted a picture of a cat hanging in there baby after the main (*coughs*) 1400-word summary. The rest is bonus material, reflecting on the links between this book and the others in the series.
‘We define policy analysis as evidence-based advice giving, as the process by which one arrives at a policy recommendation to address a problem of public concern. Policy analysis almost always involves advice for a client’ (Meltzer and Schwartz, 2019: 15).
Meltzer and Schwartz (2019: 231-2) describe policy analysis as applied research, drawing on many sources of evidence, quickly, with limited time, access to scientific research, or funding to conduct a lot of new research (2019: 231-2). It requires:
careful analysis of a wide range of policy-relevant documents (including the ‘grey’ literature often produced by governments, NGOs, and think tanks) and available datasets
perhaps combined with expert interviews, focus groups, site visits, or an online survey (see 2019: 232-64 on methods).
Meltzer and Schwartz (2019: 21) outline a ‘five-step framework’ for client-oriented policy analysis. During each step, they contrast their ‘flexible’ and ‘iterative’ approach with a too- rigid ‘rationalistic approach’ (to reflect bounded, not comprehensive, rationality):
‘Define the problem’.
Problem definition is a political act of framing, not an exercise in objectivity (2019: 52-3). It is part of a narrative to evaluate the nature, cause, size, and urgency of an issue (see Stone), or perhaps to attach to an existing solution (2019: 38-40; compare with Mintrom).
In that context, ask yourself ‘Who is defining the problem? And for whom?’ and do enough research to be able to define it clearly and avoid misunderstanding among you and your client (2019: 37-8; 279-82):
Identify your client’s resources and motivation, such as how they seek to use your analysis, the format of analysis they favour, their deadline, and their ability to make or influence the policies you might suggest (2019: 49; compare with Weimer and Vining).
Tailor your narrative to your audience, albeit while recognising the need to learn from ‘multiple perspectives’ (2019: 40-5).
Make it ‘concise’ and ‘digestible’, not too narrowly defined, and not in a way that already closes off discussion by implying a clear cause and solution (2019: 51-2).
In doing so:
Ask yourself if you can generate a timeline, identify key stakeholders, and place a ‘boundary’ on the problem.
Establish if the problem is urgent, who cares about it, and who else might care (or not) (2019 : 46).
Focus on the ‘central’ problem that your solution will address, rather than the ‘related’ and ‘underlying’ problems that are ‘too large and endemic to be solved by the current analysis’ (2019: 47).
Avoid misdiagnosing a problem with reference to one cause. Instead, ‘map’ causation with reference to (say) individual and structural causes, intended and unintended consequences, simple and complex causation, market or government failure, and/ or the ability to blame an individual or organisation (2019: 48-9).
Combine quantitative and qualitative data to frame problems in relation to: severity, trends in severity, novelty, proximity to your audience, and urgency or crisis (2019: 53-4).
During this process, interrogate your own biases or assumptions and how they might affect your analysis (2019: 50).
2. ‘Identify potential policy options (alternatives) to address the problem’.
Common sources of ideas include incremental changes from current policy, ‘client suggestions’, comparable solutions (from another time, place, or policy area), reference to common policy instruments, and ‘brainstorming’ or ‘design thinking’ (2019: 67-9; see box 2.3 and 7.1, below, from Understanding Public Policy).
Identify a ‘wide range’ of possible solutions, then select the (usually 3-5) ‘most promising’ for further analysis (2019: 65). In doing so:
be careful not to frame alternatives negatively (e.g. ‘death tax’ – 2019: 66)
compare alternatives in ‘good faith’ rather than keeping some ‘off the table’ to ensure that your preferred solution looks good (2019: 66)
think about how to modify existing policies according to scale or geographical coverage, who to include (and based on what criteria), for how long, using voluntary versus mandatory provisions, and ensuring oversight (2019: 71-3)
consider combinations of common policy instruments, such as regulations and economic penalties/ subsidies (2019: 73-7)
consider established ways to ‘brainstorm’ ideas (2019: 77-8)
note the rise of instruments derived from the study of psychology and behavioural public policy (2019: 79-90)
learn from design principles, including ‘empathy’, ‘co-creating’ policy with service users or people affected, ‘prototyping’ (2019: 90-1)
3. ‘Specify the objectives to be attained in addressing the problem and the criteria to evaluate the attainment of these objectives as well as the satisfaction of other key considerations (e.g., equity, cost, equity, feasibility)’.
Your objectives relate to your problem definition and aims: what is the problem, what do you want to happen when you address it, and why?
For example, questions to your client may include: what is your organization’s ‘mission’, what is feasible (in terms of resources and politics), which stakeholders to you want to include, and how will you define success (2019: 105; 108-12)?
In that values-based context, your criteria relate to ways to evaluate each policy’s likely impact (2019: 106-7). They should ensure:
Comprehensiveness. E.g. how many people, and how much of their behaviour, can you influence while minimizing the ‘burden’ on people, businesses, or government? (2019: 113-4)
Mutual Exclusiveness. In other words, don’t have two objectives doing the same thing (2019: 114).
Common criteria include (2019: 116):
Effectiveness. The size of its intended impact on the problem (2019: 117).
Equity (fairness). The impact in terms of ‘vertical equity’ (e.g. the better off should pay more), ‘horizontal equity’ (e.g. you should not pay more if unmarried), fair process, fair outcomes, and ‘intergenerational’ equity (e.g. don’t impose higher costs on future populations) (2019: 118-19).
Feasibility (administrative, political, and technical). The likelihood of this policy being adopted and implemented well (2019: 119-21)
Cost (or financial feasibility). Who would bear the cost, and their willingness and ability to pay (2019: 122).
Efficiency. To maximise the benefit while minimizing costs (2019: 122-3).
4. ‘Assess the outcomes of the policy options in light of the criteria and weigh trade-offs between the advantages and disadvantages of the options’.
When explaining objectives and criteria,
‘label’ your criteria in relation to your policy objectives (e.g. to ‘maximize debt reduction’) rather than using generic terms (2019: 123-7)
produce a table – with alternatives in rows, and criteria in columns – to compare each option
quantify your policies’ likely outcomes, such as in relation to numbers of people affected and levels of income transfer, or a percentage drop in the size of the problem, but also
communicate the degree of uncertainty related to your estimates (2019: 128-32; see Spiegelhalter)
Consider using cost-benefit analysis to identify (a) the financial and opportunity cost of your plans (what would you achieve if you spent the money elsewhere?), compared to (b) the positive impact of your funded policy (2019: 141-55).
The principle of CBA may be intuitive, but a thorough CBA process is resource-intensive, vulnerable to bias and error, and no substitute for choice. It requires you to make a collection of assumptions about human behaviour and likely costs and benefits, decide whose costs and benefits should count, turn all costs and benefits into a single measure, and imagine how to maximise winners and compensate losers (2019: 155-81; compare Weimer and Vining with Stone).
One alternative is cost-effectiveness analysis, which quantifies costs and relates them to outputs (e.g. number of people affected, and how) without trying to translate them into a single measure of benefit (2019: 181-3).
These measures can be combined with other thought processes, such as with reference to ‘moral imperatives’, a ‘precautionary approach’, and ethical questions on power/ powerlessness (2019: 183-4).
5. ‘Arrive at a recommendation’.
Predict the most likely outcomes of each alternative, while recognising high uncertainty (2019: 189-92). If possible,
draw on existing, comparable, programmes to predict the effectiveness of yours (2019: 192-4)
combine such analysis with relevant theories to predict human behaviour (e.g. consider price ‘elasticity’ if you seek to raise the price of a good to discourage its use) (2019: 193-4)
apply statistical methods to calculate the probability of each outcome (2019: 195-6), and modify your assumptions to produce a range of possibilities, but
note Spiegelhalter’s cautionary tales and anticipate the inevitable ‘unintended consequences’ (when people do not respond to policy in the way you would like) (2019: 201-2)
use these estimates to inform a discussion on your criteria (equity, efficiency, feasibility) (2019: 196-200)
present the results visually – such as in a ‘matrix’ – to encourage debate on the trade-offs between options
simplify choices by omitting irrelevant criteria and options that do not compete well with others (2019: 203-10)
make sure that your recommendation (a) flows from the analysis, and (b) is in the form expected by your client (2019: 211-12)
consider making a preliminary recommendation to inform an iterative process, drawing feedback from clients and stakeholder groups (2019: 212).
Policy analysis in a wider context
Meltzer and Schwartz’s approach makes extra sense if you have already read some of the other texts in the series, including:
Weimer and Vining, which represents an exemplar of an X-step approach informed heavily by the study of economics and application of economic models such as cost-benefit-analysis (compare with Radin’s checklist).
Geva-May on the existence of a policy analysis profession with common skills, heuristics, and (perhaps) ethics (compare with Meltzer and Schwartz, 2019: 282-93)
the proliferation of analysts across multiple levels of government, NGOs, and the private sector (compare with Meltzer and Schwartz, 2019: 269-77)
the historic shift of analysis from formulation to all notional stages (contrast with Meltzer and Schwartz, 2019: 16-7 on policy analysis not including implementation or evaluation)
the difficulty in distinguishing between policy analysis and advocacy in practice (compare with Meltzer and Schwartz, 2019: 276-8, who suggest that actors can choose to perform these different roles)
the emerging sense that it is difficult to identify a single client in a multi-centric policymaking system. Put another way, we might be working for a specific client but accept that their individual influence is low.
a historic tendency for economics to dominate policy analysis,
the applicability of economic assumptions (focusing primarily on individualist behaviour and markets), and
the pervasiveness of ‘rationalist’ policy analysis built on X-steps.
Meltzer and Schwartz (2019: 1-3) agree that economic models are too dominant (identifying the value of insights from ‘other disciplines – including design, psychology, political science, and sociology’).
However, they argue that critiques of rational models exaggerate their limitations (2019: 23-6). For example:
these models need not rely solely on economic techniques or quantification, a narrow discussion or definition of the problem, or the sense that policy analysis should be comprehensive, and
it is not problematic for analyses to reflect their client’s values or for analysts to present ambiguous solutions to maintain wide support, partly because
we would expect the policy analysis to form only one part of a client’s information or strategy.
Further, they suggest that these critiques provide no useful alternative, to help guide new policy analysts. Yet, these guides are essential:
‘to be persuasive, and credible, analysts must situate the problem, defend their evaluative criteria, and be able to demonstrate that their policy recommendation is superior, on balance, to other alternative options in addressing the problem, as defined by the analyst. At a minimum, the analyst needs to present a clear and defensible ranking of options to guide the decisions of the policy makers’ (Meltzer and Schwartz, 2019: 4).
Meltzer and Schwartz (2019: 27-8) then explore ways to improve a 5-step model with insights from approaches such as ‘design thinking’, in which actors use a similar process – ‘empathize, define the problem, ideate, prototype, test and get feedback from others’ – to experiment with policy solutions without providing a narrow view on problem definition or how to evaluate responses.
Policy analysis and policy theory
One benefit to Meltzer and Schwartz’s approach is that it seeks to incorporate insights from policy theories and respond with pragmatism and hope. However, I think you also need to read the source material to get a better sense of those theories, key debates, and their implications. For example:
Meltzer and Schwartz (2019: 32) note correctly that ‘incremental’ does not sum up policy change well. Indeed, Punctuated Equilibrium Theory shows that policy change is characterised by a huge number of small and a small number of huge changes.
However, the direct implications of PET are not as clear as they suggest. Baumgartner and Jones have both noted that they can measure these outcomes and identify the same basic distribution across a political system, but not explain or predict why particular policies change dramatically.
It is useful to recommend to policy analysts that they invest some hope in major policy change, but also sensible to note that – in the vast majority of cases – it does not happen.
On his point, see Mintrom on policy analysis for the long term, Weiss on the ‘enlightenment’ function of research and analysis, and Box 6.3 (from Understanding Public Policy), on the sense that (a) we can give advice to ‘budding policy entrepreneurs’ on how to be effective analysts, but (b) should note that all their efforts could be for nothing.
Meltzer and Schwartz (2019: 32-3) tap briefly into the old debate on whether it is preferable to seek radical or incremental change. For more on that debate, see chapter 5 in the 1st ed of Understanding Public Policy in which Lindblom notes that proposals for radical/ incremental changes are not mutually exclusive.
Perhaps explore the possible tension between Meltzer and Schwartz’s (2019: 33-4) recommendation that (a) policy analysis should be ‘evidence-based advice giving’, and (b) ‘flexible and open-ended’.
I think that Stone’s response would be that phrases such as ‘evidence based’ are not ‘flexible and open-ended’. Rather, they tend to symbolise a narrow view of what counts as evidence (see also Smith, and Hindess).
Further, note that the phrase ‘evidence based policymaking’ is a remarkably vague term (see the EBPM page), perhaps better seen as a political slogan than a useful description or prescription of policymaking.
Finally, if you read enough of these policy analysis texts, you get a sense that many are bunched together even if they describe their approach as new or distinctive.
Indeed, Meltzer and Schwarz (2019: 22-3) provide a table (containing Bardach and Patashnik, Patton et al, Stokey and Zeckhauser, Hammond et al, and Weimer & Vining) of ‘quite similar’ X-step approaches.
Weimer and Vining also discuss the implications of policy theories and present the sense that X-step policy analysis should be flexible and adaptive.
Many texts – including Radin, and Smith (2016) – focus on the value of case studies to think through policy analysis in particular contexts, rather than suggesting that we can produce a universal blueprint.
However, as Geva-May might suggest, this is not a bad thing if our aim is to generate the sense that policy analysis is a profession with its own practices and heuristics.
Please see the Policy Analysis in 750 words series overview before reading the summary. As usual, the 750-word description is more for branding than accuracy.
‘The basic relationship between a decision-maker (the client) and an analyst has moved from a two-person encounter to an extremely complex and diverse set of interactions’ (Radin, 2019: 2).
Many texts in this series continue to highlight the client-oriented nature of policy analysis (Weimer and Vining), but within a changing policy process that has altered the nature of that relationship profoundly.
We can use Radin’s work to present two main stories of policy analysis:
The old ways of making policy resembled a club, or reflected a clear government hierarchy, involving:
a small number of analysts, generally inside government (such as senior bureaucrats, scientific experts, and – in particular- economists),
giving technical or factual advice,
about policy formulation,
to policymakers at the heart of government,
on the assumption that policy problems would be solved via analysis and action.
Modern policy analysis is characterised by a more open and politicised process in which:
many analysts, inside and outside government,
compete to interpret facts, and give advice,
about setting the agenda, and making, delivering, and evaluating policy,
across many policymaking venues,
often on the assumption that governments have a limited ability to understand and solve complex policy problems.
As a result, the client-analyst relationship is increasingly fluid:
In previous eras, the analyst’s client was a senior policymaker, the main focus was on the analyst-client relationship, and ‘both analysts and clients did not spend much time or energy thinking about the dimensions of the policy environment in which they worked’ (2019: 59). Now, in a multi-centric policymaking environment:
It is tricky to identify the client.
We could imagine the client to be someone paying for the analysis, someone affected by its recommendations, or all policy actors with the ability to act on the advice (2019: 10).
If there is ‘shared authority’ for policymaking within one political system, a ‘client’ (or audience) may be a collection of policymakers and influencers spread across a network containing multiple types of government, non-governmental actors, and actors responsible for policy delivery (2019: 33).
The growth in international cooperation also complicates the idea of a single client for policy advice (2019: 33-4)
This shift may limit the ‘face-to-face encounters’ that would otherwise provide information for – and perhaps trust in – the analyst (2019: 2-3).
It is tricky to identify the analyst
Radin (2019: 9-25) traces, from the post-war period in the US, a major expansion of policy analysts, from the notional centre of policymaking in federal government towards analysts spread across many venues, inside government (across multiple levels, ‘policy units’, and government agencies) and congressional committees, and outside government (such as in influential think tanks).
Policy analysts can also be specialist external companies contracted by organisations to provide advice (2019: 37-8).
This expansion shifted the image of many analysts, from a small number of trusted insiders towards many being treated as akin to interest groups selling their pet policies (2019: 25-6).
The nature – and impact – of policy analysis has always been a little vague, but now it seems more common to suggest that ‘policy analysts’ may really be ‘policy advocates’ (2019: 44-6).
As such, they may now have to work harder to demonstrate their usefulness (2019: 80-1) and accept that their analysis will have a limited impact (2019: 82, drawing on Weiss’ discussion of ‘enlightenment’).
Consequently, the necessary skills of policy analysis have changed:
Although many people value systematic policy analysis (and many rely on economists), an effective analyst does not simply apply economic or scientific techniques to analyse a problem or solution, or rely on one source of expertise or method, as if it were possible to provide ‘neutral information’ (2019: 26).
Indeed, Radin (2019: 31; 48) compares the old ‘acceptance that analysts would be governed by the norms of neutrality and objectivity’ with
(a) increasing calls to acknowledge that policy analysis is part of a political project to foster some notion of public good or ‘public interest’, and
(b) Stone’s suggestion that the projection of reason and neutrality is a political strategy.
In other words, the fictional divide between political policymakers and neutral analysts is difficult to maintain.
Rather, think of analysts as developing wider skills to operate in a highly political environment in which the nature of the policy issue is contested, responsibility for a policy problem is unclear, and it is not clear how to resolve major debates on values and priorities:
Some analysts will be expected to see the problem from the perspective of a specific client with a particular agenda.
Other analysts may be valued for their flexibility and pragmatism, such as when they acknowledge the role of their own values, maintain or operate within networks, communicate by many means, and supplement ‘quantitative data’ with ‘hunches’ when required (2019: 2-3; 28-9).
Radin (2019: 21) emphasises a shift in skills and status
The idea of (a) producing new and relatively abstract ideas, based on high control over available information, at the top of a hierarchical organisation, makes way for (b) developing the ability to:
generate a wider understanding of organisational and policy processes, reflecting the diffusion of power across multiple policymaking venues
the limits to a government’s ability to understand and solve problems (2019: 95-6),
the inescapable conflict over trade-offs between values and goals, which are difficult to resolve simply by weighting each goal (2019: 105-8; see Stone), and
do so flexibly, to recognise major variations in problem definition, attention and networks across different policy sectors and notional ‘stages’ of policymaking (2019: 75-9; 84).
Radin’s (2019: 48) overall list of relevant skills include:
‘Case study methods, Cost- benefit analysis, Ethical analysis, Evaluation, Futures analysis, Historical analysis, Implementation analysis, Interviewing, Legal analysis, Microeconomics, Negotiation, mediation, Operations research, Organizational analysis, Political feasibility analysis, Public speaking, Small- group facilitation, Specific program knowledge, Statistics, Survey research methods, Systems analysis’
They develop alongside analytical experience and status, from the early career analyst trying to secure or keep a job, to the experienced operator looking forward to retirement (2019: 54-5)
A checklist for policy analysts
Based on these skills requirements, the contested nature of evidence, and the complexity of the policymaking environment, Radin (2019: 128-31) produces a 4-page checklist of – 91! – questions for policy analysts.
For me, it serves two main functions:
It is a major contrast to the idea that we can break policy analysis into a mere 5-8 steps (rather, think of these small numbers as marketing for policy analysis students, akin to 7-minute abs)
It presents policy analysis as an overwhelming task with absolutely no guarantee of policy impact.
To me, this cautious, eyes-wide-open, approach is preferable to the sense that policy analysts can change the world if they just get the evidence and the steps right.
Further Reading:
Iris Geva-May (2005) ‘Thinking Like a Policy Analyst. Policy Analysis as a Clinical Profession’, in Geva-May (ed) Thinking Like a Policy Analyst. Policy Analysis as a Clinical Profession (Basingstoke: Palgrave)
Although the idea of policy analysis may be changing, Geva-May (2005: 15) argues that it remains a profession with its own set of practices and ways of thinking. As with other professions (like medicine), it would be unwise to practice policy analysis without education and training or otherwise learning the ‘craft’ shared by a policy analysis community (2005: 16-17). For example, while not engaging in clinical diagnosis, policy analysts can draw on 5-step process to diagnose a policy problem and potential solutions (2005: 18-21). Analysts may also combine these steps with heuristics to determine the technical and political feasibility of their proposals (2005: 22-5), as they address inevitable uncertainty and their own bounded rationality (2005: 26-34; see Gigerenzer on heuristics). As with medicine, some aspects of the role – such as research methods – can be taught in graduate programmes, while others may be better suited to on the job learning (2005: 36-40). If so, it opens up the possibility that there are many policy analysis professions to reflect different cultures in each political system (and perhaps the venues within each system).
Please see the Policy Analysis in 750 words series overview before reading the summary. This post is 750 words plus a bonus 750 words plus some further reading that doesn’t count in the word count even though it does.
‘Whether you are a policy analyst, a policy researcher, a policy advocate, a policy maker, or an engaged citizen, my hope for Policy Paradox is that it helps you to go beyond your job description and the tasks you are given – to think hard about your own core values, to deliberate with others, and to make the world a better place’ (Stone, 2012: 15)
Stone (2012: 379-85) rejects the image of policy analysis as a ‘rationalist’ project, driven by scientific and technical rules, and separable from politics. Rather, every policy analyst’s choice is a political choice – to define a problem and solution, and in doing so choosing how to categorise people and behaviour – backed by strategic persuasion and storytelling.
The Policy Paradox: people entertain multiple, contradictory, beliefs and aims
Stone (2012: 2-3) describes the ways in which policy actors compete to define policy problems and public policy responses. The ‘paradox’ is that it is possible to define the same policies in contradictory ways.
‘Paradoxes are nothing but trouble. They violate the most elementary principle of logic: something can’t be two different things at once. Two contradictory interpretations can’t both be true. A paradox is just such an impossible situation, and political life is full of them’ (Stone, 2012: 2).
This paradox does not refer simply to a competition between different actors to define policy problems and the success or failure of solutions. Rather:
The same actor can entertain very different ways to understand problems, and can juggle many criteria to decide that a policy outcome was a success and a failure (2012: 3).
Surveys of the same population can report contradictory views – encouraging a specific policy response and its complete opposite – when asked different questions in the same poll (2012: 4; compare with Riker)
Policy analysts: you don’t solve the Policy Paradox with a ‘rationality project’
Like many posts in this series (Smith, Bacchi, Hindess), Stone (2010: 9-11) rejects the misguided notion of objective scientists using scientific methods to produce one correct answer (compare with Spiegelhalter and Weimer & Vining). A policy paradox cannot be solved by ‘rational, analytical, and scientific methods’ because:
We can seek order and clarity in analysis and communication, but should not expect it in political systems (2012: 10).
5-step policy analysis (identify objectives, identify alternatives, predict their effects, evaluate alternatives, choose) is pervasive, but it ‘ignores our emotional feelings and moral intuitions’ (2012: 11).
Further, Stone (2012: 10-11) rejects the over-reliance, in policy analysis, on the misleading claim that:
policymakers are engaging primarily with markets rather than communities (see 2012: 35 on the comparison between a ‘market model’ and ‘polis model’),
economic models can sum up political life, and
cost-benefit-analysis can reduce a complex problem into the sum of individual preferences using a single unambiguous measure.
Rather, many factors undermine such simplicity:
People do not simply act in their own individual interest. Nor can they rank-order their preferences in a straightforward manner according to their values and self-interest.
Instead, they maintain a contradictory mix of objectives, which can change according to context and their way of thinking – combining cognition and emotion – when processing information (2012: 12; 30-4).
People are social actors. Politics is characterised by ‘a model of community where individuals live in a dense web of relationships, dependencies, and loyalties’ and exercise power with reference to ideas as much as material interests (2012: 10; 20-36; compare with Ostrom, more Ostrom, and Lubell; and see Sousa on contestation).
Morals and emotions matter. If people juggle contradictory aims and measures of success, then a story infused with ‘metaphor and analogy’, and appealing to values and emotions, prompts people ‘to see a situation as one thing rather than another’ and therefore draw attention to one aim at the expense of the others (2012: 11; compare with Gigerenzer).
Policy analysis reconsidered: the ambiguity of values and policy goals
Stone (2012: 14) identifies the ambiguity of the criteria for success used in 5-step policy analyses. They do not form part of a solely technical or apolitical process to identify trade-offs between well-defined goals (compare Bardach, Weimer and Vining, and Mintrom). Rather, ‘behind every policy issue lurks a contest over conflicting, though equally plausible, conceptions of the same abstract goal or value’ (2012: 14). Examples of competing interpretations of valence issues include definitions of:
Equity, according to: (a) which groups should be included, how to assess merit, how to identify key social groups, if we should rank populations within social groups, how to define need and account for different people placing different values on a good or service, (b) which method of distribution to use (competition, lottery, election), and (c) how to balance individual, communal, and state-based interventions (2012: 39-62).
Efficiency, to use the least resources to produce the same objective, according to: (a) who determines the main goal and how to balance multiple objectives, (a) who benefits from such actions, and (c) how to define resources while balancing equity and efficiency – for example, does a public sector job and a social security payment represent a sunk cost to the state or a social investment in people? (2012: 63-84).
Welfare or Need, according to factors including (a) the material and symbolic value of goods, (b) short term support versus a long term investment in people, (c) measures of absolute poverty or relative inequality, and (d) debates on ‘moral hazard’ or the effect of social security on individual motivation (2012: 85-106)
Liberty, according to (a) a general balancing of freedom from coercion and freedom from the harm caused by others, (b) debates on individual and state responsibilities, and (c) decisions on whose behaviour to change to reduce harm to what populations (2012: 107-28)
Security, according to (a) our ability to measure risk scientifically (see Spiegelhalter and Gigerenzer), (b) perceptions of threat and experiences of harm, (c) debates on how much risk to safety to tolerate before intervening, (d) who to target and imprison, and (e) the effect of surveillance on perceptions of democracy (2012: 129-53).
Policy analysis as storytelling for collective action
Actors use policy-relevant stories to influence the ways in which their audience understands (a) the nature of policy problems and feasibility of solutions, within (b) a wider context of policymaking in which people contest the proper balance between state, community, and market action. Stories can influence key aspects of collective action, including:
Defining interests and mobilising actors, by drawing attention to – and framing – issues with reference to an imagined social group and its competition (e.g. the people versus the elite; the strivers versus the skivers) (2012: 229-47)
Making decisions, by framing problems and solutions (2012: 248-68). Stone (2012: 260) contrasts the ‘rational-analytic model’ with real-world processes in which actors deliberately frame issues ambiguously, shift goals, keep feasible solutions off the agenda, and manipulate analyses to make their preferred solution seem the most efficient and popular.
Defining the role and intended impact of policies, such as when balancing punishments versus incentives to change behaviour, or individual versus collective behaviour (2012: 271-88).
Setting and enforcing rules (see institutions), in a complex policymaking system where a multiplicity of rules interact to produce uncertain outcomes, and a powerful narrative can draw attention to the need to enforce some rules at the expense of others (2012: 289-310).
Persuasion, drawing on reason, facts, and indoctrination. Stone (2012: 311-30) highlights the context in which actors construct stories to persuade: people engage emotionally with information, people take certain situations for granted even though they produce unequal outcomes, facts are socially constructed, and there is unequal access to resources – held in particular by government and business – to gather and disseminate evidence.
Defining human and legal rights, when (a) there are multiple, ambiguous, and intersecting rights (in relation to their source, enforcement, and the populations they serve) (b) actors compete to make sure that theirs are enforced, (c) inevitably at the expense of others, because the enforcement of rights requires a disproportionate share of limited resources (such as policymaker attention and court time) (2012: 331-53)
Influencing debate on the powers of each potential policymaking venue – in relation to factors including (a) the legitimate role of the state in market, community, family, and individual life, (b) how to select leaders, (c) the distribution of power between levels and types of government – and who to hold to account for policy outcomes (2012: 354-77).
Key elements of storytelling include:
Symbols, which sum up an issue or an action in a single picture or word (2012:157-8)
Characters, such as heroes or villain, who symbolise the cause of a problem or source of solution (2012:159)
Narrativearcs, such as a battle by your hero to overcome adversity (2012:160-8)
Synecdoche, to highlight one example of an alleged problem to sum up its whole (2012: 168-71; compare the ‘welfare queen’ example with SCPD)
Metaphor, to create an association between a problem and something relatable, such as a virus or disease, a natural occurrence (e.g. earthquake), something broken, something about to burst if overburdened, or war (2012: 171-78; e.g. is crime a virus or a beast?)
Ambiguity, to give people different reasons to support the same thing (2012: 178-82)
Using numbers to tell a story, based on political choices about how to: categorise people and practices, select the measures to use, interpret the figures to evaluate or predict the results, project the sense that complex problems can be reduced to numbers, and assign authority to the counters (2012:183-205; compare with Speigelhalter)
Assigning Causation, in relation to categories including accidental or natural, ‘mechanical’ or automatic (or in relation to institutions or systems), and human-guided causes that have intended or unintended consequences (such as malicious intent versus recklessness)
‘Causal strategies’ include to: emphasise a natural versus human cause, relate it to ‘bad apples’ rather than systemic failure, and suggest that the problem was too complex to anticipate or influence
Actors use these arguments to influence rules, assign blame, identify ‘fixers’, and generate alliances among victims or potential supporters of change (2012: 206-28).
Wider Context and Further Reading: 1. Policy analysis
Stone (2012: 10) argues that analysts who try to impose one worldview on policymaking will find that ‘politics looks messy, foolish, erratic, and inexplicable’. For analysts, who are more open-minded, politics opens up possibilities for creativity and cooperation (2012: 10).
Punctuated equilibrium theory shows that policy change is primarily a function of such shifts in policymaker attention.
Policy is actually a collection of – often contradictory – policy instruments and institutions, interacting in complex systems or environments, to produce unclear messages and outcomes. As such, what we call ‘public policy’ (for the sake of simplicity) is subject to interpretation and manipulation as it is made and delivered, and we struggle to conceptualise and measure policy change. Indeed, it makes more sense to describe competing narratives of policy change.
Policy theories and storytelling
People communicate meaning via stories. Stories help us turn (a) a complex world, which provides a potentially overwhelming amount of information, into (b) something manageable, by identifying its most relevant elements and guiding action (compare with Gigerenzer on heuristics).
‘If the claims of philosophy to a special kind of knowledge can be shown to be without foundation, if they are at best dogmatic or else incoherent, then methodology is an empty and futile pursuit and its prescriptions are vacuous’ (Hindess, 1977: 4).
This book may seem like a weird addition to a series on policy analysis.
However, it follows the path set by Carol Bacchi, asking whose interests we serve when we frame problems for policy analysis, and Linda Tuhiwai Smith, asking whose research counts when we do so.
One important answer is that the status of research and the framing of the problem result from the exercise of power, rather than the objectivity of analysts and natural superiority of some forms of knowledge.
In other posts on ‘the politics of evidence based policymaking’, I describe some frustrations among many scientists that their views on a hierarchy of knowledge based on superior methods are not shared by many policymakers. These posts can satisfy different audiences: if you have a narrow view of what counts as good evidence, you can focus on the barriers between evidence and policy; if you have a broader view, you can wonder why those barriers seem higher for other forms of knowledge (e.g. Linda Tuhiwai Smith on the marginalisation of indigenous knowledge).
In this post, I encourage you to go a bit further down this path by asking how people accumulate knowledge in the first place. For example, see introductory accounts by Chalmers, entertaining debates involving Feyerabend, and Hindess’ book to explore your assumptions about how we know what we know.
My take-home point from these texts is that we are only really able to describe convincingly the argument that we are not accumulating knowledge!
The simple insight from Chalmers’ introduction is that inductive (observational) methods to generate knowledge are circular:
we engage inductively to produce theory (to generalise from individual cases), but
we use theory to engage in any induction, such as to decide what is important to study, and what observations are relevant/irrelevant, and why.
In other words, we need theories of the world to identify the small number of things to observe (to allow us to filter out an almost unlimited amount of signals from out environments), but we need our observations to generate those theories!
Hindess shows that all claims to knowledge involve such circularity: we employ philosophy to identify the nature of the world (ontology) and how humans can generate valid knowledge of it (epistemology) to inform methodology, to state that scientific knowledge is only valid if it lives up to a prescribed method, then argue that the scientific knowledge validates the methodology and its underlying philosophy (1977: 3-22). If so, we are describing something that makes sense according to the rules and practices of its proponents, not an objective scientific method to help us accumulate knowledge.
Further, different social/ professional groups support different forms of working knowledge that they value for different reasons (such as to establish ‘reliability’ or ‘meaning’). To do so, they invent frameworks to help them theorise the world, such as to describe the relationship between concepts (and key concepts such as cause and effect). These frameworks represent a useful language to communicate about our world rather than simply existing independently of it and corresponding to it.
Hindess’ subsequent work explored the context in which we exercise power to establish the status of some forms of knowledge over others, to pursue political ends rather than simply the ‘objective’ goals of science. As described, it is as relevant now as it was then.
How do these ideas inform policy analysis?
Perhaps, by this stage, you are thinking: isn’t this a relativist argument, concluding that we should never assert the relative value of some forms of knowledge over others (like astronomy versus astrology)?
I don’t think so. Rather, it invites us to do two more sensible things:
Accept that different approaches to knowledge may be ‘incommensurable’.
They may not share ‘a common set of perceptions’ (or even a set of comparable questions) ‘which would allow scientists to choose between one paradigm and the other . . . there will be disputes between them that cannot all be settled by an appeal to the facts’ (Hindess, 1988: 74)
If so, “there is no possibility of an extratheoretical court of appeal which can ‘validate’ the claims of one position against those of another” (Hindess, 1977: 226).
Reject the sense of self-importance, and hubris, which often seems to accompany discussions of superior forms of knowledge. Don’t be dogmatic. Live by the maxim ‘don’t be an arse’. Reflect on the production, purpose, value, and limitations of our knowledge in different contexts (which Spiegelhalter does well).
On that basis, we can have honest discussions about why we should exercise power in a political system to favour some forms of knowledge over others in policy analysis, reflecting on:
The relatively straightforward issue of internal consistency: is an approach coherent, and does it succeed on its own terms?
For example, do its users share a clear language, pursue consistent aims with systematic methods, find ways to compare and reinforce the value of each other’s findings, while contributing to a thriving research agenda (as discussed in box 13.3 below)?
Or, do they express their aims in other ways, such as to connect research to emancipation, or value respect for a community over the scientific study of that community?
The not straightforward issue of overall consistency: how can we compare different forms of knowledge when they do not follow each other’s rules or standards?
g. what if one approach is (said to be) more rigorous and the other more coherent?
g. what if one produces more data but another produces more ownership?
In each case, the choice of criteria for comparison involves political choice (as part of a series of political choices), without the ability – described in relation to ‘cost benefit analysis’ – to translate all relevant factors into a single unit.
The imperative to ‘synthesise’ knowledge.
Spiegelhalter provides a convincing description of the benefits of systematic review and ‘meta-analysis’ within a single, clearly defined, scientific approach containing high agreement on methods and standards for comparison.
However, this approach is not applicable directly to the review of multiple forms of knowledge.
So, what do people do?
E.g. some systematic reviewers apply the standards of their own field to all others, which (a) tends to produce the argument that very little high quality evidence exists because other people are doing it wrongly, and (b) perhaps exacerbates a tendency for policymakers to attach relatively low value to such evaluations.
E.g. policy analysts are more likely to apply different criteria: is it available, understandable, ‘usable’, and policy relevant (e.g. see ‘knowledge management for policy’)?
Each approach is a political choice to include/ exclude certain forms of knowledge according to professional norms or policymaking imperatives, not a technical process to identify the most objective information. If you are going to do it, you should at least be aware of what you are doing.
These posts introduce you to key concepts in the study of public policy. They are all designed to turn a complex policymaking world into something simple enough to understand. Some of them focus on small parts of the system. Others present ambitious ways to explain the system as a whole. The wide range of concepts should give you a sense of a variety of studies out there, but my aim is to show you that these studies have common themes.
