Policy analysis in 750 words: WORDLE and trial and error policymaking

I apologise for every word in this post, and the capitalised 5-letter words in particular.

WORDLE is a SIMPLE word game (in US English). The aim is to identify a 5-letter word correctly in 6 guesses or fewer. Each guess has to be a real word, and you receive informative feedback each time: GREEN means you have the letter RIGHT and in the right position; yellow means the right letter in the wrong position; grey MEANS the letter does not appear in the word.

One strategy involves trial-and-error learning via 3 or 4 simple steps:

1. Use your initial knowledge of the English language to inform initial guesses, such as guessing a word with common vowels (I go for E and A) and consonants (e.g. S, T).

2. Learn from feedback on your correct and incorrect estimates.

3. Use your new information and deduction (e.g. about which combinations work when you exclude many options) to make informed guesses.

4. Do so while avoiding unhelpful heuristics, such as assuming that each letter will only appear once (or that the spelling is in UK English).

At least, that is how I play it. I get it in 3 just over half the time, and 4 or 5 in the rest. I make 2-4 ‘errors’ then succeed. In the context of the game’s rules, that is consistent success, RIGHT?

[insert crowbar GIF to try to get away with the segue]

That is the spirit of the idea of trial-and-error learning.

It is informed by previous knowledge, but also a recognition of the benefits of trying things out to generate new information, update your knowledge and skills (the definition of learning), and try again.

A positive normative account of this approach can be found in classic discussions of incrementalism and modern discussions of policymaking informed by complex systems insights:

‘To deal with uncertainty and change, encourage trial-and-error projects, or pilots, that can provide lessons, or be adopted or rejected, relatively quickly’.

Advocates of such approaches also suggest that we change how we describe them, replacing the language of policy failure with ERROR, at least when part of a process of continuous policy learning in the face of uncertainty.

At the heart of such advice are two guiding principles:

1. Recognise the limits to centralism when giving policy advice. There is no powerful centre of government, able to carry out all of its aims successfully, so do not build policy advice on that assumption.

2. Recognise the limits to our knowledge. Policymakers must make and learn from choices in the face of uncertainty, so do not kid yourself that one piece of analysis and action will do.

Much like the first two WORDLE guesses, your existing knowledge alone does not tell you how to proceed (regardless of the number of times that people repeat the slogan of ‘evidence-based policymaking’).

Political problems with trial and error

The main political problem with this approach is that many political systems – including adversarial and/or Westminster systems – are not conducive to learning from error. You may think that adapting continuously to uncertainty is crucial, but also be wary of recommending it to:

1. Politicians who will be held to account for failure. A government’s apparent failure to deliver on promises represents a resource for its opposition.

2. Organisations subject to government targets. Failure to meet strict statutory requirements is not seen as a learning experience.  

More generally, your audience may face criticism whenever errors are associated with negative policy consequences (with COVID-19 policy representing a vivid, extreme example).

These limitations produce a major dilemma in policy analysis, in which you believe that you will not learn how to make good policy without trial-and-error but recognise that this approach will not be politically feasible. In many political systems, policymakers need to pretend to their audience that they know what the problem is and that they have the knowledge and power to solve it. You may not be too popular if you encourage open-minded experimentation. This limitation should not warn you against trial-and-error recommendations completely, but rather remind you to relate good-looking ideas to your policymaking context.

Please note that I missed my train stop while writing this post, despite many opportunities to learn from the other times it happened.

Policy Analysis in 750 Words: Political feasibility and policy success

Policy studies and policy analysis guidebooks identify the importance of feasible policy solutions:

• Technical feasibility: will this solution work as intended if implemented?
• Political feasibility: will it be acceptable to enough powerful people?

For example, Kingdon treats feasibility as one of three conditions for major policy change during a ‘window of opportunity’: (1) there is high attention to the policy problem, (2) a feasible solution already exists, and (3) key policymakers have the motive and opportunity to select it.

Guidebooks relate this requirement initially to your policymaker client: what solutions will they rule out, to the extent that they are not even worth researching as options (at least for the short term)?

Further, this assessment relates to types of policy ‘tool’ or ‘instrument’: one simple calculation is that ‘redistributive’ measures are harder to sell than ‘distributive’, while both may be less attractive than regulation (although complex problems likely require a mix of instruments).

These insights connect to Lindblom’s classic vision of:

1. Incremental analysis. It is better to research in-depth a small number of feasible options than spread your resources too thinly to consider all possibilities.
2. Strategic analysis. The feasibility of a solution relates strongly to current policy. The more radical a departure from the current negotiated position, the harder it will be to sell.

As many posts in the Policy Analysis in 750 words series describe, this advice is not entirely  useful for actors who seek rapid and radical departures from the status quo. Lindblom’s response to such critics was to seek radical change via a series of non-radical steps (at least in political systems like the US), which (broadly speaking) represents one of two possible approaches.

While incrementalism is not as popular as it once was (as a description of, or prescription for, policymaking), it tapped into the enduring insight that policymaking systems produce huge amounts of minor change. Rapid and radical policy change is rare, and it is even rarer to be able to connect it to influential analysis and action (at least in the absence of a major event). This knowledge should not put people off trying, but rather help them understand the obstacles that they seek to overcome.

Relating feasible solutions and strategies to ‘policy success’

One way to incorporate this kind of advice is to consider how (especially elected) policymakers would describe their own policy success. The determination of success and failure is a highly contested and political process (not simply a technical exercise called ‘evaluation’), and policymakers may refer – often implicitly – to the following questions when seeking success:

1. Political. Will this policy boost my government’s credibility and chances of re-election?
2. Process. Will it be straightforward to legitimise and maintain support for this policy?
3. Programmatic. Will it achieve its stated objectives and produce beneficial outcomes if implemented?

The benefit to analysts, in asking themselves these questions, is that they help to identify the potential solutions that are technically but not politically feasible (or vice versa).

The absence of clear technical feasibility does not necessarily rule out solutions with wider political benefits (for example, it can be beneficial to look like you are trying to do something good). Hence the popular phrase ‘good politics, bad policy’.

Nor does a politically unattractive option rule out a technically feasible solution (not all politicians flee the prospect of ‘good policy, bad politics’). However, it should prompt attention to hard choices about whose support to seek, how long to wait, or how hard to push, to seek policy change. You can see this kind of thinking as ‘entrepreneurial‘ or ‘systems thinking’ depending on how much faith you have in agency in highly-unequal political contexts.

Further reading

It is tempting to conclude that these obstacles to ‘good policy’ reflect the pathological nature of politics. However, if we want to make this argument, we should at least do it well:

1. You can find this kind of argument in fields such as public health and climate change studies, where researchers bemoan the gap between (a) their high-quality evidence on an urgent problem and (b) a disproportionately weak governmental response. To do it well, we need to separate analytically (or at least think about): (a) the motivation and energy of politicians (usually the source of most criticism of low ‘political will’), and (b) the policymaking systems that constrain even the most sincere and energetic policymakers. See the EBPM page for more.

2. Studies of Social Construction and Policy Design are useful to connect policymaking research with a normative agenda to address ‘degenerative’ policy design.

Policy Analysis in 750 Words: Changing things from the inside

How should policy actors seek radical changes to policy and policymaking?

This question prompts two types of answer:

1. Be pragmatic, and change things from the inside

Pragmatism is at the heart of most of the policy analysis texts in this series. They focus on the needs and beliefs of clients (usually policymakers). Policymakers are time-pressed, so keep your analysis short and relevant. See the world through their eyes. Focus on solutions that are politically as well as technically feasible. Propose non-radical steps, which may add up to radical change over the long-term.

This approach will seem familiar to students of research ‘impact’ strategies which emphasise relationship-building, being available to policymakers, and responding to the agendas of governments to maximise the size of your interested audience.

It will also ring bells for advocates of radical reforms in policy sectors such as (public) health and intersectoral initiatives such as gender mainstreaming:

• Health in All Policies is a strategy to encourage radical changes to policy and policymaking to improve population health.  Common advice includes to: identify to policymakers how HiAP fits into current policy agendas, seek win-win strategies with partners in other sectors, and go to great lengths to avoid the sense that you are interfering in their work (‘health imperialism’).
• Gender mainstreaming is a strategy to consider gender in all aspect of policy and policymaking. An equivalent playbook involves steps to: clarify what gender equality is, and what steps may help achieve it; make sure that these ideas translate across all levels and types of policymaking; adopt tools to ensure that gender is a part of routine government business (such as budget processes); and, modify existing policies or procedures while increasing the representation of women in powerful positions.

In other words, the first approach is to pursue your radical agenda via non-radical means, using a playbook that is explicitly non-confrontational.  Use your insider status to exploit opportunities for policy change.

2. Be radical, and challenge things from the outside

Challenging the status quo, for the benefit of marginalised groups, is at the heart of critical policy analysis:

• Reject the idea that policy analysis is a rationalist, technical, or evidence-based process. Rather, it involves the exercise of power to (a) depoliticise problems to reduce attention to current solutions, and (b) decide whose knowledge counts.
• Identify and question the dominant social constructions of problems and populations, asking who decides how to portray these stories and who benefits from their outcomes.

This approach resonates with frequent criticisms of ‘impact’ advice, emphasising the importance of producing research independent of government interference, to challenge policies that further harm already-marginalised populations.

It will also rings bells among advocates of more confrontational strategies to seek radical changes to policy and policymaking. They include steps to: find more inclusive ways to generate and share knowledge, produce multiple perspectives on policy problems and potential solutions, focus explicitly on the impact of the status quo on marginalised populations, politicise issues continuously to ensure that they receive sufficient attention, and engage in outsider strategies to protest current policies and practices.

Does this dichotomy make sense?

It is tempting to say that this dichotomy is artificial and that we can pursue the best of both worlds, such as working from within when it works and resorting to outsider action and protest when it doesn’t.

However, the blandest versions of this conclusion tend to ignore or downplay the politics of policy analysis in favour of more technical fixes. Sometimes collaboration and consensus politics is a wonderful feat of human endeavour. Sometimes it is a cynical way to depoliticise issues, stifle debate, and marginalise unpopular positions.

This conclusion also suggests that it is possible to establish what strategies work, and when, without really saying how (or providing evidence for success that would appeal to audiences associated with both approaches). Indeed, a recurrent feature of research in these fields is that most attempts to produce radical change prove to be dispiriting struggles. Non-radical strategies tend to be co-opted by more powerful actors, to mainstream new ways of thinking without changing the old. Radical strategies are often too easy to dismiss or counter.

The latter point reminds us to avoid excessively optimistic overemphasis on the strategies of analysts and advocates at the expense of context and audience. The 500 and 1000 words series perhaps tip us too far in the other direction, but provide a useful way to separate (analytically) the reasons for often-minimal policy change. To challenge dominant forms of policy and policymaking requires us to separate the intentional sources of inertia from the systemic issues that would constrain even the most sincere and energetic reformer.

Further reading

This post forms one part of the Policy Analysis in 750 words series, including posts on the role of analysts and marginalised groups. It also relates to work with St Denny, Kippin, and Mitchell (drawing on this draft paper) and posts on ‘evidence based policymaking’.