Policy analysis in 750 words: Three systems evaluation

There is much to be said for systems thinking or a systems-informed approach to key tasks such as evaluation. If the policy problem is complex, then we would not expect a single shot solution to work as intended in a linear or even straightforward way. If so, traditional ways to assess success or failure would be inappropriate. This broad argument has some traction, even in places associated with traditional evaluations in relation to policy cycles (such as the UK Treasury and its Magenta book supplement). However, just as systems thinking can mean 10 different things (and refer to two contradictory approaches), systems evaluation can involve very different tasks. Beware the possibility of contradictory ideas emerging from this process.

What exactly is a systems approach?

Systems thinking relates to the idea that key aspects of policy and policymaking resemble the complex systems that we find in nature, society, the brain, and information technology. Here, complex does not mean complicated. Rather, complex systems are greater than the sum of their parts, and exhibit non-linear dynamics, and prompt outcomes that emerge without central control.

Crucially, this argument can be related to many different types of policy relevant systems, including:

1. Complex policy problems. Policy problems are multi-faceted or inter-related, difficult to break down into specific parts, and not amenable to simple solutions.
2. Complex policy processes. Policymaking systems resemble complex systems
3. Complex policy mixes. ‘Policy’ is actually a policy mix, or large collection of policy instruments. The overall impact, of directing many policy instruments at one or more problems, is difficult to control or even predict. Overall policy incoherence may be a feature of such complexity.

In that context, I argue that people may be conflating three separate issues in the name of systems thinking evaluation, potentially with ironic unintended consequences. I make the distinction as follows:

1. Systems evaluations focusing on a complex policy problem, such as obesity. The aim is to identify the interconnectedness of the policy problem, such as to connect poverty, education, safe and healthy spaces, access to healthy food, commercial incentives, and so on. If so, we can adopt a ‘whole systems’ perspective to try to understand the problem, respond, and seek more meaningful ways to evaluate. This perspective may also be used to reject simplistic descriptions of problems (e.g. scroll down this page to see the ‘obesity system atlas’).
2. Systems evaluation focusing on a policymaking or governance problem, such as a lack of concerted attention or effective collaboration to solve a problem. The aim is to identify the interconnectedness of policy processes, such as: who is responsible for what, how does it all add up to systemic action, and what would a good policymaking system look like? Or, we may try to provide a map of policymaking responsibilities spread across levels of government (e.g. national/local) or sectors (e.g. health, education, transport, environment). If we intervene to change how we do things around here, we can assess the impact on things like levels of policy coherence or policymaking integration or cooperation. This perspective may also be used to counter the idea that one single centre of government can change things from the top-down.
3. Systems evaluation focusing on the overall impact of the policy mix. If many organisations use many instruments, is the policy mix coherent? Does each instrument reinforce or undermine the other? If we find limited success in relation to one instrument, is the problem really the other instruments? Or the delivery of the same instruments in different or unintended ways?

The potential irony arises if we recommend systems thinking in relation to one aspect but take a rather linear approach to others. Here, the classic mistake is to apply systems thinking to a problem but then act as if the solution can be produced by one central authority in a straightforward manner in a policymaking system. It is strange to see policy recommendations about problem complexity that do not recognise policymaking complexity. Why spend so much time identifying the interconnectedness of problems only to propose simple accounts of the responsibility for solutions?

One answer worth pursuing (because it represents a point of common ground) is that lots of people are inspired by Donella Meadows to find ‘leverage points’, ‘where a small shift in one thing can produce big changes in everything’. Yet, Meadows notes there are ‘no cheap tickets’ to a full understanding of systems. In our case, there is no easy way to find leverage points in complex policymaking systems, even if you have a good sense of a complex problem.

See also:

Complex systems and systems thinking

Policy Concepts in 1000 Words: Complex Systems 

An academic story of contemporary policy and policymaking problems

The draft of chapter 2 describes some of these concepts in more detail

Acknowledgement: I thank Dr Ravita Taheem for ongoing discussions to inform this post.