Tag Archives: Theory

Taking lessons from policy theory into practice: 3 examples

Notes for ANZSOG/ ANU Crawford School/ UNSW Canberra workshop. Powerpoint here. The recording of the lecture (skip to 2m30) and Q&A is here (right click to download mp3 or dropbox link):

The context for this workshop is the idea that policy theories could be more helpful to policymakers/ practitioners if we could all communicate more effectively with each other. Academics draw general and relatively abstract conclusions from multiple cases. Practitioners draw very similar conclusions from rich descriptions of direct experience in a smaller number of cases. How can we bring together their insights and use a language that we all understand? Or, more ambitiously, how can we use policy theory-based insights to inform the early career development training that civil servants and researchers receive?

The first step is to translate policy theories into a non-technical language by trying to speak with an audience beyond our immediate peers (see for example Practical Lessons from Policy Theories).

However, translation is not enough. A second crucial step is to consider how policymakers and practitioners are likely to make sense of theoretical insights when they apply them to particular aims or responsibilities. For example:

  1. Central government policymakers may accept the descriptive accuracy of policy theories emphasising limited central control, but not the recommendation that they should let go, share power, and describe their limits to the public.
  2. Scientists may accept key limitations to ‘evidence based policymaking’ but reject the idea that they should respond by becoming better storytellers or more manipulative operators.
  3. Researchers and practitioners struggle to resolve hard choices when combining evidence and ‘coproduction’ while ‘scaling up’ policy interventions. Evidence choice is political choice. Can we do more than merely encourage people to accept this point?

I discuss these examples below because they are closest to my heart (especially example 1). Note throughout that I am presenting one interpretation about: (1) the most promising insights, and (2) their implications for practice. Other interpretations of the literature and its implications are available. They are just a bit harder to find.

Example 1: the policy cycle endures despite its descriptive inaccuracy

cycle

The policy cycle does not describe and explain the policy process well:

  • If we insist on keeping the cycle metaphor, it is more accurate to see the process as a huge set of policy cycles that connect with each other in messy and unpredictable ways.
  • The cycle approach also links strongly to the idea of ‘comprehensive rationality’ in which a small group of policymakers and analysts are in full possession of the facts and full control of the policy process. They carry out their aims through a series of stages.

Policy theories provide more descriptive and explanatory usefulness. Their insights include:

  • Limited choice. Policymakers inherit organisations, rules, and choices. Most ‘new’ choice is a revision of the old.
  • Limited attention. Policymakers must ignore almost all of the policy problems for which they are formally responsible. They pay attention to some, and delegate most responsibility to civil servants. Bureaucrats rely on other actors for information and advice, and they build relationships on trust and information exchange.
  • Limited central control. Policy may appear to be made at the ‘top’ or in the ‘centre’, but in practice policymaking responsibility is spread across many levels and types of government (many ‘centres’). ‘Street level’ actors make policy as they deliver. Policy outcomes appear to ‘emerge’ locally despite central government attempts to control their fate.
  • Limited policy change. Most policy change is minor, made and influenced by actors who interpret new evidence through the lens of their beliefs. Well-established beliefs limit the opportunities of new solutions. Governments tend to rely on trial-and-error, based on previous agreements, rather than radical policy change based on a new agenda. New solutions succeed only during brief and infrequent windows of opportunity.

However, the cycle metaphor endures because:

  • It provides a simple model of policymaking with stages that map onto important policymaking functions.
  • It provides a way to project policymaking to the public. You know how we make policy, and that we are in charge, so you know who to hold to account.

In that context, we may want to be pragmatic about our advice:

  1. One option is via complexity theory, in which scholars generally encourage policymakers to accept and describe their limits:
  • Accept routine error, reduce short-term performance management, engage more in trial and error, and ‘let go’ to allow local actors the flexibility to adapt and respond to their context.
  • However, would a government in the Westminster tradition really embrace this advice? No. They need to balance (a) pragmatic policymaking, and (b) an image of governing competence.
  1. Another option is to try to help improve an existing approach.

Further reading (blog posts):

The language of complexity does not mix well with the language of Westminster-style accountability

Making Sense of Policymaking: why it’s always someone else’s fault and nothing ever changes

Two stories of British politics: the Westminster model versus Complex Government

Example 2: how to deal with a lack of ‘evidence based policymaking’

I used to read many papers on tobacco policy, with the same basic message: we have the evidence of tobacco harm, and evidence of which solutions work, but there is an evidence-policy gap caused by too-powerful tobacco companies, low political will, and pathological policymaking. These accounts are not informed by theories of policymaking.

I then read Oliver et al’s paper on the lack of policy theory in health/ environmental scholarship on the ‘barriers’ to the use of evidence in policy. Very few articles rely on policy concepts, and most of the few rely on the policy cycle. This lack of policy theory is clear in their description of possible solutions – better communication, networking, timing, and more science literacy in government – which does not describe well the need to respond to policymaker psychology and a complex policymaking environment.

So, I wrote The Politics of Evidence-Based Policymaking and one zillion blog posts to help identify the ways in which policy theories could help explain the relationship between evidence and policy.

Since then, the highest demand to speak about the book has come from government/ public servant, NGO, and scientific audiences outside my discipline. The feedback is generally that: (a) the book’s description sums up their experience of engagement with the policy process, and (b) maybe it opens up discussion about how to engage more effectively.

But how exactly do we turn empirical descriptions of policymaking into practical advice?

For example, scientist/ researcher audiences want to know the answer to a question like: Why don’t policymakers listen to your evidence? and so I focus on three conversation starters:

  1. they have a broader view on what counts as good evidence (see ANZSOG description)
  2. they have to ignore almost all information (a nice way into bounded rationality and policymaker psychology)
  3. they do not understand or control the process in which they seek to use evidence (a way into ‘the policy process’)

Cairney 2017 image of the policy process

We can then consider many possible responses in the sequel What can you do when policymakers ignore your evidence?

Examples include:

  • ‘How to do it’ advice. I compare tips for individuals (from experienced practitioners) with tips based on policy concepts. They are quite similar-looking tips – e.g. find out where the action is, learn the rules, tell good stories, engage allies, seek windows of opportunity – but I describe mine as 5 impossible tasks!
  • Organisational reform. I describe work with the European Commission Joint Research Centre to identify 8 skills or functions of an organisation bringing together the supply/demand of knowledge.
  • Ethical dilemmas. I use key policy theories to ask people how far they want to go to privilege evidence in policy. It’s fun to talk about these things with the type of scientist who sees any form of storytelling as manipulation.

Further reading:

Is Evidence-Based Policymaking the same as good policymaking?

A 5-step strategy to make evidence count

Political science improves our understanding of evidence-based policymaking, but does it produce better advice?

Principles of science advice to government: key problems and feasible solutions

Example 3: how to encourage realistic evidence-informed policy transfer

This focus on EBPM is useful context for discussions of ‘policy learning’ and ‘policy transfer’, and it was the focus of my ANZOG talk entitled (rather ambitiously) ‘teaching evidence-based policy to fly’.

I’ve taken a personal interest in this one because I’m part of a project – called IMAJINE – in which we have to combine academic theory and practical responses. We are trying to share policy solutions across Europe rather than explain why few people share them!

For me, the context is potentially overwhelming:

So, when we start to focus on sharing lessons, we will have three things to discover:

  1. What is the evidence for success, and from where does it come? Governments often project success without backing it up.
  2. What story do policymakers tell about the problem they are trying to solve, the solutions they produced, and why? Two different governments may be framing and trying to solve the same problem in very different ways.
  3. Was the policy introduced in a comparable policymaking system? People tend to focus on political system comparability (e.g. is it unitary or federal?), but I think the key is in policymaking system comparability (e.g. what are the rules and dominant ideas?).

To be honest, when one of our external assessors asked me how well I thought I would do, we both smiled because the answer may be ‘not very’. In other words, the most practical lesson may be the hardest to take, although I find it comforting: the literature suggests that policymakers might ignore you for 20 years then suddenly become very (but briefly) interested in your work.

 

The slides are a bit wonky because I combined my old ppt to the Scottish Government with a new one for UNSW Paul Cairney ANU Policy practical 22 October 2018

I wanted to compare how I describe things to (1) civil servants (2) practitioners/ researcher (3) me, but who has the time/ desire to listen to 3 powerpoints in one go? If the answer is you, let me know and we’ll set up a Zoom call.

2 Comments

Filed under agenda setting, Evidence Based Policymaking (EBPM), IMAJINE, Policy learning and transfer

Standing on the Shoulders of Giants?

[Updated in 2018 to just give you the article here, but I’ll leave the rest just in case I forget that it took a few years to work out how to blog well.]
My new year’s resolution was to make a blog entry for every academic article published from 2013, since the article may be behind a paywall (although if you contact me, I will see you right) and the article’s ideas may be expressed in a relatively inaccessible way (although we don’t all spew jargon-filled group-closure nonsense).  The aim is to get people interested enough to go from a short tweet to a larger blog to the high bar of reading (or the holy grail of citing) the article itself.
This article is called ‘Standing on the Shoulders of Giants’ because I wanted to give the impression that we are discussing the accumulation of scientific knowledge; our aim is to build on the insights and knowledge produced by others rather than start from scratch each time.  As stated, this is fairly uncontroversial and we might find that most people can get behind the project (in fact, they are already doing so, implicitly or explicitly).  The more problematic and debatable part of this task relates to the details: *how* do we do it?
The article focuses on this task in the policy literature, but the themes extend to political, social and, in most cases, the so called ‘hard’ sciences.  In fact, for many of us, it may be reminiscent of postgraduate discussions of the philosophy of science, in which we consider the inadequacy of most explanations of how knowledge is accumulated (from the ‘strawman’ of inductivism to the often-caricatured position of Popper (on falsification), to the idea of paradigm shift made famous by Kuhn and the rather-misleading ‘anything goes’ description of the approach by Feyerabend – a discussion captured neatly by Chalmers).  Many of us will have concluded two things: (1) we believe that we are in the business of accumulating knowledge/ we know much more about the world now than we did in the past, and we have acted accordingly; but, (2) we have no idea *how* that has happened because all of the explanations of knowledge accumulation are problematic, while some suggest that one body of knowledge *replaces* another rather than building on it.
In that broad context, the article (a) outlines three main ways in which scholars address this issue in policy studies and political science; and, (b) highlights the problems that may arise in each case:

1. Synthesis – we combine the insights of multiple theories, concepts or models to produce a single theory (in fact, the article discusses the difference between ‘synthesis’ and ‘super-synthesis’, but I don’t want to undermine my “we don’t all spew jargon-filled group-closure nonsense”).  One key problem is that when we produce a synthetic theory, from a range of other theories or concepts, we have to assume that the component parts of this new hybrid are consistent with each other. Yet, if you scratch the surface of many concepts – such as ‘new institutionalism’ or ‘policy networks’ – you find all sorts of disagreement about the nature of the world, how our concepts relate to it and how we gather knowledge of it. There are also practical problems regarding our assumption that the authors of these concepts have the same thing in mind when they describe things like ‘punctuated equilibrium’.  In other words, imagine that you have constructed a new theory based on the wisdom of five other people.  Then, get those people in the same room and you will find that they will share all sorts of – often intractable – disagreements with each other.  In that scenario, could you honestly state that your theory was based on accumulated knowledge?

2. The ‘Complementary’ Approach. In this case, you accept that people have these differences and so you accommodate them – you entertain a range of theories/ concepts and explore the extent to which they explain the same thing in different ways.  This is a popular approach associated with people like Allison (who compared three different explanations of the Cuban missile crisis) and used by several others to compare policy events.  One key problem with this approach is that it is difficult to do full justice to each theory.  Most theories have associated methods which are labour intensive and costly, putting few in the position to make meaningful comparisons.  Instead, the comparisons tend to be desktop exercises based on a case study and the authors’ ability to consider how each theory would explain it.

3. The ‘Contradictory’ Approach.  In that context, another option is to encourage the independence of such theories. You watch as different research teams produce their own studies and you simply try to find some way to compare and combine their insights.  Of course, it is impossible to entertain an infinite number of theories, so we also need some way to compare them; to select some and reject others.  This is the approach that we may be most familiar with, since it involves coming up with a set of rules or criteria to make sure that each theory can be accepted (at least initially) by the scientific community.  You may see such rules described as follows:

  • A theory’s methods should be explained so that they can be replicated by others.
  • Its concepts should be clearly defined, logically consistent, and give rise to empirically falsifiable hypotheses.
  • Its propositions should be as general as possible.
  • It should set out clearly what the causal processes are.
  • It should be subject to empirical testing and revision.
For me, this is where the task becomes very interesting because, on the one hand, most of us will find these aims to be intuitively appealing – but, on the other, they are incredibly problematic for the following reasons:
  • Few, if any, theories or research projects live up to these expectations.
  • The principles give a misleading impression of most (social?) scientific research which is largely built on trust rather than constant replication by others.
  • Many of the most famous proponents of this approach do something a bit different – such as when they subject their ‘secondary hypotheses’ to rigorous testing but insulate their ‘hard core’ from falsification.
  • The study of complex phenomenon may not allow us to falsify, since we can interpret our findings in very different ways.
  • Few theories are currently popular simply because they adhere to these principles.  In fact, science is much more of a social enterprise than the principles suggest.
Of course, by now you may have identified a key problem with this argument: it is all beginning to sound a bit ‘postpositivist’ (which, in my mind, is still more of a term of abuse than ‘you, my friend, are a positivist’).  However, it does not need to be taken this way.  It is OK to highlight problems with scientific principles and admit that science is about the methods and beliefs accepted by a particular scientific community because, if you like, you can still assert that those principles and beliefs are *correct*. Many, many, people do.  In fact, perhaps we all do it, because we have to find a way to accept some theories, approaches and evidence and reject others.  We seek a way to produce some knowledge ourselves and find a common language and set of principles to make sure that we can compare our knowledge with the knowledge of others.  We seek a way to sift through an almost infinite number of ‘signals’ from our environment, to pay attention to very few and ignore most.  That task requires rules which are problematic but necessary.
All I suggest we do (which is a bit of a bland recommendation) is to reject the unthinking and too-rigid application of rules that hold us all up to a standard that no-one will meet.  Rather, people in different disciplines might discuss and negotiate those rules with each other. This is more of an art than a science.
I also argue that (a) if we are serious about these rules, and the need to submit theories and evidence to rigorous testing; but (b) we accept that most of this is done on trust rather than replication; then (c) we should take on some of that burden ourselves by subjecting our own evidence to a form of testing, in which we consider the extent to which our findings can be interpreted in different, and equally plausible, ways.  The article talks about producing different ‘narratives’ of the same evidence, but I won’t talk about that too much in case you confuse me with the presenter of Jackanory.
Full reference: Cairney, P. (2013) ‘Standing on the Shoulders of Giants: How Do We Combine the Insights of Multiple Theories in Public Policy Studies?’ Policy Studies Journal, 41, 1, 1-21 PDF http://dx.doi.org/10.1111/psj.12000
See also:

3 Comments

Filed under public policy