Understanding Public Policy 2nd edition

All going well, it will be out in November 2019. We are now at the proofing stage.

I have included below the summaries of the chapters (and each chapter should also have its own entry (or multiple entries) in the 1000 Words and 500 Words series).

 

Putting it all together: dissertation research question and research design (POLU9RM)

Writing a dissertation can be daunting. It is likely the longest piece of work you will plan as an undergraduate (10000 words plus bibliography) but, when you are done, it will not seem long enough.

On the one hand, it is a joyous exploration of research, in which you receive supervision but are in charge. On the other, you don’t want it to go horribly wrong.

It seems unlikely that reading my blog will spark joy, but I can at least give you some tips to avoid unnecessary problems and make your dissertation manageable.

Other advice (such as the reading in your module guide) is available, and I suggest you take it. Indeed, whenever I speak with colleagues about my approach to supervision, it seems relatively conservative and joyless.

On the other hand, why not play it safe with the dissertation then use all the time you’ve saved by seeking joy in a lovely meadow or a summer’s day?

1. Ask the right research question.

Most undergraduate coursework involves answering your lecturer’s rather generic question. Your task is to produce something a bit different, with some of these characteristics:

• You should find it interesting and want to answer it.
• It should be something that you can answer.
• It should be specific enough to help you manage your time well and answer it with the resources you have.

Compare with Halperin and Heath’s (p164) criteria, in which it should be important, ‘researchable’, and it has not been ‘answered definitively’.

(also compare with Dunleavy’s call for full narrative titles)

For example, many projects that I supervise follow roughly the same format: what is policy, how much has it changed, and why?

We can then narrow it down in several ways by choosing a specific issue, political system, time period, and/ or aspect of policy change.

This narrowing can make the difference between:

(a) feeling the need to explain many theories in the literature review, versus

(b) limiting theory selection by focusing on a small number of political system dynamics.

Action point 1

Describe your initial question or theme with your supervisor, and work with them until you are both happy with the question.

2. Write the abstract and the introduction first?

Many people suggest that your first main piece of work should be the literature review, for quite good reasons:

• It allows you to gain enough initial knowledge to help you guide your research
• It allows you to get writing – often a major stumbling block – and then edit later

I suggest that your first piece of work should be the abstract and introduction for these reasons:

• Writing a half page abstract allows you to describe what your project adds up to.
• It really helps you discuss your plans with your supervisor
• Writing the introduction allows you to describe your research design in enough depth to reflect on it is coherence and feasibility.
• All going well, it will be only a small jump from your POLU9RM ‘research project design’ exercise.
• It allows you to make sense of a quite general format for research publications (in many fields): theory, method, results.

Action point 2

Write the question/ title and abstract, share it with your likely supervisor, and talk about how coherent and feasible your plan looks.

3. Identify the relevant theory or literature.

In some cases, the potentially relevant literature is vast if you have, for example:

1. A too-general question about political parties or elections.

• One good solution is to select a subfield like ‘pledge fulfilment’

2. A too-general question about policy change.

• One good solution is to ask if (say) the advocacy coalition framework or multiple streams analysis helps explain your case study.
• Or, I have supervised many good dissertations asking: what can X government learn from Y government about solving Z policy problem?

Action point 3.

Make sure to connect your research question to a well-defined literature (and do a preliminary literature search to see what is out there)

4. Identify your method to gather information.

Halperin and Heath’s chapter 7 goes into some depth about the principles of research design:

• what data collection is appropriate
• what we can deduce from certain data
• how confident you can be about cause/effect in this case (internal validity)
• and cause-and-effect more generally (external validity)
• if someone could do your research and get the same results (reliability)

They also describe the types of design you can likely not do (well, 1 and 2) in a UG dissertation, but can get the data to analyse:

1. Experimental (like an RCT)
2. Cross-sectional and longitudinal
3. Comparative (for which I did a separate post)

Then they describe data gathering strategies that you might be tempted to do (subject to ethical clearance):

• Surveys
• Interviews
• Focus groups
• Ethnographic
• Discourse analysis

In the lecture, I will put on my dour face and warn you against most of these methods, for reasons such as:

• Doing a proper survey takes a lot of time and resources, someone has likely already done a better one, and it would be a shame not to find it
• You can often find things in the public record without interviewing someone (and maybe they will only repeat what is out there)
• The ethical clearance will be a major issue with ethnographic (and other) methods

I won’t try to put you off entirely. Rather, I will encourage you to ask yourself:

• Why are you choosing this method?

1. Does it relate clearly to your research question?
2. Or, have you begun with the most interesting sounding method?
3. Or, do you have some sort of connection that gets you access, which seems a shame not to use?

• Are you prepared to do a literature review on your chosen method?
• What do you realistically expect to get from your method?
• What will you do if it goes wrong?

Action point 4

Discuss your choice of data collection with your supervisor.

5. Think about how you will analyse and interpret the results.

This part tends to make the difference between a very good or an excellent dissertation.

• Raul Pacheco-Vega gives many useful examples in Distinguishing between description and analysis in academic writing

Put most simply, simple description involves summarising things. Analysis is about telling the reader what the results mean. For example, you might:

• Evaluate the size of the results according to your expectations. Does a survey result seem unusual?
• Describe how much one should rely on the results. Does the result seem important after taking into account a margin of error?
• Describe the wider context. Does the result mark a change over time, or seem different from another country?
• Relate a case study result to your literature review. Is your case unusual, or as expected?

Action point 5

Clarify the difference between summary and analysis

6. Be clear about the conclusion.

Don’t just to the dissertation equivalent of saying ‘cheerio’ (or, my favourite thing, leaving without saying cheerio).

The conclusion differs from:

• The introduction, because you should use it to summarise your question and approach (perhaps quite briefly) and relate it in some depth to the results.
• The analysis of results, because you relate the results much more clearly to your overall project.

Don’t think of it as saying: ‘as I have said before …’

Think of it as saying: ‘here is what it all adds up to …’

7. The end.

Remember to add your bibliography and ask yourself if you need an appendix for your data (which does not count towards the word count).

PS some of my supervisees write policy analysis reports, which differ somewhat from regular dissertations. If you are keen, please see me and/ or read more here.

 

 

 

 

 

 

Taking lessons from policy theory into practice: 3 examples

Notes for ANZSOG/ ANU Crawford School/ UNSW Canberra workshop. Powerpoint here. The recording of the lecture (skip to 2m30) and Q&A is here (right click to download mp3 or dropbox link):

The context for this workshop is the idea that policy theories could be more helpful to policymakers/ practitioners if we could all communicate more effectively with each other. Academics draw general and relatively abstract conclusions from multiple cases. Practitioners draw very similar conclusions from rich descriptions of direct experience in a smaller number of cases. How can we bring together their insights and use a language that we all understand? Or, more ambitiously, how can we use policy theory-based insights to inform the early career development training that civil servants and researchers receive?

The first step is to translate policy theories into a non-technical language by trying to speak with an audience beyond our immediate peers (see for example Practical Lessons from Policy Theories).

However, translation is not enough. A second crucial step is to consider how policymakers and practitioners are likely to make sense of theoretical insights when they apply them to particular aims or responsibilities. For example:

1. Central government policymakers may accept the descriptive accuracy of policy theories emphasising limited central control, but not the recommendation that they should let go, share power, and describe their limits to the public.
2. Scientists may accept key limitations to ‘evidence based policymaking’ but reject the idea that they should respond by becoming better storytellers or more manipulative operators.
3. Researchers and practitioners struggle to resolve hard choices when combining evidence and ‘coproduction’ while ‘scaling up’ policy interventions. Evidence choice is political choice. Can we do more than merely encourage people to accept this point?

I discuss these examples below because they are closest to my heart (especially example 1). Note throughout that I am presenting one interpretation about: (1) the most promising insights, and (2) their implications for practice. Other interpretations of the literature and its implications are available. They are just a bit harder to find.

Example 1: the policy cycle endures despite its descriptive inaccuracy

The policy cycle does not describe and explain the policy process well:

• If we insist on keeping the cycle metaphor, it is more accurate to see the process as a huge set of policy cycles that connect with each other in messy and unpredictable ways.
• The cycle approach also links strongly to the idea of ‘comprehensive rationality’ in which a small group of policymakers and analysts are in full possession of the facts and full control of the policy process. They carry out their aims through a series of stages.

Policy theories provide more descriptive and explanatory usefulness. Their insights include:

• Limited choice. Policymakers inherit organisations, rules, and choices. Most ‘new’ choice is a revision of the old.
• Limited attention. Policymakers must ignore almost all of the policy problems for which they are formally responsible. They pay attention to some, and delegate most responsibility to civil servants. Bureaucrats rely on other actors for information and advice, and they build relationships on trust and information exchange.
• Limited central control. Policy may appear to be made at the ‘top’ or in the ‘centre’, but in practice policymaking responsibility is spread across many levels and types of government (many ‘centres’). ‘Street level’ actors make policy as they deliver. Policy outcomes appear to ‘emerge’ locally despite central government attempts to control their fate.
• Limited policy change. Most policy change is minor, made and influenced by actors who interpret new evidence through the lens of their beliefs. Well-established beliefs limit the opportunities of new solutions. Governments tend to rely on trial-and-error, based on previous agreements, rather than radical policy change based on a new agenda. New solutions succeed only during brief and infrequent windows of opportunity.

However, the cycle metaphor endures because:

• It provides a simple model of policymaking with stages that map onto important policymaking functions.
• It provides a way to project policymaking to the public. You know how we make policy, and that we are in charge, so you know who to hold to account.

In that context, we may want to be pragmatic about our advice:

1. One option is via complexity theory, in which scholars generally encourage policymakers to accept and describe their limits:

• Accept routine error, reduce short-term performance management, engage more in trial and error, and ‘let go’ to allow local actors the flexibility to adapt and respond to their context.
• However, would a government in the Westminster tradition really embrace this advice? No. They need to balance (a) pragmatic policymaking, and (b) an image of governing competence.

2. Another option is to try to help improve an existing approach.

• Koski and Workman build on PET to recommend ways to process information more effectively.
• Studies built on MSA tend to focus on ‘entrepreneurs’ having a technically and politically feasible solution ready for the next window of opportunity.
• Sanderson describes the skills required for ‘intelligent policymaking’.
• I sum up these approaches in ‘How Can Policy Theory Have an Impact on Policy Making?’ Teaching Public Administration, 33, 1, 22-39 

Further reading (blog posts):

The language of complexity does not mix well with the language of Westminster-style accountability

Making Sense of Policymaking: why it’s always someone else’s fault and nothing ever changes

Two stories of British politics: the Westminster model versus Complex Government

Example 2: how to deal with a lack of ‘evidence based policymaking’

I used to read many papers on tobacco policy, with the same basic message: we have the evidence of tobacco harm, and evidence of which solutions work, but there is an evidence-policy gap caused by too-powerful tobacco companies, low political will, and pathological policymaking. These accounts are not informed by theories of policymaking.

I then read Oliver et al’s paper on the lack of policy theory in health/ environmental scholarship on the ‘barriers’ to the use of evidence in policy. Very few articles rely on policy concepts, and most of the few rely on the policy cycle. This lack of policy theory is clear in their description of possible solutions – better communication, networking, timing, and more science literacy in government – which does not describe well the need to respond to policymaker psychology and a complex policymaking environment.

So, I wrote The Politics of Evidence-Based Policymaking and one zillion blog posts to help identify the ways in which policy theories could help explain the relationship between evidence and policy.

Since then, the highest demand to speak about the book has come from government/ public servant, NGO, and scientific audiences outside my discipline. The feedback is generally that: (a) the book’s description sums up their experience of engagement with the policy process, and (b) maybe it opens up discussion about how to engage more effectively.

But how exactly do we turn empirical descriptions of policymaking into practical advice?

For example, scientist/ researcher audiences want to know the answer to a question like: Why don’t policymakers listen to your evidence? and so I focus on three conversation starters:

1. they have a broader view on what counts as good evidence (see ANZSOG description)
2. they have to ignore almost all information (a nice way into bounded rationality and policymaker psychology)
3. they do not understand or control the process in which they seek to use evidence (a way into ‘the policy process’)

We can then consider many possible responses in the sequel What can you do when policymakers ignore your evidence?

Examples include:

• ‘How to do it’ advice. I compare tips for individuals (from experienced practitioners) with tips based on policy concepts. They are quite similar-looking tips – e.g. find out where the action is, learn the rules, tell good stories, engage allies, seek windows of opportunity – but I describe mine as 5 impossible tasks!
• Organisational reform. I describe work with the European Commission Joint Research Centre to identify 8 skills or functions of an organisation bringing together the supply/demand of knowledge.
• Ethical dilemmas. I use key policy theories to ask people how far they want to go to privilege evidence in policy. It’s fun to talk about these things with the type of scientist who sees any form of storytelling as manipulation.

Further reading:

Is Evidence-Based Policymaking the same as good policymaking?

A 5-step strategy to make evidence count

Political science improves our understanding of evidence-based policymaking, but does it produce better advice?

Principles of science advice to government: key problems and feasible solutions

Example 3: how to encourage realistic evidence-informed policy transfer

This focus on EBPM is useful context for discussions of ‘policy learning’ and ‘policy transfer’, and it was the focus of my ANZOG talk entitled (rather ambitiously) ‘teaching evidence-based policy to fly’.

I’ve taken a personal interest in this one because I’m part of a project – called IMAJINE – in which we have to combine academic theory and practical responses. We are trying to share policy solutions across Europe rather than explain why few people share them!

For me, the context is potentially overwhelming:

• Policy learning is a political process, and expert-driven learning is one of many possibilities, alongside dialogue, bargaining, and hierarchical models.
• Indeed, researchers often do badly because they don’t learn how to deal with the politics.
• Policy transfer can be maximal or minimal, and driven voluntarily or via coercion. A policy solution, when ‘imported’, can be mimicked, translated or transformed.
• When transfer is voluntary, receptivity to new solutions is more important than their supply.
• There are very different ‘evidence-based’ models on how to ‘scale up’ policy success.

So, when we start to focus on sharing lessons, we will have three things to discover:

1. What is the evidence for success, and from where does it come? Governments often project success without backing it up.
2. What story do policymakers tell about the problem they are trying to solve, the solutions they produced, and why? Two different governments may be framing and trying to solve the same problem in very different ways.
3. Was the policy introduced in a comparable policymaking system? People tend to focus on political system comparability (e.g. is it unitary or federal?), but I think the key is in policymaking system comparability (e.g. what are the rules and dominant ideas?).

To be honest, when one of our external assessors asked me how well I thought I would do, we both smiled because the answer may be ‘not very’. In other words, the most practical lesson may be the hardest to take, although I find it comforting: the literature suggests that policymakers might ignore you for 20 years then suddenly become very (but briefly) interested in your work.

 

The slides are a bit wonky because I combined my old ppt to the Scottish Government with a new one for UNSW Paul Cairney ANU Policy practical 22 October 2018

I wanted to compare how I describe things to (1) civil servants (2) practitioners/ researcher (3) me, but who has the time/ desire to listen to 3 powerpoints in one go? If the answer is you, let me know and we’ll set up a Zoom call.

Standing on the Shoulders of Giants?

[Updated in 2018 to just give you the article here, but I’ll leave the rest just in case I forget that it took a few years to work out how to blog well.]

My new year’s resolution was to make a blog entry for every academic article published from 2013, since the article may be behind a paywall (although if you contact me, I will see you right) and the article’s ideas may be expressed in a relatively inaccessible way (although we don’t all spew jargon-filled group-closure nonsense).  The aim is to get people interested enough to go from a short tweet to a larger blog to the high bar of reading (or the holy grail of citing) the article itself.

This article is called ‘Standing on the Shoulders of Giants’ because I wanted to give the impression that we are discussing the accumulation of scientific knowledge; our aim is to build on the insights and knowledge produced by others rather than start from scratch each time.  As stated, this is fairly uncontroversial and we might find that most people can get behind the project (in fact, they are already doing so, implicitly or explicitly).  The more problematic and debatable part of this task relates to the details: *how* do we do it?

The article focuses on this task in the policy literature, but the themes extend to political, social and, in most cases, the so called ‘hard’ sciences.  In fact, for many of us, it may be reminiscent of postgraduate discussions of the philosophy of science, in which we consider the inadequacy of most explanations of how knowledge is accumulated (from the ‘strawman’ of inductivism to the often-caricatured position of Popper (on falsification), to the idea of paradigm shift made famous by Kuhn and the rather-misleading ‘anything goes’ description of the approach by Feyerabend – a discussion captured neatly by Chalmers).  Many of us will have concluded two things: (1) we believe that we are in the business of accumulating knowledge/ we know much more about the world now than we did in the past, and we have acted accordingly; but, (2) we have no idea *how* that has happened because all of the explanations of knowledge accumulation are problematic, while some suggest that one body of knowledge *replaces* another rather than building on it.

In that broad context, the article (a) outlines three main ways in which scholars address this issue in policy studies and political science; and, (b) highlights the problems that may arise in each case:

1. Synthesis – we combine the insights of multiple theories, concepts or models to produce a single theory (in fact, the article discusses the difference between ‘synthesis’ and ‘super-synthesis’, but I don’t want to undermine my “we don’t all spew jargon-filled group-closure nonsense”).  One key problem is that when we produce a synthetic theory, from a range of other theories or concepts, we have to assume that the component parts of this new hybrid are consistent with each other. Yet, if you scratch the surface of many concepts – such as ‘new institutionalism’ or ‘policy networks’ – you find all sorts of disagreement about the nature of the world, how our concepts relate to it and how we gather knowledge of it. There are also practical problems regarding our assumption that the authors of these concepts have the same thing in mind when they describe things like ‘punctuated equilibrium’.  In other words, imagine that you have constructed a new theory based on the wisdom of five other people.  Then, get those people in the same room and you will find that they will share all sorts of – often intractable – disagreements with each other.  In that scenario, could you honestly state that your theory was based on accumulated knowledge?

2. The ‘Complementary’ Approach. In this case, you accept that people have these differences and so you accommodate them – you entertain a range of theories/ concepts and explore the extent to which they explain the same thing in different ways.  This is a popular approach associated with people like Allison (who compared three different explanations of the Cuban missile crisis) and used by several others to compare policy events.  One key problem with this approach is that it is difficult to do full justice to each theory.  Most theories have associated methods which are labour intensive and costly, putting few in the position to make meaningful comparisons.  Instead, the comparisons tend to be desktop exercises based on a case study and the authors’ ability to consider how each theory would explain it.

3. The ‘Contradictory’ Approach.  In that context, another option is to encourage the independence of such theories. You watch as different research teams produce their own studies and you simply try to find some way to compare and combine their insights.  Of course, it is impossible to entertain an infinite number of theories, so we also need some way to compare them; to select some and reject others.  This is the approach that we may be most familiar with, since it involves coming up with a set of rules or criteria to make sure that each theory can be accepted (at least initially) by the scientific community.  You may see such rules described as follows:

• A theory’s methods should be explained so that they can be replicated by others.
• Its concepts should be clearly defined, logically consistent, and give rise to empirically falsifiable hypotheses.
• Its propositions should be as general as possible.
• It should set out clearly what the causal processes are.
• It should be subject to empirical testing and revision.

For me, this is where the task becomes very interesting because, on the one hand, most of us will find these aims to be intuitively appealing – but, on the other, they are incredibly problematic for the following reasons:

• Few, if any, theories or research projects live up to these expectations.
• The principles give a misleading impression of most (social?) scientific research which is largely built on trust rather than constant replication by others.
• Many of the most famous proponents of this approach do something a bit different – such as when they subject their ‘secondary hypotheses’ to rigorous testing but insulate their ‘hard core’ from falsification.
• The study of complex phenomenon may not allow us to falsify, since we can interpret our findings in very different ways.
• Few theories are currently popular simply because they adhere to these principles.  In fact, science is much more of a social enterprise than the principles suggest.

Of course, by now you may have identified a key problem with this argument: it is all beginning to sound a bit ‘postpositivist’ (which, in my mind, is still more of a term of abuse than ‘you, my friend, are a positivist’).  However, it does not need to be taken this way.  It is OK to highlight problems with scientific principles and admit that science is about the methods and beliefs accepted by a particular scientific community because, if you like, you can still assert that those principles and beliefs are *correct*. Many, many, people do.  In fact, perhaps we all do it, because we have to find a way to accept some theories, approaches and evidence and reject others.  We seek a way to produce some knowledge ourselves and find a common language and set of principles to make sure that we can compare our knowledge with the knowledge of others.  We seek a way to sift through an almost infinite number of ‘signals’ from our environment, to pay attention to very few and ignore most.  That task requires rules which are problematic but necessary.

All I suggest we do (which is a bit of a bland recommendation) is to reject the unthinking and too-rigid application of rules that hold us all up to a standard that no-one will meet.  Rather, people in different disciplines might discuss and negotiate those rules with each other. This is more of an art than a science.

I also argue that (a) if we are serious about these rules, and the need to submit theories and evidence to rigorous testing; but (b) we accept that most of this is done on trust rather than replication; then (c) we should take on some of that burden ourselves by subjecting our own evidence to a form of testing, in which we consider the extent to which our findings can be interpreted in different, and equally plausible, ways.  The article talks about producing different ‘narratives’ of the same evidence, but I won’t talk about that too much in case you confuse me with the presenter of Jackanory.

Full reference: Cairney, P. (2013) ‘Standing on the Shoulders of Giants: How Do We Combine the Insights of Multiple Theories in Public Policy Studies?’ Policy Studies Journal, 41, 1, 1-21 PDF http://dx.doi.org/10.1111/psj.12000

See also:

Policy Concepts in 1000 Words: Combining Theories (the rewritten version)

Comparing Theories of the Policy Process: A Brief Guide for Postgraduates