Teaching policy analysis with a blog

Below is the introduction to an article that I wrote for a Special Issue paper on Teaching Policy Analysis for Gestión y Análisis de Políticas Públicas (GAPP).

Paul Cairney (2023) ‘Teaching the politics of policy analysis, aided by a blog’, Gestión y Análisis de Políticas Públicas (GAPP), https://doi.org/10.24965/gapp.11146,  PDF

When we teach policy analysis, we focus on how to be a policy analyst or how to situate the act of policy analysis within a wider policymaking context. Ideally, students would learn about both. This aim is central to Lasswell’s vision for the policy sciences, in which the analysis of policy and policymaking informs analysis for policy, and both are essential to the pursuit of human equality and dignity (Lasswell, 1951; 1956; 1971; see Cairney and Weible, 2017).

There is the potential to achieve this vision for the policy sciences. Policy analysis texts focus on the individual and professional skills required to act efficiently and effectively in a time-pressured political environment. Further, they are supported by the study of policy analysts to reflect on how analysis takes place, and policy is made, in the real world (Radin, 2019; Brans et al, 2017; Thissen and Walker, 2013; Geva-May, 2005). The next steps would be to harness the wealth of policy concept- and theory-informed studies to help understand how real-world contexts inform policy analysis insights.

First, almost all mainstream policy theories assume or demonstrate that there is no such thing as a policy cycle. It would be misleading to suggest that the policy process consists of clearly defined and well-ordered stages of policymaking, from defining problems and generating solutions to implementing solutions and evaluating their effects. If so, there is no clear route to influence via analysis unless we understand a far messier reality. In that context, how can policy analysts understand their complex policymaking environment, and what skills and strategies do they need to develop to engage effectively? These discussions may be essential to preventing the demoralisation of analysts: if they do not learn in advance about the processes and factors that can minimise their influence, how can they generate realistic expectations?

Second, if the wider aim is human equality and dignity, insights from critical policy analysis are essential. They help analysts think about what those values mean, how to identify and support marginalised populations, and how policy analysis skills and techniques relate to those aims. In particular, they warn against treating policy analysis as a technocratic profession devoid of politics. This rationalist story may contribute to exclusive research gathering practices, producing too-narrow definitions of problems, insufficient consideration of feasible solutions, and recommendations made about target populations without engaging with the people they claim to serve (Bacchi, 2009; Stone, 2012).

However, this aim is much easier described than achieved. Policy analysis texts, focusing on how to do it, often use insights from policy studies but without fully explaining key concepts and theories or exploring their implications. There is not enough time and space to do justice to every element, from the technical tools of policy analysis (including cost-benefit analysis) to the empirical findings from policy theories and normative insights from critical policy analysis approaches (e.g. Weimer and Vining, 2017 is already 500 pages long). Policy process research, focusing on what happens, may have practical implications for analysts. However, they are often hidden behind layers of concepts and jargon, and most of their authors seem uninterested in describing the normative importance of, or practical lessons from, theory-informed empirical studies. The cumulative size of this research is overwhelming and beyond the full understanding of experienced specialist scholars. Further, it is difficult to recommend a small number of texts to sum up each approach, which makes it difficult to predict how much time and energy it would take to understand this field, or to demonstrate the payoff from that investment. In addition, critical policy analysis is essential, but often ignored in policy analysis texts, and the potential for meaningful conversations between critical or interpretive versus mainstream policy scholars remains largely untapped (e.g. Durnova and Weible, 2020) or resisted (e.g. Jones and Radaelli, 2016).

In that context, policy analysis students embody the problem of ‘bounded rationality’ described famously by Simon (1976). Simon’s phrase ‘to satisfice’ sums up a goal-oriented response to bounded rationality: faced with the inability to identify, process, or understand all relevant information, they seek ways to gather enough information to inform ‘good enough’ choices. More recently, policy studies have sought to incorporate insights from individual human, social, and organisational psychology to understand (1) the cognitive shortcuts that humans use, including gut-level instinct, habit, familiarity with an issue, deeply-held beliefs, and emotions, and (2) their organisation’s equivalents (organisations use rules and standard operating procedures to close off information searches and limit analysis – Koski and Workman, 2018). Human cognitive shortcuts can be described negatively as cognitive biases or more positively as ‘thinking fast and slow’ (Kahneman, 2012) or ‘fast and frugal heuristics’ (Gigerenzer, 2001). However, the basic point remains: if people seek shortcuts to information, we need to find ways to adapt to their ways of thinking, rather than holding onto an idealised version of humans that do not exist in the real world (Cairney and Kwiatkowski, 2017).

While these insights focus on policymakers, they are also essential to engaging with students. Gone – I hope – are the days of lecturers giving students an overwhelmingly huge reading list and expecting them to devour every source before each class. This approach may help some students but demoralise many others, especially since it seems inevitable that students’ first engagement with specialist texts and technical jargon will already induce fears about their own ignorance. Rather, we should base teaching on a thoughtful exploration of how much students can learn about the wider policy analysis context, focusing on (1) the knowledge and skills they already possess, (2) the time they have to learn, and (3) how new knowledge or skills would relate to their ambitions. For example, if students are seeking fast and frugal heuristics to learn about policy analysis, how can we help?

To help answer this question, I focus on what students should learn, can learn, and how blog posts and coursework can contribute to that learning. First, I describe the valuable intersection between policy analysis, policy process research, and critical policy analysis to demonstrate the potential payoffs to wider insights. In other words, what should policy analysis students learn from mainstream policy process research and critical policy analysis? Second, I describe the rationale for the blog that I developed in tandem with teaching public policy. I taught initially at an undergraduate level as part of a wider politics programme, before developing a Master of Public Policy and contributing to shorter executive courses and one-off workshops. This range of audiences matters, since the answer to the question ‘what can people learn?’ will vary according to their existing knowledge and time. Third, I summarise the rationale for the coursework that I use to encourage the application of public policy theories and knowledge to policy analysis (as part of a wider degree programme), including skills in critical thinking about policymaking dilemmas, to accompany more specialist research and analytical skills.

Paul Cairney (2023) ‘Teaching the politics of policy analysis, aided by a blog’, Gestión y Análisis de Políticas Públicas (GAPP), https://doi.org/10.24965/gapp.11146,  PDF

See also:

500

750

1000

Español

Q. Should PhD students blog? A. Yes.

I wish I could go back and rewrite everything I have published, including my PhD. If I knew then what I know now: I would get to the point quicker and describe its importance to a far wider audience than my supervisor and a few dedicated journal readers. To do so, I would exhibit the skills you develop when you write frequently for an ‘intelligent lay’ audience.

These are the writing traits that I think you develop when just writing for academics:

1. You assume a specialist audience, familiar with key terms. So, you use jargon as shorthand without explaining its meaning. The downside is that the jargon often doesn’t have a particularly clear meaning. When you blog, you assume a non-specialist audience. You use less jargon, or you explain its meaning and value.
2. You treat the exercise as a detective novel with a big reveal: a nice, vague opening discussion (passive tense optional), a main body of text to build up the suspense, and finally the big twist at the end. Ta da! Wow, I didn’t see that coming. When you blog, you assume that people will not read your work unless you front-load the reveal. You have a catchy and tweetable title, you provide a hook in the first sentence, and you only have a few hundred words in which to show your work (and encourage people to read the longer report).
3. Or, you describe your hypotheses in a way that suggests that even you don’t know what will happen. Wow – I confirmed that hypothesis! Who knew? When you blog, it seems more sensible to use the language of hypotheses (or an equivalent) more simply, to explain what factors are most important to your explanation.

You can develop this skill by using a personal blog to describe your research progress and the value of your findings. However, it is also worth blogging in at least two other venues:

1. Somewhere like the LSE blog, or Democratic Audit, in which the editors will try to summarise your argument in a short opening statement. This is very handy for you: did they summarise the main argument? If so, good. If not, look again to see if you explained it well.
2. Somewhere like The Conversation, in which the editors will try to mess around with the title (to encourage more traffic) and wording (to make it punchier and quotable). This is a good exercise in which you can think about how far you want to go. Are you confident enough in your research to make such stark statements? Or, do you want to obfuscate and fill the argument with caveats? If the latter, you can think about the extent to which your argument is clear and defendable (it may well be – sometimes caveats and humility can be good!).

I also encourage advanced undergraduates and taught postgraduates to produce a blog post (albeit unpublished) alongside an essay or policy paper, because it is difficult to be concise, and the exercise helps develop a good life skill. Even without the blog exercise, I’d still encourage dissertation students (at the start of their research) to write up their argument/ plan/ work in a half-page document, so that we can see if it adds up to a coherent argument. You can do the same thing with a blog post, with the added (potential) benefit of some feedback from outside sources.

See also: there are resource sites which go into far more aspects of the writing process, such as medium.com/@Write4Research and patthomson.wordpress.com

Policy Concepts in 1000 Words: Framing

(podcast download)

‘Framing’ is a metaphor to describe the ways in which we understand, and use language selectively to portray, policy problems. There are many ways to describe this process in many disciplines, including communications, psychological, and sociological research. There is also more than one way to understand the metaphor.

For example, I think that most scholars describe this image (from litemind) of someone deciding which part of the world on which to focus.

However, I have also seen colleagues use this image, of a timber frame, to highlight the structure of a discussion which is crucial but often unseen and taken for granted:

1. Intentional framing and cognition.

The first kind of framing relates to bounded rationality or the effect of our cognitive processes on the ways in which we process information (and influence how others process information):

• We use major cognitive shortcuts to turn an infinite amount of information into the ‘signals’ we perceive or pay attention to.
• These cognitive processes often produce interesting conclusions, such as when (a) we place higher value on the things we own/ might lose rather than the things we don’t own/ might gain (‘prospect theory’) or (b) we value, or pay more attention to, the things with which we are most familiar and can process more easily (‘fluency’).
• We often rely on other people to process and select information on our behalf.
• We are susceptible to simple manipulation based on the order (or other ways) in which we process information, and the form it takes.

In that context, you can see one meaning of framing: other actors portray information selectively to influence the ways in which we see the world, or which parts of the world capture our attention (here is a simple example of wind farms).

In policy theory, framing studies focus on ambiguity: there are many ways in which we can understand and define the same policy problem (note terms such as ‘problem definition’ and a ‘policy image’). Therefore, actors exercise power to draw attention to, and generate support for, one particular understanding at the expense of others. They do this with simple stories or the selective presentation of facts, often coupled with emotional appeals, to manipulate the ways in which we process information.

2. Frames as structures

Think about the extent to which we take for granted certain ways to understand or frame issues. We don’t begin each new discussion with reference to ‘first principles’. Instead, we discuss issues with reference to:

(a) debates that have been won and may not seem worth revisiting (imagine, for example, the ways in which ‘socialist’ policies are treated in the US)

(b) other well-established ways to understand the world which, when they seem to dominate our ways of thinking, are often described as ‘hegemonic’ or with reference to paradigms.

In such cases, the timber frame metaphor serves two purposes:

(a) we can conclude that it is difficult but not impossible to change.

(b) if it is hidden by walls, we do not see it; we often take it for granted even though we should know it exists.

Framing the social, not physical, world

These metaphors can only take us so far, because the social world does not have such easily identifiable physical structures. Instead, when we frame issues, we don’t just choose where to look; we also influence how people describe what we are looking at. Or, ‘structural’ frames relate to regular patterns of behaviour or ways of thinking which are more difficult to identify than in a building. Consequently, we do not all describe structural constraints in the same way even though, ostensibly, we are looking at the same thing.

In this respect, for example, the well-known ‘Overton window’ is a sort-of helpful but also problematic concept, since it suggests that policymakers are bound to stay within the limits of what Kingdon calls the ‘national mood’. The public will only accept so much before it punishes you in events such as elections. Yet, of course, there is no such thing as the public mood. Rather, some actors (policymakers) make decisions with reference to their perception of such social constraints (how will the public react?) but they also know that they can influence how we interpret those constraints with reference to one or more proxies, including opinion polls, public consultations, media coverage, and direct action:

They might get it wrong, and suffer the consequences, but it still makes sense to say that they have a choice to interpret and adapt to such ‘structural’ constraints.

Framing, power and the role of ideas

We can bring these two ideas about framing together to suggest that some actors exercise power to reinforce dominant ways to think about the world. Power is not simply about visible conflicts in which one group with greater material resources wins and another loses. It also relates to agenda setting. First, actors may exercise power to reinforce social attitudes. If the weight of public opinion is against government action, maybe governments will not intervene. The classic example is poverty – if most people believe that it is caused by fecklessness, what is the role of government? In such cases, power and powerlessness may relate to the (in)ability of groups to persuade the public, media and/ or government that there is a reason to make policy; a problem to be solved.  In other examples, the battle may be about the extent to which issues are private (with no legitimate role for government) or public (and open to legitimate government action), including: should governments intervene in disputes between businesses and workers? Should they intervene in disputes between husbands and wives? Should they try to stop people smoking in private or public places?

Second, policymakers can only pay attention to a tiny amount of issues for which they are responsible. So, actors exercise power to keep some issues on their agenda at the expense of others.  Issues on the agenda are sometimes described as ‘safe’: more attention to these issues means less attention to the imbalances of power within society.

PhD Students: research is hard and it’s OK to simplify

I’m going to list this under posts for social/ political science PhD students, but it has a broader applicability. The simple message is that: if you are finding it hard to research and write a complicated PhD it’s not all your fault.

There are practical ways to make academic social science research manageable. For example, most of my published work has been an exploration of specific theories, the description of a very specific case study, or a bit of both. The down side to this focus on specific case studies is that it is difficult to produce an overall sense of what is going on in the world (or even one country). It helps to use the same theory as a language with which to consider our accumulated knowledge from many cases, but the policy literature does not have a demonstrably good record on that front: there is a weird mix of detailed case studies in different places/ times/ policy areas, or a collection of large scale comparative studies which focus on a tiny part of the policy process. So, we may find it satisfying to complete a piece of case study research but be frustrated that it is difficult to relate our findings to those of our peers.

With Emily St Denny, I have begun to write a book which aims to do something different: focus on a policy solution – summed up in the terms ‘prevention’ and ‘early intervention’ – that connects many case studies. We are using the same theories and concepts to get a sense of policymaking across 2 governments (UK and Scottish) and many (if not most) government departments. This allows us to identify common practices, policymaking constraints, and lessons that would not be as apparent if the same analysis were conducted by different people at different times, using a range of not-quite-compatible conceptual frameworks (even if we were all aware of, and interested in, each other’s work – which is not as common as we’d like to think).

So, we sort-of solved the problem I described but we also produced a whole set of others, including:

• With a focused case study or vignette you can limit your analysis to key time periods, decisions, and actors. With our study comes a sense that everything is connected; that every decision to ignore something seems to undermine the analysis arbitrarily.
• With a case study based on qualitative interviews you can often argue that, by interviewing (say) 20-30 actors you have captured the responses of a large part of the relevant policy making/ influencing ‘population’ (and perhaps much more so than a survey sample in a quantitative project). With our study comes the sense that we are scratching the surface; that our list of interviewees will seem one step up from random.
• With a detailed case study you can give the sense that you have made sense of a series of important events. With our study comes the temptation to admit that we struggle to make sense of what it going on.
• You can generally complete case studies on your own. In our case, my sense is that we are only going to achieve some success because we have been funded for two years to work as a team (as part of a far larger team) and, perhaps, because our ESRC funding/ status gives us an ‘in’ with interviewees that I don’t think I enjoyed in the past.
• If you specialise, you can generate a reputation as an expert in a niche area, and generate some simple lessons from your research. In our case, we either know a little about a lot, or will be tempted to say ‘life is complicated’ or ‘everything is connected’ or tear out our hair while we impart our wisdom (after we say ‘yes, we didn’t know what prevention policy was either’).

This is a very roundabout way of saying that research is hard, and that it is OK to produce a very specific piece of work to make it manageable enough to be completed in a set amount of time (such as a 3-year PhD). It might seem like you are ‘zooming in’ on something fairly unimportant in the grand scheme of things – like a sludgy brown piece of cardboard nothingness in a 10,000 piece jigsaw – but instead you are avoiding the problems that come with too much ‘zooming out’.

See also:

PhD chat

Key policy theories and concepts in 1000 words

Comparing Theories of the Policy Process: A Brief Guide for Postgraduates

Paul Cairney (2013) ‘Standing on the Shoulders of Giants: How Do We Combine the Insights of Multiple Theories in Public Policy Studies?’ Policy Studies Journal, 41, 1, 1-21  PDF

Paul Cairney and Michael Jones (2015) ‘Kingdon’s Multiple Streams Approach: What Is the Empirical Impact of this Universal Theory?’ Policy Studies Journal, PDF

PhD Chat: how do you write a journal article or a conference paper?

From the Phd Chat page

[Update 30.3.21 I trimmed the preamble to make it less depressing and get to the point more quickly. Please note that our PhD group was in politics and policy, and my advice is not as relevant to people in other disciplines]

Giving conference papers is a key part of the PhD process, allowing you to:

• write up your ideas in a shorter format than the PhD
• generate useful feedback
• meet people with similar interests.

So how do you do it? Here are some questions that arose in our group:

1. What is the difference between a conference paper and an article? The simple answer is that the stakes are lower at the conference, and the paper is often the first of many drafts of an eventually accepted article. Generally, in my field, once your paper is accepted in principle onto a big annual conference, people don’t really monitor its quality – and many of the people that turn up to your panel will not have read it. However, a workshop is a different matter: you don’t want to annoy people when everyone is expected to read everyone’s papers. My advice in that case is to make the paper shorter and punchier, since many of your colleagues will be reading the whole set on the train/ plane to the workshop (the same goes, perhaps, for a panel discussant at an annual conference). You might even get a pat on the back, since people will have read at least one rotten effort from a more senior colleague. It also comes in handy later, when you have to meet the short word limits of major journals.
2. Article writing is a skill that develops with practice – but is it part of the PhD training? In some fields, it is taken for granted that the data you produce will be used by your supervisor, perhaps within a research team, and that your name might go in the middle of a very co-authored paper. This seems less common in social science, and in my field you claim ownership of the material, often publish it on your own, and might end up learning by doing. If you are inexperienced, you may want to work with a more experienced colleague to help you through the often-tough process – but should it be your supervisor? I honestly don’t know the answer to that question. It is fraught with difficulty, since there is clearly an imbalance of power in the supervisor/student relationship. I know of supervisors who do it routinely, and sometimes it is bound to look like the supervisor is getting some free research. In my case, I simply offer the possibility of co-authorship to PhD students – suggesting that, if they want 4 articles from their PhD, I could contribute to two (perhaps with both of us taking the lead on one paper). It seems to me to be part of the training process and a way to help PhD students get a leg up. However, it is not for everyone and I wouldn’t push it too hard.
3. Work on the hook, structure and coherence. We talked about how to get started, discussing the idea that you should just get writing what you know (to stop you worrying over every word) and edit later. I read about this advice quite a lot. It’s not the advice that I tend to give. Instead, I recommend starting at the start: producing the 150-word abstract, and seeing if you can describe the whole thing in a short, concise way; then, producing the introduction to see if you can ‘hook’ people in with the opening rationale for the study (focusing on the theory or the case), articulate a research question in a concise way, and present a coherent structure in which you will lay out the argument. In my view, the danger with the ‘just write’ advice is that you end up with 12000 words before you try to work out how it all fits together in 8000. The advantage in starting from the start is that you become immediately aware of the need to present a short and punchy piece of work and describe it to people who don’t have the knowledge of the topic. For me, even taking a whole day to write the introduction is worth the effort, since the rest may only take a few days to draft (if you know exactly what you are doing). This is the same sort of advice that I’d give before doing a literature review: it will take half the time, and far fewer words, to explain something if you have a clear research question and small number of objectives from the start.
4. Quantity or quality? I also have my own views on this topic, and they are not shared by all of my colleagues. It’s not too controversial to say that skills develop with practice, but there is a big quality/ quantity question when you decide how many times to develop your skills. The ‘quality’ argument is that you should take your time to get the data and the argument right, even if it means a smaller number of outputs. Later in life, this approach will come in handy if you are submitted to the research assessment exercise (which will require 1-4 of your best outputs), but I’m not sure you’ll ever get to a point where you think that the submission is perfect (and then the reviewers will let you know it’s not). The ‘quantity’ arguments are maybe harder to justify: keep it simple and go for one-paper-one-argument; tie the same data to multiple research questions and relevant literatures; and, recognise that the quantity of publications on someone’s CV may have a bewitching effect (particularly if people haven’t read the articles and are not specialist enough to know the status of the publications). I also like the argument that quantity helps produce quality.
5. Develop a thick skin – and know the meaning of ‘major revision’. In my experience, conferences are generally OK and people are generally polite and helpful (and often more so for PhD students), but presenting a paper can be a bad experience if you get some arsey colleague determined to make a point. To some extent, this is useful practice for when you submit something to a journal – at least when the journal uses anonymous peer review. It is possible that the comments will be scathing or will appear to be scathing when you read it for the first time. It is difficult not to take reviews personally, but it can be done if you read them once (immediately after receiving the editor’s email) and then wait for a while and read them again. Or, better still, read the comments on someone else’s work (if they let you) and see if they seem so critical. In some cases, you will get a desktop rejection or a post-review rejection, which is just a part of normal academic life. In others, you will be asked to make a major revision. This really just means that the revision is not minor (an extra sentence or two, and fix some typos – this decision is not common). It requires you to look at how the editor boils down the comments and construct a response which addresses most of the reviewer’s concerns. You don’t have to make all of the suggested changes, but you should say why you chose to respond in a particular way (e.g. if reviewers give you contradictory advice). This takes a lot of work, but it is now standard stuff – a substantial post-referee revision is something that you should plan to do each time.
6. Choose the journal wisely. It’s an obvious point, but it is easier said than done. In my field, you might consider these factors: country (e.g. some US journals have a reputation for publishing more quantitative work); approach (e.g. some journals will expect a ‘critical’ approach or to make reference to well-established theories or concepts); history (in other words, look through some back issues to see if yours would fit, or if you would even read the articles); status (which is hard to gauge, but some journals have higher rejection rates and/ or it takes longer and more investment to get something in there); and, theory or case (for example, you can tie any case study to some general journals if you use an established theory, or just use a theory to help explain a case in a specialist journal).

See also: 3 Common Reasons Editors Reject Articles Without Review

PhD Chat: Issues arising from the research process (2) your role when conducting methods

From the Phd Chat page

Most of us are using qualitative methods, so some of our discussion of common themes won’t be as directly useful to everyone – but many points relate to many PhDs:

1. The role of external ‘gatekeepers’ to your research. In some cases, a small number of people may control your access to information and useful people. You might have to build a relationship with them to get that information. In others, you may need to work hard to fill out forms and meet the rules of an organisation. I don’t think we had a case where that wasn’t a potential obstacle. The partial-solution is to make sure that you are in the position for some of it to go wrong, and to give yourself enough time to work out how to get past the gates. In some cases, the process gets easier with experience, but in my last project it took me over a year to set up some of the interviews.
2. ‘Rapport’. We talked about how you might build up an understanding with an interviewee, but that there are trade-offs in each strategy. For example, you might be able to refer to a common background, such as gender, ethnicity or schooling (or, in Joanne McEvoy’s case, often the opposite) – but this might lead to your interviewee making the assumption that you know certain things, which means that they won’t explain them. For example, I knew someone from the US whose accent was often an advantage in UK interviews, since interviewees assumed they had to explain far more. There are also important ethical/ political questions about how detached you should be, to gain information, when participants make statements counter to your beliefs or they talk about issues that make them vulnerable: are you a detached scientific observer merely recording the exchange, a participant seeking to influence the exchange, and/or expected to engage in some way with your interviewee (discussed here or here)?
3. The bad interview. Most people I have spoken to have had a bad interview: the interviewee has agreed to be interviewed, but says very little (of use) and appears defensive. It becomes easier to deal with them (I think) when you have the experience to work out when to persevere and when to finish the interview quickly. If we are talking about 1 or 2 out of 30, it’s fine to get very little information from some people. We might also learn things from the experience, including: if they sneer at the questions, it might reveal what they think; if they are nervous, it may be because they are not senior enough in an organisation to be confident about giving answers on behalf of it; or, they may simply not want to talk while recorded. Who knows? It’s OK to leave an interview and not know what went wrong, as long as it represents an outlier.
4. How many interviews is enough? If you seek a false sense of certainty, my answer is 30. If you want the standard answer, people will say ‘it depends’ (there is a great NCRM discussion here in which the answer can be 1). Specifically, it depends on what you are doing and your approach: some talk about the idea of ‘saturation’, when you are confident that any more interviews won’t get you any more information (or it won’t be worth the extra effort); others, in (say) elite research might discuss the idea of getting enough of a proportion of an identified population (it might be 40, and you might get access to 20). The answer ‘30’ is handy since the number of interviews may have a bewitching effect on people, but you should really reflect on what information you have gathered, how you can demonstrate an adequate amount, and how you relate it to the other kinds of information available to you.
5. The idea that you are a ‘gatekeeper’ for other people. In discussions of science we often talk about conducting research in a way that makes it replicable: if someone followed your methods could they produce the same results? Yet, this does not mean that people actually do the replication – which can be rare or non-existent in some fields. In particular, case study research, in which you are piecing together disparate information from limited sources, is difficult to replicate – and people will generally take your results on trust. Similarly, with anonymised interviews, people generally have to trust that you conducted them and reproduced them faithfully. In some cases, you do that with reference to established methods (such as audio recorded interviews, transcribed in their entirety and analysed using something like NVivo). In others, you might take written notes and agree to keep everything hush hush (which tends to be my approach). In such cases, all I can recommend is that, when you present the information, you acknowledge that the outcome is not simply an ‘objective’ account that would likely be produced by someone else (I discuss this issue in relation to policy theories, methods and science in this article and post).
6. Thinking about how you fit into the research. A related issue is about the need to reflect on why you are doing the research and what part you play in it. In some cases, this issue is right up front: for example, some feminist studies may have an ‘emancipatory’ aim and be tied up in the identity of the researcher. In some cases, the student may be researching something that relates to their identity, social background, or profession (in the case of people combining a PhD with employment). In others, the link is not obvious, but the issues are similar: there is a need to think about how your aims, assumptions and biases will affect the ways in which you gather and analyse information – and if you can demonstrate that you are anticipating any problems. In some cases, there may be an open process to consider the ethics of the research, when (for example) it involves using an aspect of one’s background to access information not available to others. In others, it is a more straightforward and brief process of reflection, just in case it comes up in the viva.
7. The difficulty of saying what ‘mixed methods’ are. The chances are that, if you are doing a PhD now, you have been trained in various qualitative and quantitative methods. You may also have done a course in the philosophy underpinning methods. If you put those two kinds of training together, you may pause before providing the now-standard answer: ‘I am totally doing mixed methods’. Many examiners may not leave it at that. They might want to know how you can marry methods which, potentially, are underpinned by two very different ways of understanding the world. In my view, these problems can be overblown when people claim a particular method for their philosophy when, really, the methods are more flexible. For example, interviews can be used alongside other methods to generate meaning in interpretive research, or simply to generate information to give more depth to surveys. All you really need to do (in my view) is to provide a clear and defendable description of what methods you use and why.

PhD Chat: Issues arising from the research process (1) Mitigating risk

From the Phd Chat page

Probably the most important issue is about how to anticipate problems and incorporate them within the research design. You may find it odd when, at the very beginning of your design process, your supervisor is asking you what you would do if all or part of the process goes wrong – but it is an essential question. Can you salvage a PhD if you end up with far less data than you anticipated? There is an important balance to be struck between being ambitious and realistic. I think it’s good to make ambitious plans to, say, engage with multiple and disparate literatures, interview 50 people or do a significant piece of survey work. It’s also good to consider the end result if, say, only 20 will speak to you. In most cases, I think a good supervisor would ask you to prepare to mitigate risk.

Let’s take a hypothetical example. You want to answer the ‘what is policy?’ question, and you set up three parts: (1) documentary/ textual analysis to identify what policymakers define as policy; (2) interviews with 30-50 practitioners to determine policy in practice; and, (3) a survey to explore policy outcomes from the perspective of service users. This is super-ambitious and, while task 1 is relatively straightforward (at least if the documents are in the public domain), a lot could go wrong with 2 (it takes ages to set up, conduct, transcribe and analyse interviews, and people may not be forthcoming) and 3 (it takes a long time to design and conduct, few may respond to the survey, or data analysis becomes unmanageable). In that case, a supervisor may advise you to focus on either 2 or 3 – or to accept that, when you engage in both, only one of them may work out. The big question is: if one of them goes wrong, can you still get a PhD from what you have? Or, from the beginning, should you put your efforts into fewer tasks? This is not an easy question to answer, but it is important to ask it at the beginning.

Phd Chat: The PhD as a record of research training – not a perfect achievement

A small group of us have started meeting to talk about the PhD process, and I decided to write up some of the broad themes. The point, I think, is that, although each person is researching a different topic, they face common problems and can share experiences. I’m afraid that, more often than not, we didn’t come up with solutions – but, you know, a problem shared …

The PhD as a record of research training – not a perfect achievement

Probably the most important part of the discussion is about the end-point, which is usually the much-feared viva following thesis submission (in our case, the UK-style non-public meeting with internal/ external examiners). There are four points that I’d stress:

1. We see the PhD (at least increasingly) as a way to demonstrate proficiency in research methods, information gathering, and presentation. So, a common answer to a problem about how you do a literature review or deal with data limitations is that you should demonstrate that you have used your training and skills to produce the right outcome. There is no right answer, but there are established ways to demonstrate that you have the skills to produce an answer. This usually starts with having a clear and realistic research question. Then, it’s about showing that your engagement with the literature is geared specifically to answering that question (not a big list of points about the literature), that you have selected the most appropriate method(s), that you can write drafts and respond correctly to feedback, and that you can make oral/ conference presentations and generate more feedback.
2. This emphasis seems preferable to, for example, trying to demonstrate some sort of awesome ‘gap’ in the literature or that you are challenging the conventional wisdom (imagine every PhD challenging what came before – it would be exhausting). I wouldn’t rule gap-identification out completely, but I’d be careful about making exaggerated claims (I discuss this point in relation to policy theory here). Sometimes an examiner will end up thinking that the gap was there for a reason (the PhD does not demonstrate the topic’s importance) or that its identification of a gap is rather artificial (which is a common ploy used by more senior academics that should really set a better example). For me, a PhD will look more convincing if you provide a quite-respectful review of the relevant literature, demonstrating how it helps guide your research (and, for example, how your case compares with cases in other fields or countries) and how your study will help improve it. This can often be about adding nuance to established findings (for example, when someone uses case studies to add depth to general assumptions about political behaviour) rather than shattering them.
3. One of the most useful parts of the process is to present to an audience that doesn’t really know or care about your research. It forces you to re-think how you explain it to people other than your supervisor or people with shared interests (in both cases, you may be used to using a shared language or shorthand). I reckon you’ll find it easier to explain the case study or research question (which is often relatively clear) than your theoretical approach (which is often riddled with rotten jargon with limited meaning, or concepts like ‘power’ or ‘governance’ which might maybe mean everything and nothing).
4. The PhD will be less than perfect and that’s OK. I often tell PhD students that they will be surprised about how low the bar is to successful completion – not because the bar is too low, but because it is at a realistic level, in which examiners recognise what you can do in three years when you have just begun a research career.

See also: Phd Chat page