A Stern review for everyone?

The Stern review ‘was commissioned by the government to carry out the review of the Research Excellence Framework (REF) to ensure future university research funding is allocated more efficiently, offers greater rewards for excellent research and reduces the administrative burden on institutions’. In this post, I explain why no single policy can solve these problems uniformly: they affect scholars of different seniority, and different disciplines, very differently. The punchline is at the end.

My initial impression of the Stern review is that it has gone to great lengths to address the unintended consequences of the previous Research Excellence Framework. One of its key aims now is to try to anticipate the potential unintended consequences of its reduction of other unintended consequences! This is remarkably common in policymaking, perhaps summed up by Aaron Wildavsky’s phrase ‘policy as its own cause’: we enter a never-ending process of causing ripple effects when trying to fix previous problems.

The example of non-portability (#sternreview portable)

Take the example of one of the biggest problems:

Problem: there was a large incentive for Universities to ‘game’ the REF towards the end of the cycle: paying for 20% of the time of big name academics, or appointing them at huge salaries, to gain access to their 4 best publications. A policy of rewarding research excellence became a policy to (a) reward big transfers, undermining the efforts of other Universities and reducing their incentive to invest for the long term, and (b) boost the salaries of senior scholars (many of whom were already on 6-figure salaries), often to ridiculous levels.

Solution: non-portability. The idea is that you can move but your former employer gets to use the texts that you published while in your last job. So, there may now be less incentive to buy up the big names in the run up to the next REF.

Unintended consequence: uncertainty for early career researchers (ECRs) or scholars without permanent (open-ended) contracts. Many ECRs have expressed the concern that their incentives may suddenly change, from generating a portfolio of up to 4 excellent publications to secure a permanent post, to perhaps holding back publications and promising their delivery when in post. This could be addressed by the present review (ECRs could be included in the REF but be under no obligation to submit any publications), or perhaps by exempting ECRs since the policy is aimed at senior scholars (but the exemption would also have unintended consequences!).

Interpreting the problem through the lens of precarious positions

We will now enter a phase of debate driven by uncertainty and anxiety about the end result, and a lot of the discussion will be emotionally charged because many people will have spent maybe 8 years in education (and several years in low paid posts after it) and not know what to do next. It is relatively easy for people like me to say that the proposed new system is better, and for senior scholars to look on Stern as a big improvement, because people like me will be rewarded in either system (I will leave it to you to decide what I mean by ‘people like me’). It is more difficult for ECRs who are genuinely uncertain about their prospects.

The punchline: how does this look through the eyes of scholars in different disciplines?

The disciplinary lens is the factor that can often be most important but least discussed, for two key reasons:

1. The general differences. Scholars operate in different ways. The ‘STEM’ (science, technology, engineering and mathematics) subjects are often described in these terms: you have large teams headed by a senior scholar; there is a hierarchy; you all work on the same research question; you publish many short articles as a team (with senior authors listed at the very start and end of the list of authors); you are increasingly driven by key metrics, such as personal citations and the ‘impact factor’ of journals . The humanities is often described like this: you are a lone scholar; you work on your own research question; you publish single-author work, and the big status symbol is the research monograph (book); these journal or other metrics do not work as well in your discipline. I am verging on caricature here (many ‘STEM’ scholars will work alone, and some humanities or social science scholars will operate laboratory-style teams), but you get the idea.

These differences feed into other practices: only in some subjects can it make sense for a University to ‘poach’ a whole team or unit; only in some subjects do ECRs need to develop their own portfolio of work; in some subjects, a PhD student or ECR effectively works for a senior scholar, while in others the PhD student has a supervisor but can set their own research agenda; in some subjects, it is automatic to include the senior scholar in an article you wrote, in others it would be seen as exploitation of the work of a PhD student or ECR. In some subjects, a CV with your name on team publications is the norm, while in others it would look like you do not have your own ideas.

2. The REF reinforces these differences. You often find the impression that research exercises and metrics are there for the STEM subjects (or ‘hard sciences’) and not the humanities or social sciences: a process or review for Universities does not take into account the differences in incentives and practices across the disciplines, and some disciplines might lose out.

So, if you follow this debate on twitter, I recommend that you look at the bios of each participant to check their level of seniority and discipline because the Stern review is for all but it will affect us all in very different ways.

See also: James Wilsdon ‘The road to REF 2021: why I welcome Lord Stern’s blueprint for research assessment‘

 

Research design: Case studies and comparative research

My aim is to tell you about the use and value of comparative research by combining (a) a summary of your POLU9RM course reading, and (b) some examples from my research. Of course, I wouldn’t normally blow my own trumpet, but Dr Margulis insisted that I do so. Here is the podcast (25 minutes) which you can download or stream:

The reading for this session is Halperin and Heath’s chapter 9, which makes the following initial points about comparative research:

• ‘Comparative’ describes methods to identify and explain differences/ similarities between cases.
• These cases are often – but need not be – countries (for example, comparisons over time, or by policy area, can be just as illuminating).
• There are three main approaches: large-N (many cases), small-N (several cases), and single-N (or ‘case studies’).
• Comparison is not an end in itself. Rather, we need to justify, or clarify the value of, each comparison. This involves identifying a clear theoretical reason for a particular comparison (and, as in all studies, producing a clear research question that can be answered).
• It helps us avoid what Rose calls ‘false uniqueness’ (assuming rather than demonstrating that a case is exceptional) and ‘false universalism’ (assuming that a finding from one time/ place applies to all times or places).
• Comparative methods are most frequently used to determine the extent to which a theory generated from one set of cases applies to other cases.

Key issues when you apply theories to new cases

There are two interesting implications of the final bullet point.

First, note the tendency of a small number of countries to dominate academic publication. For example, a lot of the policy theories to which I refer in this ‘1000 words’ series derive from studies of the US. With many theories, you can see interesting developments when scholars apply them outside of the US:

• The ‘universal’ v ‘territorial’ issue. In my article with Michael Jones (summarised here) on Kingdon’s ‘multiple streams analysis’ we note that it began as a study of health/ transport policy in the US in the 1980s. It then morphed into a theory applied to many regions at many times. A large part of its success comes from the abstract nature of its key concepts, which can be applied at any place and time: for example, a lot of its discussion of agenda setting relates to ‘bounded rationality’ which applies to all people. Yet, there are also ‘territorial’ issues which apply to particular regions or types of government. For example, some studies adapt MSA’s concepts to explain the influence of supranational authorities or to describe the differences between EU and US policymaking.
• The ‘beyond the West’ issue. In one interesting application of MSA to China, Zhu argues that a key concept is not applicable: policy change in the US requires policy solutions to be technically feasible; in China, they only prompt change if ‘infeasible’. The example highlights the limits to the explanatory power of theories derived from studies of a small number of ‘Western’ countries (I know this description is loaded – what other terms could we use to describe countries like the US and UK?). In such cases, scholars are now exploring the implications in some depth: how well do these policy theories travel?

Second, consider the extent to which we are accumulating knowledge when applying the same theory to many cases. There are now some major reviews or debates of key policy theories, in which the authors highlight the difficulties of systematic research to produce a large number of comparable case studies:

• The Social Construction of Target Populations (review by Pierce et al)
• The Institutional Analysis and Development Framework (Working Together by Poteete et al)
• Multiple streams approach (review by Jones et al)
• Punctuated equilibrium theory (see the live debate on the Comparative Agendas Project)
• The Advocacy Coalition Framework (intro to special PSJ issue)

For me, the important common factor in all of these reviews is that many scholars pay insufficient attention to the theory when applying its insights to new cases. Consequently, when you try to review the body of work, you find it difficult to generate an overall sense of developments by comparing many case studies. So, in my humble opinion, we’d be a lot better off if people did a proper review of the literature – and made sure that their concepts were clear and able to be ‘operationalised’ – before jumping into case study analysis. I wrote these blog posts for established scholars and new PhD students, but the argument should also apply to you as an undergraduate: get the basics right (which includes understanding the theory you are applying) to get the comparative research right. This is just as important as your case selection.

From case study to large-N research: choosing between depth and breadth?

Case studies. It is in this context that we might understand Halperin and Heath’s point that case study research (single-N) is comparative. You might be going into much depth to identify key aspects of a single case, but also considering the extent to which it compares meaningfully with other case studies using the same theory. All that we require in such examples is that you justify your case selection. In some examples, you are trying to see if a theory drawn from one country applies to another. Or, you might be interested in how far theories travel, or how applicable they are to cases which seem unusual or ‘deviant’. In some examples, we seek the ‘crucial case study’ that is central to the ‘confirmation or disconfirmation of a theory’ (p207), but don’t make the mistake of concluding that you need a new theory because the old one can only explain so much. Further, although single case studies should not be dismissed, you can only conclude so much from them. So, be careful to spell out and justify any conclusions that you find to be ‘generalisable’ to other cases.

Example 1. In my research, I often use US-origin theories to help explain policymaking by the UK and devolved governments. For example, I used the same approach as Kingdon (documentary analysis and semi-structured interviews) to identify, in great detail, the circumstances under which 4 governments in the UK introduced the same ban on smoking in public places (the take home message: there was more to it than you might think!).

Small-N studies. The systematic comparison of several cases allows you to extend analysis often without compromising on depth. However, there is great potential to bias your outcomes by cherry-picking cases to suit particular theories. So, we look for ways to justify case selection. Halperin and Heath go to some length to identify the problems of ‘selection on the dependent variable’, which can be summed up as: don’t compare cases just because they seem to have the same outcome in common (focus instead on what causes such outcomes). Two well-established approaches are ‘most similar systems design’ (MSSD) and ‘most different systems design’ (MDSD) (p209). With MSSD, you choose cases which share key explanatory factors/ characteristics (e.g. they have the same socio-economic/ political context) so that you can track the effects of one or more difference (perhaps in the spirit of a randomised control trial, but without the substance). With MDSD you choose cases which do not share characteristics so that you can track the effect of a key similarity.

Aside from the problems of case study selection bias, it’s worth noting how difficult it is to produce a clear MSSD or MDSD research design, since you are making value judgements about which shared/ not shared characteristics are crucial (see their discussion of necessary/ sufficient factors).

Example 2. Take the example of a study I did with Karin Ingold and Manuel Fischer recently, comparing ‘fracking’ policy and policymaking in the UK and Switzerland.  We use the same theory (the ACF) and same method (documentary analysis and a survey of key actors) in both countries. We used a survey to allow us to quantify key relationships between actors: to what extent do they share the same beliefs with other actors, and how likely are they to share information frequently with their allies and competitors?

We describe the research design as MDSD because their political systems represent two contrasting political system archetypes – the ‘majoritarian’ UK and ‘consensus’ Switzerland – which Lijphart describes as key factors in their contrasting policymaking processes. Yet, central to our argument is that there are policymaking processes common to policy subsystems despite their system differences. In effect, we try to measure the effect of political system design on subsystem dynamics and find a subtle but important impact.

We also find that, although these differences exist, their policy outcomes are remarkably similar. So, ‘most different’ systems often produce very similar policymaking processes and policy outcomes. Then, we note that if we had our time again we would have extended the analysis to subnational governments in the UK. The ‘most different’ design prompted us to focus on the UK central government and Swiss Cantons (the alleged locus of power in both cases), but maybe we could have started from an assumption that they are not as different as they look. Have a look and you can see the dilemmas that still play out in (what I think is) a well-designed study.

Example 3. I face the same difficulties when comparing policy and policymaking by the UK and Scottish Governments. In some respects, they are ‘most different’: ‘new Scottish politics’ was designed to contrast with ‘old Westminster’ (particularly when it came to elections; the Scottish Parliament is also unicameral). In others, they are similar: the ‘architects of devolution’ introduced a system that seems to be of the Westminster family (particularly the executive-legislative relationships). Further, Scotland remains part of the UK, and the UK Government retains responsibility for many policies affecting Scotland. Overall, it is difficult to say for sure how similar/ different are their systems (which I discuss in a series of lectures). So, the comparison is fraught with difficulty. In such examples, they key solution is to ‘show your working’: describe these problems and state how you work within them (for example, in my case, I try to examine the extent to which policymaking reflects ‘territorial’ context or ‘universal’ drivers’, partly by interviewing policymakers in each government).

Large-N (quantitative) studies.  Halperin and Heath lay out some of the potential benefits of large-N research. For example, it is easier to come to general conclusions about many countries by studying many countries (rather than trying to generalise from a few, which might not be representative). They also highlight the pitfalls, including the problem of meaning: when you ‘operationalise’ a concept such as democracy or populism, can you provide a simple enough definition to allow you to give each system a number (to denote democratic/ undemocratic or X% democratic) that means the same thing in each case? To this problem, I would add the general issue of breadth and depth. With large-N studies you can examine the effects of a small number of variables, to explain a small part of the politics of many systems. With small-N you can study a large number of variables in a few systems. The classic trade-off is between breadth and depth. Of course, if you are doing an undergraduate dissertation the big Q is: what can you reasonably be expected to do? Maybe your highest aim should be to make sense of the studies which already exist.

 

 

 

 

Q. Should PhD students blog? A. Yes.

I wish I could go back and rewrite everything I have published, including my PhD. If I knew then what I know now: I would get to the point quicker and describe its importance to a far wider audience than my supervisor and a few dedicated journal readers. To do so, I would exhibit the skills you develop when you write frequently for an ‘intelligent lay’ audience.

These are the writing traits that I think you develop when just writing for academics:

1. You assume a specialist audience, familiar with key terms. So, you use jargon as shorthand without explaining its meaning. The downside is that the jargon often doesn’t have a particularly clear meaning. When you blog, you assume a non-specialist audience. You use less jargon, or you explain its meaning and value.
2. You treat the exercise as a detective novel with a big reveal: a nice, vague opening discussion (passive tense optional), a main body of text to build up the suspense, and finally the big twist at the end. Ta da! Wow, I didn’t see that coming. When you blog, you assume that people will not read your work unless you front-load the reveal. You have a catchy and tweetable title, you provide a hook in the first sentence, and you only have a few hundred words in which to show your work (and encourage people to read the longer report).
3. Or, you describe your hypotheses in a way that suggests that even you don’t know what will happen. Wow – I confirmed that hypothesis! Who knew? When you blog, it seems more sensible to use the language of hypotheses (or an equivalent) more simply, to explain what factors are most important to your explanation.

You can develop this skill by using a personal blog to describe your research progress and the value of your findings. However, it is also worth blogging in at least two other venues:

1. Somewhere like the LSE blog, or Democratic Audit, in which the editors will try to summarise your argument in a short opening statement. This is very handy for you: did they summarise the main argument? If so, good. If not, look again to see if you explained it well.
2. Somewhere like The Conversation, in which the editors will try to mess around with the title (to encourage more traffic) and wording (to make it punchier and quotable). This is a good exercise in which you can think about how far you want to go. Are you confident enough in your research to make such stark statements? Or, do you want to obfuscate and fill the argument with caveats? If the latter, you can think about the extent to which your argument is clear and defendable (it may well be – sometimes caveats and humility can be good!).

I also encourage advanced undergraduates and taught postgraduates to produce a blog post (albeit unpublished) alongside an essay or policy paper, because it is difficult to be concise, and the exercise helps develop a good life skill. Even without the blog exercise, I’d still encourage dissertation students (at the start of their research) to write up their argument/ plan/ work in a half-page document, so that we can see if it adds up to a coherent argument. You can do the same thing with a blog post, with the added (potential) benefit of some feedback from outside sources.

See also: there are resource sites which go into far more aspects of the writing process, such as medium.com/@Write4Research and patthomson.wordpress.com

These complaints about ignoring science seem biased and naïve – and too easy to dismiss

This is one of two posts on the use of scientific evidence in environmental policy (a broad term which can include climate change, food, land management, and energy policy). In this one I insult some of the people reacting to the Scottish Government’s decision on GM. In the next one, I present preliminary evidence on a systematic review of the use of evidence in environmental policy.

The Scottish Government recently decided to reinforce a moratorium on the use of GM crops in Scotland. It produced a strong response from many individual scientists, and groups representing scientists, which seems predictable enough. That’s what you would expect some scientists to do if you don’t act solely on their advice.

What I didn’t quite expect was a more general, and often hyperbolic, criticism by non-scientist commentators about the failings of policymakers to listen to scientists.

The general problem for non-scientist commentators is that they don’t really know what ‘the evidence’ is. It’s no great insult to predict that most commentators decrying a lack of reliance on ‘the evidence’ have never read any relevant scientific journal article, systematic review, or peer reviewed report on the science in question. In fact, that’s fine – you probably need some training to understand the technical aspects, particularly on method.

What is not fine is a tendency to accept the word of scientists when they tell you that they know the evidence. In general, they don’t. ‘The evidence’ is often remarkably patchy (a point I’ll reinforce in the next post) and highly-disputed within professions, and canny scientific groups or individuals fill those gaps with an appeal to expert-based authority. This practice should be subject to as much critical analysis and scepticism that we afford to politicians. In fact, if you are a good scientist, you welcome the challenge and the debate (with the exception of cases, such as climate change, in which debate may undermine an urgent need to act).

The general problem for the more naïve scientist commentators is that they often read the scientific reports, take a view on what they mean and what should be done, and simply expect policymakers to do the same. They fail to appreciate how scientific evidence fits into the broader picture of politics and policymaking. They fail to articulate what ‘evidence based policy making’ really means, or should mean in practice, when we have equally or more important ways to make decisions – such as representative government (we elect people to state their beliefs and make decisions on our behalf) and participatory/ deliberative democracy (we seek ways to generate a range of perspectives on a policy problem). In effect, they make an appeal to a process in which scientific knowledge or expert beliefs should be privileged above the beliefs of other people (in this case, it is accompanied by a pretentious reference to the Scottish Enlightenment).

In my view*, both of these approaches are counterproductive. The hyperbolic non-scientist criticisms are too easy to dismiss. As soon as you start equating policy decisions with, for example, witch trials (OK – I’ve only seen two people do it), you can be safely ignored as someone with an axe to grind. The narrow scientist criticisms are just as badly off. As soon as you appeal to ‘the evidence’ without showing any appreciation of the policy process or a wider appreciation of policy decisions, you can be safely dismissed as naïve or self-interested.

Update: maybe more than 2 –

SNP's witch-hunt of scientists exposed by @euanmccolm in cracking piece on GM hysteria http://t.co/kASb4GuxAK

— Murdo Fraser (@murdo_fraser) August 19, 2015

Further reading: https://paulcairney.wordpress.com/ebpm/

For a pragmatic scientific account see:

Lawton, J. H. (2007) ‘Ecology, politics and policy’, Journal of Applied Ecology, 44,3, 465–74

*These views are my own, they are stated a bit strongly, apologies to Duncan and Euan, etc.

PhD Students: research is hard and it’s OK to simplify

I’m going to list this under posts for social/ political science PhD students, but it has a broader applicability. The simple message is that: if you are finding it hard to research and write a complicated PhD it’s not all your fault.

There are practical ways to make academic social science research manageable. For example, most of my published work has been an exploration of specific theories, the description of a very specific case study, or a bit of both. The down side to this focus on specific case studies is that it is difficult to produce an overall sense of what is going on in the world (or even one country). It helps to use the same theory as a language with which to consider our accumulated knowledge from many cases, but the policy literature does not have a demonstrably good record on that front: there is a weird mix of detailed case studies in different places/ times/ policy areas, or a collection of large scale comparative studies which focus on a tiny part of the policy process. So, we may find it satisfying to complete a piece of case study research but be frustrated that it is difficult to relate our findings to those of our peers.

With Emily St Denny, I have begun to write a book which aims to do something different: focus on a policy solution – summed up in the terms ‘prevention’ and ‘early intervention’ – that connects many case studies. We are using the same theories and concepts to get a sense of policymaking across 2 governments (UK and Scottish) and many (if not most) government departments. This allows us to identify common practices, policymaking constraints, and lessons that would not be as apparent if the same analysis were conducted by different people at different times, using a range of not-quite-compatible conceptual frameworks (even if we were all aware of, and interested in, each other’s work – which is not as common as we’d like to think).

So, we sort-of solved the problem I described but we also produced a whole set of others, including:

• With a focused case study or vignette you can limit your analysis to key time periods, decisions, and actors. With our study comes a sense that everything is connected; that every decision to ignore something seems to undermine the analysis arbitrarily.
• With a case study based on qualitative interviews you can often argue that, by interviewing (say) 20-30 actors you have captured the responses of a large part of the relevant policy making/ influencing ‘population’ (and perhaps much more so than a survey sample in a quantitative project). With our study comes the sense that we are scratching the surface; that our list of interviewees will seem one step up from random.
• With a detailed case study you can give the sense that you have made sense of a series of important events. With our study comes the temptation to admit that we struggle to make sense of what it going on.
• You can generally complete case studies on your own. In our case, my sense is that we are only going to achieve some success because we have been funded for two years to work as a team (as part of a far larger team) and, perhaps, because our ESRC funding/ status gives us an ‘in’ with interviewees that I don’t think I enjoyed in the past.
• If you specialise, you can generate a reputation as an expert in a niche area, and generate some simple lessons from your research. In our case, we either know a little about a lot, or will be tempted to say ‘life is complicated’ or ‘everything is connected’ or tear out our hair while we impart our wisdom (after we say ‘yes, we didn’t know what prevention policy was either’).

This is a very roundabout way of saying that research is hard, and that it is OK to produce a very specific piece of work to make it manageable enough to be completed in a set amount of time (such as a 3-year PhD). It might seem like you are ‘zooming in’ on something fairly unimportant in the grand scheme of things – like a sludgy brown piece of cardboard nothingness in a 10,000 piece jigsaw – but instead you are avoiding the problems that come with too much ‘zooming out’.

See also:

PhD chat

Key policy theories and concepts in 1000 words

Comparing Theories of the Policy Process: A Brief Guide for Postgraduates

Paul Cairney (2013) ‘Standing on the Shoulders of Giants: How Do We Combine the Insights of Multiple Theories in Public Policy Studies?’ Policy Studies Journal, 41, 1, 1-21  PDF

Paul Cairney and Michael Jones (2015) ‘Kingdon’s Multiple Streams Approach: What Is the Empirical Impact of this Universal Theory?’ Policy Studies Journal, PDF

The Art and Skill of Academic Translation: it’s harder when you move beyond English

I have been writing about the idea of ‘translation’ in terms of ‘knowledge transfer’ or ‘diffusion’, which often suggests that there is a linear process of knowledge production and dissemination: knowledge is held by one profession which has to find the right language to pass it on to another. This approach has often been reflected in the strategies of academic and government bodies. Yet, the process is two-way. Both groups offer knowledge and the potential to have a meaningful conversation that suits both parties. If so, ‘translation’ becomes a way for them to engage in a meaningful way, to produce a common language that they can both ‘own’ and use. Examples include: the need for scientists to speak with policymakers about how the policy process works; the need for ‘complexity’ theorists to understand the limits to policymaker action in Westminster systems; the separate languages of institutions which struggle to come together during public service integration (key to local partnership and ‘joined up government’); and, the difference in the language used by service providers and users. We might also worry about the language we use to maintain interdisciplinary discussion (such as when ‘first order change’ means something totally different in physics and politics).

It’s not the same thing, but translating into another language, such as when conversing in English and Japanese, reinforces the point in an immediately visible way. In both directions, English to Japanese, and vice versa, it is clear that the recipient only receives a version of the original statement – even when people use a highly skilled interpreter. Further, if the statement is quite technical, or designed to pass on knowledge, the gap between original intention and the relayed message is wider still.

This point can be made more strongly in a short lecture using interpretation. As academics, many of us have been to conferences in English, and witnessed a presenter trying to cram in too much information in 15 minutes. They give a long introduction for 10, then race through the slides without explaining them, simply say that they can’t explain what they hoped, or keep going until the chair insists they stop. You don’t really get a good sense of the key arguments.

In another language, you have to reduce your time to less than half, to speak slowly and account for translation (simultaneous translation is quicker, but you still have to speak very slowly). You have to minimise the jargon (and the idioms) to allow effective translation. Or, you need to find the time to explain each specialist word. For example, while I would often provide an 8000 word paper to accompany a lecture/ workshop, this one is 1500. There is no visible theory, although theory tends to underpin what you focus on and how you explain it. It took 40 minutes to present, largely because I left a lot of topics for Q&A. I still had a hard time explaining some things. I predicted some (such as the difference between ‘federalism’ and ‘federacy’, and the meaning of ‘poll tax’ and ‘bedroom tax’) but realised, late on, that I’d struggle to explain others (such as ‘fracking’, or the unconventional drilling used to access and extract shale gas).

This sort of exercise is fantastically useful, to force you to think about the essential points in an argument, keep it short without referring to shorthand jargon, and explain them without assuming much prior knowledge in the audience, in the knowledge that things will just mean different things to different audiences. It is a skill like any other, and it forces on you a sense of discipline (one might develop a comparable skill when explaining complex issues to pre-University students).

Indeed, I have now done it so much, alongside writing short blog posts, that I find it hard to go back from Tokyo to jargon city. Each time I read something dense (on, for example, ‘meta-governance’), I ask myself if I could explain it to an audience whose first language is not English. If not, I wonder how useful it is, or if it is ever translated outside of a very small group.

This is increasingly important in the field of policy theory, when we consider the use of theories, developed in English and applied to places such as the US and UK, and applied to countries around the globe (see Using Traditional Policy Theories and Concepts in Untraditional Ways). If you can’t explain them well, how can you work out if the same basic concepts are being used to explain things in different countries?

Further, we don’t know, until we listen to our audience, what they want to know and how they will understand what we say. Let me give you simple examples from my Hokkaido lecture. One panellist was a journalist from Okinawa. He used what I said to argue that we should learn from the Scots; to develop a national identity-based social movement, and to be like Adam Smith (persevering with a regional accent, and a specific view of the world, in the face of snobbishness and initial scepticism; note that I hadn’t mentioned Adam Smith). Another panelist, a journalist from Hokkaido, argued that the main lesson from Scotland is that you have to be tenacious; the Scots faced many obstacles to self-determination, but they persevered and saw the results, and still persevere despite the setback (for some) of the referendum result (I pointed out that ‘the 45%’ are not always described as tenacious!). Another contributor wondered why Thatcherism was so unpopular in Scotland when we can see that, for example, it couldn’t have saved Scottish manufacturing and was perhaps proved correct after not trying to do so. Others use the Scottish experience to highlight a similar sense of central government imposition or aloofness in Japan (from the perspective of the periphery).

In general, this problem of academic translation is difficult enough when you share a common language, but the need to translate, in two ways, brings it to the top of the agenda. In short, if we take the idea of translation seriously, it is not just about a technical process in which words are turned into a direct equivalent in another language and you expect the audience to be informed or do the work to become informed. It is about thinking again about what we think we know, and how much of that knowledge we can share with other people.

Why would the UK Government take a tough line with the Scottish Government after a Yes or No vote?

Update: so far, the main reaction to the Scottish No vote has been David Cameron’s plan to address the idea of ‘English Votes for English Laws’, plus some wider discussion of how to address the English Question. From what I can tell, no one has yet told politicians in Scotland to get stuffed. Instead, The Vow, coupled with Gordon Brown’s intervention, has kept quick and ‘extensive’ further devolution high on the agenda on Scotland and it remains to be seen if constitutional change for England will get the same attention.

Judging by some of the rhetoric on the Scottish independence referendum, one thing is certain: the UK Government will take a tough line with the Scottish Government. If there is a Yes vote, it will defend UK interests to the hilt. If there is a No vote, and the Scottish Government seeks further devolution, it will assert the rights of the rest of the UK. Why will this happen? Well, because the UK Government acts in the interests of the rest of the UK, and the rest of the UK – and England in particular – is waking up to Scotland’s privileged position within the Union.

At least, that’s what I’ve heard. Beyond this assertion, how can we demonstrate that the rest of the UK is ready to put pressure on the UK Government to take a tough line? I think there are three sources of relevant information, only two of which reinforce this argument.

The first is personal testimony from politicians, either directly in interview, or indirectly through the columns of journalists, including:

• The decision by George Osborne, Ed Balls and Danny Alexander to rule out a currency union, on the basis that it is against the UK interest (Alexander also argues that it is against the Scottish interest).
• Boris Johnson’s suggestion that there is no reason to grant the Scottish Parliament more powers in the event of a No vote.
• Nigel Farage’s suggestion, in a BBC documentary, that a Yes vote would provoke the rest of the UK, and England in particular, to wake up and protect its interests (followed by vox pops making this claim in stronger terms).
• Archetypal media commentary, such as Adrian Woolridge’s caricature of an English response, describing the opportunity to ‘turn off the tap’ (end subsidies for Scotland) after a Yes vote, or invite the Scots to shut up after a No vote (note that the first three paragraphs are set up to get your attention for a more subtle argument in the remainder).

The second is quantitative survey research. For example, an IPPR press release in 2012 argues that: ‘The evidence presented here suggests the emergence of what might be called an ‘English political community’, one marked by notable concerns within England about the seeming privileges of Scotland in particular and a growing questioning of the capacity of the current UK-level political institutions to pursue and defend English interests, and one underpinned by a deepening sense of English identity. The full report by Richard Wyn Jones and colleagues suggests that a small majority of people who feel ‘English, not British’ or ‘More English than British’ believe that Scotland gets more than its ‘fair share’ (p27). It is not a majority of the English respondents as a whole – but the 44% response in 2011 is more than double the (21%) response in 2000 (see p116 of the Scottish independence chapter, by John Curtice and Rachel Ormston, in British Social Attitudes 29).

This was followed by a YouGov/ University of Cardiff survey of 3705 English adults which suggests that attitudes towards Scotland have hardened, on several measures. When asked:

• If Scotland ‘should be able to continue to use the pound’, 53% disagreed and 23% agreed (15% neither, 9% DK).
• If ‘Levels of public spending in Scotland should be reduced to the levels in the rest of the UK’, 56% agreed and 9% disagreed (21%, 13%).

Respondents were happy enough for the Scottish Parliament to take control over ‘the majority of taxes raised in Scotland’ and the welfare state, but not too keen to help it join international organizations.

The third is a mixture of qualitative and quantitative. In this case, the aim is to allow people to speak for themselves without a questionnaire setting the agenda. In other words, if you ask people , in a questionnaire, if Scots get more than their fair share, many will say ‘yes’ without having given it much thought. You don’t get a brilliant feel for the way in which they prioritise important issues – and you don’t know if they would have articulated this concern in the absence of someone asking them about it. The alternative is to simply raise a broad issue and ask people to say what they think. This approach has been done to great effect by Susan Condor (2010) in the journal article Devolution and national identity: the rules of English (dis)engagement, which draws on ‘1,652 conversational interview transcripts collected between 2000 and 2009’. The argument that is most striking to me is that: ‘most people remain unaware of the policy issues that excite so much interest among the political and intellectual elite’. In other words, politicians and journalists talk as if they represent the will of a population, without providing much evidence that the population knows or cares as much about the issues as they do.

The major caveat to this third argument is that the research took place before surveys started picking up a growing sense of English national identity (and before it started having some sort of impact on electoral behaviour via support for UKIP). Yet, it should still give us pause, to wonder: if Susan Condor repeated this research, would it support or undermine the assertion that there will be an English backlash to Scottish independence or further devolution?

 

Journal Article Acceptance(s) After 5 Rejections and 25 months

Update: the title is now less catchy but more accurate. See the italicised bits for the update. I have also added this poster:

You might have to be a glass-is-half-full kind of person to take something positive from this story of publication success after a long run of failure. After 18 months, 5 rejections, 4 substantial redrafts, 2-3 changes of journal direction, and minus 8000 words, we had it accepted (update: add another 7 months,  and three substantial redrafts and additions, for the 2nd article acceptance).

It began with our submission to World Politics, which is a high status journal, in politics and international relations, with a high rejection rate, so this was a gamble. I thought we had done the double: produced something interesting to say about ‘evolutionary’ policymaking, building on work I began for Understanding Public Policy; and, produced a wealth of new information on global tobacco policy, built on work led by Donley Studlar and Hadii Mamudu, and informing Global Tobacco Control. So, HM and I put both together to produce this paper, submitted 11^th September 2012:

World Politics Evolutionary Theory International Agreements 11September2012

It was rejected on the 4^th December (not a bad turnaround). The rejection came with substantial reviewer comments – World Politics decision letter – which we used to revise the next version substantially. My impression, from this review, was that the combination of evolutionary theory and the case study was not working. In fact, I may have been pushing us into a position that I advise PhD students and early career researchers to avoid: a paper suggesting a new theoretical angle, reinforced by a single case . In my defence, I wasn’t proposing a new theory. Instead, I was trying to present the approach as a reflection of accumulated knowledge, in both theory and case.

Still, it wasn’t working, so we separated the two elements somewhat. I chopped about 3000 words of theory – something made easier by the fact that I had submitted (February 2012) a separate paper on evolutionary theory to Policy and Politics, which was reviewed (July 2012) and accepted after a minor revise-and-resubmit (23 October) then published early 2013 – ‘What is Evolutionary Theory and How Does it Inform Policy Studies?’ Policy and Politics, 41, 2, 279-98 Paywall Green

We hummed and hawed about policy journals before I made the mistake of sending it to Public Administration and Development, partly because we were focusing on contrasts in implementation based on the simple developed/ developing country distinction, partly because it was interdisciplinary, and partly because its description seemed really close to our topic.

Cairney Mamudu Evolution Tobacco Control PAD submission 5Feb2013

It was rejected without going to review, described by the editor as ‘out of scope’.

So, we sent it, almost immediately (21 Feb 2013), to Governance, which had been HM’s (more sensible) preference. Again, this is a high status political/ policy science journal with a high rejection rate, so we were still confident enough to take the usual gamble.

Anonymous Evolution Tobacco Control Governance submission 21Feb2013

It was rejected on 26^th May after substantial review (which seems more critical than the reviewers of World Politics, so we were no further forward) –  governance rejection

We figured that we had to do two things based on the reviews: (1) strip out the discussion of evolutionary theory more and focus on the basic political science concepts (implementation, networks, agendas, etc.), shifting back the focus to the case study and evidence so far (particularly since I had now published an article separately on evolutionary theory); (2) be super-clear on key terms (leading/ laggard; developed/ developing) to anticipate future concerns, and clarify the narrative on the origins and role of the FCTC.

By this time, my University had made available some funds for Open Access, and I was keen to go this route, partly because OA seems good, and partly because I had recently co-authored an article in the OA journal Implementation Science and it was a very positive experience.

We chose Globalization and Health – based at the LSE, interdisciplinary, covering our topic and focus – and submitted on 12^th September 2013. It was rejected on 29^th October, which is a good turnaround, but the reviews were too brief to be useful – except it is still clear that our attempts to address the developed/ developing distinction are still needling our referee audience.

GH rejection letter GH referee 1 GH referee 2

Our solution was twofold: (1) to check with the editor of the next journal if there would be a problem with our approach, and (2) to get away from the developed/ developing sticking point by presenting an even more nuanced account, taking every opportunity to show that we weren’t providing naïve caricatures, and going super-conceptual to describe an ideal-type of a leading implementing country rather than identifying ‘leaders’ and ‘laggards’.

I emailed the editors of the Journal of Public Health Policy in November and got a good assurance on the developed/ developing point. The only problem is that the word limit is 4000, which is about one-third of the length of our original paper. Still, we revised the paper again.

By then, HM reckoned that Tobacco Control was a better fit, since they had begun to publish a series of papers on the ‘endgame’. We submitted there on the 20^th December.

1 TCJ-Endgame_CoverLetter-14Dec2013 2 Cairney Mamudu Checklist cover letter 3 Cairney-Mamudu_22Dec2013

They rejected it on the 8^th January 2014 without sending it to review TC rejection

We sent it to the JPHP on the 10^th January – 1 CAirney Mamuducover letter JPHP 10JAn14 2 CAirney Mamudu Submitted article JPHP 10JAn14

We got a revise and resubmit on the 17^th February – a very decent turnaround indeed. We got the classic binary response: one thought it was great, and one thought it was mince – JPHP reviews 17.2.14

We resubmitted on March 13 – 1 cover and rebuttal letter 2 resubmitted JPHP    – and got the thumbs up by the 27^th.

Update, November 2014. We submitted a much better paper on the same theme (more developed theoretical argument, more data, a better refined argument) to Public Administration (special issue on global public policy) in June. After two resubmissions (and, unusually, a referral to a member of the editorial board – to deal with comments made by the third reviewer), we had it accepted in November.

So what did we learn?

1. It is natural to blame journals, editors and reviewers for these long, drawn out processes – but I need to take some responsibility for the journal choices and the quality of submissions.
2. Even a rejection can give you useful material for a redraft, as long as it actually goes to review.
3. It is worth persevering. This is a very unusual case of 5 rejections, but it seem fairly normal to get 1 or 2 before success. For a while, I went on a good run of acceptances-after-revision, then a run of acceptances after rejection. I have almost always published each paper by the end.
4. I think the article is, in many ways, a far better paper than when it began – but it also changed so much that we reckon we can go back and submit some of the chopped material (the new data) elsewhere.
5. Final lesson – you need a thick skin for this process, particularly when you get one or two cranky anonymous reviewers, and particularly when you go interdisciplinary and invite comment from people who often don’t respect your discipline.
6. Final, final, updated lesson: don’t lose your confidence and settle for a second-best result. Our first acceptance was for an article that stripped away a lot of what was good in the original idea (partly to meet the 4000 word limit), and it was rewritten for a public health audience in a way that I don’t entirely like. The Public Administration article (9000 words) is the one I’ll send to people and be proud of. It was accepted more than two years after we first made the mistake to send it to a different journal.

 

 

 

How Can Policy Theory Have an Impact on Policy Making?

Paul Cairney (2015) ‘How Can Policy Theory Have an Impact on Policy Making?’ Teaching Public Administration, 33, 1, 22-39 PDF

Policymakers and academics often hold different assumptions about the policymaking world based on their different experiences. Academics may enjoy enough distance from the policy process to develop a breadth of knowledge and produce generalisable conclusions across governments, while policymakers/ practitioners such as civil servants may develop in-depth expertise when developing policy for a number of years. In turn, both may learn from each other about how to understand the policymaking world.

Academic-practitioner seminars and short training courses can help further that aim. Yet, there is a major barrier to such conversations: academics and practitioners may have their own language to understand policymaking, and a meaningful conversation may require considerable translation.

To examine these issues, this article relates my attempts, in a series of steps, to turn abstract policy theory into something useful for practitioners.

The first step is to identify a potential disconnect between the starting points for academic-practitioner discussions and policy theories. In the former, we may still use concepts developed to aid policymaking – such as the policy cycle, the ideal of ‘comprehensive rationality’ and the ‘top-down approach’ to implementation – because they aid discussion. In the latter, we have generally moved on from these descriptions of the world, to reflect the policy process’ complexity and our need for new theories to help explain it.

The second is to consider how to make those more realistic, but specialist, scientific concepts as meaningful to practitioners. The article considers the extent to which modern theories can provide straightforward insights to policy practitioners by condensing and articulating its ‘key tenets’.

The third is to consider how insights from those tenets, based largely on what governments do, can be used to recommend what they should do. The article contrasts how they might be used by a ‘top down minded’ government with how they might be used by scholars to recommend action. It focuses in particular on ‘complexity theory’ as an approach which combines policy theory with practical recommendations.

A final step is to consider how we can engage with policymakers to discuss those insights. The article draws on my experience of teaching civil servants in policy training seminars, using these theories to identify complex policymaking systems and encourage ‘reflexivity’ about how to adapt to, and operate within, them.

The article performs a dual role: as a way to explain the policy process in a straightforward way, and as a resource for civil servants engaged in policy training seminars.

Green version

See also: Is Evidence-Based Policymaking the same as good policymaking?

and/ or

Evidence Based Policy Making: If You Want to Inject More Science into Policymaking You Need to Know the Science of Policymaking

Policy Concepts in 1000 Words: Policy change and measurement

(podcast download)

The first thing we learn when we study public policy is that no-one is quite sure how to define it. Instead, introductory texts focus on our inability to provide something definitive. That is OK if we want to pretend to be relaxed about life’s complexities, but not if we want to measure policy change in a reasonably precise way. How can we measure change in something if we don’t know what it is?

A partial solution is to identify and measure types of public policy. For example we might treat policy as the collection of a large number of policy instruments or decisions, including:

1. Public expenditure. This includes deciding how to tax, how much money to raise, on which policy areas (crime, health, education) to spend and the balance between current (e.g. the wages of doctors) and capital (building a new hospital) spending.
2. Economic penalties, such as taxation on the sale of certain products, or charges to use services.
3. Economic incentives, such as subsidies to farmers or tax expenditure on certain spending (giving to charity, buying services such as health insurance).
4. Linking government-controlled benefits to behaviour (e.g. seeking work to qualify for unemployment benefits) or a means test.
5. The use of formal regulations or legislation to control behaviour.
6. Voluntary regulations, such as agreements between governments and other actors such as unions and business.
7. Linking the provision of public services to behaviour (e.g. restricting the ability of smokers to foster children).
8. Legal penalties, such as when the courts approve restrictions on, or economic sanctions against, organizations.
9. Public education and advertising to highlight the risks to certain behaviours.
10. Providing services and resources to help change behaviour.
11. Providing resources to tackle illegal behaviour.
12. Funding organizations to influence public, media and government attitudes.
13. Funding scientific research or advisory committee work.
14. Organizational change, such as the establishment of a new unit within a government department or a reform of local government structures.
15. Providing services directly or via non-governmental organizations.
16. Providing a single service or setting up quasi-markets.

I say ‘partial solution’ because this approach throws up a major practical problem: we do not have the ability to track and characterise all of these instruments in a satisfactory or holistic way. Rather, we have to make choices about what information to use (and, by extension, what to ignore) to build up a partial, biased, picture of what is going on. Here are some of the practical problems we face:

Depth versus breadth. Should I focus on one policy instrument or all of them (or some combination)? Should I focus on a single key event or a picture of change over decades? Should I focus on the outputs of one policymaking organization or them all, or try to track the outcomes of the system as a whole? In each case there is a major trade-off: if we ‘zoom in’ we might miss broad or long term trends; if we ‘zoom out’ we might miss important details.

Our empirical and normative expectations. When we identify policy change we link it, explicitly or implicitly, to a yardstick based on how much we expect it to change (based on, for example, the abilities of people to initiate and block change) and how much we think policy should change under the circumstances, given the size of problem or the level of public attention. Our normative expectations are difficult to separate from the empirical. Think of cases such as air pollution, environmental policy, tobacco, alcohol and drugs control, violent crime, poverty, and inequality. In each case, we have expectations about what should happen based on how important we believe the problem to be – and may often identify minor or moderate change, based on that perception, rather than compared with (say) change in other areas.

Differing perspectives. Policy change looks very different from the ‘top’ or the ‘bottom’. For example, a focus on policy choices by central governments may exaggerate change compared to long term outcomes at the ‘street level’. Indeed, it is tempting to focus on rapid, exciting changes at the top, without thinking through their long term consequences. Or, we may find reformers at the top, frustrated with a lack of progress, compared with local actors frustrated with the effects of the rapid pace of change on their organisations.

Motivation. In each case, we have to think about why a policy decision was made: what problem was it designed to solve? For example, a tax or economic sanction can be used to influence behaviour or simply to raise revenue (think, for example, of ‘sin’ taxes). Policymakers can introduce measures to satisfy a particular interest or constituency, ensure a boost to their popularity or fulfil a long-term commitment based on fundamental beliefs. The distinction is crucial if the long term political weight behind a measure determines its success.

Statistical comparisons. When we consider the use of economic measures, we need an appropriate context in which to consider its significance. We may describe spending on an issue as a proportion of GDP, a proportion of the government budget, a proportion of the policy area’s budget, and in terms of change from last year or over many years. In some cases the amount of money spent or raised by government could be compared with that spent by industry, such as when a health education budget is dwarfed by tobacco/ alcohol advertising, or a huge company receives a small fine for environmental or competition law breaches.

Contradictions. Policy as a whole may seem inconsistent, either within a single field (e.g. some governments control tobacco use but also subsidise leaf growing and encourage trade) or across government (e.g. school expulsion policies may exacerbate youth crime).

Given these problems of inevitable bias, I suggest (at the end of this post, and on p30 of the book) that we consider the extent to which our findings can be interpreted in different, and equally plausible, ways.

Comparing Theories of the Policy Process: A Brief Guide for Postgraduates

When you do a PhD in policymaking, it is likely that you have to engage significantly with the literature on policy theory. This often prompts bursts of enthusiasm, when some theories seem to be spot on, and periods of frustration, when theories seem to be deficient in some way. For example, they might only explain one part of the ‘policy process’ when you want to explain it all, or they are difficult to operationalise and apply to specific cases. This may be a particular problem for PhDs focused primarily on a substantive case study. You will find that the case study is too complicated to be explained fully by a relatively simple theory designed to be applied across a range of cases. You may then think about what to do, focusing on two main possibilities:

First, should I propose my own theory, which takes insights from other theories but puts them together in a new way, perhaps with a new terminology? My advice is: no (or not unless you are convinced that you are a genius, destined to become the new Sabatier, Kingdon, Ostrom, Schneider, Ingram, Baumgartner or Jones). Don’t confuse a better explanation for your case with a better overall explanation. Your theory is not backed up in the same way by multiple applications and long term refinement.

Second, should I use the insights from a range of established policy theories to inform my case study? My advice is: yes, but be careful. Read this Policy Concepts in 1000 Words: Combining Theories. Then, consider what you are trying to do. In most cases, your aim is to account for a significant part of the policy process. Tanya Heikkila and I describe what that might entail. In Theories of the Policy Process 3^rd ed.  we identify (with a lot of input from Chris Weible) what we think are the main elements of the policy process that a theory or case should cover:

1. “Actors making choices.  We need to simplify a policymaking world which may include thousands of people, into a set of categories and/ or discussion of the key actors involved. Actors can be individuals or collectives, and collectives can range from private companies to interest groups to governments bodies. We also need to account for the ways in which people act; their calculations and motivations. For example, most theories use ‘bounded rationality’ as a springboard for explanation, while others focus on motivations such as beliefs.
2. Institutions. These are the rules, norms, practices and relationships that influence individual and collective behaviour.  The choices of actors is explained to some extent by their understanding of, and adherence to, such rules. Those rules can be formal and widely understood, such as when enshrined in law or a constitution. Or, they can be informal and only understood in particular organisations. Policies can be considered a subset of the broad concept of institutions, but institutions at one level (e.g. constitutional rules) can also shape the policymaking activities or decisions at another level (e.g. legislation or regulation). Similarly, institutions can establish the types of venues where policy decisions are made and the rules that allow particular types of actors or information and ideas to enter into the policy process.
3. Networks or subsystems. These are the relationships between actors responsible for policy decisions and the ‘pressure participants’ such as interest groups with which they consult and negotiate.  Senior policymakers delegate responsibility for policy making to bureaucrats, who seek information and advice from groups. Groups exchange information for access to, and potential influence within, government or other collective choice processes. It is through these networks where collective action often emerges in policy processes.  Many theories describe a process in which groups exchange information for access to, and potential influence within, government. Bureaucracies and other public bodies may have particular operating procedures that favour particular sources of evidence and some participants over others.
4. Ideas. This is a broad term to describe beliefs, or ways of thinking, and the extent to which they are shared within groups, organisations, networks and political systems. It can refer to two intertwined processes. First, shared ideas (knowledge, world views, language) appear to structure political activity when they are almost taken for granted or rarely questioned – for example, as core beliefs, paradigms and monopolies of understanding. Second, new ideas or ways of thinking can be used to prompt actors to rethink their beliefs to some extent – such as when a proposed new solution challenges the way that a problem is framed or understood, and therefore how much attention it receives and how it is solved. So, for example, we may identify relative stability when shared ideas are not questioned and instability when different groups with different beliefs interact.
5. Policy Context. This is a broad category to describe the extent to which a policymaker’s environment is in her control. It can refer to the often-changing policy conditions that policymakers take into account when identifying problems and deciding how to address them, such as a political system’s: geography, biophysical and demographic profile; economy; and, mass attitudes and behaviour.  It can also refer to a sense of policymaker ‘inheritance’ – of laws, rules, institutions and programs – when they enter office.
6. Events. Events can be routine and anticipated, such as the elections which produce limited change or introduce new actors with different ideas about policy problems and solutions. Or, they can be unanticipated incidents, including social or natural crises, or major scientific breakthroughs and technological change. Their unpredictability makes them difficult to theorise, and they can often effectively be treated as ‘error’, or as external factors providing an additional source of explanation to a policy theory. Or, they can be incorporated within theories which focus on how actors interpret and respond to events. To some extent, an event is only significant if actors within political systems pay attention to them”.

From there, we can consider how each theory accounts for the nature of these elements, and their interaction, to explain policy dynamics and outcomes. You can find summaries of many of these relevant theories here: https://paulcairney.wordpress.com/1000-words/ and a copy of the 3rd and 4th edition chapters here: Cairney and Heikkila 2014, Heikkila and Cairney 2017.

For what it’s worth, when I co-authored the book Global Tobacco Control (with Studlar and Mamudu), we had chapters on the role of policy theory but structured the book according to those elements of the policy process, not specific theories. I’m not saying that you should take this simple approach. Rather, I am asking you to think about, and explain, why yours is better.

What Works (in a complex policymaking system)?

The Scottish Government and ESRC held an event yesterday to publicise their proposed new What Works centre. Its role is ‘is to deepen the impact of the emergent Scottish approach to public service delivery and reform, by evaluating evidence in delivery of that approach’. As you might expect from a series of presentations (9 in total), the speakers presented ideas which had the potential to vary in meaning and emphasis. Consequently, there seemed to be some potential tensions between things such as:

1. A government trying to ‘scale up’ from the experience of successful pilots, and a government devolving responsibility to local bodies (such as local authorities), giving them the space to choose how to meet broad Scottish Government aims – unless ‘scaling up’ simply means encouraging ‘best’ or ‘good’ practice. Scaling up may have a different meaning in different academic disciplines and different policy areas. For example, healthcare and public health may be associated more with larger and more uniform policies than ‘community’ based projects.
2. A focus on learning from successful projects, quite quickly, when the Scottish Government and ESRC are looking for examples of policies that produce good outcomes after, 10, 20 or more years. The former may involve drawing relatively quick conclusions about the success of projects, based on a mixture of their reputations and evidence limited to a small number of years. This is a general feature of ‘lesson-drawing’ or ‘transfer’ from one government to another – borrowers are often very quick to judge the success of lenders, based on some rough and ready indicators.

What struck me in particular was Harry Burns’ (Chief Medical Officer, Scottish Government) emphasis on the importance of complexity, complex systems and ‘complex systems thinking’. As I have found recently, when co-editing a book on complexity and public policy, complexity is a remarkably vague term which can mean very different things to different people. To many, it means something akin to ‘complicated’. To others, to identify a complex system is to identify a very specific set of arguments and properties, including:

1. A complex system is greater than the sum of its parts; those parts are interdependent – elements interact with each other, share information and combine to produce systemic behaviour.
2. Some attempts to influence complex systems are dampened (negative feedback) while others are amplified (positive feedback). Small actions can have large effects and large actions can have small effects.
3. Complex systems are particularly sensitive to initial conditions that produce a long-term momentum or ‘path dependence’.
4. They exhibit ‘emergence’, or behaviour that results from the interaction between elements at a local level rather than central direction.
5. They may contain ‘strange attractors’ or demonstrate extended regularities of behaviour which may be interrupted by short bursts of change.

Similarly, the meaning of ‘systems thinking’ is often unclear or unhelpful when we seek to go beyond the instantly intuitive notion of thinking about the broader context of policymaking. Here are some further examples of unresolved issues regarding complexity and public policy:

• Some people describe the natural and social world (the latter is often the thing that policymakers want, but struggle, to influence) as complex, but neglect to consider the idea that policymaking systems are complex systems. The particular relevance to What Works is that it would be a mistake to recognise the complicated set of problems to be solved by policies, but ignore the complicated set of processes to which the policies themselves are subject. This is often a key feature of wider debates on ‘evidence based policymaking’.
• One tenet of complexity theory is that law-like behaviour is difficult to identify – so a policy that was successful in one time or place may not have the same effect in another. This places important limits on the idea of ‘scaling up’ from the experience of one study. If scaling-up means encouraging good practice, adapted to local experience, all well and good. If it means ‘rolling out’ a successful policy to many areas, it may be problematic.
• One solution, of sorts, to such problems, is to accept that we don’t know the individual effects of individual policy instruments. Rather, we introduce a wide range of policy instruments and hope that they interact to produce a positive overall outcome. This sort of approach can be found in areas such as tobacco control (in which there may be six main types of policy, each with several elements), but may be less common or applicable in other policy areas. The complication is that it is difficult to say ‘what works’ if we mean ‘what particular policy instruments work?’.
• ‘Systems thinking’ may prompt us deal with uncertainty and change by encouraging trial-and-error projects, or pilots, that can provide lessons, and be adopted, amended or rejected, relatively quickly. This encourages us to think about the potentially very limited role of a central government. A lot of the literature uses complexity theory to present almost the opposite idea to the ‘Westminster model’ in which power rests, and should rest, in the hands of a small number of elected policymakers, accountable to the public via Parliament.  Many advocate relying less on central government driven targets, in favour of giving local organisations more freedom to learn from their experience and adapt to their rapidly-changing environment.
• A lot of the literature uses complexity theory to “challenge particular brands of ‘positivism’ which present a ‘vision of society based on order, laws and progress’ (Geyer and Rihani, 2010, p. 5); to suggest that ‘quantitative and reductionist methodologies’ may be useful to explain topics such as elections with ‘rules and orderly structures’, but not issues that contain unpredictable political events, significant levels of uncertainty and ambiguity (Geyer and Rihani, 2010, pp. 74–5) or factors outside the control of policy makers (Room, 2011)” (excerpt from Cairney 2012 PSR Complexity Theory). This raises the issue of methods and their underlying philosophy. At the event, he general response to a question on methods was the usual idea that we can mix them. Yet, the underlying ideas behind particular approaches may be complementary and others contradictory (see here and here Cairney 2013 PSJ Standing on the shoulder of giants for a wider discussion). A mix of methods is not a solution in itself.

A positive interpretation of these problems is that all governments face them, and some may respond better than others. This is certainly the reputation that the Scottish Government has developed, and this reputation is articulated in the What Works call:

• The Scottish Government is unusually positioned to respond favourably because successive Scottish Governments have appeared to be much more open to this sort of advice (or, at least, they have engaged in behaviour consistent with it).
• In particular, they have relied more on ‘partnership working’ and less on stringent performance management regimes linked to targets and punitive measures for not meeting them.
• The SNP Government in particular (from 2007) signalled a willingness to devolve more responsibility to local authorities, reduce the proportion of ‘ring fenced’ budgets  and develop less-top-down ‘single outcome agreements’ (Remember Alex Salmond in 2007: ‘The days of top-down diktats are over’).
• This difference of attitude might reflect cultural differences in Scotland’s political system or simply the different policymaking environment in which Scottish Governments operate.  In particular, Scotland is smaller and its policymakers have fewer responsibilities; both factors may allow them to develop quite meaningful horizontal and vertical relationships and rely less on more impersonal and inflexible policy measures.

Related posts:

Policy Concepts in 1000 Words: Policy Transfer and Learning

Policy Concepts in 1000 Words: ‘Evidence Based Policymaking’

Policy Concepts in 1000 Words: Complex Systems

Related paper: Paul Cairney 13.1.14 How Can Policy Theory Inform Policymaking

How do people consume your research? A short video, on tobacco and alcohol policy, requiring attention and feedback

This is a first draft of a simple video I am doing with Tereza Procházková @ZasCreativeBag to accompany a blog post I did on the differences between tobacco and alcohol policies in the UK. I wouldn’t mind some feedback (here or to @cairneypaul) on it before I ask to have it tweaked then embedded in the proper post (although, realistically, it’s feedback for the next one, if there is a next one). From looking at it myself, I know that I try to pack a lot of information into 3 minutes (perhaps a bit like an inexperienced lecturer trying to tell students everything) and so the pictures and audio come thick and fast. Next time, I will speak more slowly. But maybe it still works because it is accompanied by a blog post with all of the information. Maybe you listen to the 3 minutes then decide if you want to fill in the blanks by reading the full post (and then maybe the full paper). My partner tells me that I take a while to get to the point and that there need to be more punchy bullet point moments (I didn’t get too offended). Would you agree? There is also a bit of a skip in the audio towards the end (a big problem?), and I trail off at the very end (to press the stop button on the ipad). Note that I am not Glaswegian – the Irvine/ Ayrshire accent is a wee bit different. Polite comments on my voice/ pronunciation also welcome.

The post can be found here: http://blogs.lse.ac.uk/politicsandpolicy/archives/34735. If the video doesn’t play, you can get it here on youtube – http://www.youtube.com/watch?v=fujeajKKa-E or here:

UPDATE: here is the more polished version:

http://www.youtube.com/watch?v=k6pkPTPohas

For more discussion of the ‘impact’ side of the work, see: http://paulcairney.blogspot.co.uk/2013/07/how-do-people-read-your-research.html
and http://paulcairney.blogspot.co.uk/2013/05/a-picture-of-pathways-to-impact.html

How Do People Read Your Research?

People may nod at you and say ‘yes, hmm, very interesting’, but do they understand what you are saying (in a satisfying way)? I would like to know if someone could read something of mine, write down the key points and then explain them back to me in a way that I recognised. It needn’t be a regurgitation (which is not what I do when I read the work of others) but I’d like to think that they took the key points I tried to convey, with no major misinterpretations. That’s one sensible interpretation of ‘impact’, isn’t it? So, for me, these drawings by @ZasCreativeBag are excellent. A drawing also condenses an argument – and puts all the points together in one page – in a way that might take me 1000-2000 words. They may not convey the same points entirely, but they do a decent job of reinforcing the argument (I hope).

Here (with a longer explanation) are some earlier examples (I did not get the grant!) and the most recent example is below (for the blogpost, see here)

Notes From a Conference Part 1: Arrogance and Recognition

It’s hard to tell if people are (a) predisposed to arrogance at an early age and/ or (b) if they develop this trait as they age and become more powerful or better recognised in the profession. All I know is that some (generally well established) academics appear to ‘less modest’ than others. So, I have this general angst about becoming (or, at least, appearing) more arrogant when engaging with other people and, crucially, not knowing it. This is perhaps more of a concern for people in subjects like social science where some conference discussions will be about challenging the statements, methods and views of other people. There is a fine line between a positive challenge and a negative dismissal, so we need a high degree of self-awareness to reflect on our behaviour and ask ourselves if we have crossed a line. This is particularly important:

• In international conferences where people bring different levels of expectation about politeness. For example, many UK based scholars may be less likely to start their comments with ‘thank you for your interesting paper’. Instead, like me, they may see a strong (thoughtful) challenge as a strong signal of respect (since, it shows that you care enough about the paper to listen and engage in a meaningful way).
• If you are in a room with a dominant view – it is only by being open to opinions from others that you can avoid being close minded and dismissive of things you don’t agree with initially but might appreciate if you allow yourself the time to listen and reflect (something that is too easy to dismiss if you are in a room with people that largely agree with you).
• If we identify the subtext to many conference proceedings: the desire to make one’s name by presenting papers and engaging with the papers of others. I have said to a few colleagues that a large conference is really a battle for attention and recognition, wrapped up in the pretence of positive discussion, and only most of that statement is tongue-in-cheek.

It is in that context that I’d like to describe the crumbs of recognition I got at a recent conference (International Conference on Public Policy). I figure that, if arrogance comes with age, I’d better write this down before it’s too late. The thing about this profession is that it is so full of negative signals from other people: critical reviews of articles; negative signals on promotion prospects; deflating rejections for grant proposals; and, so on (if you are trying to do a PhD, we might add deflating rejections for funding that threaten the completion of the project; if you are not a white man, we might discuss further obstacles relating to relative success rates). So, when people actually come up to you and say that they have enjoyed something you’ve written (and can discuss it with you in some depth, largely proving that they are not just being polite), it’s brilliant. There will be better descriptions out there, but ‘brilliant’ will do for now. The same goes for general name recognition – there is just something about people seeing your name badge and recognising your name (it beats the quite-regular semi-sneer when people can’t be arsed with you). So, the benefit of not being fully arrogant (yet) is that you can enjoy these crumbs of comfort in a rather disproportionate way. This may be some comfort to the PhD student wondering if it’s all worth it – in some cases it might be.

See also Part 2 http://paulcairney.blogspot.co.uk/2013/06/notes-from-conference-part-2-what-are.html
See also Part 3 http://paulcairney.blogspot.co.uk/2013/06/notes-from-conference-part-3.html

Notes From a Conference Part 2: What are They For?

It’s probably not a good idea to reflect on a conference too close to its end, when you are tired and homesick, but I’m going to do it anyway. As with many large international conferences I’ve been to, my usual response is to wonder if it was worth the bother of being away from home, away from my family, (*middle class problems alert*) in a crap hotel room and faced with the need to sit, stand, listen and talk politely for such a long time – when the time could be put to better use at home (doing or writing research and/ or watching the tennis/ football). So, what are the most important benefits?

1. Meeting people, new and old. Many of us will tend to have most contact with people by email, so it is good to get the time to have an actual conversation with people (often from different countries). In my case, I had a decent mix: meeting a longstanding co-author to discuss more projects; meeting a new co-author to discuss our chapter; meeting a handful of new people that I’d like to keep in touch, and do research, with; and having a few quick discussions with one of my PhD students off campus and in a new atmosphere (and immediately before and after her paper).
2. Having your ego stroked a little bit and getting yourself known a bit better (see previous blog).
3. Giving you a deadline to complete a piece of work in a way that other deadlines can’t do (for many, if not most, people the thought of talking mince for 15 minutes in front of your peers is not an enjoyable prospect).

I think these are the least important benefits:

1. Getting new information from presentations and/ or papers. The quality of conference papers and presentations is so mixed that it’s difficult to justify the time spent reading and listening. In fact, my increasing impression is that many, if not most, people are *not* reading papers and listening (indeed, you can tell that many people are not listening because they have their laptops out and are replying to emails or having a sly look at the news and sport). This problem can be compounded by inadequate rooms (I had one seminar for 20 people in a 900 seat lecture hall; I had another in a room where you could only *just* hear the speaker if no-one moved).
2. Getting feedback on papers. Sometimes this works. In fact, for one of my papers the audience was 7 people (it was at 8.30am, the day after the conference dinner, which ended after midnight), allowing us to engage in an *actual conversation* (the other was about 30 people, which was quite good too, but in a different way – it allows you to see if you can give convincing replies). Sometimes, it doesn’t work. In fact, sometimes (for example if you are on a panel of 4) no-one will ask you a question and you will wonder why you bothered.
3. Finding that all the interesting papers are all being given at the same time (and. If you are very unlucky, at the same time as your presentation).

In other words, the benefit of a conference may not relate to the thing that seems to drive it and take up most of its time. Maybe the notional equivalent in politics is either the international summit (a set-piece event where most of the work is done in advance and the most productive discussions are ‘away from the table’) or the well-attended state funeral (which may involve fewer speeches and gives people the chance to talk without any weight of expectation).

See also part 1 http://paulcairney.blogspot.co.uk/2013/06/notes-from-conference-part-1-arrogance.html
See also part 3 http://paulcairney.blogspot.co.uk/2013/06/notes-from-conference-part-3.html

Notes From a Conference Part 3: The International Conference on Public Policy

The ICPP (Grenoble) symbolised both the best and worst aspects of scholarship. The best bits include:

• The unexpected levels of attendance (900) – which showed many of us (perhaps used to the limited focus on policymaking at general conferences) that we had many international colleagues engaged in similar research.
• The ability to see beyond your specialism and listen to plenary discussions and panels on topics you may not consider in your day-to-day research.
• The opportunities to meet people, exchange ideas and make research plans.

But, being a tired, dour Scot, I was struck mostly by the problems symbolised by the conference:

1.Are we talking *to* or *past* each other?

The plenary on the so called ‘tribes’ of policymaking (IAD, new institutionalism, ACF, etc.) involved a brief discussion, by each representative of a ‘tribe’, of the first principles of each approach – without giving much information about how they relate to each other. This is characteristic of much of the literature which involves specialisation. Such specialisation is often valuable and necessary – it is perhaps only when we immerse ourselves in, and fully understand, an approach that we can assess its merits and relate it to other approaches. However, it also seems parochial if there is a limited level of self-awareness and a tendency to ignore other approaches. Watching the event, you would struggle to identify a sense of *general purpose*. For me, the idea behind specialisation is that we are boundedly rational – we cannot produce all research ourselves. So, we produce some work and rely on others to produce the rest. Then we try to compare our experiences and: (a) explore or ability to generalise from those combined experiences; and (b) explore our ability to accumulate knowledge from a range of studies. This exchange of ideas and information will not be effective if we are all talking a different language; if we don’t know how to communicate our findings (and their significance) to each other in a meaningful way. Maybe the plenary served that purpose by reminding us of the wider world out there, but you would have to be a super-positive person to come to that conclusion.

2.Are we even talking about the same thing?

I was often struck by the relative lack of cohesion of many panels even when they came under a common banner. So, they were not only describing very different case studies but also very different ways to understand them. Again, this can produce a degree of innovative thinking when we consider new possibilities. However, it can also make you wonder if you can slip out of the room when no-one is watching.

3. Self-contradictory case study approaches.

The papers were either mainly-theoretical or contained a theoretical and case-study-based empirical section. What follows is a caricature of some presentations to make a broad point:

• First, they say that existing theories cannot fully explain their case study.
• So, they propose a ‘new’ theory which it explains it better.
• Then, they might imply that this new theory has a more general application.

The overall effect can appear to be contradictory: no theory can explain my case because it is (a) more complicated than theory suggests; and/ or (b) the case has some unusual elements that are difficult to explain. If so, such papers perpetuate the problem – we are forever seeking novel and parsimonious theories to explain many cases, only to be faced with complexity and a significant level of non-comparability when we try to apply them in different cases.

In that light, my preference is for a problem-focused approach to presentation:

• Talk about a real research problem – what do you want to explain?
• Talk about the insights that one or more theories can give you when you seek explanation.
• Accept that theories are simplifications to aid general explanation; don’t express mock surprise when they fail to explain everything. This is just not possible.
• If a key tenet of public policy studies is that politics and policymaking vary from issue to issue (and country), we should not be surprised that a theory based on some issues and countries does not map directly onto others. The same can be said for the case study – don’t just assume that the usefulness of a new or old theory in one case applies to another. Instead, reflect on the ways in which your case compares to the cases described by other studies.

We might then want to talk about the research outcomes. Such conversations require a common language – a requirement that is not served well by the constant pursuit of new theories and a rejection of the old. If we are constantly claiming to be reinterpreting the fundamental nature of policymaking, how can we communicate our findings to each other?

Instead, we can pursue a common language by focusing on what Peter John describes as the five ‘core causal processes’ in public policy. We may say that policymakers operate within the following context:

1. Institutional – they are influenced by the (written and unwritten; formal/ statutory and informal) rules and norms within systems and organisations.
2. Agenda-setting – policymakers are ‘boundedly rational’, prompting them to (a) pay more attention to some issues and solutions at the expense of most others; (b) understand issues in a biased way. So, the way in which they act follows from the way in which they understand, interpret, define or frame their problems and actions.
3. Networks/ Subsystem – policy is devolved from elected policymakers to bureaucrats who consult with groups to gather information and advice. This low level of government may be where most policy work is processed. Some groups are more powerful than others; they are considered more worthy of attention than others. Relationships develop between some groups and civil servants and these networks often represent the main arena in which information is exchanged, then given to elected policymakers (or, choices are made on their behalf by civil servants operating in these networks).
4. Socio-economic – for example, some problems may appear more pressing than others, and some solutions may be more or less attractive, because they are linked closely to the economic environment. Or, demographic change presents new problems. Or, a policymaker’s understanding of social attitudes may underpin their policy strategy. In each case, policymakers interpret a range of policy conditions, or operate in policy environments, that appear to present obstacles to, or opportunities for, action.
5. The role of ideas – policymaking is underpinned by the beliefs present within political systems, such as the world views of policymakers or the actors most influential in that system. We talk of ‘core beliefs’, ‘paradigms’ and ‘policy monopolies’ to describe the fundamental importance of a common understanding of the world that may be so dominant that it is taken for granted. We also talk about ideas as new ways of thinking about problems, and solutions, which challenge such fundamental beliefs (often following a period of ‘learning’ from the past, other issues or other political systems)..

We may have different interpretations of these concepts and they all overlap (the links between 2 and 5 may seem most obvious; we may also say that institutions are shared beliefs; that close networks are based on common understandings; that people interpret socioeconomic conditions and new ideas; and so on). Of course they do – these are analytical simplifications not present in the ‘real world’. Further, we may say that some issues transcend these factors – such as the role of gender inequalities which may be present in institutions, shape the way that people understand problems, influence the consultation process, and underpin belief systems.

However, at least they give us the chance for a common starting point for discussion and explanation. We might even say that our reference to these factors represents the product of our accumulation of knowledge in the field (or not).

4. What is a satisfactory explanation? Can we ever agree?

In a broader sense, we are talking about our ability to agree about what constitutes a satisfactory explanation. In my opinion, a convincing explanation comes from a detailed account of policymaking (stability and instability; policy continuity and change) with reference to all five of these causal factors. We discuss their individual importance – as an analytical device to aid the simplification of complex issues – and discuss the extent which outcomes are caused by the interplay between all five. So, for example, institutions alone do not explain behaviour (unless we use a ridiculously broad definition of an institution) and neither does the socioeconomic context (however pressing), the ideational context, or the strong relationships between some groups and government – but a combination of such factors may help explain why policymakers act in certain ways (and perhaps why their actions are more or less acceptable or successful).

The alternative is to specialise; to focus on certain aspects of this process to gain a better understanding of them. This is good too, but not if it comes at the expense of the bigger picture (or, if we simply try to quantify the relative effect of one factor in a naïve way – which, in many cases, misses the point of complex explanation). It would be good for presenters on particular topics to reflect, however briefly, on how these topics relate to the concerns of others – to recognise that they know a lot about the foot but that the heart might be important too.

5. Are we really talking to each other? How do we exchange information in a meaningful way?

I attended every possible session in the ICPP and so I received a concentrated dose of the tendency of presenters to give out information in an unsatisfactory way. My pet peeve is slides of very small numbers which are presented for a few seconds without explanation; without the presenter taking the time to give them meaning. For me, this tops the presenter-reads-every-word-on-the-powerpoint approach (because at least, in that case, you can close your eyes to listen). This is not good.

It is perhaps a symptom if the wider tendency to cram a ridiculous amount of presentations into short slots – either the 4 papers/ 2 discussant approach (90 minutes) of APSA or the 5 papers (2 hours) at the ICPP. Who can possibly sit through all of those presentations without daydreaming or nodding off?  It is also a symptom of the lack of awareness of the needs of an audience. If we are there to talk to each other (and not simply represent an awake audience), we need the time to discuss papers rather than just listen to them. Only then will we know if the information we present is useful, or if the round of applause is really just a symbol of audience relief.

6. Last but not least – too many men.

Even I (a male, white, middle class and increasingly privilege professor who benefits from these inequalities) am getting tired of seeing panels that are all, or predominantly, male. Most plenary sessions were embarrassingly male and, when the photos go on the web, will not serve as a good advertisement for the profession (although we cannot simply blame the organisers – http://occamstypewriter.org/athenedonald/2013/06/24/all-male-invited-speakers-its-complicated/).

See also Part 1 http://paulcairney.blogspot.co.uk/2013/06/notes-from-conference-part-1-arrogance.html
See also Part 2 http://paulcairney.blogspot.co.uk/2013/06/notes-from-conference-part-2-what-are.html

Testing, testing, podcast on independence

This is a 10-minute podcast on Scottish independence – What Does it Mean and What Are The Big Questions? – recorded on my Ipad. The production values are fairly low and, on two occasions, it seems to flicker a bit (much like in those horror or sci-fi films where things go a little bit, spookily, wrong). I also get a text which distracts me a bit. Then I sound like I am getting bored and more sarcastic from 8 minutes (any of my former students will be used to that). Other than that, it is OK, as long as you like the Andy Murray style monotone (although our accents are very, very different).

Podcast Powered By Podbean

You can also get it here: http://paulcairney.podbean.com/2013/06/18/scottish-independence/

The book is out in August and it won’t really be £25 – http://www.palgrave.com/products/title.aspx?pid=569083

See also: http://paulcairney.blogspot.co.uk/2013/06/the-indyref-and-scottish-parliament.html

A Picture of Pathways to Impact

This picture went into my recent ESRC application and it seemed a shame to just go to waste there (so to speak) …

….It was drawn by Tereza Procházková, who is a Masters student of Service Design, a course run by Hazel White at the Duncan of Jordanstone College of Art & Design. I had asked Tereza to interpret my ‘Case for Support’ so that I could demonstrate to the ESRC how I would try to work with Tereza to produce some reports using words and pictures. The point is not to make superificial or simplistic arguments about complicated topics. Rather, the idea is that the production of drawings forces you to decide what the key points of a document are (perhaps in a stronger way than an abstract or set of bullet points would make you choose). The interpretation of my reports by someone else also allows me to check if I have managed to get my point across to an audience that doesn’t understand the issues in the same way (see also http://paulcairney.blogspot.co.uk/2013/07/how-do-people-read-your-research.html).
The idea came from the CIPFA annual conference in Scotland in 2013. White’s students contributed to her lecture/ workshop on ‘Service Design’ at the event and Tereza also summarised my lecture (on complex policymaking systems) using text (found here, compare with my lecture/ blog post here) and pictures. I don’t *think* that it appealed to me simply because the picture of me was flattering. 

Using the Internet for Political Research (POL9RM 30.4.13)

Since I am giving a lecture on using the internet for political research (for POL9RM), it seems appropriate to upload it to your actual internet. This is, depending on how you view these things, either an excessively long or good-value blog. We might settle on ‘generous’.

The lecture will be split into two basic issues:

1. What type of information should you seek, and from where?

2. What are the best (or, at least, most used) tools to use to source that information?

Types of Information

Before we examine how best to use the internet for political research, we should consider why you (as a former first year student) may have been advised *not* to rely on the internet to produce essays. Why would there be such an objection to using freely available information produced by such a wide range of people? Why, perhaps, would people criticise you for favouring the democratic, as opposed to the elite, production of knowledge? The answer is not that your older lecturers – who still remember the days of punch cards, rolodexes, hard copy journals and the need to speak to someone about borrowing books – resent the fact that you can get information so quickly without going to a library or even leaving your home. It is also not a form of group closure, in which we protect and promote our own people, methods and types of information.  It is not that lecturers associate undergraduate student internet research with a small bunch of people copy-and-pasting from Wikipedia into their essays (although this does happen, sometimes).

The more important explanation is that, in a world where the amount of information seems infinite, it may be increasingly difficult to identify the information to which we should pay most or least attention.  We need to identify some sort of hierarchy of important and/ or reliable information. This is a skill like any other. Your successful development of this skill will be reflected in your grades, since it will relate to the willingness of your essay markers to accept the information you present, according to the way in which you account for your information.

One way to address this issue is to consider why lecturers tend to treat books and/ or journal articles as the gold standard of information – at least as secondary sources, before you are expected to do your own research using primary sources such as government documents.

Possible answers (though not always useful answers) include:

1. They are peer reviewed. Lecturers may recommend journals that only allow publication after a number of relevant academics have commented on the work, often anonymously. The number of peer reviewers may vary from 1 to 6; in my experience an article is usually read by 2 or 3 anonymous reviewers. Their evaluation of journals may be linked partly to the reputations that some journals have in terms of academic rigour, linked to the need for scholars to anticipate and address critical reviews before having their work accepted.

2. They are competitive and much work is rejected. Some journals have an acceptance rate below 10%, which helps them develop an image of prestige and cutting edge research. (You can get a rough idea of acceptance rates in politics and IR here – http://www.reviewmyreview.eu/ ). Some people also put some faith in proxy measures of journal reputations, such as their ‘impact factors’ (although this is a problematic faith). It is also an increasing trend to evaluate the status of individual scholars according to the extent to which the publication is cited by other scholars (see below on Google scholar). Being well cited is a proxy used increasingly to gauge respect for the information.

3. They may be based on a long period of scholarly research. This varies from discipline to discipline. Consider, for example, the historical research produced over years after painstaking attention to thousands of documents.

4. The research may be theoretically informed. Being theoretically informed means being aware of the general implications of individual pieces of information. In part, this focus on theory is based on one role of scientific research: to draw lessons from sets of single cases to produce insights that may apply to many or all cases. A focus on theory is a focus on generalisation – something that is difficult to do if we rely only on information that is produced in very particular circumstances.

5. The research may be methodologically sophisticated. Most research is required to reach some sort of level of sophistication. For example, a quantitative survey requires a certain (large) number of responses to be considered statistically significant (in other words, for us to conclude that the results could not have happened by chance). It is then subject to a series of statistical techniques (which you may learn, using programmes such as SPSS) to explore the associations between variables. Qualitative research may be judged on different criteria, but there is a similar requirement that the conduct of the research meets certain professional criteria. The data may then be subject to further techniques to gauge its meaning and significance. It is often a condition of journal article acceptance that the scholars set out clearly their methods (and often provide a copy of the data for others to use).

6. The research may be empirically rich. Much research is based on surveys of thousands of people or qualitative interviews of dozens or hundreds. The data may be combined with documentary and historical analysis to produce a wealth of information. That information may be compared with information from other studies, to help accumulate knowledge within particular fields.

7. The authors may be meticulous when they identify the source of their information. One aim of the scholarly text is to show the reader where they got their information, to allow the reader to follow up, confirm and/ or read further. Most of you will have noticed the attention that we pay to your referencing style and bibliographies (hopefully most of us are looking for a consistent style rather than a particular style, although be suspicious of people who don’t agree that Harvard is best). This is because, when people make empirical or theoretical claims, the understanding is that they show us the information on which they based their claims (or, at least, they give us the option to follow up their work).

It is according to these kinds of criteria that we may judge other sources of information. In many cases, the information may be useful even if it does not live up to many or any of these criteria. For example, it is legitimate to use newspaper stories and commentary pieces, particularly if they provide much-needed timely information and the source is seen as reliable (indeed, although some newspapers now suffer poor reputations, we can still identify a tradition of fact/ source checking as a routine part of information gathering – partly, but not exclusively, because journalists are generally proud of their reputations and newspaper managers do not want to be sued). However, we would then have to consider the trade off against the academic ‘gold standard’ (is the information likely to be theoretically informed and based on a sophisticated method?) and consider the extent to which the trade is appropriate. In many cases, this just comes down to a mix of sources – student essays could benefit from immediate sources but those sources may not be an alternative to a more comprehensive review of the relevant literature.

However, this is not to say that academic information is unproblematic. We may be meticulous when we catalogue our sources of information, but that information may still be of varying value. For example, quantitative work may be limited by the availability of information provided by other actors (such as governments) and qualitative work may be limited by access to the right sources of information and simple things like the ability of interviewees to recall or provide an honest recollection of relevant information. The more difficult task, then, is to consider in more depth how people access and present information and how we might compare and critically analyse those sources to produce what we consider to be an accurate or valuable overall assessment of the available information.

Search Tools

We might divide that search broadly intro two categories: primary and secondary sources. I will a focus on Scottish politics to tailor the advice.

Secondary sources

The starting point for most students is likely to be a secondary source: you look through the existing academic literature for your information. Those texts analyse things like government documents, and you get your information about those documents indirectly, through a secondary source.

For me, the best way to start a search for secondary sources is to use Google Scholar (http://scholar.google.co.uk/). You enter a small number of search terms and it produces a list of materials to consult. Most Universities also have the ability (particularly if you search on campus) to link the article access directly to the search. Your results are likely to reflect your search terms. For example, a search for ‘Scottish politics’ reveals a list of general texts, while a search for referendums in Scotland produces more specific texts. They tend to be organised according to the extent to which they have been cited elsewhere (which often produces a tendency for older materials to be listed first).

A good rule of thumb is that you should use Scholar more as you progress in your studies. The ‘further reading’ section of a textbook, or list of readings in a course guide, is essential when you are an early undergraduate. When you become an advanced undergraduate, and start to plan to write a relatively independent piece of work, you are expected to do your own searches for the relevant literature. This will require you to think carefully about your research question and the keywords you will have to use to get the most out of the search. You may also need to think about the sources of offer from Google scholar. There is now a wide availability of journals and books, but how do you prioritise and/ or determine the quality of the information?

Primary Sources

An independent project will also prompt you to seek primary sources of information, including government and parliamentary sources. This requires a bit more thought, since you are unlikely to get useful information unless you have first thought about what your research problem is and how you intend to address it. In other words, you think about what you want to know, what are the most appropriate methods to get the right information, and *then* do these sorts of searches.

Government and parliamentary sources

It is quite amusing to look through the Burnham et al (2008) 2^nd edition of ‘Research Methods in Politics’ because it still talks about CD ROMS. These were growing in popularity when I was a student, but you may never have used a physical, round, disk to secure information (and may never have to). Instead, sources of government and parliamentary information tend to be available online directly from them (or, if you are that way inclined, through sites that claim to provide documents that governments don’t want you to see). For example the Scottish Government has a fairly extensive site (http://www.scotland.gov.uk/Topics) which allows you to search using key terms. This tends to be less straightforward than a Google Scholar search, forcing you to be much clearer in your mind about what sort of information you want (since you will have to be fairly specific in your search unless you want to sift through a tonne of information). Similarly, the Scottish Parliament (http://www.scottish.parliament.uk/) provides a written record of virtually anything that any MSP has said in a committee or plenary discussion since 1999, as well as a full record of written and oral evidence to committees. Again, there is too much information to browse, so you first need to think about what you want to know.

Newspapers

Lexis Nexis (which can be accessed through library resources) is a key archive of UK newspapers and can track the Scottish papers to some extent (it is OK for Scottish papers like the Scotsman, Herald, Daily Record and Press & Journal, but it does not store Scottish editions of the UK papers like the Daily Mail). The archive varies, with papers like The Times providing the longest stretches of data (the Scotsman has also begun to archive its really old material). Again, to make the search manageable (below, say, 1000 stories), you need to be very specific about what you want. For example, a search for Scotland AND referendum AND independence will produce thousands of stories which will take you days to get through (unless you focus on a short space of time).

Blogs

Blogs are a minefield. Consider the extent to which they meet the gold standard criteria I outlined above. Some of them might do, but how would you know? I would treat blogs in the way that I might treat newspapers: you might trust them if they have a good reputation (but, then, whose opinions do you trust on reputations?). You should also expect biased, and often highly biased, opinions. That means that they can be a good source of information, but you might ask yourself if you can rely on a blog on its own, or as something to be compared with one or more sources. Interestingly (for me at least), along with my co-author on the 2^nd ed of Scottish Politics, Neil McGarvey, I had to come up with a list of websites to check out. The idea is that they would be relatively useful, but is this list (below) particularly reliable? Or, are they simply the ones that came to mind at the time?

SCOTTISH POLITICS BLOGS

http://www.betternation.org/

http://brightgreenscotland.org/

http://underdogsbiteupwards.blogspot.com/

http://lallandspeatworrier.blogspot.com/

http://burdzeyeview.wordpress.com/

http://www.labourhame.com/

http://andrewrunning.blogspot.com/

http://bellacaledonia.org.uk/

http://www.scotgoespop.blogspot.com/

http://joanmcalpine.typepad.com/

http://www.gerryhassan.com/

http://iainmacwhirter2.blogspot.com/

http://albamatters.blogspot.com/

http://the-universality-of-cheese.blogspot.com/

http://theshooglypeg.com/

http://scotspolitics.com

http://wingsland.podgamer.com/meltdown-man/

 

Twitter and other things

I tend to use Twitter as an alternative to the TV, as an additional source of entertainment. However, it is possible to sign up to a wide range of news and party sites and to use twitter as an alternative to reading newspapers page by page. This provides you with a new source of bias, but perhaps no more problematic than sticking with a paper like the Daily Mail. You might even simply follow specific lists (such as academics on twitter) or, if you are feeling particularly lazy, just look at the list of people/ organisations I follow and piggyback on that.  There are things that are particularly useful, such as the LSE blog sites and accounts such as ‘Writing For Research’ which you might find more useful if you progress to postgraduate work.

And finally ..

Here is a list of websites that Neil and I produced for our book. It is likely to be a bit scattergun and biased (for example, we don’t list small party websites), and some will already be out of date, but there may be some sources there that you wouldn’t otherwise consider.

EXAMPLES OF POTENTIALLY USEFUL WEBSITES ON SCOTTISH POLITICS

(from the chapters of the forthcoming 2^nd ed. of ‘Scottish Politics’ by Paul Cairney and Neil McGarvey)

The idea here is that these websites might get you started if you don’t want to rely on a scattergun search engine search. It is not a particularly well-thought-out list, so be careful!

DEVOLUTION

Scottish Parliament http://www.scottish.parliament.uk/
Report of the Consultative Steering Group http://www.scottish.parliament.uk/PublicInformationdocuments/Report_of_the_Consultative_Steering_Group.pdf 
The Economic and Social Research Council ‘Devolution and Constitutional Change’ research programme http://www.devolution.ac.uk  and ‘The Future of the UK and Scotland’ http://www.esrc.ac.uk/about-esrc/what-we-do/our-research/future-of-uk-and-scotland/index.aspx  
University of London’s Constitution Unit http://www.ucl.ac.uk/constitution-unit/    
UK Politics page http://www.ukpolitics.org.uk/

Paul Cairney’s Blog: http://paulcairney.blogspot.co.uk/

MEDIA
http://www.smg.plc.uk
http://www.allmediascotland.com  
http://www.bbc.co.uk/scotland/
http://www.theherald.co.uk/
http://www.scotsman.com
http://www.holyroodmag.com/
http://www.dailyrecord.co.uk/
http://www.scotlandonsunday.com/
http://www.sundayherald.com
http://www.thisisnorthscotland.co.uk

SCOTTISH ECONOMY

http://www.scotland.gov.uk/Topics/Economy

http://www.scottisheconomywatch.com/

http://www.cppr.ac.uk/centres/cppr/analysisofthescottisheconomy/

http://www.strath.ac.uk/fraser/

SCOTTISH SOCIETY

http://www.scottishcivictrust.org.uk/

http://www.stuc.org.uk/

http://www.churchofscotland.org.uk

http://www.scmo.org/

http://www.eis.org.uk/

http://www.cosla.gov.uk/

http://www.sns.gov.uk/

POLITICAL PARTIES

Scottish Conservative Party http://www.scottish.tory.org.uk/  
Scottish Green Party http://www.scottishgreens.org.uk/ 
Scottish Labour Party http://www.scottishlabour.org.uk/ 
Scottish Liberal-Democrats http://www.scotlibdems.org.uk/
Scottish National Party http://www.snp.org.uk

ELECTIONS

http://www.bcomm-scotland.independent.gov.uk/   
http://www.votescotland.com
http://www.electoralcommission.org.uk/scotland
http://www.fairvote.org
http://www.yougov.com
http://www.populuslimited.com  

http://www.crest.ox.ac.uk  

http://www.aboutmyvote.co.uk/

http://www.hansardsociety.org.uk/

http://ukpollingreport.co.uk/blog/archives/category/scotland

http://www.electoralcalculus.co.uk/polls_scot.html

http://www.scottishelections.org.uk/

http://www7.politicalbetting.com

PARLIAMENT

Scottish Parliament http://www.scottish.parliament.uk/
Scottish Constitutional Convention http://www.almac.co.uk/business_park/scc/ 
Consultative Steering Group http://www.scottish.parliament.uk/PublicInformationdocuments/Report_of_the_Consultative_Steering_Group.pdf 
House of Commons http://www.parliament.uk/business/commons/  

 UCL Constitution Unit – http://www.ucl.ac.uk/constitution-unit/

SCOTTISH GOVERNMENT

Scottish Government http://www.scotland.gov.uk/

Scotland Performs – http://www.scotland.gov.uk/About/Performance/scotPerforms

UK Cabinet Office http://www.cabinet-office.gov.uk 
Scottish Ministerial Code http://www.scotland.gov.uk/About/14944/684 

 GOVERNMENT BEYOND THE CENTRE

http://www.scottishhousingregulator.gov.uk/
http://www.creativescotland.com/
http://www.visitscotland.com/
http://www.scottish-enterprise.com/ 
http://www.cosla.gov.uk

http://www.improvementservice.org.uk/
http://www.show.scot.nhs.uk 
http://www.scvo.org.uk 

http://www.scottishfuturestrust.org.uk

INTEREST GROUPS

Convention of Scottish Local Authorities http://www.cosla.gov.uk/  
Friends of the Earth Scotland  http://www.foe-scotland.org.uk/     
Scottish Council for Development and Industry http://www.scdi.org.uk/
Scottish Trades Union Congress http://www.stuc.org.uk/
CBI Scotland http://www.cbi.org.uk/about-the-cbi/uk/scotland/
Scottish Council for Voluntary Organizations http://www.scvo.org.uk/ 

NFU Scotland – http://www.nfus.org.uk/

Scotch Whisky Association – http://www.scotch-whisky.org.uk/

ASH Scotland – http://www.ashscotland.org.uk/

Educational Institute of Scotland – http://www.eis.org.uk/

SCOTTISH PUBLIC POLICY

ScotHub  http://www.scothub.org/

Centre for Scottish Public Policy http://www.cspp.org.uk  
Centre for Public Policy for Regions http://www.cppr.ac.uk

For SPICe summaries of all Scottish Government bills see http://www.scottish.parliament.uk/parliamentarybusiness/Bills/12417.aspx

INTERGOVERNMENTAL ISSUES

Alan Trench http://devolutionmatters.wordpress.com 

Cabinet Office http://www.cabinet-office.gov.uk/
Scotland Office http://www.scottishsecretary.gov.uk/
Scotland Europa http://www.scotlandeuropa.com
Government EU Office http://www.scotland.gov.uk/euoffice
British/Irish Council http://www1.british-irishcouncil.org

UK Supreme Court – http://www.supremecourt.gov.uk/index.html

McKay Commission http://tmc.independent.gov.uk/

Lallands Peat Worrier http://lallandspeatworrier.blogspot.co.uk/2012/09/baccymongers-bankers-supremes.html

FINANCE

UK National Statistics http://www.statistics.gov.uk/hub/index.html

Her Majesty’s Treasury http://www.hm-treasury.gov.uk/

Scottish Government http://www.scotland.gov.uk/Topics/Economy
Scottish Parliament http://www.scottish.parliament.uk/parliamentarybusiness/17534.aspx

Improvement Service (http://www.improvementservice.org.uk/

CONSTITUTIONAL ISSUES

Scotland Act 2012 http://www.legislation.gov.uk/ukpga/2012/11/pdfs/ukpga_20120011_en.pdf 

Scotland Act 2012 Sewel Motion – Scottish Parliament Official Report 18.4.12 http://www.scottish.parliament.uk/parliamentarybusiness/28862.aspx?r=6972 

Scotland Bill committee final report http://www.scottish.parliament.uk/parliamentarybusiness/PreviousCommittees/29884.aspx

Calman Commission – http://www.commissiononscottishdevolution.org.uk/

National Conversation – http://www.scotland.gov.uk/Topics/constitution/a-national-conversation

Steel Commission – http://www.scotlibdems.org.uk/files/steelcommission.pdf

Scotland Office – https://www.gov.uk/government/news/factual-analysis-promised-as-legal-implications-of-scottish-independence-made-clear

Scottish Government – http://www.scotland.gov.uk/Publications/2013/02/8079

Scottish Government’s Council of Economic Advisers Fiscal Commission Working Group – http://www.scotland.gov.uk/Topics/Economy/Council-Economic-Advisers/FCWG

Alan Trench – http://devolutionmatters.wordpress.com/

Lallands Peat Worrier – http://lallandspeatworrier.blogspot.co.uk/

ScottishPol – http://scottishpol.blogspot.co.uk/

BBC – http://www.bbc.co.uk/news/scotland/scotland_politics/ 

Newsnet Scotland – http://www.newsnetscotland.com/index.php/scottish-opinion/4341-a-unionist-lexicon-an-a-z-of-unionist-scare-stories-myths-and-misinformation

Scots Independent – http://www.scotsindependent.org/index.htm

Constitution Unit – http://www.ucl.ac.uk/constitution-unit/

Edinburgh University Blog – http://www.referendum.ed.ac.uk/

Scottish Constitutional Futures Forum – http://www.scottishconstitutionalfutures.org/Home/tabid/1269/Default.aspx

ESRC – http://www.esrc.ac.uk/about-esrc/what-we-do/our-research/future-of-uk-and-scotland/index.aspx

 

AND FINALLY again

Maybe you have read this far to see what we can say about Wikipedia. My view is that I am sometimes pleasantly surprised about what I see on some of those pages. However, you will find very few academics that will trust your information if your source is Wikipedia (partly because it is difficult to know who is providing the information, how they got it, how well they cite that information, and how easy the information is to edit and manipulate). Maybe a good rule of thumb is that you look at it for a short cut to information, but that you do not rely on it as your definitive source.