Can you want evidence based policymaking if you don’t really know what it is?

This is an old post with a new title. It occurred to me that the phrase ‘evidence based policy making’ has a ‘bewitching’ effect. It seems impossible not to want it when you see the words in it. Yet, the problem is that not only do we not really know what it means, but we also do not know what each word means. I try to define them below, but these definitions don’t take us very far. ‘Evidence’ is assertion backed by information. ‘Based’ seems like a metaphor. ‘Policy’ is one of the worst-defined words in politics (up there with power and democracy). Policymaking implies there is a policymaker, but we don’t always know who it is.

This seems like just a semantic discussion, but I think that there is a lot of confusion in the EBPM literature simply because people begin by complaining that we don’t have it without really saying what we don’t have. A focus on things like ‘evidence informed’ doesn’t make things much better, because it is difficult to point to a policy and show how we would know if it related to the evidence, and to give it a mark out of ten. I think what people are often describing is the extent to which they feel that policymakers listen to what they have to say, and act on that basis. In some cases, people are quick to say that a policy is ‘not evidence based’ if policymakers only listen to some of what they have to say and/ or only adopt some of their recommendations (so it is less of a mark out of ten and more of a 0 or 1 code). For me, this is the most they could reasonably hope for – policy made with some regard to a wide range of the available evidence. It is less catchy, but it gives more equal status to the policymaking part of EBPM, in comparison with many accounts of EBPM which suggest that the policy process mangles *the* evidence; that we should judge the process’ merit according to how much the process dilutes scientific evidence.

…

I was re-reading this sentence …

“The term ‘evidence based policy making’ (EBPM) is in common currency in media and social media. Generally, it is a vague, aspirational term, rather than a good description of the policy process”

… and, before I knew it, I felt the need to define a whole bunch of other things in an end-note. Instead of biting my nails about these definitions, I am going to reproduce them here, and decide only to worry if anyone takes the time to object:

“Unfortunately, it is not the only term to be defined vaguely:

• Policy. There is no single accepted definition of policy. I use the working definition ‘the sum total of government action, from signals of intent to the final outcomes’ to raise important qualifications: (a) it is problematic to conflate what people say they will do with what they actually do; (b) a policy outcome can be very different from the intention; (c) policy is made routinely through cooperation between elected and unelected policymakers and actors with no formal role in the process; (d) policymaking is also about the power not to do something (Cairney, 2012a: 24-5.
• It is also important to identify the components or policy instruments that make up policies, including: the level of spending; the use of economic incentives/ penalties; regulations and laws; the use of voluntary agreements and codes of conduct; the provision of public services; education campaigns; funding for scientific studies or advocacy; organisational change; and, the levels of resources/ methods dedicated to policy implementation (2012a: 26).
• Policymakers. The intuitive definition is ‘people who make policy’, but there are two important distinctions: (1) between elected and unelected people, since actors such as civil servants also make important decisions (2) between people and organisations, with the latter used as a shorthand to refer to a group of people making decisions collectively with reference to a decision-making rule (an ‘institution’). These distinctions are crucial, to remind us that advocates would miss something important if they focused their energies only on elected politicians. They also produce a conceptual problem by blurring the dividing line between the people who make and influence In most respects, this is appropriate, since terms such as ‘policy community’ suggest that policy decisions are made, in some sense, by a collection of people with formal responsibility and informal influence. We just need to be careful to make clear what we mean by ‘policymakers’ in specific discussions.
• Evidence and scientific evidence. We can define evidence as an argument or assertion backed by information. Scientific evidence therefore describes information produced in a particular way. Some (including me) use ‘scientific’ broadly, to refer to information gathered systematically using recognised methods. Others refer to a specific hierarchy of scientific methods, with randomized control trials (RCTs) and quantitative systematic review/ meta-analysis at the top (see Nutley et al, 2013).
• Also note the difference between two kinds of evidence-based activity relating to: the size of the problem (for example, the number of smokers and the link between smoking and ill health); and, the effectiveness of the solution (for example, the effect of higher taxes and health warnings on consumption).”

The endnote will appear in The Science of Policymaking

See also Defining policy shows how messed up it seems (as part of the 1000 words series)

‘The strange new world of evidence-free government’ by Zoe Williams