Tag Archives: policy transfer

There is no blueprint for evidence-based policy, so what do you do?

In my speech to COPOLAD I began by stating that, although we talk about our hopes for evidence-based policy and policymaking (EBP and EBPM), we don’t really know what it is.

I also argued that EBPM is not like our image of evidence-based medicine (EBM), in which there is a clear idea of: (a) which methods/ evidence counts, and (b) the main aim, to replace bad interventions with good.

In other words, in EBPM there is no blueprint for action, either in the abstract or in specific cases of learning from good practice.

To me, this point is underappreciated in the study of EBPM: we identify the politics of EBPM, to highlight the pathologies of/ ‘irrational’ side to policymaking, but we don’t appreciate the more humdrum limits to EBPM even when the political process is healthy and policymakers are fully committed to something more ‘rational’.

Examples from best practice

The examples from our next panel session* demonstrated these limitations to EBPM very well.

The panel contained four examples of impressive policy developments with the potential to outline good practice on the application of public health and harm reduction approaches to drugs policy (including the much-praised Portuguese model).

However, it quickly became apparent that no country-level experience translated into a blueprint for action, for some of the following reasons:

  • It is not always clear what problems policymakers have been trying to solve.
  • It is not always clear how their solutions, in this case, interact with all other relevant policy solutions in related fields.
  • It is difficult to demonstrate clear evidence of success, either before or after the introduction of policies. Instead, most policies are built on initial deductions from relevant evidence, followed by trial-and-error and some evaluations.

In other words, we note routinely the high-level political obstacles to policy emulation, but these examples demonstrate the problems that would still exist even if those initial obstacles were overcome.

A key solution is easier said than done: if providing lessons to others, describe it systematically, in a form that describes the steps to take to turn this model into action (and in a form that we can compare with other experiences). To that end, providers of lessons might note:

  • The problem they were trying to solve (and how they framed it to generate attention, support, and action, within their political systems)
  • The detailed nature of the solution they selected (and the conditions under which it became possible to select that intervention)
  • The evidence they used to guide their initial policies (and how they gathered it)
  • The evidence they collected to monitor the delivery of the intervention, evaluate its impact (was it successful?), and identify cause and effect (why was it successful?)

Realistically this is when the process least resembles (the ideal of) EBM because few evaluations of success will be based on a randomised control trial or some equivalent (and other policymakers may not draw primarily on RCT evidence even when it exists).

Instead, as with much harm reduction and prevention policy, a lot of the justification for success will be based on a counterfactual (what would have happened if we did not intervene?), which is itself based on:

(a) the belief that our object of policy is a complex environment containing many ‘wicked problems’, in which the effects of one intervention cannot be separated easily from that of another (which makes it difficult, and perhaps even inappropriate, to rely on RCTs)

(b) an assessment of the unintended consequence of previous (generally more punitive) policies.

So, the first step to ‘evidence-based policymaking’ is to make a commitment to it. The second is to work out what it is. The third is to do it in a systematic way that allows others to learn from your experience.

The latter may be more political than it looks: few countries (or, at least, the people seeking re-election within them) will want to tell the rest of the world: we innovated and we don’t think it worked.

*I also discuss this problem of evidence-based best practice within single countries

 

1 Comment

Filed under Evidence Based Policymaking (EBPM), Prevention policy, Public health, public policy, tobacco policy

The politics of evidence-based best practice: 4 messages

Well, it’s really a set of messages, geared towards slightly different audiences, and summed up by this table:

Table 1 Three ideal types EBBP.JPG

  1. This academic journal article (in Evidence and Policy) highlights the dilemmas faced by policymakers when they have to make two choices at once, to decide: (1) what is the best evidence, and (2) how strongly they should insist that local policymakers use it. It uses the case study of the ‘Scottish Approach’ to show that it often seems to favour one approach (‘approach 3’) but actually maintains three approaches. What interests me is the extent to which each approach contradicts the other. We might then consider the cause: is it an explicit decision to ‘let a thousand flowers bloom’ or an unintended outcome of complex government?
  2. I explore some of the scientific  issues in more depth in posts which explore: the political significance of the family nurse partnership (as a symbol of the value of randomised control trials in government), and the assumptions we make about levels of control in the use of RCTs in policy.
  3. For local governments, I outline three ways to gather and use evidence of best practice (for example, on interventions to support prevention policy).
  4. For students and fans of policy theory, I show the links between the use of evidence and policy transfer.

Further reading (links):

My academic articles on these topics

The Politics of Evidence Based Policymaking

Key policy theories and concepts in 1000 words

Prevention policy

10 Comments

Filed under 1000 words, ESRC Scottish Centre for Constitutional Change, Evidence Based Policymaking (EBPM), Prevention policy, Scottish politics, UK politics and policy

Policy Concepts in 1000 Words: the intersection between evidence and policy transfer

(podcast download)

We can generate new insights on policymaking by connecting the dots between many separate concepts. However, don’t underestimate some major obstacles or how hard these dot-connecting exercises are to understand. They may seem clear in your head, but describing them (and getting people to go along with your description) is another matter. You need to set out these links clearly and in a set of logical steps. I give one example – of the links between evidence and policy transfer – which I have been struggling with for some time.

In this post, I combine three concepts – policy transfer, bounded rationality, and ‘evidence-based policymaking’ – to identify the major dilemmas faced by central government policymakers when they use evidence to identify a successful policy solution and consider how to import it and ‘scale it up’ within their jurisdiction. For example, do they use randomised control trials (RCTs) to establish the effectiveness of interventions and require uniform national delivery (to ensure the correct ‘dosage’), or tell stories of good practice and invite people to learn and adapt to local circumstances? I use these examples to demonstrate that our judgement of good evidence influences our judgement on the mode of policy transfer.

Insights from each concept

From studies of policy transfer, we know that central governments (a) import policies from other countries and/ or (b) encourage the spread (‘diffusion’) of successful policies which originated in regions within their country: but how do they use evidence to identify success and decide how to deliver programs?

From studies of ‘evidence-based policymaking’ (EBPM), we know that providers of scientific evidence identify an ‘evidence-policy gap’ in which policymakers ignore the evidence of a problem and/ or do not select the best evidence-based solution: but can policymakers simply identify the ‘best’ evidence and ‘roll-out’ the ‘best’ evidence-based solutions?

From studies of bounded rationality and the policy cycle (compared with alternative theories, such as multiple streams analysis or the advocacy coalition framework), we know that it is unrealistic to think that a policymaker at the heart of government can simply identify then select a perfect solution, click their fingers, and see it carried out. This limitation is more pronounced when we identify multi-level governance, or the diffusion of policymaking power across many levels and types of government. Even if they were not limited by bounded rationality, they would still face: (a) practical limits to their control of the policy process, and (b) a normative dilemma about how far you should seek to control subnational policymaking to ensure the delivery of policy solutions.

The evidence-based policy transfer dilemma

If we combine these insights we can identify a major policy transfer dilemma for central government policymakers:

  1. If subject to bounded rationality, they need to use short cuts to identify (what they perceive to be) the best sources of evidence on the policy problem and its solution.
  2. At the same time, they need to determine if there is convincing evidence of success elsewhere, to allow them to: (a) import policy from another country, and/ or (b) ‘scale up’ a solution that seems to be successful in one of its regions.
  3. Then they need to decide how to ‘spread success’, either by (a) ensuring that the best policy is adopted by all regions within its jurisdiction, or (b) accepting that their role in policy transfer is limited: they identify ‘best practice’ and merely encourage subnational governments to adopt particular policies.

Note how closely connected these concerns are: our judgement of the ‘best evidence’ can produce a judgement on how to ‘scale up’ success

Here are three ideal-type approaches to using evidence to transfer or ‘scale up’ successful interventions. In at least two cases, the choice of ‘best evidence’ seems linked inextricably to the choice of transfer strategy:

3 ideal types EBPM

With approach 1, you gather evidence of effectiveness with reference to a hierarchy of evidence, with systematic reviews and RCTs at the top (see pages 4, 15, 33). This has a knock-on effect for ‘scaling up’: you introduce the same model in each area, requiring ‘fidelity’ to the model to ensure you administer the correct ‘dosage’ and measure its effectiveness with RCTs.

With approach 2, you reject this hierarchy and place greater value on practitioner and service user testimony. You do not necessarily ‘scale up’. Instead, you identify good practice (or good governance principles) by telling stories based on your experience and inviting other people to learn from them.

With approach 3, you gather evidence of effectiveness based on a mix of evidence. You seek to ‘scale up’ best practice through local experimentation and continuous data gathering (by practitioners trained in ‘improvement methods’).

The comparisons between approaches 1 and 2 (in particular) show us the strong link between a judgement on evidence and transfer. Approach 1 requires particular methods to gather evidence and high policy uniformity when you transfer solutions, while approach 2 places more faith in the knowledge and judgement of practitioners.

Therefore, our choice of what counts as EBPM can determine our policy transfer strategy. Or, a different transfer strategy may – if you adhere to an evidential hierarchy – preclude EBPM.

Further reading

I describe these issues, with concrete examples of each approach here, and in far more depth here:

Evidence-based best practice is more political than it looks: ‘National governments use evidence selectively to argue that a successful policy intervention in one local area should be emulated in others (‘evidence-based best practice’). However, the value of such evidence is always limited because there is: disagreement on the best way to gather evidence of policy success, uncertainty regarding the extent to which we can draw general conclusions from specific evidence, and local policymaker opposition to interventions not developed in local areas. How do governments respond to this dilemma? This article identifies the Scottish Government response: it supports three potentially contradictory ways to gather evidence and encourage emulation’.

Both articles relate to ‘prevention policy’ and the examples (so far) are from my research in Scotland, but in a future paper I’ll try to convince you that the issues are ‘universal’

 

 

5 Comments

Filed under 1000 words, Evidence Based Policymaking (EBPM), Prevention policy, public policy

Policy Concepts in 1000 Words: Policy Transfer and Learning

(podcast download)

‘Policy learning’ describes the use of knowledge to inform policy decisions. That knowledge can be based on information regarding the current problem, lessons from the past or lessons from the experience of others. This is a political, not technical or objective, process (for example, see the ACF post). ‘Policy transfer’ describes the transfer of policy solutions or ideas from one place to another, such as by one government importing the policy in another country (note related terms such as ‘lesson-drawing’, ‘policy diffusion’ and ‘policy convergence’ – transfer is a catch-all, umbrella, term). Although these terms can be very closely related (one would hope that a government learns from the experiences of another before transferring policy) they can also operate relatively independently. For example, a government may decide not to transfer policy after learning from the experience of another, or it may transfer (or ‘emulate’) without really understanding why the exporting country had a successful experience (see the post on bounded rationality). Here are some major examples:

BOX 12.1

It is a topic that lends itself well to practical advice; the ‘how to’ of policymaking. For example, Richard Rose’s ‘practical guide’ explores 10 steps:

Rose 10 lessons rotated

The descriptive/ empirical side asks these sorts of questions:

From where are lessons drawn? In the US, the diffusion literature examines which states tend to innovate or emulate. Some countries are also known as innovators in certain fields – such as Sweden and the social democratic state, Germany on inflation control and the UK on privatization. The US (or its states) tends to be a major exporter of ideas. Some countries often learn consistently from the same source (such as the UK from the US). Studies tend to highlight the reasons for borrowing from certain countries – for example, they share an ideology, common problems or policy conditions. ‘Globalization’ has also reduced practical barriers to learning between countries.

Who is involved? Apart from the usual suspects (elected officials, civil servants, interest groups), we can identify the role of federal governments (for states), international organizations (for countries), ‘policy entrepreneurs’ (who use their experience in one country to sell that policy to another – such as the Harvard Business School professor travelling the world selling ‘new public management’), international networks of experts (who feed up ideas to their national governments), multinational corporations (who encourage the ‘race to the bottom’, or the reduction of taxes and regulations in many countries), and other countries (such as the US).

Why transfer? Is transfer voluntary? The Dolowitz/ Marsh continuum sums up the idea that some forms of transfer are more voluntary than others. ‘Lesson-drawing’ is about learning from another country’s experience without much pressure (see the book to explain why I scribbled out some of the text!). At the other end is coercion. They place ‘conditionality’ near that end of the spectrum, since the idea is that countries who are so desperate to borrow money from the International Monetary Fund will feel they have no choice but to accept the IMF’s conditions – which usually involves reducing the role/ size of the state (although note the difference between agreeing to those conditions and meeting them). ‘Obligated transfer’ is further to the left because, for example, member states sign up to be influenced by EU institutions. Indirect coercion describes countries who feel they have to follow the lead of others, simply to ‘keep up’ or to respond to the ‘externalities’ or ‘spillovers’ of the policies of the other country (they are often felt most by small countries which share a border with larger countries).

figure 12.1 DM continuum

What is transferred? How much is transferred? Transfer can range from the decision to completely duplicate the substantive aims and institutions associated with a major policy change, taking decades to complete, to the vague inspiration (or the very quick decision not to emulate and, instead, to learn ‘negative lessons’).  It can also be a cover for something you planned to do anyway – ‘international experience’ is a great selling point.

What determines the likelihood and success of policy transfer? For an importing government to be successful, it should study the exporting country’s policy – and political system – enough to know what made it a success and if that success is transferable. Often, this is not done (governments may emulate without being particularly diligent) or it is not possible, since the policy may only work under particular circumstances (and we may not always know what those circumstances are). Much also depends on the implementation of policy, particularly when the transfer is encouraged by one organization and accepted reluctantly by another (such as when the EU, with limited enforcement powers, puts pressure on recalcitrant member states).

These questions are best asked alongside the general questions we explore in policymaking studies, including:

  • Bounded rationality and Incrementalism – do governments engage in trial-and-error and learn from their own mistakes first?  Is learning and transfer restricted to the ‘most similar’ regions because there is no point in learning from countries radically different from our own?  Do some governments emulate without learning? Is transfer from another, more innovative, government a common rule of thumb?
  • Multi-level Governance – does the existence of more policymaking arenas produce more innovation and a greater demand for learning? Or, does the diffusion of power undermine the ability of a central government to adopt policies from others?
  • Punctuated equilibrium – is transfer a rare opportunity produced by the sudden and unpredictable attention to new ideas?

Further Reading:

I explore these issues (and Rose’s advice) in a paper examining what Japan can learn from the UK’s experience of regionalism. It includes a discussion (summarised from Keating et al – Paywall Green) of the extent to which policy converges in a devolved UK and how much of that we can attribute to transfer and/ or learning:

Keating et al 2012 summary from japan paper

6 Comments

Filed under 1000 words, agenda setting, Evidence Based Policymaking (EBPM), Japan, public policy, Scottish politics, UK politics and policy