The politics of evidence and randomised control trials: the symbolic importance of family nurse partnerships

We await the results of the randomised control trial (RCT) on family nurse partnerships in England. While it looks like an innocuous review of an internationally well-respected programme, and will likely receive minimal media attention, I think it has high-stakes symbolic value in relation to the role of RCTs in British government.

EBM versus EBPM?

We know a lot about the use of evidence in politics – and we hear that politicians play fast and loose with it. We also know that some professions have a very clear idea about what counts as evidence, and that this view is not shared by politicians and policymakers. Somehow, ‘politics’ gets in the way of the good production and use of evidence.

A key example is the ideal of ‘Evidence Based Medicine’ (EBM), which is associated with a hierarchy of evidence in which the status of the RCT is only exceeded by the systematic review of RCTs – particularly when the results of this work are peer reviewed and published in high-status journals or databases.

This contrasts with evidence based policy making (EBPM) in which there are competing notions of evidence value, competing sources of evidence (expertise, policymaker experience, professional opinion, service user feedback, etc.), and a greater sense that policymakers will beg, borrow and steal whatever evidence they can get their hands on quickly to address the specific problem they face – including reports that are not peer reviewed or published in outlets with recognised scientific status.

Policymakers also have to weigh up evidence on policies that are difficult (if not impossible) to compare with each other, and come up with ways of choosing between them – such as by assessing their value for money in relation to the benefits they provide.

A compromise between evidence and politics?

In some cases there may be a decent compromise between these practices. In health, expert bodies such as NICE have become responsible for combining the kinds of evidence consistent with EBM with economic and other methods (often including professional and user feedback) to produce guidance on policy choices. NICE does not quite take the politics out of health and social care choices (and nor should it) but it often acts as a standard which prompts policymakers to accept its advice or explain why they don’t.

There have also been important efforts to encourage the greater use of RCTs more widely in government, such as by the Cabinet Office’s Behavioural Insights Team and academics such as Peter John and Gerry Stoker.

Major obstacles to the uptake of RCTs

Yet, these developments do not guarantee a central role for the RCT in politics more generally (far from it). Rather, many politicians or policymakers exhibit uncertainty or scepticism about:

1. The relevance of RCT evidence – they may argue that (a) an RCT does not answer their question fully or capture the complexity of a policy problem, and (b) that RCT evidence from somewhere else does not apply to their area.
2. Practical and ethical – an RCT could require cooperation across many levels and types of government, and randomisation is a ‘hard political sell’, at least to elected policymakers who (a) rely on an image of certainty when they propose policies (why would you need to test a policy’s value – are you trying something that might fail?), and (b) struggle with the idea of giving a good intervention to one group of people and not another (if the policy works, why don’t you give everyone the benefit?).

These political concerns may combine with academic criticisms about the assumptions behind RCTs (e.g. that you can produce what can meaningfully be called ‘control’ groups in complex social interactions) to produce major obstacles to the uptake of the evidence favoured by key groups of scientists.

The next best thing: importing policies based on RCTs

Perhaps the next best thing to conducting an RCT in the relative dark is to import a programme with an international reputation for well-evidenced and impressive results. That is where the family nurse partnership (FNP) comes in (box 1).

________________________________________________

BOX 1: The Family Nurse Partnership

The FNP began in the US as the Nurse-Family Partnership – designed to engage nurses with first time mothers (deemed to be at relatively high risk of poor life chances) approximately once per month from pregnancy until the child is two. The criteria for inclusion relate to age (teenage), income (low), and partnership status (generally unmarried). Nurses give advice on how mothers can look after their own health, care for their child, minimise the chances of further unplanned pregnancy, and access education or employment. It combines intervention to address the immediate problems of mothers and early intervention to influence the longer term impact on children.

The US’ Coalition for Evidence-Based Policy gave it ‘top tier’ status, which describes ‘Interventions shown in well-designed and implemented randomized controlled trials, preferably conducted in typical community settings, to produce sizable, sustained benefits to participants and/or society’. Identifying three US-based RCTs, it describes common outcomes in at least two, including reductions in pre-natal smoking, child abuse and neglect, and second pregnancies, and improvements in their child’s cognitive function and education attainment (in follow-ups when the children reached 15-19) at a low cost. These trials have been conducted since the first project began in 1977, producing at least 18 peer-reviewed articles, including by its pioneer Professor David Olds, in elite academic journals (such as Journal of the American Medical Association), and at least two which identify new results in non-US studies.

The programme was rolled out in England to 9000 mothers, with reference to its high cost effectiveness and ‘strong evidence base’, which would be enhanced by an RCT to evaluate its effect in a new country. The FNP requires ‘fidelity’ to the US programme (you can only access the progamme if you agree to the licensing conditions) based on evaluation results which showed that the programme was most effective when provided by nurses/ midwives and using a license ‘setting out core model elements covering clinical delivery, staff competencies and organisational standards to ensure it is delivered well’. Fidelity is a requirement because, ‘If evidence-based programmes are diluted or compromised when implemented, research shows that they are unlikely to replicate the benefits’.

________________________________________________

Adopting the FNP doesn’t solve the ‘not invented here’ problem, but it helps reduce many concerns: we import a successful policy (with success demonstrated in multiple RCTs) and conduct an RCT to make sure that a programme that works somewhere else works here. Not everyone gets the programme but, unlike in the US, they still receive ‘universal’ NHS care. The use of an RCT can also be sold politically, as (a) part of the license and (b) the kind of routine evidence gathering/ evaluation that should be present in all policy interventions anyway. This RCT/ programme also relates to a fairly contained group of recipients and healthcare professionals, with not as much need for ‘joined up government’ or ‘health and social care integration’ as in many other initiatives. It is praised by NICE and the Early Intervention Foundation.

In this context, the FNP is almost perfect

That’s what makes it seem so important symbolically. It’s like a trailblazer, showing all that is right with the use of multiple RCTs, to perform meaningful tests to demonstrate the effectiveness of a public policy. It is as much an advert for the value of the RCT as for the value of the programme.

The flip side to this coin is that, if the perfect programme doesn’t produce meaningfully better results than the NHS programme it replaced, some people may get the sense that we went to a lot of bother and expense for very little reward. The idea of a ‘gold standard’ of research may take on a different connotation, particularly during a period of austerity in which governments may be reluctant to invest in new policies and their evaluation when they have to reduce public provision.

Therefore, I expect the release of the RCT results to be political, at least in the sense that they won’t be released without some thought given to how to present the findings in as positive a way as possible. That’s perhaps not in the spirit of the ideal of EBM, but it seems consistent with the reality of EBPM.

Update

The first RCT results were published in October 2015 in the Lancet. A very short summary of these developments is as follows:

• After publishing the results of the RCT, Robling et al argue that ‘Programme continuation is not justified on the basis of available evidence, but could be reconsidered should supportive longer-term evidence emerge’
• David Olds’s reply is that the FNP could be more effective if directed more accurately to the most relevant target population
• The Local Government Association, which recently became responsible for public health (alongside social services), made a broad statement about using the opportunity to look ‘closely at how to achieve maximum effectiveness for the Family Nurse Partnership programme and whether it can be adapted to achieve better value’
• The Early Intervention Foundation commends the use of good evaluation and reinforces its view that there are many well-evidenced programmes from which to choose

Further Reading

This draft paper provides some further reading on the trainspotter’s guide to evidence/ policy, while this link takes you to a draft Palgrave Pivot book with a bibliography on EBPM.

See also: Ruth Kennedy What works. Can we know?