Scientific generalisations: what’s so good about missing out all the differences?

Professor Cartwright’s entry into the RSE arena was not exactly “Daniel entering the Lion’s Den” but it was not far removed. The philosopher was not there to criticise the rigour of scientific studies in front of a group of eminent scientists – far from it – but merely, she pointed out, to highlight the importance of what is so often missed out when their results are broadcast to the world and proposed for use. That missing element was “context”, she argued, often inserted as a sentence at the very end of a paper, almost as an afterthought, despite its critical importance to what can be concluded from the results outlined in the study.

An example is widespread reports in both professional and popular press about a new method – male circumcision – that could “dramatically” reduce the spread of AIDS in sub-Saharan Africa. The reports appeared prominently in publications and broadcasts, ranging from the New York Times to the Scientific American to the BBC. Aid agencies were alerted to them by the World Health Organisation and the Cochrane Collaboration, a respected independent group which evaluates studies of medical interventions, which said three large African randomised controlled trials (RCTs) had provided “compelling evidence” that male circumcision works. Perhaps it does, said Professor Cartwright, yet what all these reports had in common was lack of context. What might be effective in one part of the world may not work in another because of local particularities.

That is why context is so important. Yet her collaborator in this work, Hakan Seckinelgin, had been struck at many scientific conferences on AIDS prevention by how many times, when new discoveries and research output are being eagerly digested, scant regard is given to a typical last sentence – “Of course context and adherence matter for getting the right outcomes.”

Despite its importance, that message is lost amongst the “euphoria” and promise of the findings and the rush to get these new and efficacious methods accepted into the global policy process.

Yet, Professor Cartwright insisted, context matters. Why? Because it is instrumental in the production of the results in the study setting. That is why, as the title of her lecture suggests, we cannot generalise from one – or even multiple – set of results. And if it is true that context matters, what does that mean for any new approach and its policy relevance and the drive to make it the immediate basis of a broad policy tool implemented across many countries and communities?

The dangers of ignoring context and drawing on generalisations are clear. In relation to male circumcision as a means of reducing AIDS, one commentator – talking to more than 2000 people at a conference – said that if we can use treatment as prevention then “we don’t even have to use condoms anymore.” But this was generalising the research output without considering its context. Returning to the earlier African example, there may be problems with trying to introduce male circumcision as a prevention method in all areas. In some regions there will be a resistance – perceived loss of manhood; fear of failed surgery; the requirement to continue using condoms after circumcision; interference with traditional circumcision practices; distrust of Western medical intervention – which could undermine the efficacy of the technique and so not bring about the hoped-for result.

So, context matters, and certainly not as a postscript. But, what is context, Professor Cartwright asked? Anthropologist Clifford Geertz describes it as “socially established structures of meaning in terms of which people do ….things.” Sociologist Tony Giddens describes structures

which “provide resources, opportunities and constraints for action.” Philosopher Pierre Bourdieu proffers the idea that “people’s reactions, choices and actions in relation to events in their lives are always part of a larger cultural and social memory that is part of them.” The lesson for policy makers and influencers is that a social intervention cannot be divorced from its social setting. Study results cannot, and should not, be generalised into a one-size-fits-all policy.

Generalisation can be described as “inductive inference” – collecting information and drawing conclusions from it. One example is Marie Curie’s induction on radium chloride and her use of the research of 19^th-Century chemists to generalise – correctly – on its properties. She generalised on some properties – which turned out to be the right ones – and not others based on her knowledge of established principles about crystal structure. Generalisation without consulting contextual knowledge like this will get us into trouble, Professor Cartwright argued. Take Bertrand Russell’s chicken, which was fed by the farmer for 364 days in a row and which was expecting the same on the 365^th. However, on the 365^th it ended up on the dinner table!

It is no surprise, she argued, that we so very often go astray trying to generalise directly from study results, since usually the concepts used in designing studies and recording their results are too narrow, too local and too context-bound to fit into general principles. Principles that hold widely tend to use highly abstract concepts; while studies should be rooted in precise, measurable concepts. This is one reason why study results so often differ and do so even when, from a more abstract view, they illustrate the same general principle.

This can be observed in the work of Duflo and Bannerjee, and other scientists, in studies on the use of insect-treated mosquito bed nets in Africa. Lauded as “RCT gurus”, Duflo and Bannerjee’s examination of the low uptake of the cheap – by western standards – nets to prevent malaria come to the conclusion that poor people do not seem to be willing to sacrifice money for preventive methods. Indeed, many fishermen used the nets not for their intended purpose but to construct a fish trap. Other studies show a host of other affordable health and nutrition measures that the poor people who were offered them failed to take advantage of. What can be concluded from all these studies? Presumably nothing that is truly general about mosquito-net use or use of water cleansing tablets. In an attempt at something truly general, Duflo and Banergee climb up to a conclusion using a far different and more abstract vocabulary than any of the studies they cite – that poor people, like rich people, suffer from weakness of will.

This difference, between the abstract concepts that figure in general truths and the more concrete ones that we know how to measure, makes trouble not only for knowing how to generalise, but also for knowing what to do with a generalisation once we have it, Professor Cartwright added. Consider the sweeping generalisation of Duflo and Banergee. Suppose we accept it? What can it predict for us about uptake of male circumcision in specific villages in Kenya or about alcohol use among 30-year-olds in Oxford? Hardly anything without a great deal of knowledge about the local circumstances.

So, Professor Cartwright concluded, context matters not just in terms of what to learn from individual study results, but equally for the application of general principles in local conditions. For scientists, the task is to be able to climb down from general principles to work out what really would happen if mosquito nets were offered in specific areas, or if male circumcision was rolled out across Africa. For studies to be of real value, scientists and policy makers have to decide: ‘What general principles can a local result support?’ Equally, ‘What local predictions can a general principle support?’ Context matters for both.