Scientific generalisations: what’s so good about missing out all the differences?
- Lectures and events
- Publication Date
- Professor Nancy Cartwright
Joint Lecture with the Royal Institute of Philosophy
Professor Nancy Cartwright discusses how we come by general truths in science – not by generalising – and explores some of the pitfalls in getting back down to the concrete, especially when we want to use science to build a laser or a better social policy.
Scientific enquiry is about making generalisations, we are told; about making things look alike: a few short, sharp general claims to deal with a wealth of different cases. Yet things, as they appear, are not much the same. We get them to look that way by blurring the details, by distorting a bit – or more than a bit, by ignoring all the instances that don’t fit in the box. The trick is often to use highly abstract concepts in our general claims, so abstract they mean a million different things in a million different contexts. So using science to control the concrete is no straightforward matter: a matter indeed that seems to escape the scientific method.
Professor Cartwright’s entry into the RSE arena was not exactly “Daniel entering the Lion’s Den” but it was not far removed. The philosopher was not there to criticise the rigour of scientific studies in front of a group of eminent scientists – far from it – but merely, she pointed out, to highlight the importance of what is so often missed out when their results are broadcast to the world and proposed for use. That missing element was “context”, she argued, often inserted as a sentence at the very end of a paper, almost as an afterthought, despite its critical importance to what can be concluded from the results outlined in the study.
An example is widespread reports in both professional and popular press about a new method – male circumcision – that could “dramatically” reduce the spread of AIDS in sub-Saharan Africa. The reports appeared prominently in publications and broadcasts, ranging from the New York Times to the Scientific American to the BBC. Aid agencies were alerted to them by the World Health Organisation and the Cochrane Collaboration, a respected independent group which evaluates studies of medical interventions, which said three large African randomised controlled trials (RCTs) had provided “compelling evidence” that male circumcision works. Perhaps it does, said Professor Cartwright, yet what all these reports had in common was lack of context. What might be effective in one part of the world may not work in another because of local particularities.
That is why context is so important. Yet her collaborator in this work, Hakan Seckinelgin, had been struck at many scientific conferences on AIDS prevention by how many times, when new discoveries and research output are being eagerly digested, scant regard is given to a typical last sentence – “Of course context and adherence matter for getting the right outcomes.”
Despite its importance, that message is lost amongst the “euphoria” and promise of the findings and the rush to get these new and efficacious methods accepted into the global policy process.
Yet, Professor Cartwright insisted, context matters. Why? Because it is instrumental in the production of the results in the study setting. That is why, as the title of her lecture suggests, we cannot generalise from one – or even multiple – set of results. And if it is true that context matters, what does that mean for any new approach and its policy relevance and the drive to make it the immediate basis of a broad policy tool implemented across many countries and communities?
The dangers of ignoring context and drawing on generalisations are clear. In relation to male circumcision as a means of reducing AIDS, one commentator – talking to more than 2000 people at a conference – said that if we can use treatment as prevention then “we don’t even have to use condoms anymore.” But this was generalising the research output without considering its context. Returning to the earlier African example, there may be problems with trying to introduce male circumcision as a prevention method in all areas. In some regions there will be a resistance – perceived loss of manhood; fear of failed surgery; the requirement to continue using condoms after circumcision; interference with traditional circumcision practices; distrust of Western medical intervention – which could undermine the efficacy of the technique and so not bring about the hoped-for result.
So, context matters, and certainly not as a postscript. But, what is context, Professor Cartwright asked? Anthropologist Clifford Geertz describes it as “socially established structures of meaning in terms of which people do ….things.” Sociologist Tony Giddens describes structures
which “provide resources, opportunities and constraints for action.” Philosopher Pierre Bourdieu proffers the idea that “people’s reactions, choices and actions in relation to events in their lives are always part of a larger cultural and social memory that is part of them.” The lesson for policy makers and influencers is that a social intervention cannot be divorced from its social setting. Study results cannot, and should not, be generalised into a one-size-fits-all policy.
Generalisation can be described as “inductive inference” – collecting information and drawing conclusions from it. One example is Marie Curie’s induction on radium chloride and her use of the research of 19th-Century chemists to generalise – correctly – on its properties. She generalised on some properties – which turned out to be the right ones – and not others based on her knowledge of established principles about crystal structure. Generalisation without consulting contextual knowledge like this will get us into trouble, Professor Cartwright argued. Take Bertrand Russell’s chicken, which was fed by the farmer for 364 days in a row and which was expecting the same on the 365th. However, on the 365th it ended up on the dinner table!
It is no surprise, she argued, that we so very often go astray trying to generalise directly from study results, since usually the concepts used in designing studies and recording their results are too narrow, too local and too context-bound to fit into general principles. Principles that hold widely tend to use highly abstract concepts; while studies should be rooted in precise, measurable concepts. This is one reason why study results so often differ and do so even when, from a more abstract view, they illustrate the same general principle.
This can be observed in the work of Duflo and Bannerjee, and other scientists, in studies on the use of insect-treated mosquito bed nets in Africa. Lauded as “RCT gurus”, Duflo and Bannerjee’s examination of the low uptake of the cheap – by western standards – nets to prevent malaria come to the conclusion that poor people do not seem to be willing to sacrifice money for preventive methods. Indeed, many fishermen used the nets not for their intended purpose but to construct a fish trap. Other studies show a host of other affordable health and nutrition measures that the poor people who were offered them failed to take advantage of. What can be concluded from all these studies? Presumably nothing that is truly general about mosquito-net use or use of water cleansing tablets. In an attempt at something truly general, Duflo and Banergee climb up to a conclusion using a far different and more abstract vocabulary than any of the studies they cite – that poor people, like rich people, suffer from weakness of will.
This difference, between the abstract concepts that figure in general truths and the more concrete ones that we know how to measure, makes trouble not only for knowing how to generalise, but also for knowing what to do with a generalisation once we have it, Professor Cartwright added. Consider the sweeping generalisation of Duflo and Banergee. Suppose we accept it? What can it predict for us about uptake of male circumcision in specific villages in Kenya or about alcohol use among 30-year-olds in Oxford? Hardly anything without a great deal of knowledge about the local circumstances.
So, Professor Cartwright concluded, context matters not just in terms of what to learn from individual study results, but equally for the application of general principles in local conditions. For scientists, the task is to be able to climb down from general principles to work out what really would happen if mosquito nets were offered in specific areas, or if male circumcision was rolled out across Africa. For studies to be of real value, scientists and policy makers have to decide: ‘What general principles can a local result support?’ Equally, ‘What local predictions can a general principle support?’ Context matters for both.
What sort of induction – going from the particular to the general – was happening in the Marie Curie example? Isn’t it a kind of ‘abduction’, in which understanding context is a matter of insight not social science?
What justifies the Marie Curie generalisation is some material facts about crystal structure . I would deny that seeing this comes just from insight; you get this from hard scientific work. So too with learning the right social facts.
I agree 100% with context, but you have made a bit of a caricature of the “RCT gurus.” They are trying to come up with pragmatic results and they do prescribe treatments in context. They are trying to understand how the world works and the causal concepts. Aren’t you being unfair to them? After all, for understanding what happens in context we can do pragmatic trials.
A pragmatic trial is one done in some ‘realistic’ setting – maybe by busy clinicians not especially trained in the treatment as they are in a more pure experiment, or on patients who would have been excluded from the original RCT due to things like age or co-morbidities. If you do a pragmatic trial on a certain population at a certain time and then provide that same treatment in the same way to that population or one just like it, I agree that would be ok. But that is not what happens. Usually, what is concluded from positive results in a good pragmatic trial is “this treatment works in real-life settings.” But ‘real-life settings’ differ, and often in ways that matter. So we still need real understanding about the context of the study to know what we can learn from it. As is often repeated, RCTs work out what is happening, but not why it is happening. So, RCTs are not the way to find out what is really going on.
I am a medical researcher. When we do our trials, we produce papers with thousands of words but then try to summarise it on one sheet of paper. But a lot of the information taken from journals into policy is from that brief summary, without all the contextual information. As we have to generalise to some extent, what advice do you have for researchers who want that information to be as useful as possible but set in the right context?
You do not have to generalise your results. Your work is scientifically sound and rigorous. But you almost certainly don’t know all the other results relevant, especially since many will come from non-medical disciplines. What’s important is that organisations such as WHO take on board studies from researchers in different places, and not just medical ones, and then be heedful that they have little directly generalisable knowledge before issuing general guidance.
The principle of a treated mosquito net reducing infection is a principle. That there are problems in applying that principle is another matter surely?
The principle is correct, but that has to be pieced together with a lot of other facts and principles to make accurate predictions about health effects In putting that principle to use, we have to take account of a lot of information about the local context.
What advice would you give to policymakers in terms of being sensitive to context?
There is some simple advice, which is that organisations such as the WHO and World Bank should not roll out policy based on a handful of RCTs. Responsibility suggests we do more studies in each place targeted, and make sure we have weighed the costs and benefits using truly local information. We need to stop generalising from three studies and at least add in research on local cultures and values; and then see what all this research suggests taken altogether before taking policy forward.