What’s wrong with policy impact?

Rethinking Policy Impact
Publication Date

The ‘impact agenda’ in higher education first gained currency around a decade ago, when ‘impact case studies’ were rolled out as part of the Research Excellence Framework (REF). At the time, one of my academic roles was to support colleagues in carrying out what we then called ‘Knowledge Transfer’ or ‘Knowledge Exchange’. And to be honest, a lot of that activity felt quite formulaic and box-ticking: organising an event to ‘disseminate’ work to policy-makers, or writing a ‘policy brief’ that would probably not see wide readership.

A woman smiling for the camera
Professor Christina Boswell FRSE is Professor of Politics at the University of Edinburgh.

Compared to this dissemination-based model of knowledge exchange, the focus on ‘impact’ was in many ways refreshing. It forced us to reflect on what kinds of changes we wanted our research to produce, and in many ways raised our game in terms of how we went about achieving this. But as a scholar of research-policy relations, like many other social scientists I also had reservations about the model. The REF guidance in particular (and UKRI ‘pathways to impact’ to some extent) appeared to adopt a rather simplistic model of how research might influence policy, which overlooked decades of rich literature on the sociology and politics of ‘knowledge utilisation’ in policy.

First, the focus on how particular pieces of ‘underpinning research’ produced identifiable change implied a rather neat, linear model. But as the literature has shown, where research does have a significant impact on policy it is more likely to be in an incremental way over a long period of time, involving a body of research rather than individual findings, and often through conceptual shifts in how issues are framed. Given this, the focus on individual outputs or projects achieving discernible impact over a relatively short period of time seemed unfeasible. More seriously, it is likely to discourage researchers from framing impact as a collaborative effort, across a wider body of research, over a longer timeframe.

Second, the focus on how research influences policy overlooks the ways in which policy and politics shape knowledge. By introducing incentives or requirements to produce impact, funders are encouraging researchers to frame their research questions and methods in a way that is likely to resonate with policy-makers. This may be desirable in many ways – most of us accept that research should yield societal benefit. But it is likely to have a fundamental effect on the research landscape, which deserves serious reflection. At worst, it means that researchers may be attempting (consciously or not) to produce research that aligns with or pre-empts expected policy changes, thereby gaining credit for contributing to such change.

Third, and linked to this, is the problem of serendipity. The dynamics of policy-making are complicated and unpredictable, punctuated by events such as changes in political leadership and exogenous events. Research may have had a significant impact on the beliefs and intentions of policy-makers, only to be scuppered by an unexpected crisis that shifts the policy agenda. Alternatively, a change in political leadership may usher in a sudden change of direction, thus fast-tracking a particular researcher to impact. And there is an equality dimension to this, in that those with better connections – perhaps because they are seen as more authoritative or have a more recognised biography – may get better access to influence than earlier career researchers or those with protected characteristics.

Fourth, such models fail to reward more complex forms of co-production. Changes in policy are likely to involve ongoing dialogue across research and policy, which produce (often unacknowledged) forms of learning over time. Paradoxically, when such exchanges go well, policy-makers may not even be aware of how their ideas are being influenced by research. And even if they are aware, they may be reluctant to attribute such shifts to the influence of researchers. As I wrote in another blog, policy-makers are far more likely to want to draw attention to how they have drawn on research where they are using it in symbolic ways, as ammunition to back-up pre-existing preferences.

This blog is part of the research project Rethinking Policy Impact

A UK-wide conversation on the principles, goals and approaches that should guide the policy impact agenda in higher education.

Find out more about the Rethinking Policy Impact project

These inadvertent effects were certainly not what funders had in mind in rolling out guidance on impact. And indeed, REF2021 guidance recognised some of these effects, and sought to adjust guidance to accommodate a more flexible approach. Many would argue that at least part of the problem lies with the way higher education institutions (HEIs) interpret impact requirements. There is a tendency for HEIs to ‘gold plate’ guidance, for example exercising caution in selecting case studies for the REF, or overstating the significance of particular pieces of research, given the high stakes in terms of funding and reputation. While we can exhort universities to support and promote more diverse forms of impact, they are unlikely to take risks in interpreting guidance. The guidance has to provide clear and unambiguous assurance that such approaches will be equally recognised and rewarded.

These are just some of the challenges we identified in our first Workshop, where our focus was very much on critiquing current approaches, and especially REF Impact. In the next workshops we will start to hone in on particular dimensions: how far such models capture diverse forms of knowledge (Workshop 2); how well they align with the needs and preferences of policy-makers (Workshop 3); and exploring lessons from other national and international research systems (Workshop 4). By Workshop 5, we hope to have come up with some more positive suggestions and practical proposals about how we can do this better.

Professor Christina Boswell is Professor of Politics at the University of Edinburgh, where she currently serves as Dean of Research for the College of Arts, Humanities and Social Sciences. Since 2021 she has been Vice President (Public Policy) of the British Academy and Fellow of the Royal Society of Edinburgh.

The RSE’s blog series offers personal views on a variety of issues. These views are not those of the RSE and are intended to offer different perspectives on a range of current issues.