Summary of discussion, ‘Concepts, principles and goals’

There is general support for the principle of policy impact: researchers and Higher Education Institutions (HEIs) should be encouraged to achieve policy impact from their research. However, there is also widespread acknowledgement that in their current form, systems for promoting and incentivising impact do not work to achieve the desired outcomes: they do not accurately reflect the processes through which impact occurs; nor do they offer the right framework for evaluating the effects that research can have. Indeed, the past 10-15 years might be considered as an experiment – and now is a good moment to reflect on and review how we might do better in supporting policy impact.

Christina Boswell opened the workshop outlining the main conceptual flaws with the current approach to impact, as exemplified in REF Impact Case Studies (see Christina Boswell and Kat Smith). Such approaches are premised on simplistic linear models that assume discreet pieces of research are taken up and generate adjustments to policy. Absent in this conceptualisation is the ‘enlightenment’ role of knowledge developed notably in the work of Carol Weiss: instances where research generates conceptual adjustments to how policy is framed and incremental shifts, often based on a body of knowledge that has built up over time, rather than discreet pieces of research.

Also absent from such models are different ways in which we think about how politics and policy shape research and research use (what questions get asked, how researchers craft projects and disseminate findings). Impact may be ascribed where a particular piece of research is deployed to substantiate what politicians and policymakers would have wanted to do. More worryingly, researchers may pre-emptively shape their research based on what they think will be taken up. These issues point to quite serious flaws in the impact model and the kind of behaviours and incentives that are being encouraged. Rather than rewarding the type of collaboration or synthesis that is more likely to generate an enlightenment effect, it rewards individualist approaches, tactical choices of topics, and luck.

Secondly, as Claire Dunlop argues in her blog, there are also serious concerns about the gap between models of impact, and how impact takes place in practice. What is meant by the term impact is subjective, and what is being done in practice is often very different from what is being presented as impact, in particular in the REF impact case studies. The boundaries between knowledge engagement, dissemination and exchange are blurred in reality. Impact is narrow and simplistic and excludes the plurality of actors involved. It also brackets out broader normative questions about the ethical and societal implications of (work to achieve) impact. As learning documents, they are difficult to use because they obscure processes of social learning and transformation.

Third, co-production was discussed. The term is widely used to describe a range of different practices, generating some confusion. From a theoretical perspective grounded in the science and technology studies, the standpoint is ontological and based on the observation of the interaction between researchers/scientists and policymakers in the knowledge production process. From this perspective, there is the methodological challenge in identifying who has produced the impact.

By contrast, co-production can also be understood as a methodology, as described in the work of Annette Boaz. Here, the aim is to conduct research that is more useful, usable and ultimately increases the use of knowledge. It includes new approaches to knowledge production which are about spaces where different partners are working together. The thinking behind this approach is that by being involved in the process of knowledge production, partners are more likely to help shape research that addresses policy issues, and also to understand the knowledge – making them more likely to apply it in practice. However, in reality, the culture in academia inhibits this form of knowledge production because of a concern for academic knowledge output. In practice, models of co-production in universities are not well set up to produce knowledge in that way. There are also challenges in attributing impact. There was also a question raised about the desirability of developing co-produced knowledge production processes at the expense of methodological and epistemological plurality throughout the process.

Fourth, the discussion turned to the absence of a link between public engagement and policy impact. Kat Smith suggested that in democracies we should be thinking about the connections between research policy impact and public engagement. The relationship between research and policy is complex and indirect and can contribute to cumulative shifts over time in how policymakers think about particular issues. But those shifts are inherently social in nature. They are unlikely to happen in an environment where policymakers and researchers talk to each entirely separate from publics, indeed they are grounded in broader public debates. More strategically, policymakers’ perceptions of the viability of particular courses of action are informed by their sense of public acceptability. Assumptions about what appears to be publicly viable or not act as a barrier to how research influences policy – and those assumptions are often questionable and not always empirically based. Moreover, public engagement is often a route to policy impact or industry impact in the longer term.

However, in the current approach to impact, policy impact risks being reified. Public engagement is not adequately valued or rewarded in REF impact terms, and is often seen by HEIs as a risky route to achieving impact. Funders also rarely make connections between policy impact and public engagement. It is also striking that scholarship on public engagement and policy impact are largely siloed, although in practice, policy actors are interested in both: they want policy to be informed by research and evidence, but also want to capture lived experience.

Fifth, over the course of the workshop, a number of points of concern were raised about models of impact codified in the REF. First, the collective dimension inherent to research is largely eliminated and there is an overemphasis on the influence of the single academic/group on policy. Secondly, they do not capture the longue duree and incremental research, the cumulative dimension of impact. In very practical terms, the story telling that derives from pre-populated fields and writing by committee compounds these problems. As a result, impact rewards serendipity, promotes individualism and has a bias towards positive outcomes, all of which do not represent – or may even contradict – what researchers actually do in practice (incremental/normal research), how research is done (collaborative) or research findings (also ‘bad news’).

On serendipity, there was agreement that it is a large factor in whether research leads to impact. As a result, serendipity and luck play too strong a role in the selection of research for impact case studies. Recognising the role of serendipity on research impact on policy is leading to an emerging literature around ‘structured serendipity’. Furthermore, funding bodies are working on creating the conditions for serendipity: their focus is on building translational infrastructure and partnerships in what is viewed as a ‘complex impact ecosystem in the UK’ in which REF is only one part of the story.

Participants also discussed what was seen as a positive bias of the current approach: an assumption that all impact was necessarily good. This points to the absence of a normative framework to think about what kind of impact is being encouraged. Furthermore, empirically, it does not reflect the complexity of the decision-making environment that impact seeks to influence nor does it engage with how ‘bad news’ is received or (not) rewarded. There is risk that researchers align their research agendas according to what they anticipate may generate output that is likely to be selected for impact cases. This could lead to skewed research, further challenging the enlightenment function of knowledge and even generating the opposite outcome – ‘grimpact’.

On the problem of individualism, participants discussed whether this resulted from funder requirements, or how HEIs (over-)interpreted such requirements. However, HEIs and funders can also aggregate individual impact examples as a means of demonstrating the value of research, and this can be an important resource to bolster support for public funding of research, and especially social sciences and humanities. More generally, there appears to be a disconnect between what kinds of impact funders want to promote and incentivise, and the often narrow and conservative way in which HEIs interpret funder guidance.

There is a strong consensus on the need to do more to understand the motivations of funders, not only of policymakers. The views and voices of the funders and funding bodies are largely absent in academic writing and thinking around impact. Impact should be understood in the framework of a triad between researchers, policymakers and funders (the view from funders will feature prominently in workshop 5).

Another aspect that needs to be taken into account is the broader higher education sector and the political economy of this sector. At a macro level, there seems to be a form of institutionalisation and instrumentalisation of concepts such as impact and a growing auditing culture in universities. The sector has witnessed the development of a prestige economy in which academics are being encouraged to act as policy entrepreneurs, in an effort to boost university reputation and funding – which can come into tension with the notion of research and academia as a public good. At the micro level, relationships and good will risk being eroded even as impact may be empowering people with networks and connections. Furthermore, the reality of having to invest time into building the networks when this is not accounted for in workloads may further exacerbate inequalities (Workshops 2 and 5 will grapple with some of these issues).