Derrick – Is peer review fit to evaluate a new breed of research excellence?
In this article, Gemma Derrick, author of The Evaluators’ Eye, discusses the suitability of peer review, from its purpose to its process and outcomes.
All academics have felt the unmistakeable pang of disappointment that comes from being rejected from a grant, promotion or award. What should sting more is the knowledge that this rejection came from not one, but of a whole group of their own academic peers. Instead, academics accept the decision not because they think it is correct, but because they believe in the integrity of the peer review system. This is despite the peer review system suffering many attacks on its reliability and ability to make unbiased decisions. Indeed, in the face of these serious drawbacks, blind faith in the peer review system runs deep.
Despite its place as the gold standard of research evaluation, surprisingly little is known about how peer review works. Most research focuses on judging the fitness of peer review solely based on how ex-post assessments match other external quantitative measures of research excellence. To do so, however, is naïve and overlooks the fact that how peer review works is more a social than scientific process.
Understanding peer review this way, changes the way that its strengths are perceived. It also changes the way it is seen as an appropriate tool for judging notions of excellence that exist beyond academic. By studying peer review as a social process, rather than the product of its outcomes alone its future as the gold-standard in evaluation is less about patching its failures and more about understanding how panels can work better, smarter and towards more equitable outcomes.
Peer review, for all wants and purposes, is an excellent tool for assessing traditional notions of academic excellence. In these situations, the peers that govern the process are not just experts in a specific topic area, but also were subjected to the same process of academic socialisation as the applicant. This results in both the applicant and evaluator sharing a common understanding of research excellence; a meeting of academic minds that guides how both the application is constructed as well as how it is evaluated.
Currently, the understanding of what constitutes research excellence is expanding to consider activities and achievements beyond the realms of academia. This has come hand in hand with governments in the UK, US, Australia, New Zealand and across Europe calling for greater transparency and accountability for the public funding of research. For research, it means that excellence is no longer constrained to the traditional expectations of writing high quality, high impact journal articles and books; collaborate internationally; have continued success in obtaining competitive grants; and producing knowledge for knowledge sake. Now, there is an expectation that researchers be rewarded and held accountable to how their research influences broader society.
There is nothing new to this view, in fact researchers have implicitly been using their research to influence society for decades. What is controversial is the idea that researchers be rewarded formally for these activities and that, using the peer review system to legitimise this new mode of research production, that such activities are incentivised. However, the question remains as to whether a process that built its reputation on assessing traditional modes of research productivity is fit for the purpose of evaluating these new, and ambiguous research criteria.
Indeed, for assessing notions of excellence that extend beyond academia, the very foundation of expertise and peer is blurred, and without certainty in this foundation, there is no longer a shared understanding to drive the process, nor validate the evaluation outcomes. To counteract this, some panels include non-academic evaluators who represent a variety of societal stakeholder groups. However, there are still problems with the extent these non-academics can function as part of an academic-dominated panel culture, and contribute to deliberations and validate the decision to use peer review to evaluate non-academic criteria.
Instead, to assess its suitability for evaluating Impact and other emerging and ambiguous criteria, peer review must be viewed not as a filtering mechanism, but as a human exercise that requires airing multiple ideas, and reaching a consensus through negotiation and capitulation on an individual level. This seems simple enough, but studying this process with the view to optimise it is another story entirely. Peer review is a black box, and there are strong political and organisations reasons for keeping it as such. This means that it is overwhelmingly difficult to “get inside” to be able to assess the assessors, and to explore in real time how decisions about notions of value are made by groups of experts. As such, old methodological tools are no longer suitable. What is needed is a level of creativity in our methodological approaches that uses thinking based on diplomacy, methodological innovation and interdisciplinary design. After all, if we judge the suitability of peer review based solely on the outcomes it produces, then we are only seeing part of the problem.