Responsible Use of Research Metrics

Conventional research metrics such as journal impact factor, citation count and H-index are common ways to assess quality of research outputs. However, the use of these metrics as a standalone measure can result in misleading conclusions. 

Some critiques about the journal impact factors and citation count include:

• Journal impact factor is a journal level metric and it is not intended to evaluate an article or individual researcher.
• Quantitative metrics cannot be used to infer positive or negative comment. Citation count does not tell you how a work is cited.
• Retracted journal articles continue to receive citation counts.
• Citations take time to accumulate; older papers tend to be more highly cited than younger ones because they have more time to accumulate citations.

Responsible Use of Research Metrics

Metrics alone cannot give a full picture of research impact. Each metric has its strengths and weaknesses. Understanding the context and limitations of metrics is one way of responsible uses of metrics for research assessment. In fact, the importance of using metrics responsibly is widely recognized by funders, research institutions and publishers. Responsible metrics should be based on a number of principles that recognize the diverse qualities of research. In 2014, a group of experts declared “The Leiden Manifesto” which aimed to advise on the use of metrics in research evaluation. The following ten principles were suggested in the Manifesto for appropriate use of metrics:

1. Quantitative evaluation should support qualitative, expert assessment
2. Measure performance against the research missions of the institution, group, or researcher
3. Protect excellence in locally relevant research
4. Keep data collection and analytical processes open, transparent, and simple
5. Allow those evaluated to verify data and analysis
6. Account for variation by field in publication and citation practices
7. Base assessment of individual researchers on a qualitative judgement of their portfolio
8. Avoid misplaced concreteness and false precision
9. Recognize the systemic effects of assessment and indicators
10. Scrutinize indicators regularly and update them

 
Watch this video, which explains the Leiden Manifesto’s principles to guide research evaluation.

Responsible use of research metrics is all about how we will use quantitative and qualitative evidence appropriately and fairly in research evaluation to best support high quality research. Think about each evidence as a single piece of puzzle, you need to put them all together to get the complete picture of research performance. 

Additional Readings

If you are interested in learning more about journal impact factor, read these articles.

• The Journal Impact Factor: A Brief History, Critique, and Discussion of Adverse Effects / by Vincent Larivière & Cassidy R. Sugimoto
• What’s wrong with the journal impact factor in 5 graphs / by Dalmeet Singh Chawla
• The impact factor fallacy / by Frieder M. Paulus; Nicole Cruz & Sören Krach

– By Kevin Ho, Library

Hits: 288