A New Way to Measure Journal Quality

How do we measure quality of scholarship? When it is common to evaluate research and researchers by where papers are published, finding good ways to rate journals becomes crucial for the global research ecosystem.

For a long time, researchers have been using journal impact factor (JIF) to judge how “good” a journal is. Many institutions adopt reward systems and research assessment that are based on whether research papers are published in high impact journals. JIF is calculated by counting citations of all articles in a journal; it cannot tell us the quality of specific articles. The problem of JIF misuse and abuse has always been clear; however, it is not easy to find a simple metric to replace it.

Recently, the Center for Open Science (COS) launched a new measure for journals, called TOP Factor. Instead of looking at articles published in journals, it assesses journal policies for how they promote research transparency and reproducibility. In other words, TOP Factor evaluates the quality of publishing process, rather than the publishing products.

The TOP Scores

TOP Factor is based on the TOP Guidelines, an array of standards for transparency and reproducibility of research. For example, the standards on “data transparency” describes how journals require published articles to ensure data is documented and available. For each standards, a journal can score from 0 to 3, depending on how strong the journal policies follow the standards. Each journal then has a total score by adding up how it performs in all standards. For example, both Science and PLOS ONE have a total of 11:

A More Complex View of Journal Quality

JIF measures citation impact of the collective of articles published in a journal. It is so simple to use; people easily extend its meaning as an indicator of research quality. TOP Factor measures how robust a journal’s publishing process in supporting transparent and reproducible research. A journal with high TOP score does not imply all its papers are high quality, but we can trust that those articles follow best practices for research rigor.

It will take some time for TOP Factor to become more widely recognized and adopted as an alternative to JIF. Nevertheless, it is a development that researchers should keep watching. Read more about it in this COS news.

— By Gabi Wong, Library

Hits: 208