Using metrics responsibly
An overview of how to use metrics responsibly, good practice and the limitations of metrics.
"Some of the most precious qualities of academic culture resist simple quantification, and individual indicators can struggle to do justice to the richness and plurality of our research"
Responsible Metrics
Limitations of metrics
Metrics can be a useful tool to help track the attention received by research outputs. Citations and online attention are relatively easy to record and measure and provide a reasonably quick and simple way to compare research.
However, Metrics on their own are not sufficient to assess research fairly. Research can impact the world in any number of ways, many of which are difficult to measure or quantify, and metrics are only part of the picture.
A controversial or fraudulent paper might receive a high number of negative citations. The impact of research isn’t always reflected in the number of citations it receives (for example, Albert Einstein has a relatively low h-index).
Metrics also reflect unconscious or implicit bias within the scholarly community. You should therefore exercise caution when using metrics.
Good practice when using metrics
- Use the right tool for the job. What question are you trying to answer? Is the metric you are using appropriate? What aspect of research performance do you want to explore? Why? Can this be measured, and if so how? Find out what each metric can tell you, and what it can't. If you’re using a metric as a proxy for something that is not directly measurable, as a minimum you should be explicit about this in your analyses.
- Always use quantitative metric-based input alongside qualitative opinion-based input. Like all statistics, metrics can be misleading without context. Metrics can be a useful tool, but they are no replacement for expert opinion.
- Get the big picture. Each metrics tool takes its data from different sources and calculates its metrics in different ways. Ensure that the quantitative, metrics part of your assessment always relies on at least two metrics to reduce bias. Using only a single measure may also encourage people to change their behaviour to game that particular measure.
Good practices
The Metric Tide review, commissioned by the Higher Education Funding Council for England (HEFCE) to examine the role of metrics in research assessment and management, identified five dimensions of responsible metrics. We've blogged about some actions you can take under these dimensions to stay on the right track.