Skip to Main Content

Academic Publishing and Impact: Measuring Impact

Mae'r dudalen hon hefyd ar gael yn Gymraeg

An Introduction to Metrics

Measuring the impact of research has become increasingly important in helping to show the value of research to your institution and funding bodies, to identify areas of research strength and weakness, to identify top journals and identify emerging areas of research. Some metrics can be used to locate potential collaborators or competitors.

Metrics are used in rankings such as Times World University Rankings and QS rankings and by some panels in the REF so whatever your views on them it is worth finding out what is recorded about you. 

There are a wide range of developing and established tools to evaluate research which you can explore using the Metrics Toolkit

The most established form of metrics is 'Bibliometrics' which is the analysis of research literature based on citations, however subject coverage does vary – arts and humanities has traditionally not been included in citation indices for example. It is important to note that citation databases have biases in coverage especially towards research written in English, and from Western countries. 

When using Research Metrics, there are two golden rules;

- Always use both qualitative and quantitative input into your evaluations

- Use a combination of indicators and acknowledge their limitations to gain a more accurate picture. 

Traditional Metrics and Altmetrics are measure of attention or engagement, not necessarily quality. There are limited methods to discover whether a citation or tweet about a work is positive or negative, so use of either or both should be contextualized where possible.

Responsible metrics

The Library Research Support Team supports the recommendations of HEFCE's Metric Tide Report. This includes the following recommendations to ensure responsible use of metrics:

  • The research community should develop a more sophisticated and nuanced approach to the contribution and limitations of quantitative indicators.
  • At an institutional level HEI leaders should develop a clear statement of principles on their approach to research management and assessment, including the role of quantitative indicators.
  • Research managers and administrators should champion these principles and the use of responsible metrics within their institution.
  • HR managers and recruitment or promotion panels in HEIs should be explicit about the criteria used for academic appointment and promotion decisions.
  • Individual researchers should be mindful of the limitations of particular indicators in the way they present their own CVs and evaluate the work of colleagues.
  • The UK research system should take full advantage of ORCID as its preferred system of unique identifiers. ORCID IDs should be mandatory for all researchers in the next REF.

Swansea University has endorsed the San Francisco Declaration on Research Assessment (DORA). Guidance for implementation of DORA principles by the Wellcome Trust has also has been released; Implementation Guidance 

Data Sources

Web of Science / InCites - you will need a login 

Scopus / SciVal -  you will need a login

Google Scholar: - you will need to set up a profile.

These are the big three in terms of sources of citation data, Scopus and Web of Science are both used by various ranking organisations. Citation data from Scopus or Web of Science can also be used as supplementary material by select REF panels. Google Scholar is not used by these organisations due to the quality of the data. 

Web of Science is the discovery tool - you can use it to search for articles and other research outputs. To access Web of Science off-campus you will need to login using your university username and password. 

InCites is the analytics tool - where you can explore Journal Impact Factors, Citation Data, and other organisational data. To access InCites and Journal Citation Reports you need to register from a PC on Campus

Scopus is the discovery tool - you can use it to explore articles and other research outputs

SciVal is the Analytics tool - you can use it to explore a range of data including but not limited to citation data, field-weighted citation impact, and other organisational data

You do not need to register to use either of these services, but if you want to save reports, create alerts etc, it is a good idea to sign up using your university email address and a password. 

Google Scholar is created without much human intervention, and the data is less reliable than Scopus & Web of Science, and is therefore not used in rankings or assessment like the REF. However, for subjects which aren't covered by Scopus or Web of Science it can give some insight into who is citing you, and can pick up citations to books and other non-journal material.

You will need to create a Google Scholar Profile like the one in the image, and see it to track citations. There is also a piece of free software called 'Publish or Perish' which can help analyse these citations

Subject Specifics: Inspec Analytics

Inspec and Inspec Analytics from the Institution of Engineering and Technology, are highly specialised discovery and research intelligence tools that provide detailed insights into research trends and patterns across physics and engineering disciplines at both local and global levels.

The granularity of the Inspec index makes it easy to analyse research output by organisation and/or specific subjects, allowing you to monitor research output over time, compare output between organisations, stay up to date with emerging trends, find collaboration opportunities and identify the most relevant  journals and conferences to publish work. Inspec Analytics is accessible via Inspec then click on the 'Go to Inspec Analytics' URL

To watch a video on how to access Inspec Analytics please follow the link; Inspec Analytics Video

Sources indexed in Inspec: https://www.theiet.org/media/6168/inspec-active-journals.pdf  

More about Inspec’s subject classifications: https://www.theiet.org/publishing/inspec/inspec-content-coverage/inspec-classification/  

 

Altmetrics

 Altmetrics are metrics based on social media interactions and attention. They are more immediate than citations but also more open to "gaming". Providers of altmetrics suggest that they are best thought of as indicators of engagement and attention.

Reasons to consider altmetrics are to:

  • Discover who may be talking about your research online
  • Discover what is being said about similar research in your field (with a view to interesting them in your own research or evaluating its impact)
  • Compile evidence of research / impact either on a personal or a project level.

There are a number of commercial providers of Altmetrics but there is also a lot freely available:

Common Metrics