In this age of accountability no one questions the idea that
data relating to university research publications is collected and reported on.
Research publications are no longer only a mechanism for disseminating research
findings but they are now also a measure of research performance.
It is hardly surprising then that discussions arising from a recent
review by PhillipsKPA are not around whether we should collect research
publication data but how we can collect it more efficiently. Australian universities
currently report research publications data through the Higher Education
Research Data Collection (HERDC) and Excellence in Research for Australia
(ERA). One of the 27 recommendations from the PhilipsKPA Review of University Reporting Requirements is to streamline the
collection of research data into a single collection. Combining the two collections into a
single collection will only be worthwhile if it improves the efficiency, integrity,
transparency and utility of the data being collected. Any consultation document
will hopefully clarify for the sector how the combined collection will achieve
this.
While both mechanisms currently collect research publication
and research income data that is really where the similarities stop. The type
of data collected, the level of detail collected and importantly the purpose of
the collections are quite different.
The purpose of the HERDC is to collect research income and publications
data to inform the distribution of research block grants to universities based
on their relative performance in each measure. The HERDC only reports on
publication volume and does not consider the field of research or the quality
of the research – at best it provides a proxy for the volume of research
activity across Australian universities.
According to the ARC’s ERA documentation the objectives of
ERA are much broader than the HERDC and are listed as, to:
·
establish an evaluation framework that gives
government, industry, business and the wider community assurance of the
excellence of research conducted in Australian higher education institutions;
·
provide a national stocktake of discipline level
areas of research strength and areas where there is opportunity for development
in Australian higher education institutions;
·
identify excellence across the full spectrum of
research performance;
·
identify emerging research areas and
opportunities for further development; and
·
allow for comparisons of research in Australia, nationally
and internationally, for all discipline areas.
If we focus on the collection of research publications data
in each collection there is one main difference. In the HERDC, publications
data is collected for all publications that acknowledge the university with
which the author is affiliated on the publication itself – for example through
an author ‘by-line’ – regardless of whether the author is currently employed at
the university or not - I will refer to this as an ‘address’ based collection. In
the ERA collection publications data is collected for all publications authored
by researchers employed by a university at a census date (usually 31 March of
the year preceding the ERA collection) – regardless of whether the university
is acknowledged within the publication or not - I refer to this as a ‘census’
based collection. The difference between an ‘address’ based collection and a ‘census’
based collection may not at first seem significant – but it is.
Consider the case of the HERDC – data are collected on publications
only where the university has been acknowledged on the publication – regardless
of whether the researcher or research group still works at the university. Once
the data have been collected and reported to the Department of Education the
numbers are used to distribute block grant funding (approximately $1700 per
publication point). This is effectively rewarding universities for the volume
of publications they can report that list the university in the byline. There
is no consideration made of the quality or the focus of the research in the
publications – just the volume. If researchers who produced the publications
have since left the university the university is still credited with their
research activity.
A big advantage of the address based collection is that it
is easy to determine which outputs are eligible for collection and which university
they belong to. An address based collection could be conducted by a third party
(for example through a citation data provider like Scopus or Web of Science).
The disadvantages are that the collection is retrospective in that the
researchers may have left the university but their output still contributes to
the block grant allocation. This is fine if you are rewarding past performance
but problematic if you are trying to profile current research strengths of
Australian universities or if you are trying to fund for future research
success.
Now consider the ERA which also collects publications data. These
publications are the ones produced by the current cohort of staff at the
university and not just the ones with a university listed byline. When the
publications are reported they are assigned to fields of research and subsequently
given a quality score by a national evaluation panel. This allows the
Department (and the public) to see where research excellence exists in
Australian universities and where research strengths may be emerging. The main
disadvantage of a census based collection is that it requires more administrative
work to collect the data as publications are not readily identifiable by a
university byline within the publication. A census based collection cannot
easily be performed by a third party. Advantages of the census based collection
are that it represents the current research profile of the university and encourages
universities to strategically recruit researchers to contribute to the research
profile of the university.
In the case of a new “combined collection” for research publications
data it is not immediately clear whether it would be done based on the ‘address’
or the ‘census’ date. Each has its pros and cons and each is useful for a different
purpose. However, I would argue that the ‘census’ based approach is more appropriate
in this case because of the following reasons:
1.
It allows universities to demonstrate a current
research profile based on researchers who are actually working at the
university rather than a retrospective profile where researchers may well have
left the university since the evaluation
2.
It allows universities to respond strategically to
changes in the research landscape and funding environment for example by
recruiting researchers to complement or strengthen their existing research
profile
3.
It trusts universities with the responsibility of
presenting their research in a meaningful way based on their own knowledge of
their researchers and their research rather than leaving it up to a third party
or a generic business process relying on accurately recorded address data
4.
It aligns with the national uptake of a
universal researcher ID in Australia such as the ORCID
I would conversely argue that using an ‘address’ based collection for a research evaluation may result in what I call ‘phantom’ units of evaluations. A publication can have multiple authors and therefore multiple universities listed in the byline – each author may also have multiple bylines. This results in each author of each publication potentially contributing to multiple university research evaluations. A ‘phantom’ evaluation would be where a university may appear to have a minimum volume of publication output (for ERA this is 50 publications to trigger an evaluation) based only on the fact that the university’s name appears at least once on 50 publications. However, while the byline appears on the publication the author may not actually work at the university – for example, if the author has since left the university or where the author has multiple bylines which include other university affiliations in addition to the one they work at. The ‘address’ based collection would also potentially disadvantage universities who have strategically invested in recruiting new researchers to their university. In this case while the new university is paying the salary of the researchers, those researchers’ publications would be contributing to another university’s research evaluation based on their previous bylines.
