Harry Potter and the Draft Engagement and Impact Guidelines

This week the Australian Research Council released for consultation their draft guidelines for the evaluation of university research engagement and impact. The engagement part of the evaluation is mainly quantitative with a shortlist of indicators around research income from industry and end-users. The impact part of the evaluation is mainly qualitative with research impact case studies providing a narrative around the benefit that university research is having outside of the university sector - including the ways that universities are fostering translation and impact from their research.

Some interesting takeaways from the draft guidelines include:

• A May/June 2018 submission deadline (which follows directly behind the ERA 2018 deadline)
• A maximum of 25 impact case studies per university which includes 23 disciplinary case studies, 1 interdisciplinary case study and 1 Aboriginal research case study
• The introduction of a low volume threshold of 150 weighted outputs (books weighted x5) over which a university must submit information and below which a university may opt-in if they so wish
• A new three point rating scale for impact (high, medium, low) which seems more sensible than the pilot ratings (mature, emerging, limited)
• Impact case studies will now receive 2 ratings each - one for the approach to impact and another for the impact itself

Adding to the sector's resource burden in complying with research evaluation is the introduction of two engagement narratives: one is an engagement indicator explanatory statement of 4,500 characters to accompany engagement indicators and the other is a 7,000 character engagement narrative to accompany each unit of assessment. Now seeing as each unit of assessment is the 2-digit field of research this results in a considerable increase in work for the sector. In ERA 2015 there was a total of 656 2-digit FORs evaluated - so if each one of these is accompanied by a 4,500 character explanatory statement and a 7,000 character engagement narrative this equates to around 7.5 million characters, or around 1.2 million words - for comparison, the entire series of Harry Potter books contain around 1.08 million words.

You can see the guidelines for yourself at the ARC website here.

Could ERA be Automated in the Near Future?

Could ERA submissions be auto-generated in the near future? The new ERA specifications released by the ARC hint perhaps yes.

Australia's national research evaluation exercise, Excellence in Research for Australia (ERA) is conducted roughly once every three years with a large investment of time and money from the university sector and the ARC. The cost of running ERA to the sector has been variously estimated to be between $30 million and $100 million.

Universities are required to submit information and data relating to their research activities over the preceding six years. This includes publications, research projects and grants, research staff, along with a raft of related indicators such as patents and commercialisation activity.

Much of the information universities submit as part of the exercise is available from other sources - either publicly available (e.g. grant outcomes from the ARC and NHMRC, HERDC income returns, ABS R&D expenditure surveys) or from third party suppliers (e.g. Scopus or Clarivate publications databases).

If we were able to link researchers, their publications and grant funding to universities and fields of research then an ERA submission could in theory be developed automatically without the time and expense incurred by universities.

The Australian Research Council (ARC) website now includes the ERA 2018 Technical Specifications and Submission Guidelines. Of note is the optional inclusion of information like unique author identifiers (ORCID) and unique article identifiers (DOI). A combination of ORCIDs, DOIs, citation data and fields of research (e.g. from the ERA Journal List) could in theory be used to auto-generate ERA submissions for  universities. Not only could this be less expensive for the sector but also offers the benefit of a more contemporary data set compared with the retrospective ERA as it currently stands.

So perhaps we will see an auto-generated ERA in 2021...

You can view the ERA guidelines for yourself at the ARC's website.