ANZSRC Review, or what are our new FOR codes?

The ARC, Australian Bureau of Statistics (ABS), Statistics New Zealand (Stats NZ), and the New Zealand Ministry of Business, Innovation and Employment (MBIE) are undertaking a joint review of the Australian and New Zealand Standard Research Classification (ANZSRC). What is the ANZSRC I hear you say - well it is the field of research codes, or FOR codes. It is also the socio-economic objectives (SEO) and type of activity (pure research, strategic research applied research etc).

It is a good time to refresh the categorization to bring it into line with current and future research activity. It is particularity pleasing to see the inclusion of a specific question around how Aboriginal research is categorized (currently it is all hidden at the '6-digit' FOR level).

The other interesting question is around interdisciplinary research and how the classification could be set up to support this better. If research is multidisciplinary then it is probably a simple matter of tagging it with more than one FOR code. If it is truly inter- or trans-disciplinary activity and of some scale then perhaps it should just have its own FOR code (maybe under a division of 'Interdisciplinary').

I sometimes think of research disciplines as fruits - a field of research might be like an apple, and another is a pear and another is a banana. When research is multidisciplinary it is like we have chopped up the fruit and tossed it together in a bowl to make a fruit salad - works well together but still separate fruits. Interdisciplinary research might be more like a fruit smoothy - we've taken all of the fruits we need but blended them together and have created something new and different. So, ANSZRC helps us classify the fruits and the fruit salad - but how will it classify the smoothy?

You can have a look at the review document yourself at the ARC website: https://www.arc.gov.au/anzsrc-review

Harry Potter and the Draft Engagement and Impact Guidelines

This week the Australian Research Council released for consultation their draft guidelines for the evaluation of university research engagement and impact. The engagement part of the evaluation is mainly quantitative with a shortlist of indicators around research income from industry and end-users. The impact part of the evaluation is mainly qualitative with research impact case studies providing a narrative around the benefit that university research is having outside of the university sector - including the ways that universities are fostering translation and impact from their research.

Some interesting takeaways from the draft guidelines include:

• A May/June 2018 submission deadline (which follows directly behind the ERA 2018 deadline)
• A maximum of 25 impact case studies per university which includes 23 disciplinary case studies, 1 interdisciplinary case study and 1 Aboriginal research case study
• The introduction of a low volume threshold of 150 weighted outputs (books weighted x5) over which a university must submit information and below which a university may opt-in if they so wish
• A new three point rating scale for impact (high, medium, low) which seems more sensible than the pilot ratings (mature, emerging, limited)
• Impact case studies will now receive 2 ratings each - one for the approach to impact and another for the impact itself

Adding to the sector's resource burden in complying with research evaluation is the introduction of two engagement narratives: one is an engagement indicator explanatory statement of 4,500 characters to accompany engagement indicators and the other is a 7,000 character engagement narrative to accompany each unit of assessment. Now seeing as each unit of assessment is the 2-digit field of research this results in a considerable increase in work for the sector. In ERA 2015 there was a total of 656 2-digit FORs evaluated - so if each one of these is accompanied by a 4,500 character explanatory statement and a 7,000 character engagement narrative this equates to around 7.5 million characters, or around 1.2 million words - for comparison, the entire series of Harry Potter books contain around 1.08 million words.

You can see the guidelines for yourself at the ARC website here.

Engagement and Impact Assessment Pilot 2017 Report

The ARC have today released their final report on the Engagement and Impact Assessment Pilot conducted in early 2017.

Changes suggested for the full assessment in 2018 include:

Field of Research 11 - Medical and Health Sciences will be split in two which means two case studies can be submitted bringing the total maximum to 25 (1 for each of the 22 FOR + an extra one for '11' plus an interdisciplinary and Aboriginal research case study)

Four engagement indicators will be used for the engagement part of the assessment:

• cash support from end-users
• total HERDC income per FTE (specified schemes)
• end-user sponsored grants: proportion of HERDC Category 1
• research commercialisation income (selected FoR codes only).

It is encouraging to finally see a definition for Aboriginal and Torres Strait Islander research:

Aboriginal and Torres Strait Islander research means that the research significantly relates to Aboriginal and Torres Strait Islander peoples, nations, communities, place, culture or knowledge.

However, the ARC will continue to consult with its Aboriginal and Torres Strait Islander stakeholders to further refine the definition.

You can read the report at the ARC's website here: http://www.arc.gov.au/ei-pilot-overview

Could ERA be Automated in the Near Future?

Could ERA submissions be auto-generated in the near future? The new ERA specifications released by the ARC hint perhaps yes.

Australia's national research evaluation exercise, Excellence in Research for Australia (ERA) is conducted roughly once every three years with a large investment of time and money from the university sector and the ARC. The cost of running ERA to the sector has been variously estimated to be between $30 million and $100 million.

Universities are required to submit information and data relating to their research activities over the preceding six years. This includes publications, research projects and grants, research staff, along with a raft of related indicators such as patents and commercialisation activity.

Much of the information universities submit as part of the exercise is available from other sources - either publicly available (e.g. grant outcomes from the ARC and NHMRC, HERDC income returns, ABS R&D expenditure surveys) or from third party suppliers (e.g. Scopus or Clarivate publications databases).

If we were able to link researchers, their publications and grant funding to universities and fields of research then an ERA submission could in theory be developed automatically without the time and expense incurred by universities.

The Australian Research Council (ARC) website now includes the ERA 2018 Technical Specifications and Submission Guidelines. Of note is the optional inclusion of information like unique author identifiers (ORCID) and unique article identifiers (DOI). A combination of ORCIDs, DOIs, citation data and fields of research (e.g. from the ERA Journal List) could in theory be used to auto-generate ERA submissions for  universities. Not only could this be less expensive for the sector but also offers the benefit of a more contemporary data set compared with the retrospective ERA as it currently stands.

So perhaps we will see an auto-generated ERA in 2021...

You can view the ERA guidelines for yourself at the ARC's website.

The ARC has released the draft ERA 2018 Submission Guidelines

The ARC has now released the draft ERA 2018 submission guidelines for consultation. You can find a copy at their website here.

There are not really too many changes to the submission which should please universities - especially as they are gearing up for the Impact and Engagement assessment at the same time. Guidelines for the impact and engagement assessment are still pending at this stage.

A couple of interesting additions include:

• Reporting of ORCID (optional)
• Reporting of DOI (optional)
• Aboriginal and Torres Strait Islander research section - although it would be nice to have a clear definition of Aboriginal research to work with. 

Also interesting to see the addition of this to the the guidelines:

Institutions agree to allow the ARC to publish any submitted data from ERA 2018. In addition, institutions must agree to publish their submission, with the exception of their staff data, on 5 February 2019.

It will be good to get some clarity on what form this would take.

Clarivate selected as citation provider for ERA 2018

Clarivate selected as citation provider for ERA 2018

Chief Executive Officer (CEO) of the Australian Research Council (ARC), Professor Sue Thomas, has today announced that the ARC has selected Clarivate Analytics to provide citation information for the 2018 round of Excellence in Research for Australia.

http://www.arc.gov.au/news-media/media-releases/clarivate-selected-citation-provider-era-2018

Web of Science used by Australian Research Council for Analysis of Benefits from University Research

According to this announcement - the Australian Research Council (ARC) will use Web of Science data as part of the next ERA and Engagement and Impact Evaluation - see release below.

http://thomsonreuters.com/en/press-releases/2016/may/web-of-science-source-data.html


Media Release:

The Australian Research Council (ARC) has recently obtained Thomson Reuters Web of Science™ Core Collection as one of the data sources to contribute to analyses that will be utilized by the ARC to support development of national impact and engagement assessment to assess the benefits derived from university research. This national assessment exercise is being introduced as part of the Australian government’s National Innovation and Science Agenda.  This was announced today by the Intellectual Property & Science business of Thomson Reuters.

In 2016 the ARC will work with the higher education research sector, industry and other end-users of research to develop quantitative and qualitative measures of impact and engagement of university research. The Web of Science Core Collection provides source data for records such as topic, title and author information which will be used by ARC to support work around sector and ERA analysis in order to derive a model for national assessment. The ARC will conduct a national assessment as a companion exercise to the Excellence in Research for Australia (ERA), the country’s national research evaluation framework which identifies and promotes excellence across the full spectrum of research activity in Australia’s higher education institutions.

Jeroen Prinsen, vice president and head of Australia & New Zealand, IP &Science, Thomson Reuters said, “As a strong advocate of research collaboration and partner of Australia’s research community, we are pleased to support this important national impact and engagement assessment of university research which will ultimately promote high-quality research that will drive Australia’s innovation and economic growth. We are honored that the ARC will utilize source data from the Web of Science Core Collection, the world’s most trusted source of citation databases.”