Developing an ‘Impact and Engagement for Australia’ (IEA) metric

ATSE has floated their idea for a research engagement and impact evaluation based purely on metrics. The metrics suggested are category 3 income and commercialization income. ATSE suggest that this evaluation be run along with the ERA to produce a combined quality and impact rating (quality 5-1 and impact A-D).

It is important for Australia to consider an impact evaluation - and examples such as the Excellence in Innovation for Australia (EIA) have shown that it is possible to evaluate research impact in Australia. The UK have also evaluated impact as part of their national research evaluation exercise. Some argue that the EIA and the REF methodology of case studies and peer review are onerous and expensive. However, this is by no means a reason not to do them - these sorts of evaluations will never be easy because the evaluation of research is not easy. Proper evaluation of research requires time and the people with the right level of expertise to carry out the evaluation.

Here are some of the issues I think the ATSE ERA-linked evaluation might have:

• It assumes that your research input, output and impact all occur in the same FoR code.
• It assumes that impact has occurred concurrently with the research – most impact is not realised that quickly.
• It won’t take into account the difference between pure research and applied research in the same FoR.
• It is focused only on economic impact – which for the commercialisation income might actually be more a measure of the success of the company selling the widget than the quality of the university.
• It also does not take into consideration the amount of gaming that occurs in the ERA.

The ATSE release can be read here: http://www.atse.org.au/atse/content/activity/innovation-content/developing-impact-engagement-australia-metric.aspx

Forum to discuss evaluation of research at Australian universities www.researchimpact.com.au

I have just launched a new forum called Research Impact for those who would like to discuss the evaluation of university research activity, quality and impact.

Feel free to drop in on the forum if you are interested in discussing topics like the Excellence in Research for Australia (ERA); the evaluation of research impact or the 'impact dividend' or maybe you are interested in discussing the growing number of article retractions in the sciences.

www.researchimpact.com.au

Public Consultation on the Draft ERA 2015 Submission Documents

The Australian Research Council (ARC) is currently seeking public feedback on the Draft ERA 2015 submission documents.

One of several changes being suggested in the new guidelines is the addition of output volume information to the published ratings (Outcome Reporting - see section 4.2.3). This means that for evaluated fields of research the ARC will not only publicly report the ERA rating (5-1) but also the volume of outputs aligned to that rating. In the past rounds of ERA the evaluation has highlighted 'excellence wherever it occurs' meaning that a small university could score an ERA rating '5' in the same field of research as a larger university and these were seen as equivalent. This suggested change to the reporting of outcomes will mean that there will be a grading or stratification within ratings - for example an ERA rating of '5' with only 50 outputs will no longer be seen as equivalent to an ERA rating '5' in the same field with 500 outputs; or, in fact, may also be seen as inferior to an ERA rating '4' with 500 outputs.

Another change noted in the consultation documents (but not specifically highlighted so you have to dig to find it) is the slight change to the 'reassignment exception'. The new "warning" in the business rules document states that "The reassignment exception must not be used to more more than 40% of the apportioned journal articles into a single UoE". This may be an attempt to put a stop to the 'gaming' that some people think is occurring in the sector. This new rule will now mean that in a single field of research at least 60% of the journal articles must be assigned a field of research as stated on the ARC's journal list

The public consultation can be accessed by the following link:

Public Consultation on the Draft ERA 2015 Submission Documents