Developing an ‘Impact and Engagement for Australia’ (IEA) metric

ATSE has floated their idea for a research engagement and impact evaluation based purely on metrics. The metrics suggested are category 3 income and commercialization income. ATSE suggest that this evaluation be run along with the ERA to produce a combined quality and impact rating (quality 5-1 and impact A-D).

It is important for Australia to consider an impact evaluation - and examples such as the Excellence in Innovation for Australia (EIA) have shown that it is possible to evaluate research impact in Australia. The UK have also evaluated impact as part of their national research evaluation exercise. Some argue that the EIA and the REF methodology of case studies and peer review are onerous and expensive. However, this is by no means a reason not to do them - these sorts of evaluations will never be easy because the evaluation of research is not easy. Proper evaluation of research requires time and the people with the right level of expertise to carry out the evaluation.

Here are some of the issues I think the ATSE ERA-linked evaluation might have:

• It assumes that your research input, output and impact all occur in the same FoR code.
• It assumes that impact has occurred concurrently with the research – most impact is not realised that quickly.
• It won’t take into account the difference between pure research and applied research in the same FoR.
• It is focused only on economic impact – which for the commercialisation income might actually be more a measure of the success of the company selling the widget than the quality of the university.
• It also does not take into consideration the amount of gaming that occurs in the ERA.

The ATSE release can be read here: http://www.atse.org.au/atse/content/activity/innovation-content/developing-impact-engagement-australia-metric.aspx

Journal Ranking

If you would like to view some of the popular and publicly available journal 'ranking' lists head on over to the researchimpact.com.au forum to check out:

• Eigenfactor.org
• ERA Journal Ranking
• Harzing - journal quality list
• ABDC - journal quality list

Make a comment on these lists or perhaps add some of the ones that you know of or use.

Why was it better to be a researcher in Design Practice and Management than Forestry for ERA 2010?

www.researchimpact.com.au

DECRA not considered a highly regarded form of accolade by the ARC's ERA 2015

The third round of the ARC's Excellence in Research for Australia (ERA 2015) is underway; and, unfortunately, if you have a Discovery Early Career Researcher Award (DECRA) it is not being considered an 'esteem' measure. However, you will be considered 'esteemed' if you hold a NHMRC Early Career Fellowship.

The ERA includes a number of measures of 'esteem' and they are defined as measures that:

constitute recognition of the quality ... and indicate that a researcher is held in particularly high regard by peers in their field of research and/or by other well qualified parties. ...embody a measure of prestige and are recognised by experts within the discipline as a highly desired, highly regarded form of accolade or acknowledgement.

The ERA requests institutions submit data on only five esteem measures (listed below) of which one is 'nationally competitive research fellowships'. Unfortunately it looks like the DECRAs did not make it onto the list this time so are not considered a measure of esteem by the ERA.

The esteem measures eligible for ERA 2015 are the following:

• editor of a prestigious work of reference;
• fellowship of a learned academy and membership of the Australian Institute of Aboriginal and Torres Strait Islander Studies (AIATSIS);
• recipient of a nationally competitive research fellowship; 
• membership of a statutory committee; and 
• recipient of an Australia Council grant or Australia Council fellowship.

And, only nationally competitive research fellowships in the following programs are eligible (where are the DECRAs?):

• ARC Discovery—Australian Laureate Fellowships; 
• ARC Discovery—Federation Fellowships; 
• ARC Discovery—Future Fellowships; 
• ARC Discovery—Indigenous Researchers’ Development; 
• ARC Discovery—Projects (including Australian Professorial Fellowships, Queen Elizabeth II Fellowships, and Australian Postdoctoral Fellowships); 
• ARC Linkage—International; 
• ARC Linkage—Projects (including Australian Postdoctoral (Industry) Fellowships); 
• NHMRC Practitioner Fellowships (Formerly Practitioner Fellowships Scheme); 
• NHMRC Research Fellowships; 
• NHMRC Australia Fellowship (Formerly Australia Fellowship Scheme);
• NHMRC Career Development Fellowships (Formerly Career Development Awards); 
• NHMRC Early Career Fellowships (Formerly Postdoctoral Training Fellowships); 
• NHMRC Sir MacFarlane Burnett Fellowship;
• NHMRC John Cade Fellowship in Mental Health Research; and 
• NHMRC Translating Research Into Practice (TRIP) Fellowships.

To check out the ERA 2015 and related documents yourself you can visit: http://arc.gov.au/era/era_2015/2015_keydocs.htm

Excellence in Research for Australia - ERA 2015

The Australian Research Council (ARC) this week released the submission documents for the Excellence in Research for Australia (ERA) 2015. The 2015 guidelines are very similar to the 2012 guidelines with only a few significant changes. 

The first change is that there is now a requirement to submit more data not directly related to the evaluation. For example institutions are now required to submit data on staff gender; which publications are available in open access; and, information on the time and cost associated with submitting the ERA data for the institution. The problem with requesting these sorts of data is that each new requirement costs an institution time and money to collect and report. This means an increased cost to the whole sector for information that is not actually going to be used for the evaluation process - it is just for information.

A second change is that institutions are now able to write to the ARC regarding staff on less than 0.4 FTE contracts and argue a case for their inclusion (and the inclusion of their publications). This is a good move as it reduces the negative impact on fractional staff - a cohort that includes more female academics than male - thus shifting towards a more equitable process (although still not ideal). I wrote about why I thought there was a gender equity issue here:

http://michaelcrisp.blogspot.com.au/2014/02/does-era-2015-present-gender-equity.html

Institutions will be required to submit their ERA 2015 data in early 2015.

http://arc.gov.au/era/era_2015/era_2015.htm

Does ERA 2015 present a gender equity issue for the higher education sector?

Academic staff employed on fractional-time contracts may be disadvantaged in the ERA by the requirement for employees on less than 0.4 FTE to have published with their new university's byline within the already passed ERA reference period.

A recent article in The Australian highlights the 'churning' of academic staff that is occurring in the sector to coincide with the ERA census date (31 March). Researchers who move (poached) to a new institution by the census date have their entire publication track record counted towards their new employer's ERA submission. This is despite the fact that the work was likely supported by their previous employer. The fact that researchers take their performance with them for the ERA results in a lot of 'horse trading' (or churning) between institutions  around the census date as they try to maximize their ERA outcomes. This means that high performing researchers, or often now groups of researchers, become a valuable commodity and can no doubt leverage this newly discovered value to their advantage through negotiations with potential employers.

However, as is often the case with these sorts of initiatives, the ERA guidelines are not quite as simple as this. There is actually a guideline that requires the FTE of the newly employed researcher to be greater than 0.4 FTE. If the contract is greater than 0.4 FTE (e.g. 1.0 FTE) then the researcher and all of their publication activity is automatically eligible for the ERA submission. BUT, if the contract is for less than 0.4 FTE there is an extra requirement that the researcher must show a "publication association" with their new institution, for example with a by-line that states the new institution as their affiliation. Now as the ERA is a retrospective evaluation it is highly unlikely that a new researcher employed close to the census date would have had time publish something listing the new university as their affiliation. With a less than 0.4 FTE contract and no affiliation byline with their new employer suddenly none of their performance counts for the ERA. In ERA terms this then makes full time researchers much more valuable than fractional researchers - and, therefore universities will likely employ full time researchers close to the ERA census date rather than look for fractional appointments.

So why is this inequitable? The process makes full-time staff more attractive to a university around the census date and fractional staff less attractive. So while full time researchers can personally and professionally benefit from the head-hunting, horse-trading and churning that goes on - the fractional researchers may not. The national higher education staff data collection shows that female academics make up a higher proportion of the fractional appointments in the sector than males (for research-only academics the ratio is almost 2:1). It is likely then that not only will fractional researchers suffer because of this but that female academics will be impacted in greater numbers than males.