Beyond the Dashboard: Surviving and Thriving in Higher Ed Analytics in the Age of AI

Remember when building a comprehensive KPI dashboard or cobbling school level metrics for a strategic review took weeks of data cleaning, late night pivot tables, and endless stakeholder alignment?

Those days are gone.

With the acceleration of AI and machine learning, anyone across the university can now generate their own descriptive reports, query data pipelines in natural language, and generate visual dashboards in a matter of seconds. For data and planning professionals working within higher education, this is a massive existential shift. If the core value of our teams historically lay in production (ie gathering, formatting, and displaying “what happened”) we are facing a new reality; production has been commoditised.

When the technical act of building a report or tracking a metric is instantaneous and accessible to all, where do data analytics teams position themselves for the mid-term? How do we move past the current wave of AI capability instead of getting swept away by it?

Here is how we need to pivot our careers and our offices to stay indispensable.

Shift from "Dashboard Producers" to "Decision Architects"

When insights are instantly accessible via AI, the bottleneck is no longer accessing data; it is deciding what matters and what to do about it.

AI excels at aggregation and pattern matching, but it completely lacks institutional memory, political nuance, and cultural context. It doesn't know why a specific college operates the way it does, or how a particular department shapes its risk tolerance.

Instead of waiting at the end of the line to visualise outcomes, we need to embed ourselves at the inception of policy and strategy. Our value shifts from building the report to framing the right questions, contextualising the output, and translating raw AI generated numbers into a coherent institutional narrative.

Move from Descriptive to Prescriptive Analytics

Institutional reporting has traditionally been backward looking answering what happened at the university, college, or school level. AI has solved backward looking analytics.

The mid-term opportunity lies in higher order thinking:

• Causal inference and intervention design: Moving beyond predicting an outcome (like applications to offers to acceptances, student attrition or research outputs) to testing and measuring what specific institutional interventions actually fix it. Don’t just admire the problem, test solutions that might work.
• Algorithmic auditing and data governance: As universities increasingly adopt automated systems for everything from resource allocation to student support, someone needs to watch the watchers. Analytics teams are uniquely positioned to act as internal watchdogs, auditing models for bias, drift, and regulatory compliance.

Become Curators of "Ground Truth" and Context

AI is only as good as the context it feeds on. In a complex university ecosystem, public models don’t inherently understand your internal data definitions, nuance, or localised KPIs.

Instead of building static dashboards, our role evolves into maintaining the proprietary context, semantic layers, and secure data pipelines that feed institutional AI tools. We become the master curators of the university's "ground truth."

Senior leaders are now drowning in fast, AI generated metrics, they need trusted guides. Position your team not simply as report builders, but as internal consultants who coach leaders on critical data thinking and protect them from misinterpreting automated noise (hallucinations!).

Embrace Ambiguity and Messy Transformation

AI thrives on structured, historical data. It stumbles on deep ambiguity, novel structural changes, and complex institutional reorganisations.

Higher education is perpetually reshaping itself, whether through shifting government funding models, multi-campus integrations, or massive restructures and mergers. These are messy, qualitative, and deeply political spaces where AI can offer a data point, but it cannot negotiate a consensus or navigate human resistance. That is where human judgment becomes irreplaceable.

The Bottom Line

The commoditisation of dashboarding is not the end for institutional analytics, it is a sign that the days of low value work should be over. By shedding the identity of "report builders" and stepping into the role of strategic sense-makers, decision architects, and governance leads, we can ensure our teams are not just surviving the AI wave, but steering it.

 

Rethinking University KPIs: What Happens When We Stop Treating Students as Sausages?

Universities love their KPIs. Applications, offers, acceptances, enrolments, they form the typical pipeline charts shown at every Council meeting. These numbers matter because, in a revenue-driven system, students are often framed (sometimes quite literally) as inputs to a financial model. I’ve even heard the phrase “sausage factory” used to describe a university suggesting that students simply enter at one end and degrees come out the other.

But is this the only way to think about the purpose of a university? How might alternate conceptions of a university’s purpose encourage us to develop more meaningful indicators than just “how full is the pipeline”?

Let’s revisit some alternative ‘ideas of a university’ and use those as lenses to reimagine what we measure.

The University as a Public Good: Beyond Revenue Pipelines

Much of our current KPI thinking comes from a “new public management” mindset where universities are quasi-businesses. Applications predict enrolments, which predict revenue, which predicts sustainability. There is nothing inherently wrong with this view; universities do require money to function.

However, universities historically exist to produce, protect and disseminate forms of knowledge that markets can’t do alone. Once we see the university as a public good, the question shifts:

• Are we attracting the right mix of students for the society we want to build?

This leads to very different indicators:

• The proportion of students entering fields of social need
• Participation of under-represented groups
• The alignment between university programs and societal needs

The pipeline is still there but it becomes a pipeline for public value, not just revenue.

The Ecological View: Students as Co-Contributors, Not Inputs

The idea of the ecological university provides another lens. In this model, the university is part of multiple ecosystems: knowledge, culture, politics, the environment, and the student lifeworld.

Students are not raw material. They are agents participating in an ecology of learning.

If we adopt this lens, enrolment KPIs broaden dramatically:

• How well are we enriching the learning ecosystem?
• How well are we sustaining student capabilities in the long term?

Measures of student wellbeing, belonging, intellectual growth, and capability development become core indicators and not optional extras.

The Democratic University: Students as Citizens-in-Formation

Universities are essential in educating democratic citizens, people who can think critically, debate respectfully, and understand the world beyond their immediate experience.

From this view of a university, student volume is less important than student development.

• What kind of thinkers and citizens are our graduates becoming?

This leads toward indicators like:

• Growth in critical reasoning
• Graduate contributions to communities
• Student exposure to diverse ideas
• Equity of access and equity of outcomes

These metrics might be harder to quantify, but they reflect a deeper purpose than simply “headcount”.

The Entrepreneurial University: Diversifying What We Value

The entrepreneurial university perspective views universities not just as centers for education and research, but as active agents that drive innovation and economic development by collaborating with government and industry

In this world, student numbers are not only about quantity but strategic composition.

• Where do our students enable innovation?
• Which cohorts create new research possibilities, industry relationships, or cultural vitality?

A university focused only on volume misses its own entrepreneurial potential.

So, What Does All This Mean for Student Number KPIs?

If we adopt some of these alternative views of what the purpose of a university is then the traditional student pipeline becomes only one dimension of performance.

A more forward-looking set of indicators might ask:

• Are we cultivating an inclusive and diverse learning community?
• Are our students developing capabilities that matter for society, democracy and culture?
• Are we creating an ecosystem in which students, staff and external partners co-create knowledge and value?
• Are our graduates contributing to the wellbeing of their communities?
• Are we strengthening the intellectual, civic and ecological fabric of our region?

None of these replace enrolment numbers but they do contextualise them. The purpose of a university is not just to convert applicants into revenue, but to sustain a complex ecosystem of knowledge, people and ideas.

Why This Matters Now

At a time when many universities are dealing with mergers, restructures, market volatility and demographic shifts, it’s tempting to remain focussed on the simplest KPIs.

But what if you had a chance to articulate a fresh purpose.

When purpose changes, measurement must change too.

And we must stop treating students as sausages!

The Illusion of Data-Driven Decision Making in Universities

“Data-driven decision making” has become a ubiquitous catchphrase in higher education. Planning and business intelligence (BI) teams often pride themselves on delivering ever-expanding datasets, dashboards, and performance metrics in service of strategic clarity and institutional improvement. Yet despite this flood of data, there’s a persistent disconnect between the volume of information available and the quality or impact of decisions being made.

In practice, universities rarely exhibit the kind of disciplined, data-led decision making they claim to champion. Most planning offices distribute large volumes of raw or lightly formatted data to a wide internal audience, few of whom are in positions of actual decision-making authority. These users may explore the data, form views, or use it in local contexts but they often lack the strategic mandate to act on the insights in any meaningful way. Meanwhile, those with the authority to make high-impact decisions, senior executives and academic leaders, tend to rely far more on heuristics, experience, and intuition than on comprehensive data analysis.

This is not a failure of character or competence. It is a well-documented feature of executive decision-making. Nobel laureate Herbert Simon’s concept of bounded rationality describes how decision makers, faced with limited time, incomplete information, and cognitive overload, opt for “satisficing”, a strategy of choosing a solution that is good enough, rather than optimal. Gerd Gigerenzer’s research further reinforces that in complex or uncertain environments, heuristics, simple, experience-based rules, are not just common but often effective. In fact, they are often superior to overly data-intensive approaches that falter in the face of ambiguity.

In the university context, empirical studies support this reality. Research by Kahneman and Lovallo, and later by Kahneman, Sibony and Lovallo, reveals that senior managers frequently fall back on pattern recognition and intuitive judgement, especially in high-stakes or uncertain decisions. Data, when used, typically plays a secondary role: it may validate a gut decision, lend legitimacy to a choice already made, or gently nudge a leader in a particular direction. Rarely does it drive the decision outright.

This presents a challenge for BI and planning professionals. The current model of data democratisation through widespread reporting assumes that more access equals better decisions. But if most data recipients are not decision makers, and if senior leaders prefer heuristics anyway, this approach may lead to diffusion without impact. Worse, it risks data fatigue and confusion, as users interpret numbers in inconsistent or unproductive ways.

A more effective model is not more data, but smarter data. This means synthesising and analysing complex datasets to produce timely, targeted insights, delivered directly to those with the authority to act. It requires planning professionals to move beyond reporting and become trusted strategic advisers, translating information into relevance, and aligning it with institutional priorities and decision cycles.

In short, the promise of data-driven decision making will not be realised by expanding access alone. It requires understanding how decisions are actually made and designing our data practices to meet decision makers where they are, not where we wish they were.

A Global Top 100 Debut: Why Adelaide University’s QS Ranking Is a Milestone Worth Celebrating

Adelaide University, the soon-to-be-launched institution born from the merger of the University of South Australia and the University of Adelaide, has made its debut on the global stage with a QS World University Ranking of 82. This result is nothing short of remarkable and represents a strong vindication of the university’s ambitious goal to be ranked among the world’s top 100 institutions.

When the merger was first proposed, many critics dismissed the idea, suggesting that combining the two institutions would weaken rather than strengthen South Australia’s university sector. Detractors argued that UniSA would dilute the quality of the more prestigious University of Adelaide. Even now, with the ink barely dry on the merger legislation, some commentators continue to talk down the achievement. A recent AFR piece labelled the result “lacklustre”, a surprising take given that fewer than 100 of the world's approximately 26,000 universities ever break into the top 100.

Others question how the new university could be ranked at all before it officially opens its doors on January 1, 2026. The answer is simple: global ranking agencies evaluate institutional identity, not bricks and mortar. They assess research output, reputation, academic strength, and global engagement, all of which are already active, measurable, and very real in the merged institution.

Rather than nit-pick or diminish the achievement, this moment deserves recognition. A debut at 82 places Adelaide University in elite company, and sends a clear message: the merger hasn’t weakened the institutions; it has elevated them.

The goal was to create a world-class university for South Australia. That goal is already being realised. Let’s celebrate that.

Want Growth? Build a University

The article titled "The Economic Impact of Universities: Evidence from Across the Globe" by Anna Valero and John Van Reenen, published in the Economics of Education Review, explores the relationship between the presence of universities and regional economic growth. Utilising a comprehensive dataset encompassing nearly 15,000 universities across approximately 1,500 regions in 78 countries, the study examines data from 1950 to 2010 to assess how the number of universities influences GDP per capita.

Key Findings:

• Positive Correlation with Economic Growth: The study finds that an increase in the number of universities within a region is positively associated with higher future GDP per capita. Specifically, a 10% increase in universities per capita correlates with a 0.4% rise in future GDP per capita.
• Spillover Effects: The economic benefits of universities extend beyond their immediate regions, positively impacting neighbouring areas within the same country.
• Mechanisms of Impact: The presence of universities contributes to economic growth not merely through direct expenditures but also by enhancing human capital and fostering innovation.
• Influence on Democratic Attitudes: Regions with a historical presence of universities tend to exhibit stronger pro-democratic attitudes, suggesting a broader societal impact.

This research underscores the multifaceted role of universities in promoting economic development and societal progress, highlighting their significance beyond education.

https://doi.org/10.1016/j.econedurev.2018.09.001

Beyond the Numbers: Reclaiming Academic Purpose from Performative Pressures

 

In today’s data-driven universities, research performance is often equated with metrics: citations, publications, grant income. But are we losing sight of what truly matters in academia?

Recent reflections from Elsevier’s Research MetricsGuidebook and a compelling paper by Visser et al. (Journal of Education Policy, 2024) point to a growing concern: the rise of performativity. That is, the pressure for academics to continuously prove their value through measurable outputs, often at the expense of deeper scholarly and educational contributions.

This performative culture distorts academic behaviour. Researchers may prioritise what is countable over what is meaningful. Critical activities such as teaching, mentoring, peer review, and community engagement can be undervalued simply because they are less visible

However,  metrics can still play a constructive role if used responsibly. The Elsevier guide promotes two simple but powerful rules: use more than one metric, and always pair metrics with expert judgment. This triangulation helps avoid simplistic rankings and ensures context-sensitive assessment.

A responsible metrics culture reframes the use of metrics but doesn't reject measurement. It acknowledges disciplinary diversity, career stages, and the rich variety of academic contributions. It supports, rather than distorts, academic integrity.

To shift the culture, institutions must lead: redesign evaluation processes, train staff in interpreting metrics critically, and celebrate contributions that metrics alone can’t capture.

Metrics should serve academic purpose; not replace it.

 

What a University Council Really Needs to Know About Research: Clarity Beyond the Metrics

The Strategic Role of Research

University research is a source of both pride and complexity. It underpins our global rankings, attracts funding, enables industry partnerships, and drives innovation and societal impact. Yet for many members of university Council, research can feel like a "black box"; full of acronyms, shifting benchmarks, and dense performance tables.

While governance bodies are not expected to be immersed in operational detail, they do need to understand the high-level performance, risks, and opportunities within research to fulfil their strategic oversight role. This short piece offers a clear view of what matters most.

The Problem: Too Many Numbers, Not Enough Insight

Research reporting to Council is often technical and fragmented. Data might include ERA results, HERDC income, grant success rates, citation metrics, rankings data, and individual initiatives, but rarely a coherent picture.

Without synthesis or trend context, it becomes hard to tell: Are we improving? Where are we strong? What should we be concerned about?

What Council Really Needs to Know

To support good governance and strategic stewardship, Council needs clear, contextual answers to five key questions:

Is our research activity growing or shrinking?

Look at trends in external research income, research-active FTE, and publication volume. Growth indicates momentum; flatlining may indicate capacity or competitiveness risks.

Is our research quality competitive?

Use field-weighted citation impact (FWCI) or citations per paper compared to sector benchmarks. Context is key, where do we stand among peers or international standards?

Are we building capability for the future?

Consider the proportion of early-career researchers, pipeline of grant applications, or internal schemes for research development. Long-term health depends on today’s investments.

How aligned is our research to strategy?

Are we publishing and attracting grants in strategic priority areas? Are we working with industry or partners in mission-aligned fields?

Are we positioned for policy and funding changes?

Anticipate the impact of ERA's replacement, the shift toward impact and translation, and potential changes to funding schemes.

From Metrics to Meaning: How to Shift the Conversation

Rather than loading Council papers with every available KPI, consider a more strategic approach:

• Use trends, not snapshots
• Provide benchmarks or context, not just figures
• Focus on signals, not noise: where performance is moving
• Link metrics to mission: how does this support our university's strategy?

A compact dashboard with visual trends and commentary can be more effective than dense tables.

Conclusion: Clarity Builds Confidence

Research is a long-term, high-stakes endeavour. Giving Council the right insights, without overwhelming detail, builds trust, improves decision-making, and strengthens advocacy beyond the university.

It’s not about oversimplifying. It’s about focusing on what truly matters.

When Published Research Gets Rejected: A Glimpse into Peer Review Flaws

 

In a fascinating and provocative experiment, researchers Peters and Ceci once resubmitted 12 already-published psychology articles to the same journals that had originally accepted them. The twist? They changed only the names and affiliations of the authors. What happened next exposed cracks in the foundations of academic peer review.

Only 3 of the 12 resubmissions were identified as duplicates. Of the 9 that underwent full peer review again, 8 were rejected, most for "serious methodological flaws." The same papers that had previously passed muster were now deemed unworthy of publication.

Psychologist John Bartko later reflected on these findings in a commentary titled "The Fate of Published Articles, Submitted Again". His takeaway? The peer-review process, while central to academic credibility, may be far less consistent and objective than many assume. Reviewer bias, institutional prestige, and systemic flaws can skew decisions and undermine trust in the system.

This experiment, now decades old, still resonates today. It reminds us that peer review is a human process, imperfect and in need of constant reflection and improvement.

Do you trust peer review? Or is it time to rethink how we judge good science?

When Opting Out Isn’t Enough: Utrecht University and the Rankings Paradox

 

In 2022, Utrecht University made headlines by stepping away from the Times Higher Education (THE) World University Rankings. The Dutch institution cited concerns that global rankings are overly reductionist, lacking in transparency, and misaligned with its values, particularly its focus on collaboration, open science, and long-term social impact.

But opting out didn’t end the conversation.

When the latest rankings were released without Utrecht’s name, the university faced a flood of questions. Students, staff, and partners wanted to know: Where would Utrecht have landed if it had taken part?

In response, the university issued a public statement explaining its absence and, somewhat paradoxically, pointed to past performance to give a sense of where it might have ranked. The message was clear: while Utrecht rejects the premise of global rankings, it still understands their pull and the need to reassure stakeholders about its standing.

The episode reveals a deeper tension: even for institutions that challenge the value of rankings, their influence is hard to ignore.

ANZSRC Review, or what are our new FOR codes?

The ARC, Australian Bureau of Statistics (ABS), Statistics New Zealand (Stats NZ), and the New Zealand Ministry of Business, Innovation and Employment (MBIE) are undertaking a joint review of the Australian and New Zealand Standard Research Classification (ANZSRC). What is the ANZSRC I hear you say - well it is the field of research codes, or FOR codes. It is also the socio-economic objectives (SEO) and type of activity (pure research, strategic research applied research etc).

It is a good time to refresh the categorization to bring it into line with current and future research activity. It is particularity pleasing to see the inclusion of a specific question around how Aboriginal research is categorized (currently it is all hidden at the '6-digit' FOR level).

The other interesting question is around interdisciplinary research and how the classification could be set up to support this better. If research is multidisciplinary then it is probably a simple matter of tagging it with more than one FOR code. If it is truly inter- or trans-disciplinary activity and of some scale then perhaps it should just have its own FOR code (maybe under a division of 'Interdisciplinary').

I sometimes think of research disciplines as fruits - a field of research might be like an apple, and another is a pear and another is a banana. When research is multidisciplinary it is like we have chopped up the fruit and tossed it together in a bowl to make a fruit salad - works well together but still separate fruits. Interdisciplinary research might be more like a fruit smoothy - we've taken all of the fruits we need but blended them together and have created something new and different. So, ANSZRC helps us classify the fruits and the fruit salad - but how will it classify the smoothy?

You can have a look at the review document yourself at the ARC website: https://www.arc.gov.au/anzsrc-review

Harry Potter and the Draft Engagement and Impact Guidelines

This week the Australian Research Council released for consultation their draft guidelines for the evaluation of university research engagement and impact. The engagement part of the evaluation is mainly quantitative with a shortlist of indicators around research income from industry and end-users. The impact part of the evaluation is mainly qualitative with research impact case studies providing a narrative around the benefit that university research is having outside of the university sector - including the ways that universities are fostering translation and impact from their research.

Some interesting takeaways from the draft guidelines include:

• A May/June 2018 submission deadline (which follows directly behind the ERA 2018 deadline)
• A maximum of 25 impact case studies per university which includes 23 disciplinary case studies, 1 interdisciplinary case study and 1 Aboriginal research case study
• The introduction of a low volume threshold of 150 weighted outputs (books weighted x5) over which a university must submit information and below which a university may opt-in if they so wish
• A new three point rating scale for impact (high, medium, low) which seems more sensible than the pilot ratings (mature, emerging, limited)
• Impact case studies will now receive 2 ratings each - one for the approach to impact and another for the impact itself

Adding to the sector's resource burden in complying with research evaluation is the introduction of two engagement narratives: one is an engagement indicator explanatory statement of 4,500 characters to accompany engagement indicators and the other is a 7,000 character engagement narrative to accompany each unit of assessment. Now seeing as each unit of assessment is the 2-digit field of research this results in a considerable increase in work for the sector. In ERA 2015 there was a total of 656 2-digit FORs evaluated - so if each one of these is accompanied by a 4,500 character explanatory statement and a 7,000 character engagement narrative this equates to around 7.5 million characters, or around 1.2 million words - for comparison, the entire series of Harry Potter books contain around 1.08 million words.

You can see the guidelines for yourself at the ARC website here.

Engagement and Impact Assessment Pilot 2017 Report

The ARC have today released their final report on the Engagement and Impact Assessment Pilot conducted in early 2017.

Changes suggested for the full assessment in 2018 include:

Field of Research 11 - Medical and Health Sciences will be split in two which means two case studies can be submitted bringing the total maximum to 25 (1 for each of the 22 FOR + an extra one for '11' plus an interdisciplinary and Aboriginal research case study)

Four engagement indicators will be used for the engagement part of the assessment:

• cash support from end-users
• total HERDC income per FTE (specified schemes)
• end-user sponsored grants: proportion of HERDC Category 1
• research commercialisation income (selected FoR codes only).

It is encouraging to finally see a definition for Aboriginal and Torres Strait Islander research:

Aboriginal and Torres Strait Islander research means that the research significantly relates to Aboriginal and Torres Strait Islander peoples, nations, communities, place, culture or knowledge.

However, the ARC will continue to consult with its Aboriginal and Torres Strait Islander stakeholders to further refine the definition.

You can read the report at the ARC's website here: http://www.arc.gov.au/ei-pilot-overview

Could ERA be Automated in the Near Future?

Could ERA submissions be auto-generated in the near future? The new ERA specifications released by the ARC hint perhaps yes.

Australia's national research evaluation exercise, Excellence in Research for Australia (ERA) is conducted roughly once every three years with a large investment of time and money from the university sector and the ARC. The cost of running ERA to the sector has been variously estimated to be between $30 million and $100 million.

Universities are required to submit information and data relating to their research activities over the preceding six years. This includes publications, research projects and grants, research staff, along with a raft of related indicators such as patents and commercialisation activity.

Much of the information universities submit as part of the exercise is available from other sources - either publicly available (e.g. grant outcomes from the ARC and NHMRC, HERDC income returns, ABS R&D expenditure surveys) or from third party suppliers (e.g. Scopus or Clarivate publications databases).

If we were able to link researchers, their publications and grant funding to universities and fields of research then an ERA submission could in theory be developed automatically without the time and expense incurred by universities.

The Australian Research Council (ARC) website now includes the ERA 2018 Technical Specifications and Submission Guidelines. Of note is the optional inclusion of information like unique author identifiers (ORCID) and unique article identifiers (DOI). A combination of ORCIDs, DOIs, citation data and fields of research (e.g. from the ERA Journal List) could in theory be used to auto-generate ERA submissions for  universities. Not only could this be less expensive for the sector but also offers the benefit of a more contemporary data set compared with the retrospective ERA as it currently stands.

So perhaps we will see an auto-generated ERA in 2021...

You can view the ERA guidelines for yourself at the ARC's website.

The ARC has released the draft ERA 2018 Submission Guidelines

The ARC has now released the draft ERA 2018 submission guidelines for consultation. You can find a copy at their website here.

There are not really too many changes to the submission which should please universities - especially as they are gearing up for the Impact and Engagement assessment at the same time. Guidelines for the impact and engagement assessment are still pending at this stage.

A couple of interesting additions include:

• Reporting of ORCID (optional)
• Reporting of DOI (optional)
• Aboriginal and Torres Strait Islander research section - although it would be nice to have a clear definition of Aboriginal research to work with. 

Also interesting to see the addition of this to the the guidelines:

Institutions agree to allow the ARC to publish any submitted data from ERA 2018. In addition, institutions must agree to publish their submission, with the exception of their staff data, on 5 February 2019.

It will be good to get some clarity on what form this would take.

Clarivate selected as citation provider for ERA 2018

Clarivate selected as citation provider for ERA 2018

Chief Executive Officer (CEO) of the Australian Research Council (ARC), Professor Sue Thomas, has today announced that the ARC has selected Clarivate Analytics to provide citation information for the 2018 round of Excellence in Research for Australia.

http://www.arc.gov.au/news-media/media-releases/clarivate-selected-citation-provider-era-2018

Web of Science used by Australian Research Council for Analysis of Benefits from University Research

According to this announcement - the Australian Research Council (ARC) will use Web of Science data as part of the next ERA and Engagement and Impact Evaluation - see release below.

http://thomsonreuters.com/en/press-releases/2016/may/web-of-science-source-data.html


Media Release:

The Australian Research Council (ARC) has recently obtained Thomson Reuters Web of Science™ Core Collection as one of the data sources to contribute to analyses that will be utilized by the ARC to support development of national impact and engagement assessment to assess the benefits derived from university research. This national assessment exercise is being introduced as part of the Australian government’s National Innovation and Science Agenda.  This was announced today by the Intellectual Property & Science business of Thomson Reuters.

In 2016 the ARC will work with the higher education research sector, industry and other end-users of research to develop quantitative and qualitative measures of impact and engagement of university research. The Web of Science Core Collection provides source data for records such as topic, title and author information which will be used by ARC to support work around sector and ERA analysis in order to derive a model for national assessment. The ARC will conduct a national assessment as a companion exercise to the Excellence in Research for Australia (ERA), the country’s national research evaluation framework which identifies and promotes excellence across the full spectrum of research activity in Australia’s higher education institutions.

Jeroen Prinsen, vice president and head of Australia & New Zealand, IP &Science, Thomson Reuters said, “As a strong advocate of research collaboration and partner of Australia’s research community, we are pleased to support this important national impact and engagement assessment of university research which will ultimately promote high-quality research that will drive Australia’s innovation and economic growth. We are honored that the ARC will utilize source data from the Web of Science Core Collection, the world’s most trusted source of citation databases.”

Stop publishing your research!

The 'Watt review' - or the Review of Research Policy and Funding Arrangements has broken the link between publications and funding. Since the mid 1990s publications have informed a competent of the research block grants for universities. In 2010 ERA provided an additional avenue for publications to inform block funding allocations. The Watt review has recommended that publications be removed from the Higher Education Data Collection (HERDC) and recommended the removal of the Sustainable Research Excellence (SRE) fund from the block grant. These recommendations mean that universities will no longer receive block funding based on publications.

When publications were introduced to the block grant allocations there was a rapid increase in the volume of publications produced - however, the quality of those publications was low - in other words the quantity went up but the quality didn't. ERA introduced a quality component to the block grant allocation, albeit a modest allocation, which saw an increase in journal article output (compared with conferences and book) and an increase in articles in 'A*' and 'A' ranked journals.

So it seems that publication behaviour changes as the policy and incentives change. It will be interesting to see what impact this newest change has on publication behaviour. Should universities tell their academics to stop publishing? Well, probably not - there are many good reasons to keep publishing, not least of which is that researchers tend to like publishing and it is still a powerful way to disseminate knew knowledge. Besides this though there are a number of other reasons - promotions and recruitments are often influenced by publication record, grant success and university rankings are also linked to publication output.

So maybe don't stop publishing just yet. But watch this space to see what happens to publishing across Australian universities.

Assessment of Impact and Engagement

We have really come full circle in a short amount of time. It wasn't all that long ago that Australia was in the midst of a research excellence and impact evaluation called the Research Quality Framework (RQF). This was to be Australia's first comprehensive evaluation of the quality and impact of its universities's research. With a change in government though came the cancellation of the RQF with concerns that it was too complex and too burdensome to the university sector. As quickly as the RQF was cancelled though it was replaced with the Excellence in Research for Australia (ERA). This would go on to become the first comprehensive evaluation of research quality of Australia's universities - note that impact was removed.

Now, thanks to the recommendations of the Watt review of research funding and policy, we find ourselves returning once again to an impact evaluation. The Watt review recommends we implement a 'companion piece' to our ERA called the Assessment of Impact and Engagement (AIE). The AIE will be a mixed methods evaluation combining quantitative and qualitative components moderated by an expert advisory group. What will the evaluation look like? Well most likely it will be informed by metrics - along the lines of the ATSE Research Engagement for Australia proposal. It will include case studies - as per the UK's REF and it will be moderated by expert review - just like the ATN/Go8 Excellence in Innovation for Australia.

Of particular importance to the evaluation will be how the terms 'impact' and 'engagement' are defined. Ask any researcher what they think the terms mean and you will almost get a different answer every time. This means there will be quite an education piece required to let us all know what the AIE is actually evaluating. And what is it evaluating? What will it tell us about the impact of university research? Most statements about Australian university research mention the same high profile impacts over and again - Cochlear, Gardasil, Atlassian - fantastic impacts, but we already know about them, we don't need an evaluation to tell us about them again. An evaluation may uncover a goldmine of unknown impacts - but what tends to happen is that the high profile impacts need very little, if any, time spent to evaluate them as top of the pile while the rest of the effort, resource, expense is consumed by the less impactful projects - the ones that never get mentioned in the media... so is it really worth it?

Journal Quality Lists: ecological fallacy or convenient and cost-effective evaluation tool?

University managers are constantly seeking simple ways to measure and evaluate the research output of their university’s academics. While peer review of scholarly research papers is arguably the best way to determine the quality of any individual research output it is also acknowledged that peer review is time consuming, expensive and subjective. Journal-level metrics, such as a journal quality list, present managers with a convenient, objective and inexpensive tool for determining the quality of scholarly articles.  However, managers relying on journal level metrics to evaluate articles may be suffering from the ecological fallacy. The ecological fallacy occurs when conclusions are made about individuals based only on analyses of group data. In this case, judging the quality of an individual article based only on the journal in which it is published.

Using journal level metrics to evaluate research quality is not a new phenomenon with the earliest examples of journal quality lists being found as far back as the late 60s and early 70s. Journal level metrics often take the form of lists of scholarly journals which have been ranked against some particular criteria. While there is no consensus on how a journal list should be compiled many lists having been created using methodologies ranging from perceptual and peer review based rankings through to objective citation based rankings.

The use and misuse of journal rankings is well documented in the literature. Within Australia, and internationally, the academic community is shifting away from the use of journal metrics to evaluate research. Australian academics were introduced to the ranked journal list as part of the national research evaluation exercise, Excellence in Research for Australia (ERA). The rankings were considered highly influential in determining a university’s ERA outcome so many institutions began to provide incentives to staff to publish in ‘A*’ and ‘A’ ranked journals. The ranked journal list quickly became the most contentious issue of the ERA and by 2011, then Minster for Education, Kim Carr, announced that it would be discontinued because its ‘existence was focusing ill-informed undesirable behaviour in the management of research’. In 2010, Australia’s other major research funding agency the NHMRC, released a statement saying that the Journal Impact Factor would no longer be accepted in applications for funding or used in the peer review of individual applications. The statement went on to say that the Journal Impact Factor is ‘not a sound basis upon which to judge the impact of individual papers’.

Internationally, the San Francisco Declaration on Research Assessment (DORA), originating from the December meeting of the American Society for Cell Biology, put forward a number of recommendations for funding agencies, universities and researchers regarding the use of metrics for research evaluation. Amongst it other recommendations DORA aims to halt the use of journal-based metrics for the research evaluation of individual researchers. As of August 2015 the declaration had over 12,500 individual and 588 institutional signatories.

While there are some compelling reasons to use journal quality lists to evaluate the research performance of academics including convenience, objectivity and cost savings there are also disadvantages. Some of the disadvantages of using journal quality lists for research evaluation include, reduced academic freedom, promotion of outlet targeting, driving research in the direction of publisher preference and disadvantaging of specialist journals and specialist fields of research.

Whether we like them or not journal quality lists have been part of research evaluation for the past 50 years and their legacy persists today. As the requirement for convenient and cost-effective research evaluation mechanisms increases it is possible that journal quality lists will continue to play a part in research evaluation into the future. For examples of journal lists from around the world visit www.researchimpact.com.au/viewforum.php?f=20.

Research Evaluation – an argument for a ‘census’ driven collection of publications

In this age of accountability no one questions the idea that data relating to university research publications is collected and reported on. Research publications are no longer only a mechanism for disseminating research findings but they are now also a measure of research performance.

It is hardly surprising then that discussions arising from a recent review by PhillipsKPA are not around whether we should collect research publication data but how we can collect it more efficiently. Australian universities currently report research publications data through the Higher Education Research Data Collection (HERDC) and Excellence in Research for Australia (ERA). One of the 27 recommendations from the PhilipsKPA Review of University Reporting Requirements is to streamline the collection of research data into a single collection. Combining the two collections into a single collection will only be worthwhile if it improves the efficiency, integrity, transparency and utility of the data being collected. Any consultation document will hopefully clarify for the sector how the combined collection will achieve this.

While both mechanisms currently collect research publication and research income data that is really where the similarities stop. The type of data collected, the level of detail collected and importantly the purpose of the collections are quite different.

The purpose of the HERDC is to collect research income and publications data to inform the distribution of research block grants to universities based on their relative performance in each measure. The HERDC only reports on publication volume and does not consider the field of research or the quality of the research – at best it provides a proxy for the volume of research activity across Australian universities.

According to the ARC’s ERA documentation the objectives of ERA are much broader than the HERDC and are listed as, to:

·         establish an evaluation framework that gives government, industry, business and the wider community assurance of the excellence of research conducted in Australian higher education institutions;

·         provide a national stocktake of discipline level areas of research strength and areas where there is opportunity for development in Australian higher education institutions;

·         identify excellence across the full spectrum of research performance;

·         identify emerging research areas and opportunities for further development; and

·         allow for comparisons of research in Australia, nationally and internationally, for all discipline areas.

If we focus on the collection of research publications data in each collection there is one main difference. In the HERDC, publications data is collected for all publications that acknowledge the university with which the author is affiliated on the publication itself – for example through an author ‘by-line’ – regardless of whether the author is currently employed at the university or not - I will refer to this as an ‘address’ based collection. In the ERA collection publications data is collected for all publications authored by researchers employed by a university at a census date (usually 31 March of the year preceding the ERA collection) – regardless of whether the university is acknowledged within the publication or not - I refer to this as a ‘census’ based collection. The difference between an ‘address’ based collection and a ‘census’ based collection may not at first seem significant – but it is.

Consider the case of the HERDC – data are collected on publications only where the university has been acknowledged on the publication – regardless of whether the researcher or research group still works at the university. Once the data have been collected and reported to the Department of Education the numbers are used to distribute block grant funding (approximately $1700 per publication point). This is effectively rewarding universities for the volume of publications they can report that list the university in the byline. There is no consideration made of the quality or the focus of the research in the publications – just the volume. If researchers who produced the publications have since left the university the university is still credited with their research activity.

A big advantage of the address based collection is that it is easy to determine which outputs are eligible for collection and which university they belong to. An address based collection could be conducted by a third party (for example through a citation data provider like Scopus or Web of Science). The disadvantages are that the collection is retrospective in that the researchers may have left the university but their output still contributes to the block grant allocation. This is fine if you are rewarding past performance but problematic if you are trying to profile current research strengths of Australian universities or if you are trying to fund for future research success.

Now consider the ERA which also collects publications data. These publications are the ones produced by the current cohort of staff at the university and not just the ones with a university listed byline. When the publications are reported they are assigned to fields of research and subsequently given a quality score by a national evaluation panel. This allows the Department (and the public) to see where research excellence exists in Australian universities and where research strengths may be emerging. The main disadvantage of a census based collection is that it requires more administrative work to collect the data as publications are not readily identifiable by a university byline within the publication. A census based collection cannot easily be performed by a third party. Advantages of the census based collection are that it represents the current research profile of the university and encourages universities to strategically recruit researchers to contribute to the research profile of the university.

In the case of a new “combined collection” for research publications data it is not immediately clear whether it would be done based on the ‘address’ or the ‘census’ date. Each has its pros and cons and each is useful for a different purpose. However, I would argue that the ‘census’ based approach is more appropriate in this case because of the following reasons:

1.       It allows universities to demonstrate a current research profile based on researchers who are actually working at the university rather than a retrospective profile where researchers may well have left the university since the evaluation

2.       It allows universities to respond strategically to changes in the research landscape and funding environment for example by recruiting researchers to complement or strengthen their existing research profile

3.       It trusts universities with the responsibility of presenting their research in a meaningful way based on their own knowledge of their researchers and their research rather than leaving it up to a third party or a generic business process relying on accurately recorded address data

4.       It aligns with the national uptake of a universal researcher ID in Australia such as the ORCID

I would conversely argue that using an ‘address’ based collection for a research evaluation may result in what I call ‘phantom’ units of evaluations. A publication can have multiple authors and therefore multiple universities listed in the byline – each author may also have multiple bylines. This results in each author of each publication potentially contributing to multiple university research evaluations. A ‘phantom’ evaluation would be where a university may appear to have a minimum volume of publication output (for ERA this is 50 publications to trigger an evaluation) based only on the fact that the university’s name appears at least once on 50 publications. However, while the byline appears on the publication the author may not actually work at the university – for example, if the author has since left the university or where the author has multiple bylines which include other university affiliations in addition to the one they work at. The ‘address’ based collection would also potentially disadvantage universities who have strategically invested in recruiting new researchers to their university. In this case while the new university is paying the salary of the researchers, those researchers’ publications would be contributing to another university’s research evaluation based on their previous bylines.