How does ERA evaluation differ between peer review and citation metrics disciplines

Evaluation of the Excellence in Research for Australia (ERA) is done by either peer review (for the humanities and social sciences) or citation metrics (for the medical, engineering and physical sciences). On average the scores across the sector are lower for the 'peer review' disciplines than they are for the 'citation metrics' disciplines. This may mean that in Australia the research quality for the humanities and social sciences is not as good as the research quality for medical sciences, engineering and physical sciences - or it may be that the process for evaluation is not equivalent between the two. Unfortunately no discipline has been evaluated using both methods in an ERA round so it is not clear whether a discipline would fair the same using both methods.

I think two simple experiments could be conducted in the ERA which could have very interesting results. Experiment 1 to determine whether the same submission evaluated by two different methods (citation metrics and by peer review) would score differently in an evaluation of research quality. Experiment 2 to determine whether an evaluation of research quality by peer review is altered by knowing the authors and the journals that the articles are published in.

Experiment 1.
Run a comparison of evaluations of the same submission in a field of research (FOR) using (a) citation metrics and (b) peer review.

a. The first group of evaluators would make a judgment based only on the metrics (e.g. citation impact, research income per faculty, and the standing of journals that articles were published in).

b. The second group of evaluators would look at the same submission but with no metrics and make a judgment purely by reading the journals articles of the submitted authors.

both groups would come up with an ERA ‘score’ – it would be interesting to see the if there was any difference.

Experiment 2.
Take group (b) above – the peer review group – and split them in to a further two groups:

c. This group would peer review the submission by reading the articles with complete knowledge of the authors of the articles and which journals the articles were published in.

d. The second group would have no information on the authors of the articles or the journals in which they were published. These evaluators would have to read the articles and make an evaluation of the quality of the research based on the articles alone.

It would be interesting to see how much the knowledge of the authors names or the knowledge of the journals that the articles were published in swayed the evaluators assessment of research quality.