(See also this mini-movie on setting up judgements).
The greater the number of judgements, the more reliable the results. The number of judgements you supply should be driven by the required reliability.
Research suggests that the number of judgements required varies from 10 to 37 per artefact. As a general rule, we would suggest carrying out 10 judgements per artefact, but in practice it depends on the nature of the artefacts, the criterion, and the relative expertise of the judges.
You would normally expect to divide the number of judgements required equally amongst the experts. To save you time, our system works out most of this information for you, based on the number of artefacts in the task.
To do this, select the Run judging session option. You will be able to amend some of the information in the calculation boxes, such as the number of judgements per artefact, or your expected number of judges:
When you are happy with the information, press the Adjust button - this will calculate the default number of judgements each judge will need to carry out.
You will see a Sign Up Link appear on screen:
To change the default number of judgements per judge for the task before signing them up as judges, you can amend the information in the calculation boxes (e.g. the number of judgements per artefact, or your expected number of judges) and press the Adjust button.
You can also adjust the quota for each existing judge individually, after they have signed up as a judge. To do this, select the judge(s) by clicking on their check-box(es), press the Update Judge Quota button and then amend the quota as appropriate:
