In particular if your school is internally judging a comparative judgement assessment, with your teachers and TAs involved, you will want to look at how the judges are getting along in their judgements. This article will explain how to do that.
At any point during the judging process, for a given task, select the Run judging session option, and then, just to make sure the judging statistics are up to date, press the 'Refresh Scores' button:
You will be able to check how the judging is progressing for your judges by examining the information in the Judges table:
The different columns show the following information:
Email - This is the email address of the judge.
Link - This is the unique link that can take you to the judging site for this judge for this task.
Infit - This statistic is an indication of the degree of consistency or inconsistency of this judge's decisions compared to those of the other judges. A low infit score indicates better consistency, a higher infit score indicates worse consistency. An infit value of 1.3 or more indicates some significant inconsistency on the part of the judge, and it might be worthwhile contacting the judge to check that they fully understand the judging process.
Local - the number of 'local' judgements the judge has made
Mod - the number of 'moderation' judgements the judge has made
Quota - the total number of judgements allocated to that judge. This information is useful if you need to remind judges to complete their judgements.
Median Time & Time Chart - Here you can see the median time taken by a judge in their judgements on this task. Quite short median times might indicate that a judge is rushing through their judgements (it is informative to compare the median time to the infit score), and a long median time might indicate that the judge is facing some problems with the judgement process. The Time Chart shows a broader picture of the range of times taken by the judge for their decisions - uneven distributions might again indicate problems with the judging process.
% Left Click - In the judging screen, scripts are presented randomly on the left or right hand side of the screen. Therefore, we would generally expect that the percentage of times that a judge clicks in favour of the 'Left' script would be around 50%. If the % Left Click value is much higher or lower than 50%, it might indicate that the judge is not thinking carefully about their judgements and just staying with the left or right hand side. Again it is informative to compare this value with the infit statistic and the median time taken.
Exclude - If there are real concerns about the standard of judging by a particular judge, you may wish to Exclude this judge's decisions from the final calculation of scores, so that they are more reliable. Only do this after you have tried to sort out any problems with the judge.