How scaled scores are calculated, and what they mean

When a set of scripts of candidates' work is judged, many judgements are made between many combinations of the candidates' scripts. The mathematical pairing algorithm that underpins our comparative judgement system looks at the outcomes of the judgements made on all of these many combinations, and levels out any inconsistencies in the judges' decisions, coming up with a 'true score' that reflects the relative level of each script (according to the judges).

This true score is set to have a mean value of 0, stretching as far as necessary positively or negatively. It therefore has no direct interpretation in the classroom, and as such we could transform it in any way we choose in order to put it on a standard scale, or even to link it to academic indicators such as national grading levels.

We initially convert these numbers into what we call our scaled scores, which are typically set with a minimum value of 0 and a maximum value of whatever we have chosen for that task, for example 100 (with a higher number representing a better performance on the task, according to the judges).

We all know that crude rank ordering can be misleading if not used carefully. (Get your class to stand in height order across the classroom, and then ask one of them in the middle to simply remove their shoes, re-order themselves, and see how far their rank order changes!)

Our scaled scores are much more illuminating than a simple rank ordering (which you could simply do by physically ordering the scripts from best to worst on your desk, assuming your judgements were reliable and unbiased!) The power of the scaled scores is that they give you an indication of not just the rank but the magnitude of the difference between two pieces of work, which can be far more informative for the classroom teacher.

A fuller explanation of how scaled scores are calculated is here.

Should you choose, you can adjust the scale that is used for a particular internal task to that of your liking.

To do this, select the task, click the 'Settings' button, adjust the Scaled Score Min and Scaled Score Range values, and then press the 'Update' button:

https://s3-us-west-2.amazonaws.com/secure.notion-static.com/464da72a-3911-476e-86d5-3631d03bd2b8/Untitled.png

In the example given, by setting Scaled Score Min to 10 and Scaled Score Range to 80, we have set the scores to range from 10 to 90.

Also important to note is that when you import Anchors from another task, your scale adapts to the task you are anchoring to.