No More Marking assumes that judging will be undertaken by a group of experts.
Judges are normally be drawn from a community that has expertise in the criterion of interest. For example, if the criterion is mathematical understanding then our experts might be mathematicians or mathematics PhD candidates or mathematics teachers.
It is important that the judging is done not by a single individual but by a group of experts so that the results reflect the collective view of the experts. This ensures that a range of expert opinion informs the results. It also minimises bias: for example one expert might prefer artefacts that, say, include both diagrams and text and another might prefer artefacts that, say, use technical language properly formatted. When both these experts contribute judgements the bias is balanced out.
You will need to decide how many experts to recruit for the judging. As a rule, the more the merrier to minimise bias. In practice we recommend a minimum of 5 experts.
There are exceptions to using a group of experts. For example, some researchers are interested in the potential of No More Marking to support peer assessment. Other studies have used non-experts as judges to investigate divergent validity. Researchers might wish to conduct study in which the focus is on the quality and nature of judging itself, rather than the artefacts on a linear scale.