How to write the most effective challenge problems for learning?

• problem: How to write the most effective challenge problems for identifying conceptual blindspots? Whereas a conventional homework problem mainly has to be quickly gradable as a numeric score, via the usual points-off rubric ("didn't say the right words"), a challenge problem on Courselets ("Open Response Concept Test", ORCT) must meet the more difficult requirement of zero blindspots: first, that if a student has a misconception it will be exposed; and second that it should be diagnosable directly from the student's answer. This is more challenging. For example, even if a student has a misconception, depending on the question they still might be able to "say the right words". This raises a clear practical question for how to write questions: are there straightforward guidelines that can best expose student misconceptions?
• best practice: our years of work on Courselets have shown that there is a straightforward best-practice for doing this, that we call translation problems: the question should require students to answer using a different representation than the question was asked in. For example, we might pose a question about a probability problem in words, using a specific vocabulary taught in class. Then we could ask students to show how to solve it by drawing pictures using a specific kind of diagram taught in class (e.g. Venn diagrams). Such translation tasks have been directly shown to improve retention. Alternatively, we could pose a problem by drawing a diagram, and ask students to show how to solve it using equations.
• example: "Draw a Venn diagram representing two independent variables, X,Y, i.e. where p(X,Y)=p(X)p(Y). To keep the diagram clear and simple, restrict each variable to just three discrete states, and draw the diagram so area represents (i.e. is proportional to) probability."

How to write translation problems

This best-practice can actually be an easy step for most instructors:

• their materials already teach students multiple representations of a concept, typically:
• words: a specific vocabulary that defines the key relationships in the concept;
• pictures: a specific way of drawing those key relationships that makes it easy to see and manipulate them;
• symbols (e.g. equations): a symbolic logic that distills those relationships in a powerful way.
• Students can't understand a complex idea if it's presented in only a single form -- such as a lecture that's 100% equations, or a textbook without a single picture or diagram -- so the use of such multiple representations is just standard practice for good teaching. Most instructors are already using this.
• in that case, all that's required is to start deliberately asking students to answer using a different one of these three representations.
• the instructor's task is made easier by the fact that translation problems don't have to be hard to be effective at exposing misconceptions. In our experience, just about any basic question "re-routed" to use translation will expose copious misconceptions: e.g. students who "know" the right words but not some of their basic meanings; students who misunderstood the original representations that were taught; students who can't use the representations to see even their basic implications; etc. For instance, the example shown above is conceptually trivial -- equivalent to asking what is the Spanish word for "hello" -- yet it exposes basic misconceptions in about 50% of students who have completed a Statistics and Probability course.

Analysis

By design, a translation problem means students can't answer the question "in the same language" it was asked in — they must perform translation:

• they must first interpret the conceptual meaning of the question representation (e.g. equation);
• then reformulate that meaning in the second representation (e.g. diagram);
• and finally reason correctly in that second representation to solve the problem.
• This translation process will immediately expose any flaw in their conceptual understanding at any point in this extended chain.
• Moreover, because of this exposure guarantee, translation problems need not be very difficult; that is, translation problems that are easy for a student with good understanding of the basic concept, can still give high probability of exposure for any student who has a misconception.