a type of learned inference that explains away your bio-clock.

Introduction

The motivation itself is simple: If there is not enough supervised data, the model family is also a choice over which q. To solve both problems, why don't we learn a reverse mapping?

Wake-sleep is Hinton's solution to this: you draw samples from both h and v and predict which h caused the present v.

• The wake phase: receptive to external output
• The sleep phase: just generating and improving from within

This tries to mimic animals' tendencies to require some resting period where the brain is still active but the external stimuli is lacking, namely for the visual cortex.

This belongs to a strange class of methods where the approximation of inference itself is learned. That is, we learn the mapping

$$ f:v \to q^* $$

where L(v,q) is maximized.

(Naturally </s>, we approximate such f with a neural net.)

Remarks

• The key distinction in these methods is that we are no longer seeing the problems as one in optimization
  • The other ELBO-related techniques treat training as an optimization procedure that increases the value of L.
• Con: Wake-sleep usually ends up only learning the ones with high probabilities, since we are super sensitive to those. The complaints thus include the method being improper, or that its power and expressiveness are lacking. (It could also be that people haven't been able to make it work very well yet!)
• Pro: Goodfellow et al in the chapter presents some trippy explanations to the duration of wake and sleep, that I will not go into.

Resources

Wake-sleep algorithm - Wikipedia

Original paper from 1995