Evidence Lower Bound / Variational Free Energy. Definitions and Math.

Introduction

Evidence Lower Bound (ELBO) / Variational Free Energy

With

$$ H[Z]=-E_q[\log q(Z)] $$

as the Shannon entropy, we have ELBO

$$

L = E_q [\log p(X, Z)] + H[Z] $$

ELBO is used as a proxy for P, (or more specifically, an augmented optimization objective).

• L provides a lower bound on the likelihood. An exact inference would be one which maximizes this perfectly. So instead of maximizing the likelihood directly, we can instead maximize L.
• Since L is always a lower bound regardless of the choice for q.

Remarks

• Compared to directly computing the log likelihood, we introduced q on top of model parameters theta.

  Recall that L depends on q, theta, and observed data v, where as log(p) only depends on v and theta.

• Gains: For some families of q, ELBO is tractable to compute.

• Losses: One more thing to decide and worry about. The choice for q has implications in our approximation

  • See approximate inference methods for more details on what this entails in practice.

• As a result, we can vary between computational intensity and speed vs. accuracy of the q we get.

• Note that this is important for approximate inference NOT because this bound is approximated, but because the methods for choosing and estimating q are approximated.

Resources