“In terms of interesting properties -- some type of test time sampling to improve generalization would be interesting (in combination with a contrastive loss to shape the energy function)”

“One big issue is that there is no unique advantage of your method — it is not better than any of the baseline methods and nor does it have any unique properties.”

“For psuedolikelihood, people have been training with it for several decades. For EBMs, you can reference the previous paper Implicit Generation with Energy Based Models and Improved Contrastive Divergence training of Energy Based Models. Both can be added as references and you can also see the paper referenced. ”

TODO:

• 回顾经典papers （EBMs&Contrastive Divergence相关）
• 提出新的具体方法（比如用对比损失+测试时采样的方法，来提高泛化能力）
  • Motivation：侧重泛化、测试时
  • Novelty：

More to Read:

• GANs
• VAEs
  • Wiki:
    • data point → distribution in latent space → data point
    • optimize a lower bound of the data likelihood
    • q-distribution: variational posterior
    • p-distribution: assumed data prior
    • two terms(both derived from the free energy expression of the probablistic model):
      • reconstruction error
      • KL-divergence: 会导致mode collapse，可以用其他的distance 函数
    • Definition：
      • Prior $p_\theta(z)$, Likelihood $p_\theta(x|z)$, Posterior $p_\theta(z|x)$
      • 由于posterior intractable, $q_\phi(z|x) \approx p_\theta(z|x)$
      • decoder学先验，encoder学后验
      • maximize the evidence lower bound(ELBO)
• Flow-Based Models
• Contrastive Divergence & Compositional EBM: Product of Experts, SVGD&SteinCD, VCD
• $f$-divergence
• EBM for discriminative tasks
• Unbalanced problems of Separate networks (VAE, GAN)
• Compositional EBMs & contrastive bakpropagation, also in RL
• Langevin dynamics
• Simulated annealing(annealed importance sampling, multimodal mcmc)
• Annealed MCMC for compositional diffusion(RRR )
• Replay buffer (RBM:PCD, )