From Physical System to Information System
Hopfield network is a model for content-addressable memory. To put simply, once the desired set of memories are stored in the network, it models the way how human recall memories given some cue. If you see some occluded object (cue), you can complete that object through your mind’s eye or if you hear a initial music of a song, you can complete that song by recalling it.
Hopfield network came after Ising model. HN is very much similar to the Ising model. First, I will give one para description of the Hopfield network, where all the key concepts are underlined. Then I will explain each of the key concepts and other related concepts in detail, from the lens of Ising model. ”Hopfield network has a fully connected recurrent architecture , where nodes can take binary values. Memories are stored by setting the weights to pairwise correlation (called as Hebb rule). It uses energy function to go from unstable state (memory cue) to stable state (memory) via asynchronous update of the node values.”
This blog follows the original paper [1]. But there exist many variants like bipolar node instead of binary node, using synchronous update instead of asynchronous update and so on.
Paper starts with showing that what properties a physical system should have so that it can be considered as content addressable memory.
Goal of Ising model and HN is completely different but they have the similar setup - similar system they deal with, same energy function, same dynamics but different interpretation of it.
Key difference between Ising model and HN is that Ising model is the model of stat mech while HN is not really. Ising model study the internal motion while HN deal with the motion of the whole system. Therefore there is no notion of microstate, macrostate and ensemble. Rather we talk about initial state, final state and dynamics between them. We study Ising model at equilibrium at different temperatures. While in HN, we are interested in creating an energy landscape (function), according to our need, making some desired states as stable, this process is also called “inverse Ising problem”. Network starts in the initial state which is typically out of equilibrium (higher energy state). And we want the network to move in energy landscape and to reach to nearby desired stable state (local minima in energy). We make the network change its state with the help of Monte Carlo. Purpose of Monte Carlo in both the model is different, in Ising model, we used it to sample from the probability distribution but here, we are using as equation of motion of the whole system. We use MC with zero temperature, which makes the motion deterministic towards local minima. Once it reaches to a stable state, network do not evolve and we say equilibrium is achieved.
I will formally define the system, HN deal with in “Architecture” section.