I'm confused. Is there a difference between Deep belief networks and Deep Boltzmann Machines? If so, what's the difference?
Although Deep Belief Networks (DBNs) and Deep Boltzmann Machines (DBMs) diagrammatically look very similar, they are actually qualitatively very different. This is because DBNs are directed and DBMs are undirected. If we wanted to fit them into the broader ML picture we could say DBNs are sigmoid belief networks with many densely connected layers of latent variables and DBMs are markov random fields with many densely connected layers of latent variables.
As such they inherit all the properties of these models. For example, in a DBN computing $P(v|h)$, where $v$ is the visible layer and $h$ are the hidden variables is easy. On the other hand computing $P$ of anything is normally computationally infeasible in a DBM because of the intractable partition function.
That being said there are similarities. For example:
- DBNs and the original DBM work both using initialization schemes based on greedy layerwise training of restricted Bolzmann machines (RBMs),
- They are both "deep".
- They both feature layers of latent variables which are densely connected to the layers above and below, but have no intralayer connections, etc.
Both are probabilistic graphical models consisting of stacked layers of RBMs. The difference is in how these layers are connected.
This link makes it fairly clear: http://jmlr.org/proceedings/papers/v5/salakhutdinov09a/salakhutdinov09a.pdf. Figure 2 and Section 3.1 are particularly relevant.
In a DBN the connections between layers are directed. Therefore, the first two layers form an RBM (an undirected graphical model), then the subsequent layers form a directed generative model.
In a DBM, the connection between all layers is undirected, thus each pair of layers forms an RBM.