I'm confused. Is there a difference between Deep belief networks and Deep Boltzmann Machines? If so, what's the difference?

  • 2
    $\begingroup$ the wikipedia article on deep belief networks is fairly clear although it would be useful/insightful to have a bigger picture of the etymology/history of the terms. basically a deep belief network is fairly analogous to a deep neural network from the probabilistic pov, and deep boltzmann machines are one algorithm used to implement a deep belief network. apparently all ANNs have probabilistic interpretations/models but they are not as easily/directly obtained as some bayesian/probabilistic-oriented "belief" models. $\endgroup$ – vzn Apr 29 '14 at 4:37

Although Deep Belief Networks (DBNs) and Deep Boltzmann Machines (DBMs) diagrammatically look very similar, they are actually qualitatively very different. This is because DBNs are directed and DBMs are undirected. If we wanted to fit them into the broader ML picture we could say DBNs are sigmoid belief networks with many densely connected layers of latent variables and DBMs are markov random fields with many densely connected layers of latent variables.

As such they inherit all the properties of these models. For example, in a DBN computing $P(v|h)$, where $v$ is the visible layer and $h$ are the hidden variables is easy. On the other hand computing $P$ of anything is normally computationally infeasible in a DBM because of the intractable partition function.

That being said there are similarities. For example:

  1. DBNs and the original DBM work both using initialization schemes based on greedy layerwise training of restricted Bolzmann machines (RBMs),
  2. They are both "deep".
  3. They both feature layers of latent variables which are densely connected to the layers above and below, but have no intralayer connections, etc.
  • 1
    $\begingroup$ How can DBNs be sigmoid belief networks?!! The layers of a DBN are RBMs so each layer is a markov random field! $\endgroup$ – Jack Twain Apr 29 '14 at 13:14
  • $\begingroup$ I think there's a typo here "This is because DBMs are directed and DBMs are undirected.". I think you meant DBNs are undirected $\endgroup$ – Jack Twain Apr 29 '14 at 13:14
  • $\begingroup$ @AlexTwain Yes, should have read "DBNs are directed". Even though you might intialize a DBN by first learning a bunch of RBMs, at the end you typically untie the weights and end up with a deep sigmoid belief network (directed). In a lot of the original DBN work people left the top layer undirected and then fined tuned with something like wake-sleep, in which case you have a hybrid. $\endgroup$ – alto Apr 29 '14 at 14:07
  • $\begingroup$ Do you mean in 3. that they do not have intralayer" connections (e.g. between nodes in the hidden layer) rather than *interlayer (e.g. from the input to the hidden layer)? $\endgroup$ – ddiez Jul 10 '15 at 6:52
  • $\begingroup$ @ddiez Yeah, that is how that should read. Thanks for correction. $\endgroup$ – alto Jul 11 '15 at 0:39

Both are probabilistic graphical models consisting of stacked layers of RBMs. The difference is in how these layers are connected.

This link makes it fairly clear: http://jmlr.org/proceedings/papers/v5/salakhutdinov09a/salakhutdinov09a.pdf. Figure 2 and Section 3.1 are particularly relevant.

To summarise:

In a DBN the connections between layers are directed. Therefore, the first two layers form an RBM (an undirected graphical model), then the subsequent layers form a directed generative model.

In a DBM, the connection between all layers is undirected, thus each pair of layers forms an RBM.

  • $\begingroup$ so a deep boltzmann machine is still constructed from RBMs? I'm basing my conclusion on the introduction and image in the paper $\endgroup$ – Marin May 29 '16 at 21:09

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.