What is the difference between a neural network and a deep belief network?

Question

I am getting the impression that when people are referring to a 'deep belief' network that this is basically a neural network but very large. Is this correct or does a deep belief network also imply that the algorithm itself is different (ie, no feed forward neural net but perhaps something with feedback loops)?

Answer 1

"Neural networks" is a term usually used to refer to feedforward neural networks. Deep Neural Networks are feedforward Neural Networks with many layers.

A Deep belief network is not the same as a Deep Neural Network.

As you have pointed out a deep belief network has undirected connections between some layers. This means that the topology of the DNN and DBN is different by definition.

The undirected layers in the DBN are called Restricted Boltzmann Machines. This layers can be trained using an unsupervised learning algorithm (Contrastive Divergence) that is very fast (Here's a link! with details).

Some more comments:

The solutions obtained with deeper neural networks correspond to solutions that perform worse than the solutions obtained for networks with 1 or 2 hidden layers. As the architecture gets deeper, it becomes more difficult to obtain good generalization using a Deep NN.

In 2006 Hinton discovered that much better results could be achieved in deeper architectures when each layer (RBM) is pre-trained with an unsupervised learning algorithm (Contrastive Divergence). Then the Network can be trained in a supervised way using backpropagation in order to "fine-tune" the weights.

Answer 2

"A Deep Neural Network is a feed-forward, artificial neural network that has more than one layer of hidden units between its inputs and its outputs. Each hidden unit, $j$, typically uses the logistic function to map its total input from the layer below,$x_j$, to the scalar state, $y_j$ that it sends to the layer above. (Ref. (1))".

That said, as mentioned by David: "deep belief networks have a undirected connections between the top two layers, like in an RBM", which is in contrast to standard feed-forward neural networks. In general, the main issue in a DNN regards the training of it that is definitely more involved that a single layer NN. (I am not working on NNs it just happened I read the paper recently.)

Reference: 1. Deep Neural Networks for Acoustic Modeling in Speech Recognition, by Geoffrey Hinton, Li Deng, Dong Yu, George Dahl, Abdel-rahman Mohamed, Navdeep Jaitly, Andrew Senior, Vincent Vanhoucke, Patrick Nguyen, Tara Sainath,, and Brian Kingsbury in the IEEE Signal Processing Magazine [82] Nov. 2012 (Link to Original Paper in MSR)