What is the difference between a neural network and a deep belief network?

I am getting the impression that when people are referring to a 'deep belief' network that this is basically a neural network but very large. Is this correct or does a deep belief network also imply that the algorithm itself is different (ie, no feed forward neural net but perhaps something with feedback loops)?

"Neural networks" is a term usually used to refer to feedforward neural networks. Deep Neural Networks are feedforward Neural Networks with many layers.

A Deep belief network is not the same as a Deep Neural Network.

As you have pointed out a deep belief network has undirected connections between some layers. This means that the topology of the DNN and DBN is different by definition.

The undirected layers in the DBN are called Restricted Boltzmann Machines. This layers can be trained using an unsupervised learning algorithm (Contrastive Divergence) that is very fast (Here's a link! with details).

The solutions obtained with deeper neural networks correspond to solutions that perform worse than the solutions obtained for networks with 1 or 2 hidden layers. As the architecture gets deeper, it becomes more difficult to obtain good generalization using a Deep NN.

In 2006 Hinton discovered that much better results could be achieved in deeper architectures when each layer (RBM) is pre-trained with an unsupervised learning algorithm (Contrastive Divergence). Then the Network can be trained in a supervised way using backpropagation in order to "fine-tune" the weights.

• Is the primary difference between the function of a DNN & DBN that DBNs function in both directions, in => out for training, out => in for predictions? – ZAR Jul 20 '18 at 0:56

"A Deep Neural Network is a feed-forward, artificial neural network that has more than one layer of hidden units between its inputs and its outputs. Each hidden unit, $j$, typically uses the logistic function to map its total input from the layer below,$x_j$, to the scalar state, $y_j$ that it sends to the layer above. (Ref. (1))".

That said, as mentioned by David: "deep belief networks have a undirected connections between the top two layers, like in an RBM", which is in contrast to standard feed-forward neural networks. In general, the main issue in a DNN regards the training of it that is definitely more involved that a single layer NN. (I am not working on NNs it just happened I read the paper recently.)

Reference: 1. Deep Neural Networks for Acoustic Modeling in Speech Recognition, by Geoffrey Hinton, Li Deng, Dong Yu, George Dahl, Abdel-rahman Mohamed, Navdeep Jaitly, Andrew Senior, Vincent Vanhoucke, Patrick Nguyen, Tara Sainath,, and Brian Kingsbury in the IEEE Signal Processing Magazine [82] Nov. 2012 (Link to Original Paper in MSR)

• A deep BELIEF network usually refers to a deep network with unsupervised pretraining (stacked restricted Boltzmann machines trained with contrastive divergence). – alfa Mar 6 '13 at 13:21
• @user11852 The paper you linked to makes a distinction between deep neural networks and deep belief networks. Deep belief networks have a undirected connections between the top two layers, like in an RBM. – David J. Harris May 24 '13 at 0:34