Multi-layer perceptron vs deep neural network This is a question of terminology. Sometimes I see people refer to deep neural networks as "multi-layered perceptrons", why is this? A perceptron, I was taught, is a single layer classifier (or regressor) with a binary threshold output using a specific way of training the weights (not back-prop). If the output of the perceptron doesn't match the target output, we add or subtract the input vector to the weights (depending on if the perceptron gave a false positive or a false negative). It's a quite primitive machine learning algorithm. The training procedure doesn't appear to generalize to a multi-layer case (at least not without modification). A deep neural network is trained via backprop which uses the chain rule to propagate gradients of the cost function back through all of the weights of the network. 
So, the question is. Is a "multi-layer perceptron" the same thing as a "deep neural network"? If so, why is this terminology used? It seems to be unnecessarily confusing. In addition, assuming the terminology is somewhat interchangeable, I've only seen the terminology "multi-layer perceptron" when referring to a feed-forward network made up of fully connected layers (no convolutional layers, or recurrent connections). How broad is this terminology? Would one use the term "multi-layered perceptron" when referring to, for example, Inception net? How about for a recurrent network using LSTM modules used in NLP?   
 A: One can consider multi-layer perceptron (MLP) to be a subset of deep neural networks (DNN), but are often used interchangeably in literature.
The assumption that perceptrons are named based on their learning rule is incorrect. The classical "perceptron update rule" is one of the ways that can be used to train it. The early rejection of neural networks was because of this very reason, as the perceptron update rule was prone to vanishing and exploding gradients, making it impossible to train networks with more than a layer.
The use of back-propagation in training networks led to using alternate squashing activation functions such as tanh and sigmoid.
So, to answer the questions,

the question is. Is a "multi-layer perceptron" the same thing as a "deep neural network"?

MLP is subset of DNN. While DNN can have loops and MLP are always feed-forward, i.e.,
A multi layer perceptrons (MLP)is a finite acyclic graph

why is this terminology used?

A lot of the terminologies used in the literature of science has got to do with trends of the time and has caught on.

How broad is this terminology? Would one use the term "multi-layered perceptron" when referring to, for example, Inception net? How about for a recurrent network using LSTM modules used in NLP?

So, yes inception, convolutional network, resnet etc are all MLP because there is no cycle between connections. Even if there is a shortcut connections skipping layers, as long as it is in forward direction, it can be called a multilayer perceptron. But, LSTMs, or Vanilla RNNs etc have cyclic connections, hence cannot be called MLPs but are a subset of DNN.
This is my understanding of things. Please correct me if I am wrong.
Reference Links:
https://cs.stackexchange.com/questions/53521/what-is-difference-between-multilayer-perceptron-and-multilayer-neural-network
https://en.wikipedia.org/wiki/Multilayer_perceptron
https://en.wikipedia.org/wiki/Perceptron
http://ml.informatik.uni-freiburg.de/former/_media/teaching/ss10/05_mlps.printer.pdf
A: Good question: note that in the field of Deep Learning things are not always as well-cut and clearly defined as in Statistical Learning (also because there's a lot of hype), so don't expect to find definitions as rigorous as in Mathematics. Anyway, the multilayer perceptron is a specific feed-forward neural network architecture, where you stack up multiple fully-connected layers (so, no convolution layers at all), where the activation functions of the hidden units are often a sigmoid or a tanh. The nodes of the output layer usually have softmax activation functions (for classification) or linear activation functions (for regression). The typical MLP architectures are not "deep", i.e., we don't have many hidden layers. You usually have, say, 1 to 5 hidden layers. These neural networks were common in the '80, and are trained by backpropagation.
Now, with Deep Neural Network we mean a network which has many layers (19, 22, 152,...even > 1200, though that admittedly is very extreme). Note that


*

*we haven't specified the architecture of the network, so this could be feed-forward, recurrent, etc.

*we haven't specified the nature of the connections, so we could have fully connected layers, convolutional layers, recurrence, etc.

*"many" layers admittedly is not a rigorous definition.


So, why does it still make sense to speak of DNNs (apart from hype reasons)? Because when you start stacking more and more layers, you actually need to use new techniques (new activation functions, new kind of layers, new optimization strategies...even new hardware) to be able to 1) train your model and 2) make it generalize on new cases. For example, suppose you take a classical MLP for 10-class classification, tanh activation functions, input & hidden layers with 32 units each and output layer with 10 softmax units $\Rightarrow 32\times32+32\times10 = 1344$ weights. You add 10 layers $\Rightarrow 11584$ weights. This is a minuscule NN by today's standards. However, when you go on to train it on a suitably large data set, you find that the convergence rate has slowed down tremendously. This is not only due to the larger number of weights, but to the vanishing gradient problem - back-propagation computes the gradient of the loss function by multiplying errors across each layers, and these small numbers become exponentially smaller the more layers you add. Thus, the errors don't propagate (or propagate very slowly) down your network, and it looks like the error on the training set stops decreasing with training epochs.
And this was a small network - the deep Convolutional Neural Networks called AlexNet had 5 layers but 60 millions weights, and it's considered small by today's standards! When you have so many weights, then any data set is "small" - even ImageNet, a data set of images used for classification, has "only" about 1 million images, thus the risk of overfitting is much larger than for shallow network.
Deep Learning can thus be understood as the set of tools which are used in practice to train neural networks with a large number of layers and weights, achieving low generalization error. This task poses more challenges than for smaller networks. You can definitely build a Deep Multilayer Perceptron and train it - but (apart from the fact that it's not the optimal architecture for many tasks where Deep Learning is used today) you will probably use tools which are different from those used when networks used to be "shallow". For example, you may prefer ReLU activation units to sigmoid or tanh, because they soften the vanishing gradient problem.
A: I wanna add that according to what I have read from many posts : 
There are many different architecture through DNN like : MLPs (Multi-Layer Perceptron) and CNNs (Convolutional Neural Networks).So different type of DNN designed to solve different types of problems. 
MLPs is classical type of NN which is used for :


*

*Tabular Data-sets (contain data in a columnar format as in a database table ).

*Classification / Regression , prediction pbs.


MLPs are very and can be used generally to lean mapping from in put to outputs.
But you can try for other format like image data as base line point of comparison
to confirm that other models are more suitable.
CNNs designed to map image data to an output variable. it's used for :


*

*Image data,

*classification/Regression prediction pbs,


It work well with data that has Spacial relationships.
It's traditionally used for 2D data but it can be used for 1D data, CNNs achieves the state of the art on some 1D pbs.
You have first to "define clearly" what you aim to solve as problem (what kind of data to work with, classification/regression problem ...etc) to know which type of architecture to use. 
You can refer to those links that have been so useful to me to understand more about those concepts :).


*

*MLPS.

*CNNs.

*When to Use MLP, CNN, and RNN Neural Networks.
Hope this add will be useful :p.
