The advantages of recurrent neural network(RNN) over feed-forward neural network (MLP) Assume we use both RNN and MLP for the same task, and each network is well trained. Since RNN uses more information than MLP, theoretically its performance should be better than MLP. Are there any exceptions? If so, why?
 A: For purposes of discussion I'll assume you are using RNN for the typical use case of time series analysis, where the recurrence operation allows response to depend on a time-evolving state; for example the network can now detect changes over time. This is exactly the added capability you'd want a Recurrent Neural Network for, in this example. That part it sounds like you know.
The downside, is it can be much more difficult to train and has multiple issues with convergence. For example, the backpropegation "signal" tends to decay exponentially over "time". The choice of learning algorithm can be more limited. (SGD can't obviously throw out intermediate timesteps without serious modifications).
There are other methods that address some of these issues; for example Long-Short Term Memory, which basically uses a gating approach to build a recurrent circuit that can be "set" or "cleared". 
Even if you aren't talking time series (e.g. Recurrent neural network have also been used with convolutional layers to extend the effective pixel neighborhood.) you will still have similar convergence and exponential backprop decay issues.
A: Theoretically, MLP can approximate any function, to an arbitrary precision, therefore there is no need for RNN. However that doesn't mean it is usable in a wild. Assuming we are talking about time series input, textbook answer would be that you can feed your time series in feed forward network, by having a input layer containg also inputs from previous time points. Therefore effectively transforming time series problem, into feed forward problem. However you will have to choose length of your input beforehand, and you will not be able to learn functions that depends on the inputs happening long time ago. You can solve this problem by having a RNN, that can theoretically, store information from arbitrarily long time ago, in it;s context layer. In practice however, you will have gradient exploding/vanishing problem.
