6
$\begingroup$

Assume we use both RNN and MLP for the same task, and each network is well trained. Since RNN uses more information than MLP, theoretically its performance should be better than MLP. Are there any exceptions? If so, why?

$\endgroup$
1

2 Answers 2

11
$\begingroup$

Theoretically, MLP can approximate any function, to an arbitrary precision, therefore there is no need for RNN. However that doesn't mean it is usable in a wild. Assuming we are talking about time series input, textbook answer would be that you can feed your time series in feed forward network, by having a input layer containg also inputs from previous time points. Therefore effectively transforming time series problem, into feed forward problem. However you will have to choose length of your input beforehand, and you will not be able to learn functions that depends on the inputs happening long time ago. You can solve this problem by having a RNN, that can theoretically, store information from arbitrarily long time ago, in it;s context layer. In practice however, you will have gradient exploding/vanishing problem.

$\endgroup$
3
  • $\begingroup$ Great answer! Recently I found in my task RNN is slight worse than MLP but the gradient of the params are not exploding/vanishing during training, so I guess there might be some other reasons that hurt RNN's performance. Do you have any suggestions? $\endgroup$
    – fishiwhj
    Nov 9, 2015 at 13:05
  • $\begingroup$ You might be just overfitting/underfitting $\endgroup$
    – rep_ho
    Nov 9, 2015 at 14:42
  • $\begingroup$ When I increased the training batches for RNN, the performance improved. Does that mean underfitting? Is there any other method to avoid it? $\endgroup$
    – fishiwhj
    Nov 10, 2015 at 9:01
3
$\begingroup$

For purposes of discussion I'll assume you are using RNN for the typical use case of time series analysis, where the recurrence operation allows response to depend on a time-evolving state; for example the network can now detect changes over time. This is exactly the added capability you'd want a Recurrent Neural Network for, in this example. That part it sounds like you know.

The downside, is it can be much more difficult to train and has multiple issues with convergence. For example, the backpropegation "signal" tends to decay exponentially over "time". The choice of learning algorithm can be more limited. (SGD can't obviously throw out intermediate timesteps without serious modifications).

There are other methods that address some of these issues; for example Long-Short Term Memory, which basically uses a gating approach to build a recurrent circuit that can be "set" or "cleared".

Even if you aren't talking time series (e.g. Recurrent neural network have also been used with convolutional layers to extend the effective pixel neighborhood.) you will still have similar convergence and exponential backprop decay issues.

$\endgroup$
1
  • $\begingroup$ The hidden units of RNN contains historical information from previous states, while MLP only uses local contexts. That's why I said "uses more information". $\endgroup$
    – fishiwhj
    Nov 4, 2015 at 1:24

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.