What are the benefits of convolutional neural networks structures and recurrent neural network structures compared to the dense feedforward network when there would be enough computing power to represent any arbitrary function?
$\begingroup$
$\endgroup$
Add a comment
|
2 Answers
$\begingroup$
$\endgroup$
0
This is an interesting question, let be just rephrase it a bit differently:
Fully connected (FC) Neural Networks are known to be unifersal function approximators (i.e. they can approximate any function). If we had infinite computation power, would there be any reason to use Convolutional Neural Networks (CNNs) or Recurrent Neural Networks (RNNs)?
Even if we had enough "computing power" and we weren't at all interested in efficiency (i.e. solving the same task quicker with less parameters), there is still the issue that Fully Connected Neural Networks tend to overfit very easily. Actually I answered a similar question the other day on why "CNNs are less prone to overfitting than FC networks". Besides that CNNs have some useful properties relating to images, the most notable is translation invariance (i.e. the network is invariant to translations in the image). This is very useful in image classification where the object that we want to classify can be anywhere in the image.
A similar case can be made for RNNs: Because they can exploit the sequential nature of the data, they can solve problems that wouldn't be possible with FC networks. Through the context of its previous states it can extract information and enhance its performance. This gives it the ability to identify trends and seasonalities in the data, which are impossible to identify in a FC network. Even if we had an arbitrarily-large FC network and we passed it the full sequence in every time step (so that it doesn't miss out on any of the context), that would mean that the RNN requires less information (i.e. fewer features) about that given time step (because they can obtain this information from another time step). This makes less dimensions for the data so less room to overfit.
As a rule of the thumb the more parameters in a model, the more data required to train it without overfitting!
So, generally speaking, if we can get the same work done with fewer parameters, it is advantageous (faster and more efficient computation, less memory required, less prone to overfitting).
$\begingroup$
$\endgroup$
0
CNNs are very sparse compared to dense networks, meaning that there are fewer parameters to estimate. In addition to this being nice for computing times, it reduces the potential to overfit to training data. CNNs don't just drop parameters the way that you can specify in Keras with a dropout command. CNNs are smart about which parameters are forced to be zero, exploiting structure present in the data. Which parameters get dropped by a dropout command in Keras will depend on the particular training data. CNNs will drop the same parameters every time. Further, CNNs force some connections to share parameters, again by being smart and exploiting structure present in the data. Together, these reduce the variability of parameter estimates (some can't vary from 0) to combat overfitting.
Remember that performance on training data isn't interesting. It's all about the ability to generalize to unseen data.
If you've never drawn out what a couple of 2x2 filters do to a 3x3 image, I suggest drawing out what happens. CNNs are typically thought of in terms of putting squares on top of bigger squares. However, they're absolutely neural networks with predictable numbers of parameters and nodes. As an exercise, if you have two 2x2 filters and a 3x3 image, how many nodes are in the next layer, and how many parameters are there? You can check your answer in Keras.