0
$\begingroup$

In http://cs231n.github.io/convolutional-networks/, it states that a "huge number of parameters would quickly lead to overfitting" in neural networks. I don't think I quite understand this.

The thought that I have is more model parameters = ability to represent more complex functions. So when training, having more model parameters could mean that you're fitting a function to the training data very closely such that it leads to overfitting and loss of generalization. Is this the idea or am I totally off?

$\endgroup$
  • 2
    $\begingroup$ Yes, that's it. $\endgroup$ – Matthew Drury Jan 24 at 15:40
0
$\begingroup$

It is true that having the ability to represent more complicated functions is a necessary condition for learning more complex functions. However the goal of the learning process is to select a good function from the set of all functions the learning machine can represent. If that set is large then the problem of learning is intrinsically harder. This is called the curse of dimensionality in the pattern recognition literature. Not only must statistical regularities exist in the data for discriminating among members of a large set of functions but the learning algorithm needs to be powerful enough to exploit such statistical regularities.

If the learning machine can represent any arbitrary function and the learning algorithm for the machine is capable of learning any arbitrary function then the learning machine will simply memorize the training data set resulting in essentially a fancy way to implement a lookup table!

To prevent memorization and have good generalization performance prior knowledge should be provided to the learning machine that restricts the class of functions the neural net learning machine can learn. Examples of such restrictions are using a linear regression or logistic regression model rather than a nonlinear regression model, having fewer parameters, adding regularization such as L1 or L2.

Finally note that having a lot of parameters does not always mean that the function space is unconstrained. There are many thousands of parameters in deep learning neural networks but such learning machines have a very specific and highly constrained hierarchical and convolutional architecture which greatly restricts the class of functions they can represent.

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.