How large number of parameters with small datasets leads to overfitting?

Question

If you have the large number of parameters in your model and have very small datasets , it must leads toward overfitting , I want to know that what is the reason it is leading towards it , because there must be too complex functions if you have large number of parameters ?

Answer 1

First, your statement

If you have the large number of parameters in your model and have very small datasets , it must leads toward overfitting

Is incorrect.

We can fit a model with large number of parameters without overfitting. Just add strong regularization (note, there are many parameters just for regularization).

Here is a simple example: we can fit few points with high order polynomial without overfitting. And the number of parameters can be even more than number of data points. Just make sure the regularization $\lambda$ is big enough. In following example, we are fitting 5 data points with 4th order polynomial, but the output can like a line.

library(glmnet)
set.seed(0)
x = runif(5)
y = runif(5)
par(cex=1.5)
plot(x,y)
grid()

A=matrix(cbind(1,as.matrix(poly(x,4))),ncol=5)
fit = glmnet(A,y,lambda=0.15)

x_new  = seq(0,1,0.01)
A_new =  matrix(cbind(1,as.matrix(poly(x_new,4))),ncol=5)
lines(x_new,predict(fit,A_new),type='l',col='red')

---

To you question, if we have large number of parameters and not enough regularization, it may leads overfitting.

You can think in this way: overfitting means the model is too specific for the training data set and has poor generalization.

In fact we can easily build such model with a hash map: just hardly memorize all the x,y pairs. In this case, "number of the parameters" is equal to number of data points. Such model will have many parameters and you can easily see why it is overfitting.

On the other hand, suppose we require the model can only have only 1 parameter. In such case, we may use the average value of response variable as the model parameter / output. You can also see, why this will lead to under-fitting.