3
$\begingroup$

I was watching professor Andrew Ng's lecture on Logistic Regression and Regularisation. While teaching overfitting professor said that, "If we choose a hypothesis function with high degree then it tends to overfit and does not generalise well. This is because such a polynomial can fit almost any function and we don't have enough data to constraint it to give us a good hypothesis."

My questions are:

  1. Why will a hypothesis function with high degree overfit? What actually is meant by "such a polynomial can fit almost any function and we don't have enough data to constraint it to give us a good hypothesis."?
  2. If this hypothesis function "can fit almost any function" then why does it have high variance? Since it "can fit almost any function" should it not fit unseen data points as well? So why does it fail to generalise?
$\endgroup$
5
  • $\begingroup$ How can you fit to unseen data points if they are unseen? $\endgroup$ Commented Jan 18, 2018 at 17:15
  • $\begingroup$ A polynomial with high degree can fit almost any function so it should be able to fit any random distribution of data point as the distribution too likely to have some function that can be fitted to it. Since the premise is that high degree polynomial can fit almost any function, so, it should fit the new data points as well. $\endgroup$ Commented Jan 18, 2018 at 17:21
  • $\begingroup$ You need to be able to see something to fit to it. $\endgroup$ Commented Jan 18, 2018 at 17:30
  • $\begingroup$ By unseen I meant the data-points that were not on the training set. Once we get a overfit hypothesis function form the training set, we apply the hypothesis function on test set data-points(which are what I called unseen). ML states that this hypothesis function is going to perform poorly on test set(or it has high variance) but according to our premise it can fit almost any function, so it should fit test set data-point well when applied to them. $\endgroup$ Commented Jan 18, 2018 at 17:36
  • 3
    $\begingroup$ Since the algorithm has not seen those data points, there is nothing guiding the fit towards them. Since you are using a flexible class of functions, your argument shows that is is possible for the result to be close to the unseen data, but it does not follow that this is very likely. There are many more ways for the result to be far from the unseen data than for it to he close, so it is in fact very unlikely to happen. $\endgroup$ Commented Jan 18, 2018 at 17:46

1 Answer 1

1
$\begingroup$

One way of thinking about overfitting is consider number of free parameters in the model vs. available data.

Very roughly speaking, we may need ~10 data points per parameter (this number is coming from my own experience...) to get a good estimation.

Now, think about how number of parameters will increase respect to the order of the polynomial. It grows very fast and usually we cannot have enough data for it.

$\endgroup$
2
  • $\begingroup$ Am I wrong somewhere? Does high variance not mean "Low error on training set but high error on test set"? If number of parameters are large for training set data-points shouldn't they be large for test set data-points as well? In such a case, by definition, high degree polynomial should fit test set well and thereby have low error(and thus low variance). $\endgroup$ Commented Jan 18, 2018 at 17:28
  • 1
    $\begingroup$ That's not what high variance means. High variance means that the predictions from the model for a fixed input will undergo large changes when the training data undergoes small changes. A symptom of this is often that the training and testing errors are divergent, but that is not the meaning of the phrase. $\endgroup$ Commented Jan 18, 2018 at 17:31

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.