# Why do hypothesis functions with high degree or large number of features have high variance(tend to overfit)?

I was watching professor Andrew Ng's lecture on Logistic Regression and Regularisation. While teaching overfitting professor said that, "If we choose a hypothesis function with high degree then it tends to overfit and does not generalise well. This is because such a polynomial can fit almost any function and we don't have enough data to constraint it to give us a good hypothesis."

My questions are:

1. Why will a hypothesis function with high degree overfit? What actually is meant by "such a polynomial can fit almost any function and we don't have enough data to constraint it to give us a good hypothesis."?
2. If this hypothesis function "can fit almost any function" then why does it have high variance? Since it "can fit almost any function" should it not fit unseen data points as well? So why does it fail to generalise?
• How can you fit to unseen data points if they are unseen? Commented Jan 18, 2018 at 17:15
• A polynomial with high degree can fit almost any function so it should be able to fit any random distribution of data point as the distribution too likely to have some function that can be fitted to it. Since the premise is that high degree polynomial can fit almost any function, so, it should fit the new data points as well. Commented Jan 18, 2018 at 17:21
• You need to be able to see something to fit to it. Commented Jan 18, 2018 at 17:30
• By unseen I meant the data-points that were not on the training set. Once we get a overfit hypothesis function form the training set, we apply the hypothesis function on test set data-points(which are what I called unseen). ML states that this hypothesis function is going to perform poorly on test set(or it has high variance) but according to our premise it can fit almost any function, so it should fit test set data-point well when applied to them. Commented Jan 18, 2018 at 17:36
• Since the algorithm has not seen those data points, there is nothing guiding the fit towards them. Since you are using a flexible class of functions, your argument shows that is is possible for the result to be close to the unseen data, but it does not follow that this is very likely. There are many more ways for the result to be far from the unseen data than for it to he close, so it is in fact very unlikely to happen. Commented Jan 18, 2018 at 17:46