# Polynomial regression underfits data when degree becomes large

I was trying to show that in polynomial regression, the model overfits the data when the degree of polynomial $$k$$ increases. To demonstrate this, I had 30 2D datapoints and $$k = 1,\dots,18$$. I plotted the MSE training error against $$k$$.

However the plot shows that the training error kept decreasing until $$k=10$$, after which point the error rate goes up and down randomly.

From this answer I realise the problem might be that my design matrix is ill-conditioned so the solution becomes numerically unstable.

How can I get around this problem?

Thanks.

• Have you used orthogonal polynomials?
– Dave
Commented Nov 6, 2019 at 16:43
• @Dave Hi, I haven't used them. In fact I don't think I learned that in class. Commented Nov 6, 2019 at 16:49
• I found two links on CV: stats.stackexchange.com/questions/241703/… and stats.stackexchange.com/questions/258307/…. My suspicion is that the issue is numerical instability.
– Dave
Commented Nov 6, 2019 at 17:09
• That's a truly eluding issue that is not directly obvious unless you delve into the hairy math behind these models. The ill-conditioned matrix is a suspect, since you could end up with more parameters than observations if you are using interaction parameters too (i.e. $x^5y^5, x^4y^6, x^3y^7$ ...and so on). Notice also that more parameters does not necessarily mean higher marginal likelihood, which will result in suddenly underfitting the data after some point. I explained this a bit in the following answer > answer Commented Jun 3, 2022 at 0:05