0
$\begingroup$

I was trying to show that in polynomial regression, the model overfits the data when the degree of polynomial $k$ increases. To demonstrate this, I had 30 2D datapoints and $k = 1,\dots,18$. I plotted the MSE training error against $k$.

However the plot shows that the training error kept decreasing until $k=10$, after which point the error rate goes up and down randomly.

From this answer I realise the problem might be that my design matrix is ill-conditioned so the solution becomes numerically unstable.

How can I get around this problem?

Thanks.

Improve this question
$\endgroup$
5
  • 1
    $\begingroup$ Have you used orthogonal polynomials? $\endgroup$
    – Dave
    Commented Nov 6, 2019 at 16:43
  • $\begingroup$ @Dave Hi, I haven't used them. In fact I don't think I learned that in class. $\endgroup$
    – FrankieYin
    Commented Nov 6, 2019 at 16:49
  • 1
    $\begingroup$ I found two links on CV: stats.stackexchange.com/questions/241703/… and stats.stackexchange.com/questions/258307/…. My suspicion is that the issue is numerical instability. $\endgroup$
    – Dave
    Commented Nov 6, 2019 at 17:09
  • $\begingroup$ That's a truly eluding issue that is not directly obvious unless you delve into the hairy math behind these models. The ill-conditioned matrix is a suspect, since you could end up with more parameters than observations if you are using interaction parameters too (i.e. $x^5y^5, x^4y^6, x^3y^7$ ...and so on). Notice also that more parameters does not necessarily mean higher marginal likelihood, which will result in suddenly underfitting the data after some point. I explained this a bit in the following answer > answer $\endgroup$
    – Ammar Rashed
    Commented Jun 3, 2022 at 0:05
  • $\begingroup$ No "hairy math" is needed. Simply notice that in double precision arithmetic, $1^{10}$ compared to $18^{10}$ is indistinguishable from zero. Your real problem stems from attempting to use such a high-degree polynomial in the first place. $\endgroup$
    – whuber ♦
    Commented Nov 6 at 13:58

0

Reset to default

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Browse other questions tagged or ask your own question.