I was playing around with polynomial regression and the idea of overfitting. So I decided to approximate $\sin(x)$ between $-\pi$ and $\pi$ through a polynomial function of x. All fairly standard stuff. I was able to get better and better fit from polynomial degrees 3 to polynomial degree 13.

Fit for polynomial of degree 3

enter image description here

Fit for polynomial of degree 13

enter image description here

However, when I tried fitting a polynomial of degree 15, the fit became poorer. This is counter-intuitive to me.

Fit for polynomial of degree 15

enter image description here

For polynomials of higher degree, the fit became poorer and poorer. I was under the impression that as long as I keep increasing the degree of the polynomial, I'll keep getting better and better fit with the in-sample data. Can someone please tell me if that's not true? And if it's not true, what is the reason?

  • 2
    $\begingroup$ How are you fitting the polynomial? Specifically are you evaluating the squared error only at specific points or are you computing the area (sqaured) between the two curves using the integral? Because if you do the former there's nothing to penalise the high order polynomial between the points you evaluate at. $\endgroup$ – David Waterworth Oct 17 '19 at 5:27
  • $\begingroup$ You have pushed this procedure far past its breaking point. In the last graphic, the fit and the function agree at 13 places, suggesting you are using just 13 points to fit a polynomial. Starting with degree 12, you can make the polynomial fit perfectly. Beyond degree 12, the very same solution will work, but the additional flexibility afforded by the new high-power terms means an entire manifold of perfect-fitting solutions is available. The fit to the data remains perfect, but the procedure doesn't know about the sine function and has no obligation to conform to it. $\endgroup$ – whuber Oct 17 '19 at 10:41
  • $\begingroup$ I’ll add more details and my code when I am at a computer later. I am generating 400 points between -$\pi$ and $\pi$. Then I use least squares to fit a polynomial of degree n to the curve. The procedure works well for n = 13 but fails for n = 15. $\endgroup$ – ragster Oct 18 '19 at 15:24

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Browse other questions tagged or ask your own question.