I was playing around with polynomial regression and the idea of overfitting. So I decided to approximate $\sin(x)$ between $-\pi$ and $\pi$ through a polynomial function of x. All fairly standard stuff. I was able to get better and better fit from polynomial degrees 3 to polynomial degree 13.
Fit for polynomial of degree 3
Fit for polynomial of degree 13
However, when I tried fitting a polynomial of degree 15, the fit became poorer. This is counter-intuitive to me.
Fit for polynomial of degree 15
For polynomials of higher degree, the fit became poorer and poorer. I was under the impression that as long as I keep increasing the degree of the polynomial, I'll keep getting better and better fit with the in-sample data. Can someone please tell me if that's not true? And if it's not true, what is the reason?