I am taking an online Deep learning class from Andrew Ng and it starts with optimising a classifier based on Logistic Regression. During the online assignment there was one paragraph which does not make sense to me:
You can see the cost decreasing. It shows that the parameters are being learned. However, you see that you could train the model even more on the training set. Try to increase the number of iterations in the cell above and rerun the cells. You might see that the training set accuracy goes up, but the test set accuracy goes down. This is called overfitting.
I cannot understand why increasing the number of iterations will result in overfitting? I can understand that increasing model complexity can result in overfitting but cannot understand why increasing the number of gradient descent iterations for the logistic regression cost function can overfit.
Is the statement wrong or have I failed to understand some important concept?