I am building several models using GBM.
My first 2 models (happened to be the same response variable) fit very nicely.
My last 2 models (different than the first 2 models and from each other) seem to be fitting somewhat okay but I am concerned about my model because the RMSE seems to be very "jumpy".
To give you an overview of whats going on, I have divided my data into training and testing sets (80/20). As I tune the model, the test and training RMSE decreases as I would expect. However, I get to certain point (what some would call the optimal model - where test RMSE is at its lowest), where the test RMSE reverses direction and starts to increase, while the training RMSE continually decreases.
First question: Am I incorrect in saying that the model is now starting to overfit?
Moving on, as I continue to fit more models, the training RMSE continues to decrease but the test RMSE starts to jump around. It starts to go back up and then will jump below the "optimal" RMSE and then slowly increase, then another jump (sometimes RMSE will increase and sometimes decrease - it seems random). See screenshot below.
Second question: Is the RMSE jumping around a problem I should worry about? What could be causing this, and is there a solution I can try?
I hope this is enough information - if not let me know and I'll see what more I can contribute.