# RMSE jumps around - is this a sign of a greater problem?

I am building several models using GBM.

My first 2 models (happened to be the same response variable) fit very nicely.

My last 2 models (different than the first 2 models and from each other) seem to be fitting somewhat okay but I am concerned about my model because the RMSE seems to be very "jumpy".

To give you an overview of whats going on, I have divided my data into training and testing sets (80/20). As I tune the model, the test and training RMSE decreases as I would expect. However, I get to certain point (what some would call the optimal model - where test RMSE is at its lowest), where the test RMSE reverses direction and starts to increase, while the training RMSE continually decreases.

First question: Am I incorrect in saying that the model is now starting to overfit?

Moving on, as I continue to fit more models, the training RMSE continues to decrease but the test RMSE starts to jump around. It starts to go back up and then will jump below the "optimal" RMSE and then slowly increase, then another jump (sometimes RMSE will increase and sometimes decrease - it seems random). See screenshot below.

Second question: Is the RMSE jumping around a problem I should worry about? What could be causing this, and is there a solution I can try?

I hope this is enough information - if not let me know and I'll see what more I can contribute.

• I see consistency in RMSE to 2 d.p. Sounds stable to me. I don't think I've ever felt able to discuss differences of that magnitude. – Nick Cox Sep 20 '16 at 19:02
• @NickCox The RMSE is these model are decreasing very slowly by 0.00X or even 0.000X. Do think that is still stable? Thanks. – Danib90 Sep 20 '16 at 20:11
• I don't have specific experience with GBMs. I am just reacting to the idea of jumping around. – Nick Cox Sep 20 '16 at 20:33
• Your intuition about overfitting is probably correct but I do not understand what you mean by "Moving on, as I continue to fit more models..." Didn't you stop modelling after you reached your optimal RMSE from your previous step? In general, I have the feeling you are using your test data as validation data so I am not completely certain your approach is correct. Which program are you using for modelling? The R package caret has some great cross-validation convenience training functions. – usεr11852 Sep 20 '16 at 21:28
• @usεr11852 Sorry for the terminology, I meant as I put more trees into the model. I am using the gbm package in R, while also verifying in caret that I get similar parameters (number of trees, shrinkage, depth). The problem I find with caret, and why I'm not using it, is that the train function doesn't seem to deal with missing data the same as gbm. I am using the test set more as a validation set to find the optimal parameters. After, I use the whole dataset with 10-fold CV for the final model. What approach would you use? – Danib90 Sep 21 '16 at 10:47