I am using R to develop a multiple linear regression model for some data I have. I do not have a lot of data points (about 30, sorry very hard to collect the data) and am trying different regression models. When I use a very large regression model (see below) I get R2 of 0.974 and adjusted R2 of 0.965. The RMSE is 0.339. When I use 5-fold cross validation the RMSE for the cross validation is 0.584. 10-fold and 2-fold cross validation also give similar larger RMSE values.

How do I interpret this? Does this mean the model is overfitting? Should I aim to have the cross validation RMSE about equal to the full model RMSE?

mobig.fit <- lm(y ~ x1+x2+x3+x1:x2+x2:x3+x1:x3+x1:x2:x3, data=datas)

#Print r-squared values

rss.mobig <- c(crossprod(mobig.fit$residuals))
mse.mobig <- rss.mobig / length(mobig.fit$residuals)
rmse.mobig <- sqrt(mse.mobig)

#Cross validate results
cv.mobig <- CVlm(datas,mobig.fit,m=5,plotit=FALSE)
cv.mobig.rmse <- sqrt(attr(cv.mobig,"ms"))
cat("RMSE for full model: ",rmse.mobig)
cat("RMSE for CV: ",cv.mobig.rmse)
  • $\begingroup$ Is the "non-crossvalidation" RMSE in-sample? $\endgroup$ Jul 10 '18 at 13:45
  • $\begingroup$ @StephanKolassa Do you mean is the RMSE value (0.339) from the full data set/full regression model? If so then yes. $\endgroup$
    – che_kid
    Jul 10 '18 at 13:50
  • $\begingroup$ In-sample RMSE does not tell you anything about predictive accuracy. Don't even bother looking at it. And it's not enlightening to compare in-sample and out-of-bag errors, either. $\endgroup$ Jul 10 '18 at 13:54
  • $\begingroup$ Why the downvote? Is this question off-topic? Some explanation would be useful. $\endgroup$
    – che_kid
    Jul 10 '18 at 15:47

It looks like your "full model" is trained on the whole dataset and your RMSE is the error on the training set. This will almost always be lower than the error you'll find when doing a train/test split or CV, since the algorithm was not trained on the data you're using to evaluate RMSE. A high error on the test/CV set can be indicative of overfitting, but you'll virtually always have at least somewhat worse performance on the test set.

The key here is that your "full model" RMSE is a terribly biased evaluation of the model. Intuitively, the model could just "memorize" the training data and spit out pefect predictions for those cases, but it wouldn't generalize well for cases that it hasn't seen (i.e. a held-out test set).

  • $\begingroup$ Thanks. A follow up question is how much worse can the test/CV RMSE be to be considered "unacceptable"? $\endgroup$
    – che_kid
    Jul 10 '18 at 13:47
  • 2
    $\begingroup$ @che_kid That question is highly problem-specific and comes down to the application - is your model good enough to be useful? A cheap medical screening test, for example, might be useful even with a fairly high RMSE, but an expensive confirmatory test should have low RMSE to justify its cost and fit its purpose. $\endgroup$ Jul 10 '18 at 13:52

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.