# Interpreting Cross Validation with Multiple Linear Regression

I am using R to develop a multiple linear regression model for some data I have. I do not have a lot of data points (about 30, sorry very hard to collect the data) and am trying different regression models. When I use a very large regression model (see below) I get R2 of 0.974 and adjusted R2 of 0.965. The RMSE is 0.339. When I use 5-fold cross validation the RMSE for the cross validation is 0.584. 10-fold and 2-fold cross validation also give similar larger RMSE values.

How do I interpret this? Does this mean the model is overfitting? Should I aim to have the cross validation RMSE about equal to the full model RMSE?

mobig.fit <- lm(y ~ x1+x2+x3+x1:x2+x2:x3+x1:x3+x1:x2:x3, data=datas)

#Print r-squared values
summary(mobig.fit)$r.squared summary(mobig.fit)$adj.r.squared

#Get RMSE
rss.mobig <- c(crossprod(mobig.fit$residuals)) mse.mobig <- rss.mobig / length(mobig.fit$residuals)
rmse.mobig <- sqrt(mse.mobig)

#Cross validate results
cv.mobig <- CVlm(datas,mobig.fit,m=5,plotit=FALSE)
cv.mobig.rmse <- sqrt(attr(cv.mobig,"ms"))
cat("RMSE for full model: ",rmse.mobig)
cat("RMSE for CV: ",cv.mobig.rmse)

• Is the "non-crossvalidation" RMSE in-sample? – Stephan Kolassa Jul 10 '18 at 13:45
• @StephanKolassa Do you mean is the RMSE value (0.339) from the full data set/full regression model? If so then yes. – che_kid Jul 10 '18 at 13:50
• In-sample RMSE does not tell you anything about predictive accuracy. Don't even bother looking at it. And it's not enlightening to compare in-sample and out-of-bag errors, either. – Stephan Kolassa Jul 10 '18 at 13:54
• Why the downvote? Is this question off-topic? Some explanation would be useful. – che_kid Jul 10 '18 at 15:47