I am using R to develop a multiple linear regression model for some data I have. I do not have a lot of data points (about 30, sorry very hard to collect the data) and am trying different regression models. When I use a very large regression model (see below) I get R2 of 0.974 and adjusted R2 of 0.965. The RMSE is 0.339. When I use 5-fold cross validation the RMSE for the cross validation is 0.584. 10-fold and 2-fold cross validation also give similar larger RMSE values.
How do I interpret this? Does this mean the model is overfitting? Should I aim to have the cross validation RMSE about equal to the full model RMSE?
mobig.fit <- lm(y ~ x1+x2+x3+x1:x2+x2:x3+x1:x3+x1:x2:x3, data=datas) #Print r-squared values summary(mobig.fit)$r.squared summary(mobig.fit)$adj.r.squared #Get RMSE rss.mobig <- c(crossprod(mobig.fit$residuals)) mse.mobig <- rss.mobig / length(mobig.fit$residuals) rmse.mobig <- sqrt(mse.mobig) #Cross validate results cv.mobig <- CVlm(datas,mobig.fit,m=5,plotit=FALSE) cv.mobig.rmse <- sqrt(attr(cv.mobig,"ms")) cat("RMSE for full model: ",rmse.mobig) cat("RMSE for CV: ",cv.mobig.rmse)