0
$\begingroup$

I think I must be missing some fundamental part of the logic of cross-validation, or machine learning in general.

Using the caret package in R, I ran a repeated k-fold cross-validation and compared the resulting coefficients to an identical model fit using lm():

trainControl <- trainControl(method = "repeatedcv", number = 10, repeats = 30
fit <- train(Petal.Length ~ ., data = iris, trControl = trainControl, method = "lm")
fit$FinalModel

fit.lm <- lm(Petal.Length ~ ., data = iris)
fit.lm

They are identical, all the way out the the 4th decimal place. Why? I thought cross validation uses resampling to calculate coefficients, get the average, and these will perform better on future data than coefficients from a standard linear model. The point is that the latter coefficients are usually overly optimistic. Am I misunderstanding this process, or did I just use the incorrect code, miss a step or something?

Thank you in advance for your time.

$\endgroup$
0
$\begingroup$

Ah. I realized that I missed the point of cross-validation. Cross-validation, in the context with which I'm using it, is just to estimate out-of-sample prediction error. But with the model fit to the entire dataset.

What I actually wanted were bootstrapped coefficients, which use resampling to estimate bias of the coefficients. Related, but different. This link provides a nice walk-through: http://www.sthda.com/english/articles/38-regression-model-validation/156-bootstrap-resampling-essentials-in-r/#bootstrap-procedure

$\endgroup$
1
  • $\begingroup$ The bootstrap, like cross-validation, is intended to tell you how an estimator performs, not to fix the estimator. The original estimates are as good as the average bootstrap estimates in most cases. Before of overfitting in any case. Be sure to do a bootstrap or repeated cross-validation smooth continuous calibration curve. Details are in the model validation chapter in RMS course notes. $\endgroup$ Jul 25 '21 at 11:19

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.