1
$\begingroup$

I want to estimated residuals (actual - predicted) within a k-fold cross-validation scheme (i.e. predicted residuals) in a regression problem.

The aim is to get a reasonable estimate as the data is high dimensional with p (~ 1000) >> n (< 100). If I fit a model (I am using LASSO) on the whole data then it over-fits and the residuals are close to 0. So I thought using k-fold CV can help, which it does but comes with many questions.

  • What should I use as the value of k? Leave-one-out is over optimistic and thus the residuals will be closer to 0. Should I use the standard 10-fold, how about 5-fold? Fewer folds (e.g. 2) will create bad models due to small training samples and inflate the residuals. Hard to determine a sensible choice of k.
  • Should I repeat the CV procedure and if yes then how to combine the residuals from the repeats? If I just average across repeats then again the residuals will be closer to 0 as it will be something like a bagging and approach LOO in limit.

Any suggestions are welcome. Thanks!

Improve this question
$\endgroup$
7
  • $\begingroup$ just to clarify, by residuals you are referring to MSE/MAE right? Normally residuals refer to observed - predicted $\endgroup$
    – StupidWolf
    Commented Apr 16, 2020 at 10:10
  • $\begingroup$ @StupidWolf I am referring to observed - predicted, edited the question to make it clear! $\endgroup$
    – Krrr
    Commented Apr 16, 2020 at 10:14
  • $\begingroup$ thank you for clarifying.. that's quite a bit of data to collect. If you do cross-validation, you get 1 residual for each data point (regardless of k) because each fold is only predicted once. you need to repeat the CV procedure $\endgroup$
    – StupidWolf
    Commented Apr 16, 2020 at 10:17
  • $\begingroup$ @StupidWolf thanks! Yes I am aware of that but any choice of k and number of repeats is problematic, so the question is which choices are sensible? $\endgroup$
    – Krrr
    Commented Apr 16, 2020 at 16:22
  • $\begingroup$ Not very sure i understand the problem, but naively, you can do one round of k-fold validation to see whether the overfitting (if any starts). i.e just compare mse across different k-fold validation. Not very sure for your data, but hopefully you find something sensible this way $\endgroup$
    – StupidWolf
    Commented Apr 16, 2020 at 16:28

1 Answer 1

Reset to default
1
$\begingroup$

For choice of $k$, see Choice of K in K-fold cross-validation

Repeating:

  • yes, I definitively recommend at least a few repetitions since this allows you to determine stability (e.g. as standard deviation of predictions for the same test case in different repetitions).

    If the variation there is negligible, your models are stable, and further repetitions would not change anything.

    If the variation is substantial, the [surrogate] models are unstable. More repetitions then give a better estimate of the average error of the surrogate models, but that does not necessarily help with estimating generalization error of the model trained on the whole data set since that is likely subject to almost the same instability.

  • Calculating pooled error across residuals from folds and repetitions is not like an out-of-bag error: out-of-bag error would pool/average the predictions for the same case and then calculate one residual (average prediction - refererence) per case.

Improve this answer
$\endgroup$
1
  • $\begingroup$ Thanks for your answer. Perhaps my question is not clear. The selection of k you describe is good for estimating the generalization error but would that be the same for estimating the expected residuals of the data at hand? $\endgroup$
    – Krrr
    Commented Apr 16, 2020 at 19:36

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.