2
$\begingroup$

In a binary response setting (data matrix D with N rows) I have performed LOOCV and obtained a final lambda*. The average CV error for this lambda* is also, as I understand it, an unbiased estimator for my out-of-sample error. I use this to train my final model using all the data. There are three pathways I can take to generate an ROC. Now I've only seen where we can estimate the error, not the ROC or AUC, in an unbiased manner. So I'm just not sure which ROC or AUC is the real out-of-sample character. Typical customers for the model have no intuition to translate out-of-sample error (deviance) into performance. They do, however, like to look at the ROC and AUC (AUC is just another summary statistic).

The first is to use the final model with all the data to get the scores. I think this may be the in-sample-ROC (and hence the in-sample AUC). This takes only N*M (N x number of lambda candidates) training events.

The second is to get the final model as I did above and go back to each LOO data set, say D(i), and retrain with the final model lambda* and calculate the score for x(i), the left out row. This is sort of an out-of-sample-ROC. This takes N*(M + 1)

The third is to get the final model as above and go back through the LOO data sets, D(i), and find the optimal lambda*(i) using something like K-fold CV and use this to train on D(i) and then calculate the score for x(i). This would seem to be yet another version of the out-of-sample ROC. This takes NMK training events.

I'm not sure what to call these three ROC curves or if there is another standard way to generate an in-sample ROC estimate and an out-of-sample ROC estimate.

Improve this question
$\endgroup$
3
  • 1
    $\begingroup$ What is lambda*? What is final lambda*, and how did you obtain/finalize it? $\endgroup$
    – cbeleites unhappy with SX
    Dec 19, 2018 at 16:51
  • 1
    $\begingroup$ Sorry about that. Lambda is the tuning parameter, say for LASSO. When we look at all candidate values for the tuning parameter, lambda, we calculate the average error for either K-fold CV or at the extreme, LOOCV. The lambda corresponding to the minimum CV error is selected as the final tuning parameter, lambda*. This is then used to train all the data to obtain the final model...in the case of LASSO, the final coefficients. $\endgroup$
    – Gene
    Dec 19, 2018 at 19:15
  • 1
    $\begingroup$ That's what I suspected. Will answer accordingly. $\endgroup$
    – cbeleites unhappy with SX
    Dec 19, 2018 at 19:22

1 Answer 1

Reset to default
3
$\begingroup$

The average CV error for this lambda* is also, as I understand it, an unbiased estimator for my out-of-sample error.

No. It is an optimistically biased estimate.

To get an unbiased* estimate of out-of-training error, you need to wrap your whole training procedure (including the optimization of λ) in another independent cross validation. See nested (aka double) cross validation.

Now there are 2 ways/approaches to get from nested cross validation to the final model.:

  1. Use the outer cross validation to check/ensure the λ* found for each of the outer surrogate models is the same (read: sufficiently similar) to take this as the λ* for the final model.
    This seems to be what you are planning. However, you'll then need to specify (beforehand!) what variability for λ* is acceptable.
  2. You can treat the inner cross validation (i.e. all you've done so far) as training method. You then say that the out-of-sample performance obtained by doing cross validation for this training method on the data set at hand is just what your outer cross validation loop measures.
    From that point of view, you'd run the λ* optimization again during training on the whole data set. (This is the point of view I prefer - it saves the difficulty what to do if λ* variability in the outer cross validation is just outside your specified target - while you can still evaluate and interpret this variability).
    From that point of view, you have now basically finished training the final model, and still need to do the measurement of out-of-sample error.

ROC /AuROC with cross validation

Again 2 possibilities:

  • You can always calculate separate ROCs for each of the cross valiation surrogate models. For LOO, however, they'd be too crude to be useful: only one point besides the (1/0) and (0/1) trivial points.
  • If your models are set up in a way that the predicted scores are on the same scale for all surrogate models, you can pool the predicted scores just like you pool the dichotomized predictions for calculating cross validation error. This gives you one ROC.

See my answer here for more details and a picture

The first is to use the final model with all the data to get the scores. I think this may be the in-sample-ROC (and hence the in-sample AUC). This takes only N*M (N x number of lambda candidates) training events.

Yes, that's a training error estimate.

The second is to get the final model as I did above and go back to each LOO data set, say D(i), and retrain with the final model lambda* and calculate the score for x(i), the left out row. This is sort of an out-of-sample-ROC. This takes N*(M + 1)

That's also a training error estimate (although it is a common error to mistake this for a generalization error): D(i) has been used to determine λ*, so it is not independent of the model.

The third is to get the final model as above and go back through the LOO data sets, D(i), and find the optimal lambda*(i) using something like K-fold CV and use this to train on D(i) and then calculate the score for x(i). This would seem to be yet another version of the out-of-sample ROC. This takes NMK training events.

If I got you correctly, that's the nested cross validation I've been talking about :-) => do this.

Note that the computation may be drastically reduced:

  • LOO doesn't pay well for the increased computational effort over k-fold cross validation (unless N is so small that anyways only 1 or 2 samples can be left out without totally changing the model)
    k-fold with k between maybe 5 and 10 is usually the way to go, if you can spend some more computation, it may be better spent on iterations/repetitions of k-fold in order to check stability - in particular as stability of the solution would be an important criterium when validating the optimization of λ.
  • You may want to have a closer look at LASSO λ optimization, e.g. Tibshirani and Taylor: The Solution Path of the Generalized Lasso (Disclaimer: haven't read or used that, it's just a hint I have in the back of my mind - so take with a grain of salt)

* slightly pessimistically biased

Improve this answer
$\endgroup$
5
  • 1
    $\begingroup$ I have a quick question regarding final model selection using nested cross validation. How I usually do the final model fitting is to repeat the inner validation process on the entire dataset to find the hyperparameters and then set these hyperparameters that optimize the objective function as the hyperparameters to use when fitting the final model on the entire dataset. Are there any concerns, however, with changes in the optimal hyperparameters due to the extra data being used in the final, selected model? For example, in the xgboost algorithm, if one uses early stopping rounds it is... $\endgroup$
    – aranglol
    Dec 19, 2018 at 20:54
  • 1
    $\begingroup$ ...common that the estimated rounds to use will be too low when fitting the model with more data so I have seen people multiply the final rounds found by some arbitrary factor like 1.10. $\endgroup$
    – aranglol
    Dec 19, 2018 at 20:56
  • 1
    $\begingroup$ I don't see any concern running the same optimization (I wouldn't change its parameters though!) for the final model. In fact that is what approach no. 2 suggests to do. Yes, the additional data may allow for a somewhat more complex model. But then we assume the cross validation models to be essentially equivalent to the model on the whole data and this assumption should IMHO be used consistently. A 2nd reason is that the usually employed "take the hyperparameter where inner CV looks best" rule can already cause overfitting in itself. $\endgroup$
    – cbeleites unhappy with SX
    Dec 19, 2018 at 22:07
  • 1
    $\begingroup$ I appreciate the thoughtful answer. Let me validate my understanding. I use K-fold on the whole data set, D (of size N), to get lambda*. This is the λ (grid size M) with the smallest error. Then train on D using λ* for the final model parameters and scores. I should not use the scores from this model to make an ROC/AUC, it is optimistic. For OOS error (OOSE) we repeat the same procedure on every LOO D(i). Each D(i) gives λ*(i), the score s(i) and prediction error E(i) for the left out row. OOSE = average(E(i)*) and ROC comes from {s(i)}. Also check distribution of {λ*(i)} against λ* $\endgroup$
    – Gene
    Dec 20, 2018 at 16:31
  • 1
    $\begingroup$ yes, that's it. Conceptually, I find it easier to wrap the finding of λ* together with the [low level] training using this λ* for any given dataset (D or D(i)) into a high-level training function. I can then validate the output of this traning function like any other simple (= no hyperparameter optimization) training function. And inside that "black box" I can do whatever I like on the data that is handed to this function without compromising independence of the validation. This isn't doing anything differently, it is just looking at the same procedure in a different way. $\endgroup$
    – cbeleites unhappy with SX
    Dec 20, 2018 at 17:23

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge that you have read and understand our privacy policy and code of conduct.

Not the answer you're looking for? Browse other questions tagged or ask your own question.