The average CV error for this lambda* is also, as I understand it, an unbiased estimator for my out-of-sample error.
No. It is an optimistically biased estimate.
To get an unbiased* estimate of out-of-training error, you need to wrap your whole training procedure (including the optimization of λ) in another independent cross validation. See nested (aka double) cross validation.
Now there are 2 ways/approaches to get from nested cross validation to the final model.:
- Use the outer cross validation to check/ensure the λ* found for each of the outer surrogate models is the same (read: sufficiently similar) to take this as the λ* for the final model.
This seems to be what you are planning. However, you'll then need to specify (beforehand!) what variability for λ* is acceptable.
- You can treat the inner cross validation (i.e. all you've done so far) as training method. You then say that the out-of-sample performance obtained by doing cross validation for this training method on the data set at hand is just what your outer cross validation loop measures.
From that point of view, you'd run the λ* optimization again during training on the whole data set. (This is the point of view I prefer - it saves the difficulty what to do if λ* variability in the outer cross validation is just outside your specified target - while you can still evaluate and interpret this variability).
From that point of view, you have now basically finished training the final model, and still need to do the measurement of out-of-sample error.
ROC /AuROC with cross validation
Again 2 possibilities:
- You can always calculate separate ROCs for each of the cross valiation surrogate models. For LOO, however, they'd be too crude to be useful: only one point besides the (1/0) and (0/1) trivial points.
- If your models are set up in a way that the predicted scores are on the same scale for all surrogate models, you can pool the predicted scores just like you pool the dichotomized predictions for calculating cross validation error. This gives you one ROC.
See my answer here for more details and a picture
The first is to use the final model with all the data to get the scores. I think this may be the in-sample-ROC (and hence the in-sample AUC). This takes only N*M (N x number of lambda candidates) training events.
Yes, that's a training error estimate.
The second is to get the final model as I did above and go back to each LOO data set, say D(i), and retrain with the final model lambda* and calculate the score for x(i), the left out row. This is sort of an out-of-sample-ROC. This takes N*(M + 1)
That's also a training error estimate (although it is a common error to mistake this for a generalization error): D(i) has been used to determine λ*, so it is not independent of the model.
The third is to get the final model as above and go back through the LOO data sets, D(i), and find the optimal lambda*(i) using something like K-fold CV and use this to train on D(i) and then calculate the score for x(i). This would seem to be yet another version of the out-of-sample ROC. This takes NMK training events.
If I got you correctly, that's the nested cross validation I've been talking about :-) => do this.
Note that the computation may be drastically reduced:
- LOO doesn't pay well for the increased computational effort over k-fold cross validation (unless N is so small that anyways only 1 or 2 samples can be left out without totally changing the model)
k-fold with k between maybe 5 and 10 is usually the way to go, if you can spend some more computation, it may be better spent on iterations/repetitions of k-fold in order to check stability - in particular as stability of the solution would be an important criterium when validating the optimization of λ.
- You may want to have a closer look at LASSO λ optimization, e.g. Tibshirani and Taylor: The Solution Path of the Generalized Lasso (Disclaimer: haven't read or used that, it's just a hint I have in the back of my mind - so take with a grain of salt)
* slightly pessimistically biased