I am applying Lasso regression and the R function glmnet::cv.glmnet() to obtain a prediction model based on 90% of the data. I have set aside 10% as a hold-out set and obtain predicted probabilities for the hold-out set. I repeat this for each 90%-10% split of my data (10-fold cross validation). This gives me a predicted value for each observation in my data s.t. each predicted value is obtained from a model fit using training data where that observation was not included. (This is sometimes called "nested cross validation" to obtain an out-of-sample prediction error estimate.)
I run a repeated 10-fold cross-validation to obtain 5 out-of sample predicted values for each observation. Is it ok to plot the prediction errors corresponding to these 5 vectors against some of the feature variables (for example age) to demonstrate the expected predictive performance of my model for new data corresponding to persons of different ages? This could for example reveal that predictions are more uncertain for old people. I would find this extremely useful. But I am a bit concerned that I may be ignoring some subtle issues similar to those discussed below.
These are some posts discussing important (often neglegted) issues when utilising out-of-sample predicted valued from cross-validation: __ | __ Does cross validation with 10 folds give one e.g. RMSE or 10? __ | __ Appropriate way to get Cross Validated AUC __ | __ Appropriate way to get cross validated performance metrics __ | __ Average ROC for repeated 10-fold cross validation with probability estimates __ | __ The conclusion appears to be that out-of-sample performance metrics (RMSE and in particular AUC) should be averaged from the the 10 metrics calculated separately from the 10 folds and should not be calculated together for the combined set of out-of-sample predicted values. Obviously, for measures like MAE (mean abs error) there is no difference and for RMSE there may not be much difference, but the same does not apply to AUC.
