This question is regarding cross validation and prediction with regularized logistic regression, so by parameters here I mean the beta-coefficients for each predictor variable, for output I get predicted probabilities of belonging to a group, and for performance measure I use AUC. I am using glmnet on matlab.
For my training set I have group A (normal group) and group B (patient group), and for my validation set I have group C (high-risk group that later converts to patient group) and group D (high-risk group that later does not convert). I want to take parameters that best differentiate group A and B to predict conversion to patient group among people in group C and D.
To this end, I have done 10-fold cross validation on group A and B to find the best lambda (lambda_1se to be exact), and I am taking the parameters at that lambda value and applying them to Group C and D to get predicted probability values (via cvglmnet and cvglmnetPredict).
Here, it seems that:
- The best lambda is the only thing that will be searched for from the CV, much like hyperparameter optimization that would happen in an inner loop of a nested cross validation.
After finding the best lambda value from CV, my whole training dataset (Group A,B) is retrained to get parameters for that specific lambda value, and those parameters are used to predict outcome in my validation dataset.
Is my understanding correct?
What I would like to do is make a boxplot of predicted probabilities of groups A~D so that I can see the trend of predicted values across those groups (ideally the values would be gradiently descending from patient-> highrisk-later convert -> high risk-not convert -> normal).
Here is my main question:
I have already gotten the predicted probability values from groups C and D, but how do I get the predicted probability values from group A and B (the training set on which I did CV on)?
I first thought of taking the predicted probabilities calculated from the left-out test set in each of the 10 folds. However, my current understanding is that if I were to do that, each of the test sets' predicted probabilities would be based on different lambda values, since each training set in each fold would have had different optimal lambda values.