I am trying to make a classifier that effectively distinguishes between control group and patient group, and then I want to use that classifier to distinguish high-risk patients who convert to patient group versus high-risk patients that do not (highrisk-c vs highrisk-nc).
Thus, I have a control-patient dataset as the training set, and the highrisk-c highrisk-nc dataset as the independent test set.
I want to first see the performance (ROC area under the curve) of my classifier on the control-patient dataset, so to avoid overfitting I have used nested 10-fold cross validation and have gotten an overall performance estimate.
However, this means that I do not have one final optimal set of parameters (instead, I had 10 different parameters sets for each of the 10 outer loops..which were obtained from grid search on each internal loop). So, I am confused on what parameters I would use on the independent test set?
1) Do I simply use svmtrain on the whole control-patient dataset and use the model from there to do svmpredict on the high-risk patient dataset?
2) or do I get an optimal parameter from the control-patient dataset through cross validation (not nested) and apply that to the svmpredict on the high-risk patient dataset?
3) or do I select the best hyperparameter from among the external folds (from the nested CV) and use that on the high-risk patients? In this case though, the hyperparameters would have been based off of only a part of the entire training set, so would that be okay? Any comments on this would be greatly appreciated, Thanks.