I am trying to make a classifier that effectively distinguishes between control group and patient group, and then I want to use that classifier to distinguish high-risk patients who convert to patient group versus high-risk patients that do not (highrisk-c vs highrisk-nc).

Thus, I have a control-patient dataset as the training set, and the highrisk-c highrisk-nc dataset as the independent test set.

I want to first see the performance (ROC area under the curve) of my classifier on the control-patient dataset, so to avoid overfitting I have used nested 10-fold cross validation and have gotten an overall performance estimate.

However, this means that I do not have one final optimal set of parameters (instead, I had 10 different parameters sets for each of the 10 outer loops..which were obtained from grid search on each internal loop). So, I am confused on what parameters I would use on the independent test set?

1) Do I simply use svmtrain on the whole control-patient dataset and use the model from there to do svmpredict on the high-risk patient dataset?

2) or do I get an optimal parameter from the control-patient dataset through cross validation (not nested) and apply that to the svmpredict on the high-risk patient dataset?

3) or do I select the best hyperparameter from among the external folds (from the nested CV) and use that on the high-risk patients? In this case though, the hyperparameters would have been based off of only a part of the entire training set, so would that be okay? Any comments on this would be greatly appreciated, Thanks.

  • $\begingroup$ What is the difference between 2) and 3)? If i understand you correctly, for 3) you perform multiple cross-validations (thus the outer loop) as opposed to 2) where you rely on a single crossvalidtion, right? $\endgroup$ – Christopher Schröder Dec 12 '16 at 11:11
  • $\begingroup$ Hi Christopher, yes that is true. In 3), the hyperparameter of each outer fold would be derived from parameter optimization from its inner loop, while in 2), the parameter optimization would take place on the outer loop itself (and no inner loop involved)? $\endgroup$ – Michelle Dec 12 '16 at 16:14
  • $\begingroup$ When I read other papers on a similar topic, they all just vaguely say that a "single classifier built to separate controls and patients was made and its hyperparameters from 20-fold CV was applied to score the high-risk patient group?" , or "only controls and patients were used as training and validating of the classifier, and that high-risk patients were not invovled in neither training nor testing", which I guess implies that like the first paper, they applied some hyperparameters from the control-patient classifier to the high-risk group -- but I am confused about which parameter exactly. $\endgroup$ – Michelle Dec 12 '16 at 16:18

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Browse other questions tagged or ask your own question.