3
$\begingroup$

I cross-validated a model using the classification accuracy using leave-one-out-cross-validation (proportion of correctly classified cases). Below is a matrix of accuracies typical to what I see. The rows and columns represent different combinations of two hyper-parameters that are cross-validated. For lower rows and more right columns, the model becomes more flexible, whereas it is far less flexible in the upper left corner of the matrix.

The accuracies are quite similar. Therefore I am hesitant to simply choose the parameter combination that maximizes the fit (lower right). I am too concerned that I may go wrong in choosing one of the extreme categories, as their fit does not differ much from other options.

How should one choose the model hyper-parameters in this case?

      [,1]  [,2]  [,3]  [,4]  [,5]
[1,] 0.699 0.673 0.693 0.686 0.693
[2,] 0.706 0.699 0.686 0.686 0.719
[3,] 0.699 0.699 0.686 0.693 0.719
[4,] 0.686 0.699 0.686 0.693 0.732
[5,] 0.693 0.699 0.680 0.706 0.732
[6,] 0.693 0.673 0.693 0.706 0.732
Improve this question
$\endgroup$
6
  • $\begingroup$ Can you please clarify how big is your sample? In general, repeated CV or repeated bootstrap are more common-place than LOOCV. $\endgroup$ – usεr11852 Mar 13 '18 at 21:29
  • $\begingroup$ @usεr11852 It is about n=300. I wanted to use as much data for training as possible, but I am not sure actually whether this is ideal. $\endgroup$ – tomka Mar 13 '18 at 21:31
  • 1
    $\begingroup$ Well, we all do. :) Anyway, see the answer here I think addresses your question quite closely. In short, probably do a 100x 5-fold CV or bootstrap and you are fine. Also you show accuracy (and OK 3% change is not small) but there are other classification measurements that might help you tease out the best model eg. the parameters at [4,5] might be quite better in terms of Cohen's $k$ than the parameters in [6,5]. $\endgroup$ – usεr11852 Mar 13 '18 at 21:39
  • $\begingroup$ @usεr11852 Interesting; I will try the repeated 5-fold CV. However, I am really not sure about the type of fit measure either. There are tons of options. Another one would be AUC for example. It seems they all may suggest a different model is best. In terms of objective I need a model that predicts the underlying class probabilities closely. Classification itself is less crucial. Do you happen to know which fit measure is best for this objective? (This is actually a new question perhaps) $\endgroup$ – tomka Mar 13 '18 at 21:49
  • $\begingroup$ This means you want a well-calibrated model. Yes, in that case one would use a proper scoring rule like Brier score. $\endgroup$ – usεr11852 Mar 14 '18 at 0:07
1
$\begingroup$

That's where Occam's razor comes in: "simpler theories are preferable to more complex ones."

Put into your context: within a similar performance range, the least flexible model is preferable. This makes your model overfit less and thus hopefully more general.

More concretely, this selection is performed using the one standard error rule. Choose the least flexible model within a one standard error range of performance compared to the best model.

The above assumes that you've chosen the performance metric which is adapted to your problem at hand. In addition, leave one out CV has drawbacks on its own but they do not appear to be part of your question.

Improve this answer
$\endgroup$
6
  • $\begingroup$ Good idea. I remember the one standard error rule from the glmnet package (LASSO). It seems useful here too. However I wonder if I can simply apply to it any fit statistic. For example, I could have also used AUC (area under the receiver operating curve) instead of classification accuracy. Would it apply then as well? $\endgroup$ – tomka Mar 13 '18 at 20:52
  • $\begingroup$ I don't think that the metric would matter in this context, though I do not know that for sure. $\endgroup$ – g3o2 Mar 13 '18 at 21:08
  • $\begingroup$ And your concerns about LOOCV? The problem I have is that the sample is not large n=300, so I wanted to use as much data for training as possible. $\endgroup$ – tomka Mar 13 '18 at 21:27
  • $\begingroup$ You can still use 10 fold CV, which uses 270 out of 300 observations for training each run. Regarding the concerns about LOOCV, check this post. $\endgroup$ – g3o2 Mar 13 '18 at 22:32
  • $\begingroup$ Hmm in that link you attach they divide by $\sqrt(K-1)$, but shouldn't this be $\sqrt(K)$? $\endgroup$ – tomka Mar 14 '18 at 9:53

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.