I cross-validated a model using the classification accuracy using leave-one-out-cross-validation (proportion of correctly classified cases). Below is a matrix of accuracies typical to what I see. The rows and columns represent different combinations of two hyper-parameters that are cross-validated. For lower rows and more right columns, the model becomes more flexible, whereas it is far less flexible in the upper left corner of the matrix.
The accuracies are quite similar. Therefore I am hesitant to simply choose the parameter combination that maximizes the fit (lower right). I am too concerned that I may go wrong in choosing one of the extreme categories, as their fit does not differ much from other options.
How should one choose the model hyper-parameters in this case?
[,1] [,2] [,3] [,4] [,5]
[1,] 0.699 0.673 0.693 0.686 0.693
[2,] 0.706 0.699 0.686 0.686 0.719
[3,] 0.699 0.699 0.686 0.693 0.719
[4,] 0.686 0.699 0.686 0.693 0.732
[5,] 0.693 0.699 0.680 0.706 0.732
[6,] 0.693 0.673 0.693 0.706 0.732