2
$\begingroup$

I am trying to utilize LOOCV in the data partition in R. The idea of LOOCV is to train the model on n-1 set and test the model on the only remaining one set. Then, is to repeat this process n times

Now suppose that I am dealing with KNN. That means on each repetition of LOOCV, I will get the Confusion Matrix to assess my model, which I want.

Since the dimension of my dataset is (569) (32). It means by using LOOCV, I will have 569 the confusion matrix as a total. The reason for calculating the confusion matrix is to find the number of correct class.

Am I right or I have something wrong?

If I am right. How to assess the model?

Improve this question
$\endgroup$
  • 3
    $\begingroup$ Note that Leave-One-Out Cross-Validation (LOOCV) can be viewed as a special case of "$k$-fold CV" where $k = n = \text{sample size}$. Further, for this special case there often exist efficient closed-form expressions. So, always look for those first, instead of trying to re-invent the wheel yourself. $\endgroup$ – Jim Jul 4 '18 at 15:03
  • 1
    $\begingroup$ loocv is discouraged by basically any paper that compares different resampling methods. Are you sure you want to use it? $\endgroup$ – rep_ho Jul 5 '18 at 11:40
  • 1
    $\begingroup$ @rep_ho, yes, the first reason is KNN with different distances, not the default one. The second reason is to replicate some results and to make sure about them. I will also try to do k-fold cross-validation. $\endgroup$ – jeza Jul 5 '18 at 12:31
4
$\begingroup$

Each one of the 569 leave-one-out CV's will create 1 prediction, e.g. P(+) = 0.43. Then you need to apply a threshold to this probability value which will binarise it to 0 or 1. You then compare this binary prediction with the actual label, as a result of which your prediction for that fold will result in one of {TP, TN, FP, FN}. Since you have only one prediction per fold, constructing a confusion matrix does not sound intuitive.

You can calculate the mean accuracy, mean recall, and mean other confusion matrix based metrics by taking the average of all LOOCV results, but for the reasons explained above, constructing a confusion matrix per fold does not make much sense.

As for the merits of LOOCV and what to be aware of, I won't repeat what is given in this link, which I think is detailed.

Improve this answer
$\endgroup$
  • 1
    $\begingroup$ I am calculating the confusion matrix because I want to find the correct number of class. $\endgroup$ – jeza Jul 4 '18 at 16:29
3
$\begingroup$

I'll post two links that might be useful.

link 1 link 2

It means by using LOOCV, I will have 569 Confusion Matrix as a total.

Technically yes, but an already built-in function like knn.cv would take care of that and show you the mean performance over the "global" LOO. So you'll see only one performance.

Improve this answer
$\endgroup$
  • 1
    $\begingroup$ I do not use any built function because of the Euclidean distances is the default. For example, I want to use L_0.1 norm. $\endgroup$ – jeza Jul 4 '18 at 16:28
  • 2
    $\begingroup$ Build your confusion matrix on all the observation you leave out. Repeat n times, the process, this will force a new "test" set each time, compute the confusion matrix (or one appropriate metric for model evaluation). You'll end up with n values, pick the best, that's your optimal model. $\endgroup$ – RLave Jul 5 '18 at 6:48

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.