I am doing KNN classfication on a dataset composed of 1040 instances. I have 40 subejcts each having 26 samples. I want to do a LOSO validation in WEKA. I divided my data in a way that each time a subject is left out for testing and the rest of subjects are used for training the model. that is, I have a training set consisting of samples for all the subjects except subject 1, and a testing set consisting of only subject1. Then I ran the classifier and supplied the test set and re-evaluated the model. After doing all the 40 tests, I found the mean accuracy over all the accuracies, but, unfortunately the result I found does not match the result presented in the base paper. Can anyone help me to handle this?? Thank you.

  • $\begingroup$ I don't think you have shown us enough so that we can be very helpful. As an historical note in the 1960s, Tony Lachenbruch introduced the leave-one-out method in discriminant analysis to greatly reduce the bias in of the resubstitution estimate of the classifier's error rate. In 1983 Brad Efron did simulations to show improvement over the leave-one-out method when comparing it to a variety of bootstrap estimates. The 632 bootstrap appeared to be the best when the features had multivariate Gaussian distributions. $\endgroup$ – Michael Chernick Dec 15 '16 at 0:56
  • $\begingroup$ Subsequently I show in a series of three papers how this extended to a more general set of distributions. Later Efron and Tibshirani produced a refinement to 632 called 632+. $\endgroup$ – Michael Chernick Dec 15 '16 at 0:58
  • $\begingroup$ It seems that it should relate to k nearest neighbor classification but I don't know much about what has been done in the machine learning literature. $\endgroup$ – Michael Chernick Dec 15 '16 at 0:59
  • $\begingroup$ I dont want to do a leave-one-object out cross validation. I want to do leave-one-subject-out one. The difference is that in loo data is divided randomly. And in weka k-fold cross validation k value is set to number of instances. But in leave-one-subject-out data is not partitioned in a random way. Each time the samples about one subject is left out for testing. $\endgroup$ – Niousha Karimi Dec 15 '16 at 1:07
  • $\begingroup$ I don't know WEKA or LOSO. So the objective of classification is to classify subjects instead of observations? $\endgroup$ – Michael Chernick Dec 15 '16 at 1:51

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Browse other questions tagged or ask your own question.