I am learning how to use classification in Matlab and have a question on calculating the training and testing accuracy. I am testing my data on SVM algorithm. I am using crossval and found two ways of calculating the training accuracy.

Method 1 After creating a partitioned model using crossval function using the formula (1 - kfoldLoss(partitionedModel, 'LossFun', 'ClassifError') to calculate the accuracy.

Method 2 Using the trained classifier after crossval to predict the responses for the predictors that I used for training. trainedClassifier.predictFcn(TrainingPredictors) and then using the classperf function to compare the predicted values to my actual responses and then calculating the CorrectRate from the classperf function.

In both cases, I am using same predictors and responses, but getting different accuracy numbers.

My questions:

  1. Why do I get different accuracy from these two methods?
  2. Which one is a correct representation of the training accuracy?
  3. Are there better ways to calculate the training accuracy?

In method 1, you calculate accuracy as 1-CV_Error which is reliable and makes sense. In method 2, you give samples to trained classifier for the second time. The obtained accuracy is not much reliable as the method 1. Of course, you get different accuracy from method 1 and method 2. And I think the first method for caculating accuracy is enough.

Finally, I hope you know that in the k-fold CV procedure we construct k distinct classifiers.

  • $\begingroup$ Thanks for the response. The issue i am running into is that the training accuracy i calculate using method 1 is lower than the testing accuracy when i use method 2 to test the accuracy of the model using a new set of data. It is not intuitive that the testing accuracy can be higher than training accuracy. Is there a different way to test the testing accuracy that will be comparable to method 1? $\endgroup$ – texas12345 Dec 5 '15 at 23:48
  • $\begingroup$ When your training data is not sufficient or your model is not good enough to learn, you will get such result, i.e. training accuracy is lower than testing accuracy on new sets of data. I suggest you play with parameters in your model to tune it, maybe you get better testing accuracy. $\endgroup$ – user4704857 Dec 6 '15 at 7:41

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.