A friend of mine has a confusion about how to validate a proposed classification method while I am not familiar with this topic.

The data has 6 independent columns (A,B,C,D,E,F) and 1 categorical respond (G which accepts only class3 and class4).

The following is a snap shot of her training data set. G is the output column and as you can see it is categorical with 3 and 4 being the correspondent classes .

A MATLAB classifier toolbox and has come up with two classifiers with best accuracies. These classifiers are tested with the testing data set and did some predictions and plotted the confusion matrix for the predictions.

Here is a snapshot of the data

enter image description here

Now the question is whether are these two classifiers really significantly different (need the p-value to prove this e.g. p<0.05 ). I am not sure how to fulfill this. I want to make sure the success of one classifier has not been randomly.

First of all, what tests can be used? (ttest2 in MATLAB ,McNemar's test ( [h,p,e1,e2] = testcholdout() .This is the function) or any other tests which is suitable for this particular problem). If t-test is not Gaussian distributed, is it fine to use t-test?

If ttest2 should be used, what are the inputs of ttest2, as ttest2 accepts x,y and I am not sure what they are .

  • $\begingroup$ If you're using cross validation to estimate accuracy, this thread may be helpful for choosing a statistical test $\endgroup$
    – user20160
    Jun 5 '18 at 6:41
  • $\begingroup$ @user20160, removed cross validation tag. I am not sure what it means in statistics. A friend recommended AUC (area under curve) method. But I am not sure if it is the best criteria. $\endgroup$
    – Adams
    Jun 5 '18 at 7:40

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Browse other questions tagged or ask your own question.