Statistical tests for classification problems

Question

I have a binary classification problem in weka where I get the detailed results as either 1 or 0 for all of the 100 observations. Now I need to perform a statistical test to determine which of the two algorithms have better results. I want to ask which test I need to perform in this case. I can't use Wilcox test because that supports continuous values like 0.3,0.45,1.6 etc

Also adding to my query, if I have to perform baysian test how can I do it? I saw an online calculator for the baysian test but couldn't understand because it have had four different text boxes and not sure what to enter in those boxes.

Answer 1

Let's say you have two algorithms. The first gets 60 out of 100 correct, the other 55 out of 100. You can perform a significance test using a test of proportions.

If you're using R...

>prop.test(c(60,55), c(100,100), correct = F)

    2-sample test for equality of proportions without continuity
    correction

data:  c(60, 55) out of c(100, 100)
X-squared = 0.51151, df = 1, p-value = 0.4745
alternative hypothesis: two.sided
95 percent confidence interval:
 -0.08684704  0.18684704
sample estimates:
prop 1 prop 2 
  0.60   0.55 

You can see that the 95% confidence interval for the difference in proportions spans 0, so we fail to reject the null hypothesis (that both algorithms perform equally well).

If you don't have access to R, there are plenty of examples on the internet. You might want to look for "test of proportions"