I'm trying to assess performance of a supervised machine learning classification algorithm. The observations fall into nominal classes (2 for the time being, however I'd like to generalize this to multi-class problems), drawn from a population of 99 subjects.
One of the questions I'd like to be able to answer is, if the algorithm exhibits a significant difference in classification accuracy between the input classes. For the binary classification case I am comparing mean accuracy between the classes across subjects using a paired Wilcoxon test (since the underlying distribution is non-normal). In order to generalize this procedure to multi-class problems I inteded to use a Friedman test.
However, the p values obtained by those two procedures in case of a binary IV vary wildly, with the Wilcoxon test yielding
p < .001 whereas
p = .25 for the Friedman test. This leads me to believe I have a fundamental misunderstanding of the structure of the Friedman test.
Is it not appropriate to use a Friedman test in this case to compare the outcome of the repeated measures of the accuracy across all subjects?
My R code to obtain those results (
subject is the subject identifier,
acc the accuracy DV and
expected the observation class IV):
> head(subject.accuracy, n=10) subject expected acc 1 10 none 0.97826087 2 10 high 0.55319149 3 101 none 1.00000000 4 101 high 0.68085106 5 103 none 0.97826087 6 103 high 1.00000000 7 104 none 1.00000000 8 104 high 0.08510638 9 105 none 0.95121951 10 105 high 1.00000000 > ddply(subject.accuracy, .(expected), summarise, mean.acc = mean(acc), se.acc = sd(acc)/sqrt(length(acc))) expected mean.acc se.acc 1 none 0.9750619 0.00317064 2 high 0.7571259 0.03491149 > wilcox.test(acc ~ expected, subject.accuracy, paired=T) Wilcoxon signed rank test with continuity correction data: acc by expected V = 3125.5, p-value = 0.0003101 alternative hypothesis: true location shift is not equal to 0 > friedman.test(acc ~ expected | subject, subject.accuracy) Friedman rank sum test data: acc and expected and subject Friedman chi-squared = 1.3011, df = 1, p-value = 0.254