I need a little bit guidance. I have to compare the classification performance of multiple algorithms using simple or paired t-test.

Let's say I have four datasets (A,B,C) with training and test samples. I am running 3 algorithms (SIFT, SURF, ORB) and compute the classification accuracy such 0.9 means 90% of images correctly match from the test dataset.

Let' say I get the following table:

Dataset (A,B,C,D)

  • SIFT (0.90, 0.84, 0.90,0.45)
  • SURF (0.84, 0.67, 0.45,0.34)
  • ORB (0.34,0.45,0.45,0.23)

Can you please guide me how I can compare the performance of these algorithms using some statistical analysis such as simple t-test?

Any guidance will be really appreciated. Thanks.


I suggest using a paired t-test because the accuracies on different data sets should not be compared directly. Each data set you test on should form a pair in your t test.

Based on your example, you would be doing something like this in R to compare SIFT and SURF:

SIFT <- c(0.90, 0.84, 0.90, 0.45)
SURF <- c(0.84, 0.67, 0.45, 0.34)
SIFT_v_SURF <- t.test(SIFT,SURF,paired=TRUE,alternative="greater")

Note: by using a t-test you are assuming normality which may not be the case.

| cite | improve this answer | |
  • $\begingroup$ what is R ? plz $\endgroup$ – user1388142 Jan 23 '14 at 22:59
  • $\begingroup$ R is a (free) statistical programming package. You can find more information here: r-project.org. $\endgroup$ – Marc Claesen Jan 23 '14 at 23:01
  • $\begingroup$ cay u plz also tell me how to interprete values i.e. how I can claim SURF is better than SIFT . . The output is ! !! ! data: SIFT and SURF t = -2.9785, df = 3, p-value = 0.9707 alternative hypothesis: true difference in means is greater than 0 95 percent confidence interval: -0.6310159 Inf sample estimates: mean of the differences -0.3525 $\endgroup$ – user1388142 Jan 24 '14 at 0:33
  • $\begingroup$ hi . . . marc plz waiting for ur reply $\endgroup$ – user1388142 Jan 28 '14 at 4:06
  • $\begingroup$ A large $p$-value indicates the null hypothesis is not rejected. Based on your results you tested whether the mean in SIFT was not smaller than the mean of SURF. You get a large $p$-value indicating that your results are explained well by the null hypothesis (e.g. they are likely to occur when the null hypothesis is true). Testing the other way around would have been a better choice (since the values for SIFT appear to be larger than those of SURF). Another solution would have been to use a two-tailed t test. $\endgroup$ – Marc Claesen Jan 28 '14 at 9:12

The t-test is for comparing 2 groups (or one group to a theoretical value). With 3 groups (tests) you would need ANOVA and since there is blocking (the generalization of pairing) due to the different datasets you would be using randomized block ANOVA or a mixed effects model.

However, these methods depend on approximate normality and with the nature of your data, it is not likely to be approximately normal and your sample size is not large enough to invoke the CLT. A permutation test is probably your best option given your data.

Here is R code for one possible way to do a permutation test:

SIFT <- c(0.90, 0.84, 0.90, 0.45)
SURF <- c(0.84, 0.67, 0.45, 0.34)
ORB <- c(0.34, 0.45, 0.45, 0.23)

tmpdat <- rbind( SIFT, SURF, ORB )

tmpfun <- function(m) diff( range( rowMeans(m) ) )

out <- c( tmpfun(tmpdat), 
    replicate( 9999, tmpfun( apply(tmpdat, 2, sample) ) ) )
mean( out >= out[1] )
| cite | improve this answer | |
  • $\begingroup$ do u know any softwares which I can use to do that let's say some snippet in excel or tool ? $\endgroup$ – user1388142 Jan 23 '14 at 23:05
  • $\begingroup$ hi thanks a lot. just one question i am not a statistics guy. I have seen the histogram? How I can infer which one is best from such kind of histogram? plz ur guidance will be really nice $\endgroup$ – user1388142 Jan 24 '14 at 1:42
  • $\begingroup$ I haven't worked it out for this case, but with few observations and the existence of several ties, the achievable significance levels on a permutation test may be somewhat limited. $\endgroup$ – Glen_b Jan 24 '14 at 5:56
  • $\begingroup$ Further, if data are dependent (akin to "paired") the number of available combinations is further reduced. $\endgroup$ – Glen_b Jan 24 '14 at 6:29

Usually you don't summarise since performance of specific algorithm is related to the characteristics of the specific dataset. In literature you see phrases like "Algorithm X won in 5 out of 10 datasets".

However, if numbers are correct, in your case there is a clear winner and that's SIFT: it beats all other algorithms in all datasets.

| cite | improve this answer | |
  • $\begingroup$ u r right but i have been asked to use it anyway some sort of examiner comments on my paper. $\endgroup$ – user1388142 Jan 23 '14 at 23:09
  • $\begingroup$ I raised a question here because I really want to know if this a correct approach from a statistical point of view. $\endgroup$ – iliasfl Jan 23 '14 at 23:57

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.