use of t-test to compare performance of algorithms I need a little bit guidance. I have to compare the classification performance of multiple algorithms using simple or paired t-test.
Let's say I have four datasets (A,B,C) with training and test samples. I am running 3 algorithms (SIFT, SURF, ORB) and compute the classification accuracy such 0.9 means 90% of images correctly match from the test dataset.
Let' say I get the following table:
Dataset (A,B,C,D)          


*

*SIFT (0.90, 0.84, 0.90,0.45)

*SURF (0.84, 0.67, 0.45,0.34)

*ORB (0.34,0.45,0.45,0.23)


Can you please guide me how I can compare the performance of these algorithms using some statistical analysis such as simple t-test? 
Any guidance will be really appreciated. Thanks.
 A: The t-test is for comparing 2 groups (or one group to a theoretical value).  With 3 groups (tests) you would need ANOVA and since there is blocking (the generalization of pairing) due to the different datasets you would be using randomized block ANOVA or a mixed effects model.
However, these methods depend on approximate normality and with the nature of your data, it is not likely to be approximately normal and your sample size is not large enough to invoke the CLT.  A permutation test is probably your best option given your data.
Here is R code for one possible way to do a permutation test:
SIFT <- c(0.90, 0.84, 0.90, 0.45)
SURF <- c(0.84, 0.67, 0.45, 0.34)
ORB <- c(0.34, 0.45, 0.45, 0.23)

tmpdat <- rbind( SIFT, SURF, ORB )

tmpfun <- function(m) diff( range( rowMeans(m) ) )

out <- c( tmpfun(tmpdat), 
    replicate( 9999, tmpfun( apply(tmpdat, 2, sample) ) ) )
hist(out)
abline(v=out[1])
mean( out >= out[1] )

A: I suggest using a paired t-test because the accuracies on different data sets should not be compared directly. Each data set you test on should form a pair in your t test.
Based on your example, you would be doing something like this in R to compare SIFT and SURF:
SIFT <- c(0.90, 0.84, 0.90, 0.45)
SURF <- c(0.84, 0.67, 0.45, 0.34)
SIFT_v_SURF <- t.test(SIFT,SURF,paired=TRUE,alternative="greater")

Note: by using a t-test you are assuming normality which may not be the case.
A: Usually you don't summarise since performance of specific algorithm is related to the characteristics of the specific dataset. In literature you see phrases like "Algorithm X won in 5 out of 10 datasets".
However, if numbers are correct, in your case there is a clear winner and that's SIFT: it beats all other algorithms in all datasets. 
