# Statistical analysis of multiple algorithms over a single dataset

I have a dataset $X = \{x_1, ..., x_n\}$. I also have three algorithms, $A_1, A_2, A_3$, that each take a single data point as input and produce some measure of how well they performed. If I apply each of the algorithms to each of the data, I get a matrix of performance measures, $M$, where $M_{a,i}$ is the value of $A_a(x_i)$. This matrix is therefore $3 \times n$.

What statistical methods are most appropriate to analyse the differences in performance of these algorithms and the significance of any differences?

If it matters, the "better" algorithm is that which gives performance measures closer to zero. I'm interested in determining whether any algorithm is significantly better than another.

• Looks like a [few] simple t-test[s] would work just just fine here Apr 24 '13 at 20:51
• @Affine Are paired t-tests most appropriate here? My reasoning is that the dataset is the same for each algorithm so is probably a repeated measure. Apr 24 '13 at 21:26