1
$\begingroup$

I have a dataset $X = \{x_1, ..., x_n\}$. I also have three algorithms, $A_1, A_2, A_3$, that each take a single data point as input and produce some measure of how well they performed. If I apply each of the algorithms to each of the data, I get a matrix of performance measures, $M$, where $M_{a,i}$ is the value of $A_a(x_i)$. This matrix is therefore $3 \times n$.

What statistical methods are most appropriate to analyse the differences in performance of these algorithms and the significance of any differences?

If it matters, the "better" algorithm is that which gives performance measures closer to zero. I'm interested in determining whether any algorithm is significantly better than another.

$\endgroup$
2
  • $\begingroup$ Looks like a [few] simple t-test[s] would work just just fine here $\endgroup$
    – Affine
    Apr 24 '13 at 20:51
  • $\begingroup$ @Affine Are paired t-tests most appropriate here? My reasoning is that the dataset is the same for each algorithm so is probably a repeated measure. $\endgroup$ Apr 24 '13 at 21:26
0
$\begingroup$

Partially answered in comments:

Looks like a [few] simple t-test[s] would work just just fine here – Affine

Are paired t-tests most appropriate here? My reasoning is that the dataset is the same for each algorithm so is probably a repeated measure. – Joseph Mansfield

For both answers above: Since data is duration's, (close to) normal distribution is not a reasonable assumption. And, for some cases of data, different algorithms would have a multiplicative effect, so log transformation before analysis could be indicated! After that, t-tests or some non-parametric analogue.

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.