I am looking for the correct statistical test to compare the performance of several algorithms:

Let us assume, we have S subjects and one to M measurements recorded by C cameras (sensors). For each camera, A algorithms were applied to calculate signals which were compared with a reference R. Each algorithm is identified by c categories (factors). For each category, there are more than two groups/levels.

We calculated a measure of similarity (the temporal Coverage which ranges from 0-100 %) for each observation and the reference to have a performance metric to compare the algorithms.

Question 1: I now want to compare the algorithms based on the median values.

  • Null Hypotheses H0: Median algorithm results (temporal coverage) for the population are all the same.
  • Alternative Hypotheses H1: There is a difference in the algorithm results.

For this, I calculated the median values of the algorithms based on all observations (2640).Hence, for each of the 55 algorithms, there are 48 observations. The following plots show violin plots for the temporal coverage. violin plots of algorithms

I think that I have to use a non-parametric test (Friedman?) and that the observations are not independent (i.e. there are observations from the same subject). What is the correct test to be applied (and why)?

Question 2: I also want to determine whether there is a difference in the levels of the categories.

  • Null Hypotheses H0: Median results per level for the population are all the same.
  • Alternative Hypotheses H1: There is a difference in the results per level. Can I apply the same test as in question 1? After the test, can I infer that the level with the highest median value performs best, or do I need to perform further tests?

There is a different number of observations per level. All factors are categorical (except for one category where the factors are ordinal). To give an example for one category, the algorithms can be classified as one of four variants (levels) A1 - A4. The following violin plots show the results for the variants.violin plots variants

  • $\begingroup$ About Q1: Compare algorithms in what sense? Speed, results, ...? Which hypothesis do you want to test? We cannot tell which is correct without knowingb the hypotheses! Maybe show us some plots, sample sizes, ... $\endgroup$ – kjetil b halvorsen May 4 at 1:37

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Browse other questions tagged or ask your own question.