0
$\begingroup$

I'm analyzing the performance of 3 parameterizations of the same algorithm under the same dataset. This performance is user-evaluated under a single Likert-type item (5-values). Given the ordinal nature of this data, I'm using a non-parametric test.

I'm trying to address the question: are there statistically significant preferences between the different (3) parameterizations?


For the general analysis, should I use a repeated-measures test (e.g. Friedman Test) or an independent test (e.g. Kruskal-Wallis Test)?

Furthermore, in the presence of a statistically significant difference among the groups, what post-hoc should I perform?

$\endgroup$
  • $\begingroup$ What is your research question? $\endgroup$ – user2974951 Jul 17 at 13:34
  • $\begingroup$ @user2974951I just updated the question (the research question is "are there statistical significant preferences between the different (3) parameterisations?". Thanks. $\endgroup$ – MrT77 Jul 17 at 13:46
1
$\begingroup$

Since you are evaluating more than two treatments (parametrizations of your algorithm), you should use a repeated measures test blocked by subject, i.e., Friedman's test.

For a post-hoc procedure, I (and several studies in the literature) usually perform a Nemenyi's Test.


You can read some good papers in the literature that discuss how to choose a statistical test, see this and this.

$\endgroup$
  • $\begingroup$ Thanks for the answer and references. However, I still have onde major, as I've seen similar studies (same domain, very similar problem) to use Kruskal-Wallis followed by post-hoc Wilcoxon signed-rank. In my study the variable of interest is ordinal: does this alter any of the assumptions, and may justify the use of (unpaired) omnibus test (Kruskal-Wallis) followed by paired post-hoc test (Wilcoxon Signed-Rank)? $\endgroup$ – MrT77 Jul 22 at 15:03
  • 1
    $\begingroup$ There is no problem if your variable is ordinal. The Kruskal-Wallis test should be used if you have repetitions of your measurement. However, you are evaluating several different treatments (each parametrization of your algorithm is a treatment). Thus, your results table blocked by algorithms, which can be correctly evaluated through the Fridman's test. See calcscience.uwe.ac.uk/w2/am/Ch12/12_5_KWFriedman.pdf and stats.stackexchange.com/questions/12030/… $\endgroup$ – Iago Carvalho Jul 22 at 15:11
  • $\begingroup$ Sorry, in your answer you said "Since you are evaluating more than two treatments (parametrizations of your algorithm), you should use a repeated measures test, i.e., Friedman's test.", but now you say "The Kruskal-Wallis test should be used if you have repetitions of your measurement.". Could you please explain? $\endgroup$ – MrT77 Jul 23 at 10:49
  • 1
    $\begingroup$ I'm sorry for my incomplete answer. Both KW and Friedman's are repeated measures tests. However, you need a test which is blocked by subject, which is the case of Friedman's test. Therefore, I edited my answer just to include this sentence. $\endgroup$ – Iago Carvalho Jul 23 at 12:45
  • 1
    $\begingroup$ Yeah. The first factor is the rows, i.e., the different instances or datasets used in your experiment. The second factor is the subjects (treatments, algorithms), which are displayed in each column. KW considers only one factor (just the rows), assuming that the values within each row are repreated measures of your data. On the other hand, Friedman considers each column as another factor (blocks), thus being a two factors test. $\endgroup$ – Iago Carvalho Jul 23 at 13:56

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.