0
$\begingroup$

I'm analyzing the performance of 3 parameterizations of the same algorithm under the same dataset. This performance is user-evaluated under a single Likert-type item (5-values). Given the ordinal nature of this data, I'm using a non-parametric test.

I'm trying to address the question: are there statistically significant preferences between the different (3) parameterizations?

For the general analysis, should I use a repeated-measures test (e.g. Friedman Test) or an independent test (e.g. Kruskal-Wallis Test)?

Furthermore, in the presence of a statistically significant difference among the groups, what post-hoc should I perform?

Improve this question
$\endgroup$
2
  • $\begingroup$ What is your research question? $\endgroup$
    – user2974951
    Commented Jul 17, 2019 at 13:34
  • $\begingroup$ @user2974951I just updated the question (the research question is "are there statistical significant preferences between the different (3) parameterisations?". Thanks. $\endgroup$
    – MrT77
    Commented Jul 17, 2019 at 13:46

1 Answer 1

Reset to default
1
$\begingroup$

Since you are evaluating more than two treatments (parametrizations of your algorithm), you should use a repeated measures test blocked by subject, i.e., Friedman's test.

For a post-hoc procedure, I (and several studies in the literature) usually perform a Nemenyi's Test.

You can read some good papers in the literature that discuss how to choose a statistical test, see this and this.

Improve this answer
$\endgroup$
6
  • $\begingroup$ Thanks for the answer and references. However, I still have onde major, as I've seen similar studies (same domain, very similar problem) to use Kruskal-Wallis followed by post-hoc Wilcoxon signed-rank. In my study the variable of interest is ordinal: does this alter any of the assumptions, and may justify the use of (unpaired) omnibus test (Kruskal-Wallis) followed by paired post-hoc test (Wilcoxon Signed-Rank)? $\endgroup$
    – MrT77
    Commented Jul 22, 2019 at 15:03
  • 1
    $\begingroup$ There is no problem if your variable is ordinal. The Kruskal-Wallis test should be used if you have repetitions of your measurement. However, you are evaluating several different treatments (each parametrization of your algorithm is a treatment). Thus, your results table blocked by algorithms, which can be correctly evaluated through the Fridman's test. See calcscience.uwe.ac.uk/w2/am/Ch12/12_5_KWFriedman.pdf and stats.stackexchange.com/questions/12030/… $\endgroup$
    – Iago Carvalho
    Commented Jul 22, 2019 at 15:11
  • $\begingroup$ Sorry, in your answer you said "Since you are evaluating more than two treatments (parametrizations of your algorithm), you should use a repeated measures test, i.e., Friedman's test.", but now you say "The Kruskal-Wallis test should be used if you have repetitions of your measurement.". Could you please explain? $\endgroup$
    – MrT77
    Commented Jul 23, 2019 at 10:49
  • 1
    $\begingroup$ I'm sorry for my incomplete answer. Both KW and Friedman's are repeated measures tests. However, you need a test which is blocked by subject, which is the case of Friedman's test. Therefore, I edited my answer just to include this sentence. $\endgroup$
    – Iago Carvalho
    Commented Jul 23, 2019 at 12:45
  • 1
    $\begingroup$ Yeah. The first factor is the rows, i.e., the different instances or datasets used in your experiment. The second factor is the subjects (treatments, algorithms), which are displayed in each column. KW considers only one factor (just the rows), assuming that the values within each row are repreated measures of your data. On the other hand, Friedman considers each column as another factor (blocks), thus being a two factors test. $\endgroup$
    – Iago Carvalho
    Commented Jul 23, 2019 at 13:56

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.