1
$\begingroup$

I'd like to test if three data sets are statistically similar, but I'm not sure which test is the right one (because the examples on the internet are all very simple). I start with a list of values (1 variable). These values are put through a model (it's basically a function), which returns a new list of the same dimensions, but with transformed values. I do this three times for three diferent models, while the input list is always the same and the output list retains the sequence (output of the first inputted value can be found in the first place). How do I now test if the three models produce significantly similar/different results?

The best possibility I have found so far was the Wilcoxon signed-rank test. Is this test appropriate? I think my problem originates from the lack of exact understanding of repeated measures, paired measurements, when can a certain assumption be violated, etc.- they all seem very similar to me and I can’t be 100% which way to go… :)

Any help would be appreciated and if need be I can provide more information ;).

Improve this question
$\endgroup$
2
  • $\begingroup$ What does "different" mean? If you want to test whether the median differs, the ranksum test is ok. If you want to test variances, or distribution shapes, or means, you need a different test. Also: are you testing whether all three models are the same (in which case you'd need a global test first), or is one your standard model which you compare the others to? $\endgroup$
    – Jonas
    Commented Mar 22, 2016 at 10:16
  • $\begingroup$ I'm almost certain the models (I just simplified it to 3) will not be the same, but I was planning to do an overall test anyway, I just didn't see the need to write about it in the question. So, I'll end up testing them pair by pair and hopefully the models with similar results will show to have some properties in common (the models have characteristics). I think the best thing would be to test the means, testing variances could also work, while medians and distribution shapes would be pointless. $\endgroup$
    – kronos
    Commented Mar 22, 2016 at 13:27

1 Answer 1

Reset to default
1
$\begingroup$

It is not 100% clear what you need, but an overall Friedman test, followed by pairwise Wilcoxon tests, the latter with with multiplicity adjustment (eg with Bonferroni correction), could be a reasonable approach.

Improve this answer
$\endgroup$
1
  • $\begingroup$ Thank you very much for this, it's good to know I didn't completely misunderstand something and made a stupid mistake ;). What you have described was exactly what I was planning on doing, although I'm not yet sure about the Bonferroni correction, because I actually have somewhere between 10 and 20 models (not sure at the moment) and the α in such a case gets tiny. $\endgroup$
    – kronos
    Commented Mar 22, 2016 at 13:36

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.