I have run some unsupervised domain adaptation algorithms on two data sets. (TJM, JDA, GFK, and ARTL). I read this paper that states that the Wilcoxon test is better than parametric tests. I am trying to figure out how to carry out this test on my experiments. I found a tutorial on Matlab but it speaks of data samples which I did not understand how it relates to classifiers.
$\begingroup$
$\endgroup$
2
-
$\begingroup$ The Wilcoxon signed rank test might be used for comparing two algorithms over many data sets$^*$, rather than many algorithms over two data sets ... $*$(at least, under certain assumptions about how the data sets relate to the notional population of sets that you wish to perform inference over. Are there such assumptions that would make sense though? E.g. a convenience sample would not constitute a random sample of that population. There's also no assignment-to-treatment over which a randomization argument could be used. Is there even a basis on which one could really perform inference here?) $\endgroup$– Glen_bCommented Feb 24, 2018 at 23:02
-
$\begingroup$ @Glen_b I understand. Thanks for clarifying. What test would be optimal for my situation? $\endgroup$– MichaelMMeskhiCommented Feb 25, 2018 at 1:46
Add a comment
|
1 Answer
$\begingroup$
$\endgroup$
5
The Wilcoxon signed rank test might be used for comparing two algorithms over many data sets*, rather than many algorithms over two data sets ...
*(at least, under certain assumptions about how the data sets relate to the notional population of sets that you wish to perform inference over. Are there such assumptions that would make sense though? E.g. a convenience sample would not constitute a random sample of that population. There's also no assignment-to-treatment over which a randomization argument could be used. Is there even a basis on which one could really perform inference here?)
Thanks for clarifying. What test would be optimal for my situation?
What are we optimizing?
On just two data sets?
That's a rather different question to the posted one. I probably wouldn't consider a nonparametric test for that because you can't attain typical significance levels unless there are a lot of algorithms being compared (even if you could overcome the above concerns about whether it makes sense to apply statistical inference at all in this situation). If you plan to apply post hoc testing to identify which pairs of algorithms can be said to differ, you are going to have even greater difficulty attaining reasonable significance levels with only two data sets.
So I think it would come down to choosing a suitable parametric model for the response (e.g. if you're dealing with proportion correct, you might perhaps look at some form of logistic regression model; other scoring mechanisms might need other models). Once there, and with a formal null and alternative, coming up with a reasonable test wouldn't be so difficult.
Edit: With four algorithms and two data sets you cannot even attain a 10% significance level with a Friedman test. If you want to do something with so little data, I'd suggest you need to choose a model.
-
$\begingroup$ I understand to some point what you are talking about. My scenario is simple, I ran 4 algorithms on 2 data sets and I have a table of their accuracies. Now I need a statistical test to measure which algorithm performed better. We already know their accuracies yes, but having a statistical test might show differently. $\endgroup$ Commented Feb 25, 2018 at 14:05
-
$\begingroup$ So from what I understood from your comments is that it might not even be necessary or useful to run a test. $\endgroup$ Commented Feb 25, 2018 at 14:25
-
$\begingroup$ You still did not answer my question. $\endgroup$ Commented Feb 25, 2018 at 15:32
-
$\begingroup$ Michael, I think your question, in its present form, is too vague to be answerable in any more detail than Glen_b has already been kind enough to provide. It is singularly ungrateful and counterproductive to flag for moderator attention advice that is so obviously informed and intended to be helpful. $\endgroup$– whuber ♦Commented Feb 25, 2018 at 16:35
-
1$\begingroup$ @whuber I apologize, I was trying to get more clarification and it was wrong on my side to do so. I might not have formulated my question well. After reading Glen_b's answer and doing some more research on my side I understood his point. And thus my current question would be more like, "What statistical test would be appropriate to compare multiple classifiers over 2 datasets? Would a simple accuracy comparison suffice?" $\endgroup$ Commented Feb 26, 2018 at 0:22