I have several classifiers $f_i\ (i=1, \cdots, N)$ and calculated performance measures on multiple domains $(D)$ for each. Thus, there are $N \times D$ values.
I want to find out (increasing complexity):
- Is a particular classifier significantly better than the baseline, given all domains?
- Is there a particular classifier which is better than any of the others, given all domains?
- Is there a ranking of classifiers $f_i > f_k > f_l >$, where
>means “significantly better”, given all domains?
I cannot assume them to be distributed according to a normal, thus I'm looking for non-parametric tests. From (JAPKOWICZ/SHAH, 2001), I inferred the following procedure:
- Perform Friedman's test to determine whether at least one classifier performs significantly better than the others for all domains
- If this applies, perform the Nemenyi post-hoc test to identify pairwise “significantly better than” relationships.
- Is there any rule of thumb considering the minimum amount of domains required for giving Friedman's / Nemenyi's test sufficient power? Are there other tests better suited for a low number of domains (<5)?
- Is feasible to say that classifier $f_i$ is significantly better than any other, if pairwise results $f_i>f_k, f_k>f_j$ or $f_k = f_j$, where
=means “not significant better”, exist?
- Is it feasible to infer such a ranking using pairwise Nemenyi tests? Is there a need to perform Friedman's test on subsets?
Please note that this question is related to another, unanswered one. A