# Friedman's test to identify best of multiple classifiers on multiple domains

I have several classifiers $$f_i\ (i=1, \cdots, N)$$ and calculated performance measures on multiple domains $$(D)$$ for each. Thus, there are $$N \times D$$ values.

I want to find out (increasing complexity):

1. Is a particular classifier significantly better than the baseline, given all domains?
2. Is there a particular classifier which is better than any of the others, given all domains?
3. Is there a ranking of classifiers $$f_i > f_k > f_l >$$, where > means “significantly better”, given all domains?

I cannot assume them to be distributed according to a normal, thus I'm looking for non-parametric tests. From (JAPKOWICZ/SHAH, 2001), I inferred the following procedure:

1. Perform Friedman's test to determine whether at least one classifier performs significantly better than the others for all domains
2. If this applies, perform the Nemenyi post-hoc test to identify pairwise “significantly better than” relationships.

Questions:

1. Is there any rule of thumb considering the minimum amount of domains required for giving Friedman's / Nemenyi's test sufficient power? Are there other tests better suited for a low number of domains (<5)?
2. Is feasible to say that classifier $$f_i$$ is significantly better than any other, if pairwise results $$f_i>f_k, f_k>f_j$$ or $$f_k = f_j$$, where = means “not significant better”, exist?
3. Is it feasible to infer such a ranking using pairwise Nemenyi tests? Is there a need to perform Friedman's test on subsets?

Please note that this question is related to another, unanswered one. A