3
$\begingroup$

I have several classifiers $f_i\ (i=1, \cdots, N)$ and calculated performance measures on multiple domains $(D)$ for each. Thus, there are $N \times D$ values.

I want to find out (increasing complexity):

  1. Is a particular classifier significantly better than the baseline, given all domains?
  2. Is there a particular classifier which is better than any of the others, given all domains?
  3. Is there a ranking of classifiers $f_i > f_k > f_l >$, where > means “significantly better”, given all domains?

I cannot assume them to be distributed according to a normal, thus I'm looking for non-parametric tests. From (JAPKOWICZ/SHAH, 2001), I inferred the following procedure:

  1. Perform Friedman's test to determine whether at least one classifier performs significantly better than the others for all domains
  2. If this applies, perform the Nemenyi post-hoc test to identify pairwise “significantly better than” relationships.

Questions:

  1. Is there any rule of thumb considering the minimum amount of domains required for giving Friedman's / Nemenyi's test sufficient power? Are there other tests better suited for a low number of domains (<5)?
  2. Is feasible to say that classifier $f_i$ is significantly better than any other, if pairwise results $f_i>f_k, f_k>f_j$ or $f_k = f_j$, where = means “not significant better”, exist?
  3. Is it feasible to infer such a ranking using pairwise Nemenyi tests? Is there a need to perform Friedman's test on subsets?

Please note that this question is related to another, unanswered one. A

$\endgroup$
0
$\begingroup$

with k algorithms and n domains (datasets):

1 - You can use chi-square Table with k-1 degrees of freedom for large n (usually > 15) and k (usually > 5).

2 - In the case of smaller n and k, the chi-square approximation is imprecise and a table lookup is advised from tables of Friedman values approximated specifically for the Friedman test. Minimal values for n and k is 3;

recommended read: Evaluating Learning Algorithms - A classification perspective 2011

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.