4
$\begingroup$

I am trying to compare some metrics on the "same data set". So, I calculated some performance measures and I got the results for each metric.

I have only one dataset. I build a classification model using KNN, but with KNN (non-parametric method ) algorithm I used 7 distance metric for example Euclidean distance and so on.

My question is how to know if there are significant differences between the results. Is there any statistical test can help to find the statistically significant difference of each row in the table below. Do the t-test and ANOVA works for that.

For example, in the table below. Is there a statistically significant difference between accuracy 95.43, 95.78, 96.66 ,... and so on for other performance measures such as Sensitivity, F1 score etc. I am also not familiar with Kappa and Mcnemar's Test p-value from classification results.

Note: I have checked other related questions, but I did not find a helpful answer. Also, my question is not only about accuracy but also for other performance measures.

I will really appreciate an informative detailed answer with an application (in R if possible).

enter image description here

$\endgroup$
  • 3
    $\begingroup$ Possible duplicate of Checking whether accuracy improvement is significant $\endgroup$ – Marcin Sep 13 '18 at 18:58
  • 2
    $\begingroup$ If you are trying to select a model based on the scores, then you should select the simplest model. As to the question of how to know if there are differences between the scores, you can't really tell, not with these scores, maybe if you had AIC / logLikelihood or something similar. $\endgroup$ – user2974951 Sep 14 '18 at 6:02
  • 2
    $\begingroup$ The scores that are in the table (acc., spec., sens., f1) cannot be compared or tested for differences, you need other scores. $\endgroup$ – user2974951 Sep 14 '18 at 11:40
  • 2
    $\begingroup$ If these are results from models then maybe you could get other scores from them, as I mentioned, such as AIC, and you could use that to get p-values. $\endgroup$ – user2974951 Sep 14 '18 at 12:42
  • 2
    $\begingroup$ Avoid improper accuracy scores. And building a model is not as simple as removing apparently unimportant variables. $\endgroup$ – Frank Harrell Sep 15 '18 at 11:38
3
$\begingroup$

I don't think that you can accomplish exactly what you want with respect to the set of KNN models based on different distance metrics on your single data set, but you can try to evaluate the relative performance of the modeling approaches based on the different distance metrics. You will, however, have to make two adjustments.

Much of what follows is informed by the discussion on this page.

First, you should evaluate performance with a proper scoring rule like the Brier score instead of accuracy, specificity, sensitivity, and F1 score. Those are notoriously poor measures for comparing models, and they make implicit assumptions about the cost tradeoffs between different types of classification errors.* The Brier score is effectively the mean square error between the predicted probabilities of class membership and actual membership. You will have to see how your KNN software provides access to the class probabilities, but this is typically possible as in this sklearn function.

Second, instead of simply fitting the model one time to your data, you need to see how well the modeling process works in repeated application to your data set. One way to proceed would be to work with multiple bootstrap samples, say a few hundred to a thousand, of the data. For each bootstrap sample as a training set, build KNN models with each of your distance metrics, then evaluate their performances on the entire original data set as the test set. The distribution of Brier scores for each type of model over the few hundred to a thousand bootstraps could then indicate significant differences, among the models based on different distance metrics, in terms of that proper scoring rule.

Even this approach has its limits, however; see this answer by cbeleities for further discussion.


*Using accuracy (fraction of cases correctly assigned) as the measure of model performance makes an implicit assumption that false negatives and false positives have the same importance. See this page for further discussion. In practical applications this assumption can be unhelpful. One example is the overdiagnosis and overtreatment of prostate cancer; false-positives in the usual diagnostic tests have led to many men who were unlikely to have died from this cancer nevertheless undergoing life-altering therapies with frequently undesirable side effects.

The F1-score does not take true negative cases/rates into account, which might be critical in some applications. Sensitivity and specificity values depend on a particular choice of tradeoff between them. Sometimes that tradeoff is made silently by software, for example setting the classification cutoff in logistic regression at a predicted value of $p>0.5$. The explicit or hidden assumptions underlying all of these measures mean that they can be affected dramatically by small changes in the assumptions.

The most generally useful approach is to produce a good model of class membership probabilities, then use judgements about the costs of tradeoffs to inform final assignments of predicted classes (if needed). The Brier score and other proper scoring rules provide continuous measures of the quality of a probability model that are optimized when the model is the true model.

$\endgroup$
  • $\begingroup$ why Brier score is a good performance measure? $\endgroup$ – jeza Oct 1 '18 at 19:31
  • $\begingroup$ @jeza because it is what is called a proper scoring rule, for which the best score is provided by the true probability distribution. The scores you propose don't always have that property. Also see the link contained in Frank Harrell's comment on your question. $\endgroup$ – EdM Oct 1 '18 at 22:15
  • $\begingroup$ I have read both links. But, unfortunately, I do not understand it fully. If you could explain it and why it is good in a simple way, please. $\endgroup$ – jeza Oct 2 '18 at 22:42
  • 1
    $\begingroup$ @jeza I have added an extensive footnote to the answer, which I hope makes clearer the issues involved in using these various measures of model quality. $\endgroup$ – EdM Oct 3 '18 at 15:33
3
+25
$\begingroup$

Disclaimer: I think this answers OP's questions only to some extent.

I had seen Friedman post hoc test used in these settings. For example: let us say on particular dataset A - Algorithm X gives $A_x$ % accuracy, Algorithm Y gives $A_y$ % accuracy and Algorithm Z gives $A_z$ % accuracy. Similarly on Dataset B - Algorithm X gives $B_x$ % accuracy, Algorithm Y gives $B_y$ % accuracy and Algorithm Z gives $B_z$ % accuracy. Let us say we have 5 such datasets(A,B,C,D and E) on which these algorithms were run.

In such a setting, Friedman post hoc test can be used to check if Algorithm X's accuracy is significantly different from others (Y and Z). Friedman's test is similar to ANOVA but without the assumptions of normality. Luckily in R, it is fairly straightforward to implement this test.

$\endgroup$
  • $\begingroup$ Yes, it can be applied to any performance measure. $\endgroup$ – kasa Sep 18 '18 at 3:21
  • $\begingroup$ The Friedman test is the non-parametric alternative to the one-way ANOVA with repeated measures. It is used to test for differences between groups when the dependent variable being measured is ordinal. How would this work here? $\endgroup$ – user2974951 Sep 19 '18 at 11:06
  • $\begingroup$ @user2974951 @ jeza I am sure it can also be used when dependent variable is measured in interval / scale measures. The main advantage is that it removes the normality assumption which is required for ANOVA. As an example implementation of this kind is given in page 12 of this PDF: cran.r-project.org/web/packages/PMCMR/vignettes/PMCMR.pdf Here is a research paper which has used this comparing performance of models : ijcai.org/Proceedings/16/Papers/281.pdf $\endgroup$ – kasa Sep 19 '18 at 12:16
  • $\begingroup$ @kasa, thanks. ok, do you think I can use this test in R even I have only 7 accuracies to compare? Also, the accuracy I got is from only one data set but from different distance metric I used. I need to specify whether a matrix or vector of values, groups, and blocks. $\endgroup$ – jeza Sep 20 '18 at 12:19
  • $\begingroup$ @jeza: you mean, you have the models tested over 7 datasets? If yes, then you can go ahead and use this test. $\endgroup$ – kasa Sep 20 '18 at 14:00

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.