Timeline for Statistical Significance of multiple classifiers by using p-value
Current License: CC BY-SA 3.0
6 events
| when toggle format | what | by | license | comment | |
|---|---|---|---|---|---|
| Oct 17, 2016 at 17:22 | comment | added | Ramalho | If found @cbeleites explanation to be much more comprehensive, I think I have learned a thing or two! :) I would advise you to follow it. To my experience I never had the problem of non-normality, but sure it can happen, in which case I would advise you to lookup for a non parametric test. Think of it, the problem of non normality will surely be more preeminent if you use higher complexity classifiers, those will very likely be more sensitive to the folds and in turn bring higher variance to your sample of statistics. Pick your statistic/metric wisely. | |
| Oct 16, 2016 at 22:43 | comment | added | Memin | Thank you Ramalho for replying, do you think the comment I have for @user7019377 post, will be right approach to get p-values. | |
| Oct 16, 2016 at 10:01 | comment | added | Ramalho | Think of the problem as the problem of testing if two classes of students that were subject to different teaching methods show significative differences. In practice here will always some overlap, will that be limitation? In practice, probably not. | |
| Oct 16, 2016 at 9:44 | comment | added | Ramalho | I dont know if Im following your concern, what I meant is to use the training folds as training data and test it according to a desired metric. This will give you a sample of the performance of your model and hyperparameters. Training folds overlap yes and that can lead to violation of independence yes, but you apply the same procedure to all classifiers, they will be subject to this less ideal scheme. Sorry I dont have a paper on this. | |
| Oct 16, 2016 at 9:35 | comment | added | air | Uhm unless you can provide a reference, I really don't think it's that straight-forward. Correlations across folds in cross-validation (because the training folds always overlap) makes variance estimation and consequent testing a very difficult problem, as far as I am aware. | |
| Oct 16, 2016 at 0:20 | history | answered | Ramalho | CC BY-SA 3.0 |