Timeline for Permutation test null and correct label distribution
Current License: CC BY-SA 3.0
10 events
| when toggle format | what | by | license | comment | |
|---|---|---|---|---|---|
| Jun 15, 2013 at 0:17 | comment | added | Glen_b | What you quote merely describes permutation tests. None of that quote suggests any of what you're trying to do, which is use a distribution of values rather than the mean of them, and none of it relates to the several assertions in your first paragraph. | |
| Jun 14, 2013 at 10:27 | comment | added | dashi | To quote from textbook: "The number of resamples on which a permutation test is based determines the number of decimal places and accuracy in the resulting P-value. Tests based on 999 resamples give P-values to three places (multiples of 0.001), with a margin of error 2 P(1 − P)/999 equal to 0.014 when the true one-sided P-value is 0.05. If high accuracy is needed or your computer is sufficiently fast, you may choose to use 9999 or more resamples." | |
| Jun 14, 2013 at 10:17 | comment | added | Glen_b | No. You seem to be reading something other than what I wrote. Let's pass back to the question: what is the basis on which you make the assertions in your first paragraph? | |
| Jun 14, 2013 at 10:14 | comment | added | dashi | So, if I compare the 97.5 confidence value from the correctly labelled distribution I also use the 97.5 confidence interval from the null distribution? And if I compare the mean of the correct distribution, I compare with the mean of the null? | |
| Jun 14, 2013 at 10:11 | comment | added | Glen_b | Whatever you use to measure your accuracy is what you test - but you must treat the permuted samples in exactly the same way. | |
| Jun 14, 2013 at 10:10 | comment | added | dashi | The question I had was whether this single mean value should represent the mean of the 'correctly labelled' distribution, or whether it can come from any position in the distribution 'correctly labelled' distribution. Thanks for your post. | |
| Jun 14, 2013 at 10:07 | comment | added | Glen_b | In which case, that single mean value can be compared with the distribution of single mean values generated via resampling procedures such as permutation tests. | |
| Jun 14, 2013 at 10:05 | comment | added | dashi | a single mean value = 1 accuracy composed of the average of k-fold crossvalidation. Clarified in question now. | |
| Jun 14, 2013 at 9:45 | history | edited | Glen_b | CC BY-SA 3.0 |
added 173 characters in body
|
| Jun 14, 2013 at 9:31 | history | answered | Glen_b | CC BY-SA 3.0 |