Timeline for Suspiciously small p-values from randomization (Fisher-Pittman) test
Current License: CC BY-SA 3.0
13 events
| when toggle format | what | by | license | comment | |
|---|---|---|---|---|---|
| Apr 7, 2016 at 13:46 | comment | added | Matt Krause | I added a little summary that makes the whole thing explict--feel free to roll it back or modify it if you'd like. | |
| Apr 7, 2016 at 13:46 | history | edited | Matt Krause | CC BY-SA 3.0 |
added 391 characters in body
|
| Apr 7, 2016 at 13:29 | vote | accept | Matt Krause | ||
| Apr 7, 2016 at 13:29 | comment | added | Matt Krause | You....may have stumbled onto my secret plan. I was going to be nice and submit a patch with it, but haven't quite had time yet. Thanks! | |
| Apr 7, 2016 at 13:28 | comment | added | Roland | I agree that a p-value of exactly zero from a MC simulation could have a warning attached. Maybe you should direct the package maintainer to this discussion. | |
| Apr 7, 2016 at 13:25 | comment | added | Matt Krause | COIN could do any number of things in this situation, including refusing to emit a p-value at all, warning that the p-value printed is unreliable (as it does in other situations), showing a confidence interval for the p-value, or calculating it differently. I guess part of what I wanted to ask in this question is whether returning a p-value of (effectively) zero is also reasonable, or at close enough that users should be expected to look out for it. I was also interested in whether Monte Carlo p-values are better calculated as k/N or (k+1)/(N+1) but perhaps that's a separate question. | |
| Apr 7, 2016 at 13:09 | comment | added | Matt Krause | The zero case seems especially pathological. Suppose you have 1/1000 successes, and thus p=1/1000. The binomial CDF (which generates the confidence interval) says that there's only a ~5% chance of seeing 15+ successes out of 10,000 iterations (and likewise for <5), so there is a fairly small chance that the (rounded) p-value will change with more iterations. This is not at all the case when observing 0/1000 successes, as your answer shows. | |
| Apr 7, 2016 at 6:52 | comment | added | Roland | OK. And what about exactly one "success"? We shouldn't really trust such a result either. And I wouldn't feel comfortable with two "successes". Are three "successes" sufficient? Now I would ask for some kind of uncertainty/power analysis. And I assume that the confidence interval is based on such an analysis. | |
| Apr 6, 2016 at 19:19 | comment | added | Matt Krause | I was thinking something like "No 'successes' in any MC replicate. Consider increasing the number of replications." (which showing the confidence interval would also imply, I guess). | |
| Apr 5, 2016 at 15:27 | comment | added | Roland |
It's not obvious to me what appropriate conditions for a warning might be. Instead I would prefer if the confidence interval of the p-value (or some other measure of uncertainty) was reported in the print output.
|
|
| Apr 5, 2016 at 15:11 | comment | added | Matt Krause | In a situation where none of the replicates are as extreme as the true value, should it actually be returning zero here? As I wrote at the beginning of my question, I would have expected it to return a slightly-too-conservative $p$-value of 1/(B+1) instead of a wildly-incorrect zero. A warning or diagnostic message wouldn't seem out of place here either. Am I totally off base here? | |
| Apr 5, 2016 at 14:48 | history | edited | Roland | CC BY-SA 3.0 |
added 101 characters in body
|
| Apr 5, 2016 at 14:40 | history | answered | Roland | CC BY-SA 3.0 |