Timeline for Not sure if standard error of p-values makes sense in Fisher Exact Test
Current License: CC BY-SA 3.0
7 events
| when toggle format | what | by | license | comment | |
|---|---|---|---|---|---|
| Apr 8, 2012 at 0:04 | comment | added | ely | Yes, indeed it is blatantly obvious. Again, this is why I was puzzled. It's effectiveness of a particular unemployment program that is known to be extremely effective. The test statistic in question is extremely different for two populations. Just histogramming the outcomes shows you it's unreasonable under the Fisher Exact Test, which assumes a sharp null hypothesis that the treatment is ineffective for literally all participants. | |
| Apr 7, 2012 at 21:18 | comment | added | whuber♦ | @EMS, are you sure you're getting an SE of $10^{-9}$? That's incredibly low. You can get such SEs only when $p \approx 10^{-18} n$. If the code is fast, indicating $n$ is small, this suggests $p$ itself is so small that no formal testing should be necessary: the significance of the result ought to be blatantly obvious. | |
| Apr 7, 2012 at 21:06 | comment | added | ely | I should also add that the code is already very fast too. So learning that we could reduce the iterations doesn't allow us to do things faster. We'd be saving a fraction of a second on something we'll only run a few dozen times, if that. I think this s.e. was requested more out of a force of habit than anything else, which is totally fine and probably a good habit for me to develop too. | |
| Apr 7, 2012 at 21:03 | comment | added | ely | Right, I get that. But in this case, it should have been pretty obvious that we were using far more iterations in the m.c. simulation than necessary. The s.e. I am computing is smaller than 10^(-9) for a p-value that realistically only needs a few digits of accuracy. If the simulation wasn't for a p-value, but for something truly needed many digits of accuracy, I would totally see the merit in needing to know a standard error, as has been the case for the many m.c. simulations I've created for other studies. | |
| Apr 7, 2012 at 20:27 | comment | added | whuber♦ | @EMS, a Monte Carlo calculation is an approximation. It therefore behooves one to know how accurate that approximation is, which is why it is reasonable for your advisor to ask for some assessment of the quality of that approximation. More to the point, once you know how to quantify the connection between the number of MC iterations and precision of the p-value, you can determine (in advance) how many iterations may be necessary. | |
| Apr 7, 2012 at 3:28 | comment | added | ely | I totally agree. It threw me for a loop when my adviser asked for it. I thought the same thing... it doesn't give a confidence that the p-value is telling us something, just that we did enough iterations to believe our Monte Carlo approximation to FET is good. | |
| Apr 7, 2012 at 3:26 | history | answered | ALV | CC BY-SA 3.0 |