Timeline for When does Fisher's "go get more data" approach make sense?
Current License: CC BY-SA 4.0
10 events
| when toggle format | what | by | license | comment | |
|---|---|---|---|---|---|
| Jul 17, 2019 at 18:00 | comment | added | Yakk | "Collecting more data based on the p-value is only p-hacking if you compute a new p-value" -- compute a new p-value based on what? Anything? Compute any new p-value? Or a specific kind of p-value? If I do an experiment on dogs and cat rainfall correlation, and use the 3rd significant digit of the p-value to determine if I should do an experiment on the relative speeds of eggs and chickens (do I posit they are the same, or different?), is that p-hacking? (I'm asking for less ambiguity, I'm not asserting you are claiming that is p-hacking; it is not, however, ruled out by anything you said) | |
| Jul 16, 2019 at 18:53 | comment | added | usεr11852 | I do not think one would concede (because there is no element of loss or victory, the data is what it is) but rather realise. (Nice answer though +1.) | |
| Jul 15, 2019 at 22:05 | comment | added | Upper_Case | @jsk Ah, I understand your point better now. Thank you for the clarification. | |
| Jul 15, 2019 at 21:13 | comment | added | jsk | @Upper_Case I was commenting on a very small section of the post in regards to p-hacking, which is why I included that section in quotes. You are reading way too much into my statement. My point is that ANY decision rule that is used to decide to collect more data must be incorporated into calculating the p-value. As long as you incorporate the decisions made into the calculation of the p-value, you can still conduct a valid NHST if you so desire. This does not in any way mean that I am advocating for a stopping rule that says, "collect more data until you find a significant result." | |
| Jul 15, 2019 at 20:53 | comment | added | Upper_Case | @jsk I think it's less that subsequently calculated p-values are in some way invalid, and more that you are using an arbitrary and non-data-driven standard to judge when your experiment is "correct" and your research on that project is "done". Deciding that all non-significant p-values are wrong, and gathering data until you get one that is significant and then stopping because you've gotten the "right" result is the opposite of experimental science. | |
| Jul 15, 2019 at 18:16 | comment | added | jsk | "It's only p-hacking if you compute a new p-value." Doesn't this actually depend entirely on the method used to calculate the p-value? Ignoring the sequential analysis and decision to collect more data will result in an inaccurate p-value. However, if you incorporate the decision rule to collect more data into the calculation of the p-value, then you will produce a valid p-value. | |
| Jul 15, 2019 at 16:56 | comment | added | nalzok | So basically, say I want to test if the mean of population A is equal to that of population B. Initially, I get some data, conduct a test for $H_0$: "the means are equal", and I fail to reject it. In this case, I should not conduct another test for $H_0$: "the means are NOT equal". All I can do is estimating the confidential intervals of the means, is that correct? What if there is no overlap between the two intervals? | |
| Jul 15, 2019 at 8:24 | history | edited | Frans Rodenburg | CC BY-SA 4.0 |
added 10 characters in body
|
| Jul 15, 2019 at 6:39 | history | edited | Frans Rodenburg | CC BY-SA 4.0 |
deleted 94 characters in body
|
| Jul 15, 2019 at 6:29 | history | answered | Frans Rodenburg | CC BY-SA 4.0 |