Timeline for $p$-value: Fisherian vs. contemporary frequentist definitions
Current License: CC BY-SA 4.0
13 events
| when toggle format | what | by | license | comment | |
|---|---|---|---|---|---|
| Dec 2, 2023 at 16:20 | comment | added | Richard Hardy | That reminds me of Aris Spanos' treatment of Fisherian vs. Neyman-Pearson hypothesis testing in some of his papers and textbook(s), e.g. section 14.5.2 from his "Probability theory and statistical inference" (1st edition, 1999). I am not sure if this is included in the 2nd edition from 2019. | |
| Dec 2, 2023 at 15:33 | comment | added | Christian Hennig | @RichardHardy $H_0$ does never hold anyway, nothing in reality is continuous and nothing is i.i.d. Tests are not about whether $H_0$ is true, but rather whether the data suggest specific meaningful deviations from $H_0$. This will always depend on what kind of deviations are relevant to us. This means that meaning of data, interpretation of model, aim of analysis are always important, and no $H_0$ on its own can say how it should be tested. (Fisher did one-sided tests without referring to an $H_1$ by the way.) | |
| Dec 2, 2023 at 15:25 | comment | added | Richard Hardy | @ChristianHennig, this argument is approaching a tautology, because the motivation for choosing a one-sided test comes from $H_1$, not from $H_0$. From $H_0$, the logical thing to do is to use a low density region. Things that are uncommon under $H_0$ suggest $H_0$ does not hold. | |
| Dec 2, 2023 at 14:04 | comment | added | Christian Hennig | @RichardHardy Thinking about it, actually any standard one-sided test with symmetric distribution of the test statistic would be an example where we wouldn't reject everywhere where the density is lowest. | |
| Dec 2, 2023 at 14:02 | comment | added | Christian Hennig | @RichardHardy No, I meant to refer to the distribution of the test statistic. A test statistic in such applications can normally be interpreted as measuring aggregate quality, so one would reject on one side (the "bad" side) and not elsewhere, density low or not. Of course this was hypothetical and abstract as I didn't have a specific test statistic and H0 in mind that fulfiulls your requirements, see the comment before that. | |
| Dec 2, 2023 at 13:58 | comment | added | Richard Hardy | In the last comment you are probably referring to the density of the data instead of a test statistic but the latter is key. | |
| Dec 2, 2023 at 13:41 | comment | added | Christian Hennig | @RichardHardy Think about quality control applications for example. Regardless of the value of the density you want to reject where the quality of products is too bad and not where the density is low if in that place the quality is good. | |
| Dec 2, 2023 at 13:40 | comment | added | Christian Hennig | @RichardHardy Well, a test with distribution of the test statistic so that there is low density somewhere in the middle is a strange thing in the first place. Maybe if you can give me an example where that occurs I could tell you an application where you then still want to reject only in big distance from the center regardless. ;-) | |
| Dec 2, 2023 at 10:43 | comment | added | Richard Hardy | It it hard to argue against in some situations we may want something else :) I would be curious to see an example where a value that has low density under $H_0$ can be considered less "weird/extreme" than one with high density. I wonder if that can be achieved without violating the intrinsic meanings of "weird" and "extreme". Here, I define "weird/extreme" in relation to $H_0$ without a reference to $H_1$. If we introduce $H_1$, things can change. | |
| Dec 1, 2023 at 20:58 | comment | added | Shawn Hemelstrand | I think your point about the implicit alternative is true. Fisher seemed to completely reject Neyman-Pearson's ideas of an alternative hypothesis, but seemed to have no issue however with accepting claims like $\mu_1 \neq \mu_2$ when rejecting the null (Hurlbert & Lombardi, 2009). | |
| Dec 1, 2023 at 13:40 | comment | added | Christian Hennig | @RichardHardy In real data analysis I don't think the concept "intrinsically meaningful" gets us very far. We have to define what we mean by "extreme" or "weird" and the definition will depend on what exactly we know and what exactly we're interested in. Being in a low density region is "extreme" according to any definition that measures extremity as inverse proportional to density, and I'm fine with that; but in some situations we may want something else. | |
| Dec 1, 2023 at 11:16 | comment | added | Richard Hardy | From gung's comment: Imagine a test statistic whose distribution was bi-modal under the null (I can't think of one, but nothing prevents a test static from having a weird distribution). In that case, a "low density" could occur in the middle of the distribution Does your "no contradiction" statement hinge on this not being the case (i.e. low density at the tails, not in the middle)? I find low density to be a very intuitive measure of what is atypical under $H_0$ while I think other measures (such as how far in a tail) derive from that but otherwise might not be intrinsically meaningful. | |
| Dec 1, 2023 at 11:11 | history | answered | Christian Hennig | CC BY-SA 4.0 |