Hot answers tagged

29

The frequentist paradigm is a conflation of Fisher's and Neyman-Pearson's views. Only in using one approach and another interpretation do problems arise. It should seem strange to anyone that collecting more data is problematic, as more data is more evidence. Indeed, the problem lies not in collecting more data, but in using the $p$-value to decide to do so,...


28

I don't think the objection is to just the term "statistically significant" but to the abuse of the whole concept of statistical significance testing and to the misinterpretation of results that are (or are not) statistically significant. In particular, look at these six statements: P-values can indicate how incompatible the data are with a specified ...


17

A confidence interval for a t-test is of the form $\bar{x}_1 - \bar{x}_2 \pm t_{\text{crit}, \alpha}s_{\bar{x}_1 - \bar{x}_2}$, where $\bar{x}_1$ and $\bar{x}_2$ are the sample means, $t_{\text{crit}, \alpha}$ is the critical $t$ value at the given $\alpha$, and $s_{\bar{x}_1 - \bar{x}_2}$ is the standard error of the difference in means. If $p=1.0$, then $\...


14

Caveat: I am NOT an expert on climatology, this is not my field. Please bear this in mind. Corrections welcome. The figure that you are referring to comes from a recent paper Santer et al. 2019, Celebrating the anniversary of three key events in climate change science from Nature Climate Change. It is not a research paper, but a brief comment. This figure ...


14

For the purpose of this answer I'm going to assume that excluding those few participants was fully justified, but I agree with Patrick that this is a concern. There's no meaningful difference between p ~ 0.05 or p = 0.06. The only difference here is that the convention is to treat the former as equivalent to 'true' and the latter as equivalent to 'false'. ...


12

There are a dozen of continues probability distributions There are an infinite number of continuous probability distributions. The ones that have been discussed enough to be named and included in the space of a couple of pages are nevertheless sufficient to fill numerous books (and indeed they do - see, for example, the many books by Johnson, Kotz and other ...


12

The p-value is the probability of seeing what you saw or something more extreme if the null hypothesis was true. The p-value is not the probability that the null hypothesis is true. So yes, interpreting a p-value as the probability that the null hypothesis is true is akin to the prosecutor's fallacy. If you want that probability, you need to assume a ...


10

A few things: The BF gives you evidence in favor of a hypothesis, while a frequentist hypothesis test gives you evidence against a (null) hypothesis. So it's kind of "apples to oranges." These two procedures, despite the difference in interpretations, may lead to different decisions. For example, a BF might reject while a frequentist hypothesis test doesn'...


10

Given a big enough sample size, a test will always show significant results, unless the true effect size is exactly zero, as discussed here. In practice, the true effect size is not zero, so gathering more data will eventually be able to detect the most minuscule differences. The (IMO) facetious answer from Fisher was in response to a relatively trivial ...


10

Being super-lazy, using R to solve the problem numerically rather than doing the calculations by hand: Define a function that will give normally distributed values with a mean of (almost!) exactly zero and a SD of exactly 1: rn2 <- function(n) {r <- rnorm(n); c(scale(r)) } Run a t-test: t.test(rn2(16),rn2(16)) Welch Two Sample t-test data: ...


9

2.2e-16 is the scientific notation of 0.00000000000000022, meaning it is very close to zero. Your statistical software probably uses this notation automatically for very small numbers. You may be able to change this in the settings. The notation alone is no reason to be suspicious. The result itself might be, but you will have to be the judge of that. < ...


8

The Bayes factor $B_{01}$ can be turned into a probability under equal weights as $$P_{01}=\frac{1}{1+\frac{1}{\large B_{01}}}$$but this does not make them comparable with a $p$-value since $P_{01}$ is a probability in the parameter space, not in the sampling space its value and range depend on the choice of the prior measure, they are thus relative rather ...


8

The CI can have any limits, but it is centered exactly around zero For a two-sample T-test (testing for a difference in the means of two populations), a p-value of exactly one corresponds to the case where the observed sample means are exactly equal.$^\dagger$ (The sample variances can take on any values.) To see this, note that the p-value function for ...


8

There are an awful lot of issues raised in your question, so I will try to give answers on each of the issues you raise. To frame some of these issues clearly, it is important to note at the outset that a p-value is a continuous measure of evidence against the null hypothesis (in favour of the stated alternative), but when we compare it to a stipulated ...


8

In my opinion one of more honest yet non-technical phrasing would be something like: The obtained result is surprising/unexpected (p = 0.03) under the assumption of no mean difference between the groups. Or, permitting the format, it could be expanded: The obtained difference of $\Delta m$ would be quite surprising (p = 0.03) under the scenario of two ...


7

Thanks. There are a couple things to bear in mind here: The quote may be apocryphal. It's quite reasonable to go get more / better data, or data from a different source (more precise scale, cf., @Underminer's answer; different situation or controls; etc.), for a second study (cf., @Glen_b's comment). That is, you wouldn't analyze the additional data ...


7

There is no intercept term in this model, so the best-fit line must go through the origin. A best-fit line that goes through the origin will clearly have a positive slope for the data you're showing. It seems including x_ran = sm.add_constant(x_ran) will add the constant term. You should then find that your intercept is significantly diffrent from zero, ...


6

The famous seminal Benjamini & Hochberg (1995) paper described the procedure for accepting/rejecting hypotheses based on adjusting the alpha levels. This procedure has a straightforward equivalent reformulation in terms of adjusted $p$-values, but it was not discussed in the original paper. According to Gordon Smyth, he introduced adjusted $p$-values in ...


6

...the concept of the null hypothesis is basically associated with Student's t-distribution. Not really. The null hypothesis is associated with a corresponding null distribution, which varies depending on the model and test statistic. In classical hypothesis tests for unknown linear coefficients or mean values, one generally uses a test statistic that is ...


6

What we call P-hacking is applying a significance test multiple times and only reporting the significance results. Whether this is good or bad is situationally dependent. To explain, let's think about true effects in Bayesian terms, rather than null and alternative hypotheses. As long as we believe our effects of interest come from a continuous ...


6

You are right that the documentation is wrong. Note that p values are defined somewhat differently from what you write. They do not measure the probability of a decision, such as the decision to reject the null or to fail to reject the null. They measure the probability of test statistics. Whether or not to reject a null hypothesis is a subsequent decision ...


6

In-sample information like what you are presenting has only a very weak relationship with out-of-sample predictive performance. For instance, a larger model, like your Model 6, will always have a larger $R^2$, i.e., proportion of variance explained, than a smaller model. But that may well be due only to overfitting. And come with worse predictive ...


6

It is actually used quite often (in fact, Bonferroni and Bonferroni-Holm can be shown to be valid tests using the closed testing principle - see below). Part of the reason why some simple procedures like Bonferroni are still so popular is of course, that they are very easy to implement and it is easy to communicate what you did. E.g. even for the simple ...


6

I agree with the answer by Peter Flom, but would like to add an additional point on the use of the term "significance" in statistical hypothesis testing. Most hypothesis tests of interest in statistics have a null hypothesis that posits a zero value for some "effect" and an alternative hypothesis that posits a non-zero (or positive, or negative) value for ...


6

Likelihood applies to data once observed. If you were describing a property of this data set, you would be speaking of a likelihood. Probability describes a property of hypothetical data sets not yet observed. Against the arbitrary standard of this data set: the probability of observing another data set which departs from the null hypothesis as much as or ...


5

Why would you expect anything else? You don't need a simulation to know this is going to happen. Look at the formula for the t-statistic: $t = \frac{\bar{x_1} - \bar{x_2} }{\sqrt{ \frac{s^2_1}{n_1} + \frac{s^2_2}{n_2} }}$ Obviously if you increase the true difference of means you expect $\bar{x_1} - \bar{x_2}$ will be larger. You are holding the variance ...


5

This idea of the uniform distribution for P-values is fairly new in statistics education and practice. I don't know if anyone has yet made up a name for the related histograms that has come into general use. Below I just call them "Null P-value" histograms. It is important to note that this uniform distribution for P-values holds only if the null hypothesis ...


5

To expand on what Ben Bolker notes in a comment on another answer, the issue of what a frequentist p-value means for a regression coefficient in LASSO is not at all easy. What's the actual null hypothesis against which you are testing the coefficient values? How do you take into account the fact that LASSO performed on multiple samples from the same ...


5

When we want to test a hypothesis, we need a test statistic with a known probability distribution. This usually involves standardisation of the data. For example, if we collect a random sample $X_1, \dots, X_n$ with mean $\mu$ and variance $\sigma^2$, and the data is assumed to be normally distributed. Then we would standardise it as $$Z_n = \frac{\bar{X}_n-...


5

As whuber has commented: the Kolmogorov-Smirnov test is only valid as a comparison against a fully specified distribution. You cannot use it to compare an observed distribution against a distribution whose parameters have been estimated based on your observed sample. If you do so, your p-values will not be uniformly distributed under the null hypothesis, but ...


Only top voted, non community-wiki answers of a minimum length are eligible