How do you know if an insignificant P value is because of chance error or because the sample size of the data is too small?

Question

I have always thought that an insignificant P value is the product of chance causing variation in the data. However when I was watching a talk on a study, the presenter mentioned how the insignificant P value meant that the sample size of the data was too small to make any conclusions rather the being the product of chance error. How can you tell if an insignificant P value is caused chance error or small sample size then?

Answer 1

The question basically asks what the truth is behind the observed p-value. But if we knew the truth we wouldn't need to compute a p-value in the first place!

I leave technical considerations regarding power to the other answers, but here's a bit about significance test logic.

The idea is that there is an underlying data generating process that we cannot observe directly, therefore we can't say what it truly is. We only see the data.

The p-value is a measure computed from the data that measures compatibility of the data with the null hypothesis. (I think that you use the term "chance" to refer to the null hypothesis, which often but not always interprets as something like "only random variation is going on".)

An insignificant p-value means that if the null hypothesis is in fact true, the data look pretty much as they should be expected to look like (regarding the specific test statistic at least; they may look different in other respects). They are compatible with the null hypothesis, they don't deliver evidence against it.

It does not mean that the null hypothesis is true, however. Data generating processes deviating from the null hypothesis may also produce data that look like this. Note also that most statisticians would agree with Box's famous quote that "all models are wrong but some are useful", so actually at best the null hypothesis may be a very good model (i.e. idealised mathematical description) for what's really going on, but never literally true.

The role of the sample size now is this: The larger the sample size, the more information in the data, and the better we can use the data to actually distinguish the null hypothesis from alternative models. If you have a low sample size, the data may not only be compatible with the null hypothesis, but also with models that are really quite different (and in a real context would have a very different interpretation). This means that a large p-value doesn't allow to say that we are at least close to the null hypothesis (meaning "chance variation" in some circumstances). If indeed an alternative model is true, larger sample size will make it more likely to observe a significant p-value (that is called "power" of the test). This implies that with a larger sample size, observing an insignificant p-value, we have some evidence that we're at least close to the null hypothesis.

However, note that if in fact the sample size is very small, we cannot tell these apart, i.e. you cannot say whether the p-value is as it is because we're in fact very close to the null hypothesis, or whether it is only because our sample size is too small and in fact an alternative model is appropriate that with a larger sample size would lead to significance. You can only know this collecting a larger sample! Post hoc power calculations do not address this question! What they tell you is, if you want, the degree of distinctiveness of the sample you have, i.e., how far away from the null you may have been without being able to find it. This doesn't mean that this is indeed the case, as in fact your small sample doesn't allow you to distinguish these cases!

Unfortunately there is a catch in significance test logic, which is that if you collect a very large sample, you will quite likely find a significant p-value even if the truth is, though slightly different from the null hypothesis, so close to it that regarding its meaning in the real situation you'd say that it is not in any relevant manner different from the null hypothesis (like, a true parameter may be 0.02 rather than 0, but you'd think that only parameters larger than 0.5 are worth bothering; obviously this depends on the exact situation, meaning of the data etc.). So indeed large samples may lead to significance often even when this is actually meaningless, which means that you always need to look at estimated effect sizes and possibly confidence intervals to say something more relevant than just whether something is "significant".

Answer 2

Ideally, you would have run a 'power and sample size' procedure before doing your experiment. Then you you will have known all along whether you had a good chance of rejecting $H_0$ if an effect of the desired size leads to rejection.

Of course, when you are working at the 5% level of significance, there is by definition a 5% probability of rejecting a true null hypothesis. So you can never really be absolutely sure you reach a correct decision.

If you did not do a power and sample size procedure at the design phase of experimentation, then doing an ad hoc power analysis is probably better than just wondering. But you need to understand that an ad hoc power determination cannot be exactly the same thing as one done at the proper time. Power analyses at the design phase almost always involve making some guesses, and you will likely have to make fewer guesses ad hoc---or at least think so.

Perhaps the main reason for avoiding ad hoc power analyses arises when you fail to reject when you think you are 'entitled' to a rejection. This can tempt some people into "P-hacking" procedures such as 'suddenly realizing' that you really intended a one-sided test all along, or dredging through the data with unwarranted ad hoc tests looking for 'something like' the 'unfairly withheld' rejection. At any stage of the experimental procedure, a power analysis is not a guarantee. It is a prudent precaution. If you could figure out whether to reject via a power analysis, you wouldn't have to do the experiment.

If no effect is found, many of the experimenters (and thier bosses) may be disappointed and intent on finding the reason for no rejection. Somebody (often the project statistician) has to have the guts to say, the real reason no effect was found may be that no effect exists.

Answer 3

The presenter mentioned how the insignificant P value meant that the sample size of the data was too small

This is very wrong.

• The p-value does not indicate whether the sample size was too small.

• It is the estimate of the error/deviation in the measured effects that indicates this.

  (Along with a relative comparison of the size of that error and the size of the suspected effect that one hoped to be able to measure. The error should be smaller than the effect that you hope to measure.)

What this presenter is saying is presumptive. An insignificant p-value can occur when the sample is too small. But... it can also occur because there is no effect. It sounds like this presenter presupposes the presence of an effect and bases the story of the discussion and conclusion around this

It is also a bit silly to state this as a conclusion. It is a bit like saying 'the experiment failed' while hiding that it already failed before it started because the sample is too small (instead it looks like the presentation makes the 'too small sample' is a discovery). In order to know that the sample is too small you do not need to observe an insignificant p-value. You can already determine beforehand whether a sample is going to be too small to be able to detect a certain magnitude of effect. The only situation when this is not the case is when there is no reasonable a priori estimate of the variance that might be expected on the measurements. But even then, one should not say that the insignificant p-value means that the sample is too small.

Answer 4

You do a post hoc / a posteriori power analysis. It answers the question: how likely was I going to find an effect given the size of my sample and the observed effect size (roughly speaking). There is a WIDE range of methods to calculate this given your statistical methods. If you find that you had very little chance of finding an effect with your sample size given the data, your sample size may have been too small.

There are plenty of resources on this:

Differences and relation between retrospective power analysis and a posteriori power analysis?

Also look at G* Power, it’s a piece of software often used for this.

Answer 5

There are two main types of error: Type I error, where we reject the null hypothesis despite it being true, and Type II error, where we fail to reject the null hypothesis even though it is false. The probability of getting a Type I error, conditioned on the null hypothesis being true is called $\alpha$, and $1-\alpha$ is the sensitivity of the test. The probability of getting a Type II error, conditioned on the null hypothesis being false, is called $\beta$, and $1-\beta$ is the specificity or power of the test. Methods to estimate $\beta$ are called "power analysis".

$\alpha$ can be calculated from the null hypothesis; one of the requirements for a proper null hypothesis is that it is precisely stated enough that $\alpha$ is known. Calculating $\beta$, however, requires a precise alternative hypothesis; an alternative hypothesis that consists of merely saying that the null is false is not enough. For instance, if the null hypothesis is $\mu =0$, then we would get different $\beta$ for an alternative hypothesis of $\mu = 1$ versus an alternative hypothesis of $\mu = 2$.

How much a parameter differs from the null hypothesis is the effect size (this is often further qualified as "true effect" to distinguish it from "observed effect", with the former being the difference in the actual parameter, while the latter is the difference in the statistic measured from the sample). The main factors affecting power are sample size and effect size. The larger these factors are, the higher the power. For a small sample sizes, a failure to reject the null can be caused the small sample size, the null being true, or chance. But as the sample gets larger, the probability of failure to reject being due to chance gets smaller and smaller, and it becomes easier to draw conclusions.

Since the actual effect size is unknown, we can't know for certain what the power is. But we can, given bound on the effect size, give bounds on the power. For instance, if a drug killing one in a million people who take it is an acceptable risk, we can ask "Given an effect size of one in a million, what's the probability that a drug trial would find no statistical significant difference in fatality rates of those who took the drug?"

Answer 6

The long answers are mostly correct, but only address your specific question indirectly. The short and direct answer is that you can gather evidence that will help you decide if your sample size is too small or if your data are misleading (due to error or due to natural sampling variation) ONLY by gathering another sample of data.

The reasons that you need another set of data can be had from the longer answers.