I know it's probably a basic question, but I'm relatively new to statistical analysis. When you have to reject a null hypothesis, or verify the design of a study, can you base your assumption on p-values?


1 Answer 1


Well, I am not entirely sure what your question is. Your subject question asked whether p-values were different (or what the difference was) from error rate. But, in the body of your post you ask if you can use p-values to reject the null or verify the design of the study.

So, you might get better answers if you clarify what it exactly it is that you're asking.

That said, I'll offer a few thoughts that might be relevant to what you're asking.

First, you certainly can't use p-values to verify study design. In fact, I'm not sure what that would even look like. For example, you could have a terrible study design that completely fails to target the construct you are attempting to investigate and still wind up with p < 0.05. And, of course, non-significance doesn't indicate a bad study design. It certainly can indicate a lack of statistical power, but it can also indicate (as it is supposed to!) that your hypothesis is wrong.

Second, within the frequentist tradition, rejecting or accepting the null is exactly what p-values are for. So, if your asking: 'do people use p-values to reject null hypotheses?', then yes, they absolutely do. However, if you're asking this question at a higher level, maybe with a hint of, 'should people use p-values to reject null hypotheses?', that's a rather different question.

As maybe you've read, p-values represent a probability that is associated with your data. However, they do not represent the probability that your hypothesis is correct or anything along those lines. Rather, they represent the probability that you would have observed data at least as extreme as the data that you observed, given that the null hypothesis is true.

For example, if you are using a t-test to compare two means, the associated p-value tells you the probability of observing a difference between those means at least as large as the difference you observed, given that there is actually no difference between the groups from which those means were sampled.

So, a p of 0.05 or less indicates that the probability of observing data at least as extreme as yours is 0.05 (or less).

As for the relationship between p-values and error rates, you can probably see that as p decreases, the likelihood of a false positive decreases, holding other things constant. That is, as the probability of observing my data or data more extreme, given that the null is true, decreases, accepting the alternative hypothesis (rejecting the null) becomes a safer and safer bet.

Something to keep in mind, though, is that the null is practically always false. Two means, for example, are never going to be exactly equal in your sample. This means that, given a sufficiently large sample size, even the smallest difference could be statistically significant. So, while lower p-values generally indicate a lower chance of committing a false positive, it does not mean that the observation that is significant is meaningful or noteworthy.

  • $\begingroup$ The answer you provided is very useful and fits perfectly - even if in the question I made is not perf. clear - all the doubts I had. I've got some little points to clarify. What does "more extreme data" means exactly? $\endgroup$
    – Liv
    Commented Nov 16, 2016 at 11:04
  • $\begingroup$ Good question! Another way to think about this is in terms of a test statistic. If you're comparing means, this would translate into a difference between means equal to or greater than the one observed. However, because the test statistic (e.g. difference between means) is ultimately a function of the data, people tend to use these terms/definitions interchangeably. $\endgroup$
    – Joe Hoover
    Commented Nov 16, 2016 at 19:54

Not the answer you're looking for? Browse other questions tagged or ask your own question.