I just learned about p-value. My understanding of p-value is as follows:

If we take a sample dataset and find out that sample's result, the p-value basically indicates how likely it is to get that (or similar) result for a dataset with similar size and variance, given that the null hypothesis is true.

If it's lower than the significance level, we reject the null hypothesis because there's a very low chance that such a result would come if the null hypothesis were true.

If it's equal to or higher, we accept the null hypothesis because it's likely that such a result would come given that it's true.

Now my question is how exactly is the p-value used in real life? The dataset taken could be an exceptional dataset and could give a completely different result than what is correct. For large datasets, even doing this 5 times could result in incorrect results (by this I mean, datasets with exceptions would hence have p-values that suggest something different than what is actually correct). Basically, how do we ensure that the sample dataset taken is accurate enough for our results and for us to accept/reject the null hypothesis completely?

  • 4
    $\begingroup$ In your second paragraph, replace "or similar" with "or more extreme" and you have the right definition. $\endgroup$ – Frans Rodenburg Jun 30 '19 at 8:39

The level of significance is the chance of a false positive. You cannot ensure that your sample will be representative of the population, because the sample is but a random draw of the population. Such is the nature of empirical research.

You can however, take certain precautions to avoid an inflated false positive rate. For example, you can randomize both your sampling method and/or your treatment allocation to avoid sample bias beyond that of the luck of the draw. You can also stratify your sampling scheme to account for imbalances. If you run multiple tests, you should apply a correction, such as Bonferroni or FDR.

You can indeed be unlucky, and draw a sample that generates a false positive 5 times in a row, but (granted your study design and assumptions are not flawed) consider the chance of this actually happening: $\alpha^5$! For the commonly used $\alpha = 0.05$, this is already so phenomenally low, that this really shouldn't be your primary concern. Even two false positives in a row at $\alpha=0.05$ occur at a rate as low as $0.25\%$.

Of course, it never hurts to look beyond $p$-values. Always try plotting your data, consider the magnitude of the effect and compute a confidence interval. And of course, avoid anything that will inflate the type I error rate, such as running multiple tests and considering only significant ones, or performing stepwise regression followed by $p$-value based inference.

| cite | improve this answer | |

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.