# p-value in real life

I just learned about p-value. My understanding of p-value is as follows:

If we take a sample dataset and find out that sample's result, the p-value basically indicates how likely it is to get that (or similar) result for a dataset with similar size and variance, given that the null hypothesis is true.

If it's lower than the significance level, we reject the null hypothesis because there's a very low chance that such a result would come if the null hypothesis were true.

If it's equal to or higher, we accept the null hypothesis because it's likely that such a result would come given that it's true.

Now my question is how exactly is the p-value used in real life? The dataset taken could be an exceptional dataset and could give a completely different result than what is correct. For large datasets, even doing this 5 times could result in incorrect results (by this I mean, datasets with exceptions would hence have p-values that suggest something different than what is actually correct). Basically, how do we ensure that the sample dataset taken is accurate enough for our results and for us to accept/reject the null hypothesis completely?

• In your second paragraph, replace "or similar" with "or more extreme" and you have the right definition. – Frans Rodenburg Jun 30 '19 at 8:39

You can indeed be unlucky, and draw a sample that generates a false positive 5 times in a row, but (granted your study design and assumptions are not flawed) consider the chance of this actually happening: $$\alpha^5$$! For the commonly used $$\alpha = 0.05$$, this is already so phenomenally low, that this really shouldn't be your primary concern. Even two false positives in a row at $$\alpha=0.05$$ occur at a rate as low as $$0.25\%$$.
Of course, it never hurts to look beyond $$p$$-values. Always try plotting your data, consider the magnitude of the effect and compute a confidence interval. And of course, avoid anything that will inflate the type I error rate, such as running multiple tests and considering only significant ones, or performing stepwise regression followed by $$p$$-value based inference.