# Does size of p-value matter beyond knowing if it is greater/less than a predetermined significance level?

In my understanding, hypothesis tests are equivalent to confidence intervals. Because of this, I initially believed $$p$$-values should only be reported in a binary sense i.e. is $$p<\alpha$$ or $$p>\alpha$$ so you can either reject or fail to reject the null respectively.

But what happens when you do your test at the 5% level and is not significant at 5% level but at the 10% level? That means we are 90% confident that the true $$\mu$$ lies outside the 90% confidence interval, which seems quite high.

Is it not valid to reject at the 10% level in that case? Maybe I was stringent with the 5% significance.

Also since these cases only occur when $$p$$-values are between 2 significance levels (at least in the case of symmetric tests), doesn't that mean the size of $$p$$-values actually matters?

• Yes, size of p-value does matter. No, you should not change your $\alpha$, which you fixed before the experiment, after the fact, because the results are significant for a different $\alpha$. Also, your interpretation of the confidence interval is wrong. Jan 6, 2022 at 9:39
• What if both $\alpha$'s are acceptable for the scientist? There might not be much difference between 90% and 95% confidence for the application or test at hand. If before I worked with 95% confidence but then I can reject at the 90%, what's the harm? Jan 6, 2022 at 9:43
• Your understanding of p-values, $\alpha$'s, and confidence intervals is lacking. This is not a simple difference of 5% in probability. Changing $\alpha$ from 0.05 to 0.1 is not about saying that you are 95 or 90% confident that the true value lies inside/outside this range. It is much more nuanced and, some would say (those whose name starts with B) non-intuitive. But to answer your question, what's the harm? The harm is that this procedure is completely arbitrary, subjective, and non-scientific. This is how you get non-reproducible results which are of no use to anyone. Jan 6, 2022 at 10:09
• I am not a p-value convert and they continue to be poorly used, even misused and abused. IMHO, the worst mistake in using p-values is to classify them into arbitrary significance classes of 10%, 5% or 1% - with anything else being non-significant. The actual p-value should be reported every time so the reader can intepret the meaningfulness (not necessarily the significance) of the results. Having such transparency is why the size of the p-value matters. Jan 6, 2022 at 10:11
• The p-value is just a measure of the effect size in terms of a probability. What you do with it is subjective. Depending on your situation you might create a more objective formulation for getting to a decision (e.g. optimise the expected profit of a certain decision). Say you have to gamble on some rejection (and it costs you money to reject) would it matter whether you have a 5% or a 10% probability of being false? Jan 6, 2022 at 11:20

• In the Neyman-Pearson approach, you have a clear set of decision rules (e.g. rejecting the null hypothesis if $$p < \alpha$$), which, if followed correctly, will limit the rate at which you make type 1 and type 2 errors.
So, you can either take Fisher's approach, and worry about the actual p-value, or Neyman and Pearson's, and worry about whether or not $$p < \alpha$$, but you can't do both.