# How are p values misinterpreted in logistic regression and how should we interpret them?

I am currently going through the book Practical Guide to Logistic Regression by Joseph Hilbe and there is this part I came across:

For whatever reason, it gives this warning, but sorta trails off and doesn't explain why this is the case. As far as it is my understanding, the p values are calculated in a similar fashion as those in linear regression, so I was curious as to why this warning was given.

• I can add my two cents here: in general, $p$ values are indeed misinterpreted for all the wrong reasons. However, the definition is tenable. So as long as the concept is clear, there is nothing to get bothered about. Aug 21, 2022 at 4:56
• The ways to mis-interpret p-values are as many as they are common. Check out this article for a (complete?) list. Greenland, S., Senn, S.J., Rothman, K.J. et al. Statistical tests, P values, confidence intervals, and power: a guide to misinterpretations. Eur J Epidemiol 31, 337–350 (2016). doi.org/10.1007/s10654-016-0149-3 Aug 21, 2022 at 6:34
• The issues with p-values are not specific to logistic regression. For a lot of discussion of common misinterpretations, see the American Statistical Association's (ASA) statement on p-values and the accompanying articles in The American Statistician. The statement itself starts on page 131 here: tandfonline.com/doi/pdf/10.1080/00031305.2016.1154108 Aug 21, 2022 at 7:02
• Dr. Hilbe is a good statistician and author (I recall he had a statistical computing column at TAS for a long time), but there are several statements in the quotation that lend themselves to misinterpretation, of which the most notable is the last assertion that the test will fail to reject the null one in twenty times. That is, of course, assuming the null hypothesis (and all the other attendant assumptions) are true and that the null hypothesis is "simple" and not "compound." I hope (and trust) the remainder does not just taper off and that it clarifies these points!
– whuber
Apr 20, 2023 at 21:09
• This text is rather confused itself. Values greater than 0.05 do not indicate that "the predictor does not contribute to the model". "the coefficient is in fact not significant when we thought it was" - nope, the term "significant" refers to the test result, not to the underlying truth. Saying that "values less than 0.05 indicate that the null hypothesis is false" suppresses the uncertainty (possibility of error) in this conclusion. I think "contribute to the model" is rather unclear wording, as the model is not the truth we are interested in. Apr 20, 2023 at 22:03

I do not see anything here unique to logistic regression. Plenty of people misinterpret p-values, whether for linear regression coefficients, logistic regression coefficient, coefficients for other generalized linear models, or from other tests. It’s basically all the same combination of misinterpreting the p-value to be the effect size and the posterior probability of the null hypothesis.

(Note that the p-value neither (exclusively) measures effect size nor is the (posterior) probability of the null hypothesis being true.)

• +1 for explicitly mentioning the last part in parenthesis. Aug 21, 2022 at 5:09
• Thanks. I think based off what you and @User1865345 have said, its just a general p value problem rather than something unique to logistic regression. Aug 21, 2022 at 5:13

OK, here's my take on "how should we interpret them?"

The p-value generally is the probability, assuming the null hypothesis (H0) were true, that we would observe something that is as far or further away from what can be expected under H0, than what we actually observed.

If the p-value is very small, we have observed something that indicates so strongly against H0 that it would very rarely happen, were H0 in fact true. This makes the H0 look incompatible with the data, and can in this way count as evidence against it (depending on how small the p-value actually is, we can differentiate between weak, moderate, strong, (...) evidence against it).

"As far or further away from what is expected" is defined in terms of the test statistic, i.e., the test statistic defines a direction of "critical deviation" from the H0. This has three implications.

Firstly, even though a certain test may not provide evidence against the H0, it is still possible that the H0 is wrong in different ways than captured by the test statistic, which could in principle be found by other tests (i.e., violations of other aspects of the probability model chosen as H0, usually framed as "violation of the model assumptions").

Secondly, probability models are idealisations and one should not think that they can be literally "true" in reality. This means that non-rejection should never be interpreted as indication that the H0 is true. At best, it indicates that the data look more or less like data generated by the H0 can be expected to look like, considering the specific aspect measured by the test statistic. As p-values do not measure effect size, it is advisable to look at a confidence interval (CI) to see how "close" to the H0 the data are in terms of the parameter of which the CI is considered (many tests correspond to CIs, so this can be very closely connected to the test statistic).

Thirdly, rejection of the H0 can be interpreted in terms of the direction encoded in the test statistic, i.e., a significant positive z-value in logistic regression (one-sided test) indicates (with the usual error probability; the stronger the smaller the p) that it is in fact more likely to observe a positive relation between explanatory variable and outcome than encoded in the H0 ("no relation"). Note that this statement avoids the implicit assumption that the model is true, which in reality won't be the case (see above).

The value of testing an H0 if we don't believe that models are true anyway is that we need models to formally encode our thinking so that we can compare it systematically with the data. Even though the H0 is not literally true, one could think about the situation as "variable x not having any impact on the outcome" (given other variables in the model if this applies), which is encoded by the H0, and not rejecting the H0 certainly means that data are compatible with this idea (for which one possible formalisation is the exact H0 we are testing).

• That all aligns with what I would normally think a p value "does". Thanks for clarifying. The text just confused me on this point (and probably shouldnt have). Apr 21, 2023 at 12:52
• @ShawnHemelstrand As I wrote in the comments earlier, I do find the text genuinely confusing. Apr 21, 2023 at 13:02