Chi-squared versus logistic regression

Question

I used summary.formula from Hmisc with continuous Age and binary outcome O with test=TRUE. This returned a p-value for Age predicting O (if I understand this correctly).

I then ran a glm using Age and O (univariate logistic regression), which returned a different p-value. I thought that the p-value should be the same?

Edit:

library(Hmisc)

summary.formula(Outcome ~ cut2(Age, seq(15,75,10)), method="reverse", test=TRUE)

p-value = 0.8

x = glm(Outcome ~ Age, family=binomial(link="logit"))

p-value = 0.4

(I'm getting quite confused: the example above which gives the 0.8 p-value is binning the ages, but I was told that the p-value it returns is for all the ages. However, if I run the summary without the binning, the p-value is 0.5.)

Answer 1

First, you need to know what a p-value is. A p-value is the probability that you would observe results as extreme, or more extreme, than the ones you have, if the null hypothesis was in fact true.

The reason you aren't getting the same p-value in your two tests, is that you aren't examining the same null hypotheses under the same assumptions. Pick up a stats textbook.

If a relationship between two variables is approximately linear, binning will reduce statistical power, which probably explains why you get a lower p-value without binning. Think of it this way, binning treats different values within a bin as identical. The information about the differences within a bin may be valuable, but this information is ignored by statistical tests.

Answer 2

$p-value=Pr(E||H)$ where E is the "data at least as extreme as what was observed" event, and $H$ is the hypothesis, usually of the form "some set of the parameters are zero". I have used the double lines $||$ to indicate that it is not a conditional probability per se, rather a probability based on the the assumption that the null hypothesis is true.

You have not explicitly said with the null hypothesis was is both examples. If you take the time to carefully state exactly what the null hypothesis is in each case, you may find that it makes perfect sense for them to give different p-values. You may also find that the event $E$ is not the same in the two cases either. If the "extreme event" is different, then you would in general expect to see different p-values.

p-values are usually used to "reject" models rather than compare models. At least that's how I use them. A small p-value usually means (at least to me anyways) "there is a better explanation for the observed data than the null hypothesis". I usually take the size of this p-value to be a rough measure of "how much easier" such an explanation would be to find. For me, p-values of the size $0.4$ and $0.8$ just indicate that the data is fairly consistent with either model.

I'm not so sure that it says that the "binned" model is better than the "line" model. This is because there is no explicit comparison or reference to the other model made when either of these two p-values are calculated. The reason I say this is that, generally there is an implicit class of alternative hypothesis to which a p-value refers. If both models are in this implicit class when calculating the p-value based on the other, then it is does indicate that the binned model is better. But determining this implicit class is not always straight forward, unless the p-value is based on a sufficient statistic for the null hypothesis.