2
$\begingroup$

I used summary.formula from Hmisc with continuous Age and binary outcome O with test=TRUE. This returned a p-value for Age predicting O (if I understand this correctly).

I then ran a glm using Age and O (univariate logistic regression), which returned a different p-value. I thought that the p-value should be the same?

Edit:

library(Hmisc)

summary.formula(Outcome ~ cut2(Age, seq(15,75,10)), method="reverse", test=TRUE)

p-value = 0.8

x = glm(Outcome ~ Age, family=binomial(link="logit"))

p-value = 0.4

(I'm getting quite confused: the example above which gives the 0.8 p-value is binning the ages, but I was told that the p-value it returns is for all the ages. However, if I run the summary without the binning, the p-value is 0.5.)

$\endgroup$
2
  • $\begingroup$ What exactly does "summary.formula" do? What test? $\endgroup$
    – Andi F
    Mar 24, 2011 at 18:09
  • $\begingroup$ @AndyF In this particular case, a classical $\chi^2$ test (and not a trend test) as the factor Age as not been specified as an ordered one. $\endgroup$
    – chl
    Mar 24, 2011 at 23:06

2 Answers 2

1
$\begingroup$

First, you need to know what a p-value is. A p-value is the probability that you would observe results as extreme, or more extreme, than the ones you have, if the null hypothesis was in fact true.

The reason you aren't getting the same p-value in your two tests, is that you aren't examining the same null hypotheses under the same assumptions. Pick up a stats textbook.

If a relationship between two variables is approximately linear, binning will reduce statistical power, which probably explains why you get a lower p-value without binning. Think of it this way, binning treats different values within a bin as identical. The information about the differences within a bin may be valuable, but this information is ignored by statistical tests.

$\endgroup$
1
  • 1
    $\begingroup$ I like your first paragraph and about the last one, there's a nice summary about Problems Caused by Categorizing Continuous Variables on Frank Harrell's website. Should you have some more time to invest in your response, you could expand on the "why the difference" (summary.formula offers either standard $\chi^2$ test or trend test in case the variable on the RHS is ordinal; the $p$-value associated to Age treated as continuous in the GLM is derived from a Wald test). $\endgroup$
    – chl
    Mar 23, 2011 at 19:30
0
$\begingroup$

$p-value=Pr(E||H)$ where E is the "data at least as extreme as what was observed" event, and $H$ is the hypothesis, usually of the form "some set of the parameters are zero". I have used the double lines $||$ to indicate that it is not a conditional probability per se, rather a probability based on the the assumption that the null hypothesis is true.

You have not explicitly said with the null hypothesis was is both examples. If you take the time to carefully state exactly what the null hypothesis is in each case, you may find that it makes perfect sense for them to give different p-values. You may also find that the event $E$ is not the same in the two cases either. If the "extreme event" is different, then you would in general expect to see different p-values.

p-values are usually used to "reject" models rather than compare models. At least that's how I use them. A small p-value usually means (at least to me anyways) "there is a better explanation for the observed data than the null hypothesis". I usually take the size of this p-value to be a rough measure of "how much easier" such an explanation would be to find. For me, p-values of the size $0.4$ and $0.8$ just indicate that the data is fairly consistent with either model.

I'm not so sure that it says that the "binned" model is better than the "line" model. This is because there is no explicit comparison or reference to the other model made when either of these two p-values are calculated. The reason I say this is that, generally there is an implicit class of alternative hypothesis to which a p-value refers. If both models are in this implicit class when calculating the p-value based on the other, then it is does indicate that the binned model is better. But determining this implicit class is not always straight forward, unless the p-value is based on a sufficient statistic for the null hypothesis.

$\endgroup$
1
  • 1
    $\begingroup$ Any particular reason for the down vote? Seems a bit harsh to me - what is wrong with my answer? $\endgroup$ Mar 25, 2011 at 22:45

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge that you have read and understand our privacy policy and code of conduct.

Not the answer you're looking for? Browse other questions tagged or ask your own question.