3
$\begingroup$

I tried a regression in the form ${\rm logit}(Y) = {\rm coefficient}\times X + 0 + e$, where $Y$ is a binomial variable and $X$ is a factor variable with $n$ levels. I noticed that removing the intercept yields higher $p$ values. I'm wondering how to interpret it though.

Since removing the intercept makes it equal to $0$, I believe that the coefficients returned are relative to a $0$ probability of the event $Y$ and that all $X$ factors are in the $0$ state. But I think this is impossible isn't it?

$X$ are mutually exclusive factors, therefore it's impossible to have a case where no factor is $1$, at least in the presented observations. And it cannot be interpreted like the coefficient is relative to hypothetical cases in which really no one of the factors is present, because we have no data like that.

Regarding $Y$ having a $0$ intercept, wouldn't it mean forcing the probability of the event to $0$ when none of the factors is present? Again, this is an impossible case.

Nonetheless this kind of regression would allow me to retrieve pure probability range of the event Y given a factor by transforming the coefficients in the confidence intervals given as $\exp({\rm coefficient})/(1 + \exp({\rm coefficient}))$, and the $p$ values would test whether this probability is not $50\%$. This could also be a valuable result, since it would give independent probabilities for each factor.

Am I wrong?

$\endgroup$
  • 1
    $\begingroup$ unclear: $Y$ is a random variable not an event. If it takes only integer values, the regression model $Y=\beta X+\epsilon$ is inappropriate. $\endgroup$ – Xi'an Dec 29 '14 at 19:57
  • $\begingroup$ Y is a event that happens or not for an observation, say disease or not disease. It's a binomial variable that can be modeled as 0 or 1, or yes or no, or whatever. $\endgroup$ – Bakaburg Dec 30 '14 at 0:46
  • 2
    $\begingroup$ Although I gather English is not the OP's first language, & the OP is clearly confused about the nature of logistic regression & categorical data, this question seems perfectly clear & answerable to me. $\endgroup$ – gung - Reinstate Monica Dec 30 '14 at 16:39
  • 1
    $\begingroup$ @gung I will vote to reopen with some reluctance: there is so much confusion and misinformation expressed within the body of this question that whoever answers will have to take great care not to be misunderstood! $\endgroup$ – whuber Dec 30 '14 at 22:04
  • 1
    $\begingroup$ @whuber, thanks. I have tried to address the misunderstandings in the question. Let me know if you think I missed any, or if my answer may lead to other (different) misunderstandings. $\endgroup$ – gung - Reinstate Monica Dec 31 '14 at 2:41
6
$\begingroup$

The problem is that you have several misunderstandings about logistic regression and the representation of categorical data in regression models. First, we don't generally think of there being an error term in logistic regression1, so the "$e$" in your description of your model should not be there. (Also, since the predictor is a multi-level categorical variable, $\bf X$ will have to be a row vector and the $\rm coefficient$ will have to be a column vector by which it is post multiplied.) Thus, your model should be written: ${\rm logit}(Y=1|\bf X) = \bf X\boldsymbol\beta$.

Next, we need to be clear on what the logit is. Specifically, it is a transformation:
$$ {\rm logit}(y_i = 1) = \ln\bigg(\frac{p_i}{1-p_i}\bigg) $$ Here $p_i$ is the probability that $y_i = 1$, the fraction within the parentheses is the odds2 that $y_i = 1$, and $\ln$ is the natural logarithm. Thus, ${\rm logit}(y_i = 1)$ is the log odds that $y_i = 1$. When the right hand side in your model is $0$, the log odds is $0$, so the probability is $.5$, not $0$.

Lastly, when you have a categorical variable with multiple (i.e., $>2$) levels, care must be taken to represent the variable appropriately. There are a number of ways such variables can be represented; the most common is called "reference level coding" (RLC; this is the default in R). Another way is called "level means coding"3 (LMC). In R, if you suppress the intercept when you have a multi-level factor (y ~ x + 0), you will get level means coding by default. Let's consider a simple case where there is a factor with three levels, with 10 observations within each level, and where the observed probabilities of success are .2, .5, .8 (meaning that the logits will be: -1.386, 0, 1.386).

With RLC4, your design matrix will be your data matrix with a column of $1$s attached on the left side to represent the intercept, which is always present, but the column for x1 will be missing:
\begin{array}{rrrr} {\rm obs.} &{\rm (Intercept)} &x2 &x3 \\ 1 &1 &0 &0 \\ \vdots &\vdots &\vdots &\vdots \\ 10 &1 &0 &0 \\ 11 &1 &1 &0 \\ \vdots &\vdots &\vdots &\vdots \\ 20 &1 &1 &0 \\ 21 &1 &0 &1 \\ \vdots &\vdots &\vdots &\vdots \\ 30 &1 &0 &1 \end{array} The result of this strange-seeming arrangement is that the (Intercept) will represent x1 (this is the "reference level"), while x2 and x3 will represent the differences between the estimates for these levels and the reference level in your output.

If we use LMC, we won't have an intercept column, with nothing but $1$s, but you will see your missing $x1$ column, which will indicate the observations that were in the first level of the factor:
\begin{array}{rrrr} {\rm obs.} &x1 &x2 &x3 \\ 1 &1 &0 &0 \\ \vdots &\vdots &\vdots &\vdots \\ 10 &1 &0 &0 \\ 11 &0 &1 &0 \\ \vdots &\vdots &\vdots &\vdots \\ 20 &0 &1 &0 \\ 21 &0 &0 &1 \\ \vdots &\vdots &\vdots &\vdots \\ 30 &0 &0 &1 \end{array} In your output, the intercept will be missing but you will see each level represented. In this case, the estimates are the log odds of the outcome in each level directly.

Another important change to be aware of is that with RLC, the test (e.g., $p$-value) associated with the intercept is a test of whether that parameter (i.e., the log odds of 'success' in the reference level, x1) is equal to $0$ (i.e., if the probability is $.5$), and the tests associated with the other levels (x2 and x3 here) are tests of whether they each are equal to whatever x1 is. On the other hand, with LMC all tests are whether the parameter in question is equal to $0$ (again, if the probability equals $.5$).

Here is a quick R demo:

x   = as.factor(rep(c(1,2,3), each=10))
y   = c(rep(0,8), 1, 1, 
        rep(c(0,1), each=5), 
        0,0, rep(1,8)       )
RLC = glm(y~x, family=binomial)
summary(RLC)
# Call:
# glm(formula = y ~ x, family = binomial)
# 
# Deviance Residuals: 
#     Min       1Q   Median       3Q      Max  
# -1.7941  -0.6681   0.0000   0.6681   1.7941  
# 
# Coefficients:
#             Estimate Std. Error z value Pr(>|z|)  
# (Intercept)  -1.3863     0.7906  -1.754   0.0795 .
# x2            1.3863     1.0124   1.369   0.1709  
# x3            2.7726     1.1180   2.480   0.0131 *
# ... 
#     Null deviance: 41.589  on 29  degrees of freedom
# Residual deviance: 33.879  on 27  degrees of freedom
# AIC: 39.879
LMC = glm(y~x+0, family=binomial)
summary(LMC)
# Call:
# glm(formula = y ~ x + 0, family = binomial)
# 
# Deviance Residuals: 
#     Min       1Q   Median       3Q      Max  
# -1.7941  -0.6681   0.0000   0.6681   1.7941  
# 
# Coefficients:
#      Estimate Std. Error z value Pr(>|z|)  
# x1 -1.386e+00  7.906e-01  -1.754   0.0795 .
# x2 -3.511e-17  6.325e-01   0.000   1.0000  
# x3  1.386e+00  7.906e-01   1.754   0.0795 .
# ...
#     Null deviance: 41.589  on 30  degrees of freedom
# Residual deviance: 33.879  on 27  degrees of freedom
# AIC: 39.879

Notice that the residuals, the deviances, the degrees of freedom, and the AIC are identical between the two models. Importantly, notice that the Estimate and Pr(>|z|) for (Intercept) and x1 are identical in RLC and LMC. Further, notice that the Estimates for x2 and x3 in RLC are equal to the differences in the estimates in LMC: viz., 0--1.386 = 1.386, and 1.386--1.386 = 2.7726. Lastly, notice that the Pr(>|z|)s differ for x2 and x3 in RLC and LMC, as they are testing different null hypotheses.

1. See, e.g., here and here.
2. For more on probabilities and odds, see my answer here.
3. There are many more, see here for some.
4. I cover some material related to this here.

$\endgroup$
  • $\begingroup$ Many thanks gung! very clear! two more questions: using a factor bX with many levels, isn't the same of having a multiple regression with b1x1 + b2x2 + b3x3 but in this case with every x mutually exclusive? so the coefficient estimated is relative to each single level being zero, independently from the others. But if you have another, unrelated covariate, say SEX (the usual), the coefficient for this covariate is relative to what? $\endgroup$ – Bakaburg Jan 2 '15 at 11:57
  • $\begingroup$ Second: if LMC allows me to compare to a condition in which P(Y) is 50%, that is can or cannot happen equally, I see that this could be very useful in clinical studies. But then why everyone use RML? $\endgroup$ – Bakaburg Jan 2 '15 at 12:02
  • $\begingroup$ @Bakaburg, I'm not sure I follow all your questions. You can't have all levels of a categorical variable represented w/ dummies & have an intercept without having perfect multicollinearity (see my answer here). If you include a continuous covariate in a model, it is tested against 0, just as normal. Re: why RLC is the default, I don't know, you can use any number of coding schemes that are all equally valid, it may be part tradition, it may be part that there is often a natural reference level (ie, the control) against which you want to compare. $\endgroup$ – gung - Reinstate Monica Jan 2 '15 at 15:03
  • $\begingroup$ You don't really want to interpret the individual p-values from either coding scheme anyway. You test a categorical covariate by dropping the entire variable (all levels / dummies) and fitting a nested model. $\endgroup$ – gung - Reinstate Monica Jan 2 '15 at 15:05
  • 2
    $\begingroup$ I very sympathetic, @Bakaburg. & you seem to be doing well. It's just that some topics are harder to explain. $\endgroup$ – gung - Reinstate Monica Jan 6 '15 at 18:47

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.