5
$\begingroup$

Correlation and linear regression are sometimes distinguished in statistics books by saying that the former is symmetric and the latter is asymmetric in the following sense: in the case of correlation, no distinction is made between dependent and independent variables, whereas it makes a difference which variables are treated as dependent and independent variables in a regression equation. To be sure, it is possible to treat $Y$ as a predictor of $X$ instead of treating $X$ as a predictor of $Y$ by using the following equations:

$X$ as a predictor of $Y$:
\begin{align} \text{Slope: }b_1 &= r_{XY} \cdot \frac{(sY)}{(sX)} \\ ~\\ ~\\ \text{Intercept: }b_0 &= \bar{y} - b_1 \bar{x} \end{align} $Y$ as a predictor of $X$:
\begin{align} \text{Slope: }b_1 &= r_{XY} \cdot \frac{(sX)}{(sY)} \\ ~\\ ~\\ \text{Intercept: }b_0 &= \bar{x} - b_1 \bar{y} \end{align} However, the regression lines to which the scatter plot will be fitted will differ depending on whether $X$ is treated as a predictor of $Y$ or $Y$ is treated as a predictor of $X$.

My question is whether something similar holds in the case of logistic regression and whether it is possible to formulate similar equations for the regression coefficients in logistic regression for when $Y$ is used as a predictor of $X$.

$\endgroup$
  • $\begingroup$ Although you already seem to understand the correlation / linear regression case, you may (or may not) still find it interesting to read my answer here: What is the difference between linear regression on y with x and x with y?, which discusses that issue. $\endgroup$ – gung - Reinstate Monica Aug 3 '13 at 14:57
  • $\begingroup$ @gung thanks for the reference! I found your explanation of the linear case very lucid, but my question is how much of it carries over to the non-linear case of logistic regression. $\endgroup$ – Guest Aug 3 '13 at 18:02
  • 1
    $\begingroup$ Just to be clear: it seems the question makes sense only in the case where (a) there is a single predictor $X$ (in addition to the implicit constant) and (b) $X$ is a binary variable. Otherwise, logistic regression of $X$ against $Y$ cannot be done. $\endgroup$ – whuber Aug 3 '13 at 18:48
5
$\begingroup$

Yes, there is a similar relationship: for circumstances where it makes sense and where both variables are coded by $0$ and $1$ (the analog of standardization), the "slope" in the logistic regression of $Y$ against $X$ equals the slope in the logistic regression of $X$ against $Y$.

Recall that (univariate) logistic regression models a binary response $Y$ in terms of a variable $x$ and a constant, using two parameters $\beta_0$ and $\beta_1$, by stipulating that the chance of $Y$ equaling one of its values (generically termed "success") can be modeled by

$$\mathbb O(Y=\text{success}) = \beta_0 + \beta_1 x$$

where "$\mathbb O$" refers to the log odds, equal to the logarithm of the odds $\Pr(\text{success}) / \Pr(\text{not success})$.

The only circumstance under which it makes sense to switch the roles of $Y$ and $x$, then, is when $x$ also is binary. That compels us to view its outcomes now as draws from a random variable $X$. The values of $Y$ must be encoded as fixed (nonrandom) values $1$ for "success" and $0$ otherwise. We might as well assume, then, that the encoding $1$="success" and $0$="not success" has been used all along for both variables.

Notice that the data in this situation can be considered a two-by-two contingency table in which the counts of all four possible combinations of $x$ and $y$ are displayed. Let the counts for $x=i$ and $y=j$ be written $n_{ij}$, for $i=0,1$ and $j=0,1$.

The conventional estimator of the parameters is obtained in the maximum likelihood theory by finding values for which the gradient of the log likelihood equals zero. In the first case, viewing $Y$ as the dependent variable, the likelihood equations are

$$\cases { 0 = n_{01} + n_{11} - \frac{n_{00}+n_{10}}{1+\exp{\beta_0}} - \frac{n_{10}+n_{11}}{1+\exp(\beta_0+\beta_1)} \\ 0 = n_{11} - \frac{n_{10} + n_{11}}{1+\exp(\beta_0+\beta_1)} }$$

When all the $n_{ij}\ne 0$ the solution is

$$\cases{ \beta_0 = \log(n_{00}) - \log(n_{01}),\\ \beta_1 = \log(n_{01}) + \log(n_{10}) - \log(n_{00}) - \log(n_{11}).}$$

Switching the roles of the variables merely permutes the subscripts of the $n$'s (although now $\beta_0$ and $\beta_1$ have different meanings, for they multiply the $y$ values instead of the $x$ values). But the symmetry of the solution for $\beta_1$ shows that it remains unchanged. This is the "slope" term and it is the perfect analog of the regression coefficient in ordinary least squares regression.


Example

Software will confirm this result. Here, for instance, are the results of the two logistic regressions in R using the following two-way table:

    Y=0   Y=1
X=0:  1     3
X=1:  2     4

Regressing $Y$ against $X$ gives $(\hat\beta_0, \hat\beta_1)$ = $(\log(1/3), \log(3/2))$ = $(-1.0986, 0.4055)$ while regressing $X$ against $Y$ gives $(\hat\beta_0, \hat\beta_1)$ = $(\log(1/2), \log(3/2))$ = $(-0.6931, 0.4055)$.

y <- matrix(c(1,2,3,4),nrow=2)
(fit <- glm(y ~ as.factor(0:1), family=binomial))
(fit.t <- glm(t(y) ~ as.factor(0:1), family=binomial))

The output suggests that both the slopes and the null deviances remain the same upon switching $X$ and $Y$:

Coefficients:
    (Intercept)  as.factor(0:1)1  
        -1.0986           0.4055  

Degrees of Freedom: 1 Total (i.e. Null);  0 Residual
Null Deviance:      0.08043 
Residual Deviance: 2.22e-16     AIC: 7.948 


Coefficients:
    (Intercept)  as.factor(0:1)1  
        -0.6931           0.4055  

Degrees of Freedom: 1 Total (i.e. Null);  0 Residual
Null Deviance:      0.08043 
Residual Deviance: 4.441e-16    AIC: 8.072 
$\endgroup$
  • $\begingroup$ +1, nice. I was going to talk about the case when X was {0, 1}, but I had to run. You beat me to it. $\endgroup$ – gung - Reinstate Monica Aug 3 '13 at 22:21
6
$\begingroup$

An important distinction here is that Pearson's product-moment correlation, the linear regression of $Y$ on $X$, and the linear regression of $X$ on $Y$ (assuming $X$ is continuous) are all linear models.

On the other hand, logistic regression is a nonlinear model / an instance of the generalized linear model. If you were to regress a continuous $X$ variable onto a binary $Y$ variable, that would be a t-test1. (The t-test, in turn, is a special case of regression / the general linear model2.) Using logistic regression to model a binary $Y$ is a different animal because there is a nonlinear transformation between the left hand side and the right hand side of the equation, namely the link function (specifically, the logit3).


(I had wanted to address the case that @whuber discusses, when $X$ constitutes only two categories coded as $0$, $1$, earlier, but didn't have time so I had to leave off. @whuber has done a good job with that topic, but I'll go ahead and explain it again, because I'll come at it from a slightly different direction, which may help some to understand it more easily, and I'll add one more detail.)

In this situation, your data consist of four counts: $n_{00}$ (the number of observations where: $X=0,~Y=0$), $n_{01}$ (where: $X=0,~Y=1$), $n_{10}$ (where: $X=1,~Y=0$), $n_{11}$ (where: $X=1,~Y=1$). The thing to remember at this point is that logistic regression is linear in the log odds of the response, and when exponentiated, the intercept is the odds of success when $X=0$, and the slope is the odds ratio4 associated with a one-unit change in $X$. Hence, $$ \exp(\hat\beta_{0;\text{ Y on X}})=\frac{n_{01}}{n_{00}}\quad\text{ and }\quad\exp(\hat\beta_{0;\text{ X on Y}})=\frac{n_{10}}{n_{00}} $$ Thus, the two intercepts will be equal if and only if $n_{01}=n_{10}$. In addition, $$ \exp(\hat\beta_{1;\text{ Y on X}})=\frac{\frac{n_{11}}{n_{10}}}{\frac{n_{01}}{n_{00}}}\quad\text{ and }\quad\exp(\hat\beta_{1;\text{ X on Y}})=\frac{\frac{n_{11}}{n_{01}}}{\frac{n_{10}}{n_{00}}} $$ But in both cases these equal $\frac{n_{11}}{n_{10}}\cdot\frac{n_{00}}{n_{01}}$, so the slopes must always be equal (as @whuber explained).


The subscripts that @whuber and I are using to index the $n$'s are switched around. Also, in @whuber's R example, he seems to be using Y=0 as success, whereas I would call Y=1 success. For example, note that for $\hat\beta_{0;\text{ Y on X}}$, he has $\log(1/3)$, whereas using my convention, $\exp(\hat\beta_{0;\text{ Y on X}})=3/1$. I duplicate @whuber's R example below; both work.

y = c(0,0,0,1,1,1,1,1,1,1)
x = c(0,1,1,0,0,0,1,1,1,1)
t(table(y,x))      # these data are the same as @whuber's
  y
x   0 1
  0 1 3            # using my conventions, exp(b0[YonX]) would be 3/1 = 3
  1 2 4            # using my conventions, exp(b0[XonY]) would be 2/1 = 2

fit.YonX = glm(y~x, family=binomial(link="logit")) 
fit.XonY = glm(x~y, family=binomial(link="logit"))

coef(fit.YonX)
(Intercept)           x 
  1.0986123  -0.4054651 
exp(1.0986123)
[1] 3  

coef(fit.XonY)
(Intercept)           y 
  0.6931472  -0.4054651
exp(0.6931472)
[1] 2

Footnotes:
1 Strictly speaking, running a t-test 'in the other direction' wouldn't quite be a logistic regression. A stronger analogy would be Fisher's linear discriminant analysis. That's because in logistic regression there is no assumption about the distribution of $X$, but LDA does assume $X$ is normally distributed and the t-test likewise assumes the residuals are. Nonetheless, given that we're starting from logistic regression, for a quick way to think about what you're doing if you were to switch (a continuous) $X$ and $Y$, calling it a t-test is close enough.
2 For help with understanding how the t-test is a special case of regression, see here: How are regression, the t-test, and the ANOVA all versions of the general linear model?
3 For help with understanding link functions and the logit transformation, it may help to read my answer here: Difference between logit and probit models, although it was written in a different context.
4 For more about logistic regression and how it's related to odds and odds ratios, see my answer here: interpretation of simple predictions to odds ratios in logistic regression.

$\endgroup$
  • $\begingroup$ thanks for the nice responses! One thing that nags me though is that it doesn't seem that they are equivalent. It seems that gung is saying that b0;YonX = ln(n01) - ln(n00), whereas whuber is saying that b0;YonX = ln(n00) - ln(n01), and that gung is saying that b1;YonX = ln(n11) + ln(n00) - ln(n10) - ln(n01), whereas whuber is saying that b1;YonX = ln(n01) + ln(n10) - ln(n00) - ln(n11). $\endgroup$ – Guest Aug 4 '13 at 11:46
  • $\begingroup$ Sorry about the confusion (perhaps I should have just left this part out--it doesn't seem to have made it easier to follow to have it explained two different ways). It's too long for a comment to explain; I edited my answer above. $\endgroup$ – gung - Reinstate Monica Aug 4 '13 at 14:37
  • $\begingroup$ On the contrary, your response was very helpful! $\endgroup$ – Guest Aug 6 '13 at 5:07

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.