2
$\begingroup$

I'm guessing this is a pretty basic question, but I am having a hard time wrapping my head around it.

So my understanding with linear regression, is that it shows how much a change in X, will cause a change in Y. And the same with multiple linear regression.

But can the same be said about logistic regression? What if both of the variables are nominal? Can you do logistic regression this way?

I am currently running an ordinal variable against a nominal one, and I get similar results when I alternate independent vs dependent. So, my question is should logistic regression be viewed as explaining causal relationships as we do with linear regression? Or is it possible that causality is overwritten by multicollinearity? Leaving us with strictly correlational inferences? Thanks

$\endgroup$
2
  • 9
    $\begingroup$ Linear regression is not causal in the usual sense of causal $\endgroup$
    – Henry
    Aug 9 '19 at 19:08
  • $\begingroup$ @Henry Care to elaborate a little bit more? That does causal mean in this context then? $\endgroup$
    – nalzok
    Aug 10 '19 at 16:20
13
$\begingroup$

Causality has nothing to do with regression. You can regress any variables that are not causally linked. Better way of thinking of regression is "response of Y to X", or "relationship of Y and X". And in this regard it does not matter if the link function is identical (as in the normal regression) or logit function (as in the logistic regression). In logistic regression the response of Y to X will just have different shape due to the logit link function than it would have in normal regression.

So the answer is no, there are no differences in causality.

$\endgroup$
6
$\begingroup$

Neither of them establish causality - unless we talk about specific experimental set-ups (e.g. randomized studies). They show correlation. E.g. a linear model for maximum daytime temperature with the ice cream sales as a predictor cannot show that selling ice cream causes higher temperature, even if higher ice cream sales correlate with higher temperatures. Of course, I could come up with a whole convoluted story of how this works (e.g. all those freezers increase the outdoors temperature in order to produce ice cream), but that would all be post-hoc reasoning. In my example, we can of course immediately tell that I probably got my causality the wrong way around, but in many practical examples that is a lot less clear.

Just the same thing is happening, if I do a logistic regression for "Was it a very hot day" (yes/no).

$\endgroup$
1
$\begingroup$

A causal relationship is defined by a structural model that relates an outcome to its causes. Regression is one way of estimating the parameters of the structural causal model (there are other ways). If the structural model takes the form of a logistic regression model, then a logistic regression model is one way of recovering the true causal parameter. If the structural model takes the form of a linear regression model, then a linear regression model is one way of recovering the true causal parameter. With a binary outcome, a logistic regression model makes more sense because the the output of a logistic regression model can be interpreted as a probability, and part of the outcome-generating process may involve drawing a 0 or 1 with a given probability. With a continuous outcome, a linear regression may make more sense. The choice of model you use should depend on the form of the structural causal model you are trying to approximate. (Note that both linear and logistic regression can be used for both binary and continuous outcomes.)

What many of the posters here are arguing is that regression (logistic or linear) are not inherently causal methods; they can be used to extract a causal parameter from data, but they can be used for other purposes, too. My view is that as long as the predictor precedes the outcome temporally, regression is inherently causal, regardless of your study design. The difference is that the parameter you may estimate from your model may be a biased estimate of the causal parameter. The degree of bias depends on qualities of your design (e.g., whether you have randomized your treatment, whether you are implicitly conditioning on a consequent of treatment, whether you have collected enough variables to remove confounding). Including covariates in a linear or logistic regression is one way to attempt to remove the bias of a causal effect estimate.

When the structural causal model is perfectly reproduced by the model you specify, the causal parameter of interest will be estimated without bias. This tends not to be the case, and so estimating causal parameters using regression can leave you with a biased estimate (if bias is induced by your design). It is for this reason that other answers have been strong in claiming regression is not an inherently causal method; in a bias-inducing design, it's almost impossible to specify a regression model that correctly reproduces the structural causal model that underlies the data.

A final note: people often say correlation does not imply causation, but correlation without confounding does imply causation. Regression is one way to remove confounding. The effectiveness of logistic regression or linear regression at doing so depends on generally unknown qualities of the data-generating structural causal model.

$\endgroup$
3
  • $\begingroup$ "My view is that as long as the predictor precedes the outcome temporally, regression is inherently causal, regardless of your study design." --- I think you will find that this is the post hoc ergo propter hoc fallacy. $\endgroup$
    – Ben
    Aug 10 '19 at 23:10
  • $\begingroup$ That fallacy is just the failure to control for confounding. If what follows is not caused by what precedes, then an appropriate causal analysis or design will demonstrate that (i.e., when an unbiased estimate of the causal parameter is zero). $\endgroup$
    – Noah
    Aug 11 '19 at 4:01
  • $\begingroup$ I'm afraid I don't agree with your framing of the issue. In most cases it is not possible to observe all possible confounding variables. The purpose of controlled randomised designs is to avoid having to do this. Saying that regression is inherently causal, except when we don't properly estimate the confounders, is the exception that swallows the rule. $\endgroup$
    – Ben
    Aug 11 '19 at 8:23
0
$\begingroup$

In linear regression, we can speak about the change in the conditional mean of one variable due to a change in the other variable. That's the closest thing to "causality" we get. (It has the rather amusing consequence that a change in either variable's $z$-score is associated with a smaller mean change in the other variable's $z$-score, which, like the twin paradox, is explicable in terms of different variables-held-constant definitions of partial derivatives.)

Logistic regression is just linear regression where one variable has been transformed, so we get $y=\sigma(Wx+b)$ instead of $y=Wx+b$. Thus a change in $X$ "causes" a change in the conditional mean of $\Sigma:=\sigma^{-1}(Y)$, and vice versa. But this can't be restated in terms of changes in $X$ and $\Bbb EY$, because nonlinear transformations don't commute with expectations.

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.