1
$\begingroup$

Background

I'm trying to measure the causal impact of action on outcome (sorry for the vague names, but trying to keep this general). My data consists of the following for each record:

  • m other_features [the result of TruncatedSVD to a much larger number of features]
  • action which is a binary
  • outcome which is continuous [in all cases here, I use the log of the actual value]

Based on business knowledge of the situation, I'm confident in a causal model that states:

  • the other_features have a causal effect on action
  • the other_features also have a causal effect on outcome
  • the action have a causal effect on outcome

I tried two techniques and got different results - I wanted to know how to interpret these results.

Attempt 1: Vanilla Linear Regression

First, I tried to regress outcome against other_features and action. When I did this, I got the following results (I'm using $\beta$ to refer to linear coefficients):

  • $R^2 = 0.793$
  • $\beta_{action} = 0.0943$

This implies that the action has the effect of ~9.5% increase on outcome.

Attempt 2: Looking at Action vs Residuals

Second, I tried to regress outcome against other_features (i.e. without action). When I did this, I got the following results:

  • $R^2 = 0.793$ [very little difference with the original regression).

Then, I looked at the residuals of those with and without action and got the following:

enter image description here

This seems to indicate that action has about a ~-8.3% impact on outcome.

Questions

  1. Is attempt 2 a valid way to proceed?
  2. If yes, how should I interpret the difference between the two approaches?
  3. Is there anything I ought to be weary/careful/aware of in using this approach?

(Please let me know if it additional data/results would be helpful in interpreting these results, I can supplement the information here).

$\endgroup$
  • 2
    $\begingroup$ You are employing two different models, so it's no surprise you get different results. If you want to be consistent in Model 2, you need to explore the relationship between the residuals and the residuals after regressing action against other_features. See stats.stackexchange.com/a/46508/919 for an explanation. $\endgroup$ – whuber Sep 19 at 21:22
  • $\begingroup$ @whuber thanks, that makes sense. I understand that different models would give different results, but I'm wondering what the best way to interpret this difference is. If: (a) i've accounted for all causal factors on outcome (a big assumption, I know); (b) action is something that is decided after, and based on, other_features, then attempt 2 gives a better estimate of the causal effect of action? $\endgroup$ – roundsquare Sep 20 at 15:17
3
$\begingroup$

Your question has nothing to do with causality per se, it is actually about the mechanics of OLS. So let me leave the causal part to the end.

Thus, first, regarding the two approaches of fitting an OLS model, your second approach of regressing residuals on the variable of interest is related to what we call the FWL theorem.

Your only mistake is that you also need to regress "action" on "other_features", and only then you would regress the residuals of "outcome" on the residuals of "action". As it is, your "attempt 2" is not correct, since it is not fully adjusting for your "other_features". If you residualize both treatment and outcome, then "attempt 2" will give you numerically the same results as your "attempt 1" (barring corrections on the actual degrees of freedom).

Now, regarding the causal interpretation, first you need to decide whether "other_covariates" satisfy the back-door criterion. Second, you need to make sure a linear model is good working approximation, otherwise linear regression will not do the proper adjustment (or you need to reinterpret the coefficient as weighted average of causal effects, see Mostly Harmless Econometrics, Chapter 3).

$\endgroup$
  • $\begingroup$ Thanks! But what I am trying to do is a bit different. I'm trying to completely account for the effect of other_features on outcome and then, after doing that, see what remaining effect action has. The reason is that the causal graph would look something like other_features --> action, other_features --> outcome, and action --> outcome so it makes sense to me account completely for other_features before accounting for action. Am I doing it wrong? $\endgroup$ – roundsquare Sep 25 at 22:12
  • $\begingroup$ @roundsquare yes this is wrong. Just work out a simple linear structural model with three variables, and you will see that you need to remove the effect of "other_features" on "action" as well to identify the causal effect. $\endgroup$ – Carlos Cinelli Sep 26 at 2:10
0
$\begingroup$

Trying to establish causality is the hardest part of statistics. The short answer is no, you cannot be confident you've established causality, unless $action$ was randomly assigned. If you have customers coming to your business, and some of them have a 0 and some have a 1, that doesn't necessarily imply imposing 0 or 1 on random individuals will have the effect reported from a statistical model. Same holds if it's your business assigning the action to customers, unless you just blindly take the action, with no other considerations (A/B testing). Most likely the action is being done due to some expectation the action will be productive, i.e. $action$ is correlated with some features, seen or unseen, of the customer. If you do have randomized assignment, then your linear regression approach should pick up the causal effect in the $\beta$ coefficient. If you just want to make predictions, given that we know someones $action$ status, we can make some estimate about his outcome.

$\endgroup$
  • $\begingroup$ Unfortunately, action is decidedly not randomly assigned. However, my understanding is that as long as I've accounted for all confounders, which the deconfounder algorithm makes easier (arxiv.org/abs/1805.06826), then I can estimate the causal effect of action on outcome. $\endgroup$ – roundsquare Sep 20 at 15:21
  • $\begingroup$ At least in academia, no one will ever accept those results, because you simply can't credibly control for all cofounders in most cases. Depending on your case, a regression might be "good enough" or "better than nothing". $\endgroup$ – JPErwin Sep 20 at 18:53

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.