12
$\begingroup$

I am currently working on a problem in which we have a small dataset and are interested in the causality effect of a treatment on the outcome.

My advisor has instructed me to perform a univariate regression on each predictor with the outcome as the response, then the treatment assignment as the response. Ie, I am being asked to fit a regression with one variable at a time and make a table of the results. I asked "why should we do this?", and the answer was something to the effect of "we are interested in which predictors are associated with the treatment assignment and the outcome, as this would likely indicate a confounder". My advisor is a trained statistician, not a scientist in a different field, so I'm inclined to trust them.

This makes sense, but it's not clear how to use the result of the univariate analysis. Wouldn't making model selection choices from this result in significant bias of the estimates and narrow confidence intervals? Why should anyone do this? I'm confused and my advisor is being fairly opaque on the issue when I brought it up. Does anyone have resources on this technique?

(NB: my advisor has said we are NOT using p-values as a cut off, but that we want to consider "everything".)

$\endgroup$
  • 6
    $\begingroup$ If by "univariate regression" your instructor would include drawing a scatterplot, then it's wise advice indeed. And since no regression you care about ever should be conducted without plotting, you will get some useful information. Do it all at once, if you can, with a scatterplot matrix, and show some robust smooths with them. The advantages will be obvious when you see the various ways in which your variables can depart from exhibiting linear relationships. $\endgroup$ – whuber Jan 23 at 23:09
  • 1
    $\begingroup$ What if the response data are binary, and we are using a glm with a logit link? Your explanation is certainly clarifying for the linear case, and now that I think about it, the use of scatter plots would be natural $\endgroup$ – Marcel Jan 23 at 23:26
  • 5
    $\begingroup$ I was worried you might ask that :-). Actually, a good smooth can still provide great insight. It helps to jitter the response so you can make out its distribution. Here's an example of such a plot: stats.stackexchange.com/a/14501/919. I illustrate another solution at stats.stackexchange.com/a/138660/919. $\endgroup$ – whuber Jan 23 at 23:29
  • 3
    $\begingroup$ This univariate regression before multivariable regression technique is called "purposeful variable selection" in Hosmer and Lemeshow's book "Applied Logistic Regression" $\endgroup$ – Great38 Jan 24 at 3:29
  • 7
    $\begingroup$ Beware -- a variable may show no relationship in a unvariate regression, yet be important in the multivariate relationship. $\endgroup$ – Glen_b Jan 24 at 6:57
2
+25
$\begingroup$

The causal context of your analysis is a key qualifier in your question. In forecasting, running univariate regressions before multiple regressions in the spirit of the "purposeful selection method" suggested by Hosmer and Lemenshow has one goal. In your case, where you are building a causal model, running univariate regressions before running multiple regression has a completely different goal. Let me expand on the latter.

You and your instructor must have in mind a certain causal graph. Causal graphs have testable implications. Your mission is to start with the dataset that you have, and reason back to the causal model that might have generated it. The univariate regressions he suggested that you run most likely constitute the first step in the process of testing the implications of the causal graph you have in mind. Suppose that you believe that your data was generated by the causal model depicted in the graph below. Suppose you are interested in the causal effect of D on E. The graph below suggests a host of testable implications, such as:

  • E are D are likely dependent
  • E and A are likely dependent
  • E and C are likely dependent
  • E and B are likely dependent
  • E and N are likely independent

enter image description here

I mentioned that this is only the first step in the causal search process because the real fun starts once you start running multiple regressions, conditioning of different variables and testing whether the result of the regression is consistent with the implication of the graph. For example, the graph above suggest that E and A must be independent once you condition on D. In other words, if you regress E on D and A and find that the coefficient on A is not equal to zero, you'll conclude that E depends on A, after you condition on D, and therefore that the causal graph must be wrong. It will even give you hints as to how to alter your causal graph, because the result of this regression suggests that there must be a path between A and E that is not d-separated by D. It will become important to know the testable dependence implications that chains, forks, and colliders have.

$\endgroup$
1
$\begingroup$

Before I try to answer I'd like to point out that type of data and its distribution can affect the way you evaluate/regress/classify it.

Also you might want to look here for the method that your advisor might want you to use.

A bit of background. While using a model selection tool is a possibility, you still need to be able to say why a predictor was used or left out. Those tools can be a black box. You should fully understand your data and be able to state why a particular predictor was selected. (Especially, I'm assuming for a thesis/master's project.)

For example, look at the price of houses and age. The price of houses generally decreases with age. Therefore when you see an old house with a high price in your data it would look like an outlier to be removed but that's not the case.

As to (NB: my advisor has said we are NOT using p-values as a cutoff, but that we want to consider "everything".) p-values aren't the be all and end all of everything but they can be helpful. Recall algorithms/programs are limited and cannot view the whole picture.

As to why you might univariate regression on each predictor/treatment assignment.

This could be to aid in selecting the predictors to include in the basic multivariate model. From that basic model, you would then look to see if those predictors are significant and should remain or if they should be removed with the aim to get a parsimonious model.

Or it could be for you to better get an understanding of the data.

$\endgroup$
  • 1
    $\begingroup$ My wife and I bought an old house but could not afford a historic house, so your example has an easy counter-example. $\endgroup$ – Nick Cox Jan 30 at 15:56
  • $\begingroup$ True. I meant to talk about the price of houses actually. How the prices of houses generally decrease with age. Therefore when you see an old house with a high price that it would look like an outlier to be removed. I'll edit that point. Thx. $\endgroup$ – Apocryphon Jan 30 at 16:12
0
$\begingroup$

I think your supervisor is asking you to perform a first analysis of the data with the objective of identifying if any of the variables can explain a significant fraction of the variance in the data.

Once you concluded if any of the variables can explain some of the variability, then you will be able to assess how they work together, if they are colinear, or correlated between each other, etc. In a purely exploratory phase to have a multivariate analysis could make a first assessment harder, because by construction each variable you would be removing the effect of the others. It could be harder to assess if any of the variables could explain any of the variation.

$\endgroup$
0
$\begingroup$

That may be an approach to understand data, but experience shows that predictions will vary when you use all predictors combined and each one predictor one by one. That's just something we do understand predictability of data and understand what needs to be done for future steps.
I have seen many times when with all variables the p-value says some variables are not significant but with those non-significant variables alone, they were significant enough. That's due to mixed effect: it's not that your supervisor is wrong, but to understand data we have to do this.

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.