1
$\begingroup$

If you have a variable you want to investigate but you realize you should probably control for a number of variables, is it okay to do this for every variable you find interesting? Say you have 5 variables of interest and all have different variables you want to control for, would it be okay to run 5 multiple regressions or would this be p hacking? Do I need to fit them all into 1 multiple regression?

$\endgroup$
2
$\begingroup$

You ask a couple of questions here so let's unpack.

1.) "If you have a variable you want to investigate but you realize you should probably control for a number of variables, is it okay to do this for every variable you find interesting?"

Normally, you would use the literature to determine which variables you should examine that need to be controlled for. However, just because these variables are controlled for in other studies in the literature, does not mean they need to be controlled for in your data. You should test for confounding between your variable of interest and any extra variables. If there is shown to be no confounding it is okay to remove the variables you had originally thought to control for. If there is confounding, you should leave the variables in your model to control for them. Here is a link to an article which explains confounders and when to control for them in detail. I generally go with the 10% rule. Just click on view PDF once you click on the link. Hernan 2002 You do not want to control for everything you find interesting because then there is no limit on how many variables you could put in your model. From a research perspective, the simpler the better, as it makes your results easier to understand and easier to draw conclusions.

2.) "Say you have 5 variables of interest and all have different variables you want to control for, would it be okay to run 5 multiple regressions or would this be p hacking? Do I need to fit them all into 1 multiple regression?"

This depends on your research question. If your goal is to build a predictive model, and these 5 variables are what are used in the literature and you have justification to use them, feel free to use them to build your model. In this case, you don't really have to worry about controlling for anything else, because you are only interested in explaining as much about your outcome variable as you can. For instance, say you wish to build a predictive model, and income is one of the predictor variables in your model. In another study, it may be worth it to control for race or ethnicity, as it could cloud your interpretation surrounding income. However, in a predictive model, if income is statistically significant, that's all you really need to know about it.

If you are attempting to build an associative model, which is a model that examines how one variable influences another, you would want to control for any potential confounders or other variables that could cloud your results. If you are interested in 5 variables as you state in the question, and your goal is to examine the variable rather than to build a model that explains as much of your outcome as possible, you would build 5 different regression models, and in each of them include variables that should be controlled for (confounders).

Hope this helps.

$\endgroup$
  • $\begingroup$ Thank you! I found the reference: "A second common approach, strategy 2, compares adjusted and unadjusted effect estimates. If the relative change after adjustment for certain variable(s) is greater than 10 percent, for example, then the variable(s) is selected" When the authors mention adjusted and unadjusted "effect estimates" do they mean the coefficients or the p values? Or neither? Sorry English is not my first language. $\endgroup$ – Paze Jan 11 at 18:47
  • $\begingroup$ No worries! In the case of the 10% rule you can use a 10% change in the coefficient if using linear regression or a 10% change in the odds ratio if you are using logistic regression. Most of the time both will result in a 10% change as the odds ratio is just the exponentiated coefficient. As the 10% rule is a loose rule (much like a p-value is) there is some flexibility with how it is applied. $\endgroup$ – coconn41 Jan 11 at 19:15
  • $\begingroup$ Thank you for the explanation! $\endgroup$ – Paze Jan 11 at 20:47
  • $\begingroup$ Sorry one more thing, does the 10% change inspection reset every time you add a variable or is it 10% from the baseline (the first variable that was added)? Example: I have 10 independent variables and I add 1 variable, 2 variables not much happens, but a bit, 3 variables, still not much but again a bit, 4th variable and a bit happens again, 5th variable and now the change is more than 10% from the first variable I added. Should I consider it a collinear variable according to this tactic? $\endgroup$ – Paze Jan 11 at 20:50
  • $\begingroup$ Also do they mean 10% across every variable, so if a 10% change in ANY variable occurs when adding a variable, it is a collinear to that variable? $\endgroup$ – Paze Jan 11 at 20:52

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.