Control variables and other independent variables

Question

I'm trying to do a multiple-regression analysis in Stata. I'm new to this subject, so I need someone to explain it to me in "simple words".

I did an multiple-regression analysis: my control variables turned out to be "not significant", but I still want to include them in my analysis to show that I have controlled for them, because they are expected variables. How can I explain the relationship between the controlling variables and my other significant independent variables? What do the insignificant controlling variables say about the significant independent variables? And how do I know if my significant independent variables are "good" or not, when all the controlling variables are insignificant?

LOJ = Dependent variable DI, TI = Independent variabels

Answer 1

When you say "control", I suspect you mean that you have a primary variable of interest, and then you have other variables that are potential confounders.

In the presence of a confounder, the effect size of the primary variable may appear higher or lower than it actually is (Simpson's Paradoxon / omitted variable bias). To "control" for this effect (see also here), the confounder must be added to the multiple regression (otherwise you lose the ability to infer the causal effect of the primary variable).

Note, however, that not all variables should be added to a regression. In some cases, adding a variable can even produce bias (collider). The causal structure determines which variables should go into the regression, regardless of significance or how they affect the estimates of other variables. See more comments here and in the excellent paper by Lederer et al., 2019.

Answer 2

While it is common practice to "control" (put another independent variable in regression) for any potential confounders, this isn't always the best case. Sometimes "controlling" for variables introduces confounders into the regression.1 It all depends on the underlying relationship between your variables.

As @Florian Hartig and @boulder allude to, you are likely interested in explaining (causal inference), not just predicting. (Although, often there is lots of overlap.) To Explain or Predict

First, I would recommend drawing a causal graph 3 of your system. It won't be perfect, because well, life is messy. Just a simple diagram with all relevant variables and arrows that show how they are connected. This graph will be based on the scientific theory and allows one to show others how they think the system operates (generates data). After you have this causal graph then you can consider which variables would reduce confounding and which variables would introduce confounding.

Example of causal graph:

EDIT:

If you wanted to understand what produces traffic jams, you might use rush hour and accidents. However, if you don't include bad weather you might overestimate the effect of accidents. Because bad weather happens to increase traffic directly (from slower speeds presumably) and indirectly by increasing accidents which then go onto increase traffic. So, given this graph of the system you would want to include ("control for") bad weather. Then you could interpret your coefficients as given my causal graph of the system and given my statistical model choice and and given my data and given a certain amount of bad weather and given a certain amount of accidents the effect of rush hour is ___ on traffic. But don't let this example lead you to believe that adding extra variables is harmless. As stated earlier, depending on the structure of your graph/system sometimes it hurts inference.

If you want more plain language discussion of this topic I highly recommend Richard McElreath's book 6 and lectures 7. If you have access to academic journals you may find a copy of his book there for free.

Answer 3

My suggestion

1. Model: Run a model only with the independent variables you are interested in.

2. Model: Add the variables you want to control for into the model.

Reason for the suggestion

If you add all the variables together this might change the coefficients of your predictors, i.e. increase/ decrease or even change the direction of the coefficients. If there is no contradiction between borh models you can interpret the coefficient of the independent variable from the second model.

Example

I'll use an example that I was taught by a statistic professor of my university. Let's assume you want to predict success at work (variable: success; higher is better) and you collected different variables to do this. Among others, you know what the average grades of the people are (variable: grades; higher is better) and how much people are afraid of exams (variable: fear; higher is worse, i.e. more fear). You run a multiple regression and you get the following results:

Variable   Estimate   Std. Error   t value   p
Intercept  0.00       0.12         0.00      1.000  
fear       0.33       0.14         2.42      0.019  
grades     0.67       0.14         4.85      < 0.001

Incredible: The more someone was afraid of exams the more successful he/ she is at work. Who would have thought that! But wait: Actually, I made up the data and in fact the correlation matrix looks like this (see R code below):

         success   fear   grades
success  1         0      .5
fear     0         1      -.5
grades   .5       -.5     1

As you can see, actually there is no correlation between fear and success. Keep in mind the relationship between correlation and regression: If the correlation is zero, the estimate of the regression will be zero, too. Why does the estimate of fear incease in the multivariate regression? We have a suppression effect here: fear does not explain variance in the dependent variable but in the other independent variable grades. This means you would draw a wrong conclusion if you would only look at the multivariate regression! Altough the estimate of fear is positive and significant in the multivaraite regression you can not say that having more fear increases success at work. And you can only understand that this conclusion would be wrong if you compare the univariate (1. model) and the multivariate (2. model) regression.

Real world examples and a bigger picture

The example above shows why I disagree with @Florian Hartig who wrote that "suppressors are secondary, and seldom crucial for the scientific conclusions". To stress this point, please note that "Reversal in association (magnitude or direction) are examples of Simpson's Paradox, Lord's Paradox and Suppression Effects. The differences essentially relate to the type of variable" (here). Thus not only the suppression but Simpson's and Lord's paradox can occure in multivariate regression, too, hence, the suggestion to run two models in order to avoid misleading results applies here, too. You can find real world examples here or here. Also, looking for suppression on cv shows that people do encounter this problem. See here or here, for example.

Last notes

Also keep in mind that there are concepts that are close to suppression. So "in order to interpret these results you will have to think about / have some theory of why the relationships manifest as they do. That is, you will need to think about the pattern of causal relationships that underlies your data" (here).

R Code

require(MASS)
df <- data.frame(mvrnorm(50, mu = c(0, 0, 0),
               Sigma = matrix(c(1, 0, 0.5,
                                0, 1, -0.5,
                                0.5, -0.5 ,1), ncol = 3),
               empirical = TRUE))

names(df) <- c("success", "fear", "grades")

cor(df)

summary(lm(success ~ fear, data= df))

summary(lm(success ~ fear + grades, data= df))

Answer 4

Keep in mind the goal of regression through this simple example:

Let's say you want to determine the relationship between the price of homes in the state you live in and the size of the home. And you are clever, so you also include a control variable which indicates whether the home is in a rural or urban district of the state. To study this problem, you randomly pick 311 homes in your state and record those three measurements for each.

Now the question that regression is trying to answer is as follows: given the relationships between the variables observed (which we obtain through the coef. column of the output), are these relationships likely to be simply a coincidence of the particular 311 homes we randomly picked, or is it likely that regardless of which homes ended up in the sample, you would still be likely to observe the same relationships.

(slight tangent on what we mean by significance)

How do we define what "likely" means in this context? This is where the p > |t| column comes in. In general, you will often see a significance level of 5% being used. That is, you want your p > |t| column to be less than .05 and if it is you call the variable "statistically significant". The exact reason for this starts to get technical, but the general idea is as follows: suppose there is no relationship between the things we are studying. If we were to repeatedly take random samples and perform the regression, how often would the regression indicate there was a significant relationship between the things? The answer is around 5% of the time.

OK, back to the concept of control variables.

In this scenario, the reason for including a control variable is rather obvious-- real estate prices are highly influenced by location so by including a location control variable you will greatly increase your understanding of the relationship between price and size. In fact, by not including a control variable, it is possible that you completely miss the true relationship between your variables of interest. Consider this situation:

Let's say you have a population of interest and you want to know the relationship between Y and X. In this population, people are 50-50 split between membership in some group (this group membership is our control variable). Suppose that if people belong to the group, the coefficient between Y and X is a large positive number and if not it is large negative number. If we ran a regression without including this group membership we would find that our regression would tell us there is no relationship between Y and X! The large positive association that exists for people in the group would be completely negated by the large negative association between Y and X that people not in the group have.

But it can get even worse! Look at https://en.wikipedia.org/wiki/Simpson%27s_paradox . Not only can relationships be negated by not including the correct control variables, doing so may give you relationships that are the opposite of what they are in reality. You think Y and X are positively correlated, but in reality after proper controls they are negatively correlated. This is why we sometimes see 2 studies on the same subject that come to opposite conclusions!

Now, back to your questions.

How can I explain the relationship between the controlling variables and my other significant independent variables? What do the insignificant controlling variables say about the significant independent variables?

Given your results, it would appear that location (I am assuming this is what the categorical variable BO represents) has no impact on the relationship between your dependent and independent variables. Essentially, because the p > |t| is so large, it is highly likely that another random sample of data on these variables would produce entirely different coefficients for Oppland Oslo and so on. Which is to say, it would not at all be unusual to see that another sample produced a slightly positive coefficient for Oppland instead of a slightly negative one.

Because the p > |t| values are so large for this variable it should almost certainly be excluded and instead you should just note that you attempted to control for location but found no relationship.

Now AB is a different story because one level is highly significant while the others are not. I would guess the improvement you see (according to r^2 and adj r^2) in the bigger model is almost entirely because of the >= 10 ganger level of AB. One thing you could do here is instead turn the variable into a binary variable and see what happens. But since this is an ordinal categorical variable (there is a natural way to order the levels) sometimes slightly non-significant levels are kept if the coefficients "make sense".

To specifically answer your question: holding everything else constant we would expect an observation with 10+ ganger to have a higher LOJ than one with <10 ganger, on average.

And how do I know if my significant independent variables are "good" or not, when all the controlling variables are insignificant?

It could just be the case that X and Y have a highly causative relationship and that not many other things influence the relationship between them very much. Or you just picked bad control variables. It really doesn't mean much on its own. It can be a cause for concern, though. Like if you are doing the house price study and it comes out that location isn't significant I would be very concerned about the data quality.

This question really gets down to the heart of the issues with regression. Why do we see studies that say x causes cancer, and then another that says the exact opposite? Because the whole thing only works when we have included all variables relevant to the problem. In nutrition studies, as an example, this is incredibly difficult. Because in reality we can never know for certain that we haven't missed an incredibly influential lurking variable that, if included, would change all the results.