1
$\begingroup$

I have a linear regression model with 3 independent variables (let's say A1, A2, A3) and 2 different dummy variables, one for the gender (d1) and the other one for the location (d2).

When I estimate the model with all the variables included, some of independent variables are not significant, but when I add just one of the dummy variables, all of the independent variables are significant. Could you please let me know why?

And when I estimate the model in the form of f(A1, A2, A3, d1), I get different coefficients for the independent variables in comparison with the ones for f(A1, A2, A3, d2). Why?

In general, should I add all the possible dummy variables to the model at once, or it is possible to have different similar models with just one of the dummy variables included? Thanks.

$\endgroup$
2
  • 2
    $\begingroup$ Are you adding interaction terms when you add your dummies? What software are you using, & what output are you looking at? Often there is an ANOVA table, & a parameter estimates table; eg in R, there is anova(model) & summary(model). These two can provide different information & so be potentially confusing. $\endgroup$ Commented Mar 3, 2014 at 22:42
  • 1
    $\begingroup$ @gung Thanks for your reply. I don't have interaction terms and I am using STATA for the analysis. The output is a real value variable indicating the average salary of a football player. $\endgroup$
    – Ashkan
    Commented Mar 3, 2014 at 22:54

1 Answer 1

3
$\begingroup$

To get started, let's look at an example of what your regression output might look like.

Pred  Estimate  StdErr  t      p        sig
A1     1.0      0.2      5.00   0.0005  *
A2    -1.9      2.0     -0.95   0.1850
A3     4        0.1      40.0  <0.0001  *
d1    -2        1.1     -1.81   0.0539
d2     0.5      0.1      5.00   0.0005  *

Of special interest to you is the sig column, which as an * if and only if the p-value for the corresponding variable is statistically significant given all the other variables in the model.

When I estimate the model with all the variables included some of independent variables are not significant but when I add just one of the dummy variables all of the independent variables are significant.

Think of each variable as carrying some information about the response Y, and a variable being significant if it carries "enough" of that information, in some sense. You can think of the * in the table to mean that if we dropped that one variable and left the others, we would lose a significant amount of information. A lack of a * would then mean that we could drop that variable and as long as we kept the rest we wouldn't lose too much information.

Now let's say you dropped d1 because it wasn't significant, and your table now looks like this:

Pred  Estimate  StdErr  t      p        sig
A1     1.1      0.2      5.50   0.0003  *
A2    -4.1      1.2     -3.42   0.0045  *
A3     4.2      0.1      42.0  <0.0001  *
d2     0.4      0.1      4.00   0.0020  *

Let's pretend A2 is weight and d1 is sex. It might be that weight and sex carry much of the same information about Y, especially since they are correlated. Therefore when we had weight (A1) and sex (d1) in the model, each was a bit redundant when the other was present, and we could drop one as long as we kept the rest. Now once we've dropped sex, all the information that was present in both weight and sex is now present only in weight, and if we now drop weight, we will lose that information. Thus weight (A2) has become significant.

And, when I estimate the model in the form of f(A1,A2,A3,d1) I get different coefficients for the independent variables in comparison with the ones for f(A1, A2, A3, d2).

Recall that the regression model looked like this: $$ \hat{\bar{Y}}_i = \hat{\beta}_0 + \hat{\beta}_1 A_{1,i} + \hat{\beta}_2 A_{2,i} + \hat{\beta}_3 A_{3,i} + \hat{\beta}_4 d_{1,i} + \hat{\beta}_5 d_{2,i}. $$

Now once we've dropped d1, it looks like this: $$ \hat{\bar{Y}}_i = \hat{\beta}_0 + \hat{\beta}_1 A_{1,i} + \hat{\beta}_2 A_{2,i} + \hat{\beta}_3 A_{3,i} + \hat{\beta}_5 d_{2,i}. $$ If we kept the $\hat{\beta}_{p,i}$ the same, each $\hat{\bar{Y}}_i$ would now be decreased by $\hat{\beta}_4 d_{1,i}$, which would be the difference between the right-hand sides of the two equations. That doesn't really make sense, though, so the estimates of the coefficients have to be changed. The rest of the changes in the table will follow from those new estimates.

$\endgroup$
2
  • $\begingroup$ @P Schnell Many thanks for your comment. Just one question... Can I split the model into two models in a way that each model just contains one dummy variable and all the IVs? And, then interpret them separately or I should have all the possible dummy variables in just one model? $\endgroup$
    – Ashkan
    Commented Mar 4, 2014 at 0:08
  • $\begingroup$ There are a few things you can do depending on your research question, but they will have different interpretations. If you use gender only, you'll have a model for the effects of the variables marginally over the location variable. If you include both, you will have the effects each of gender and location adjusted for the other. Finally, you could fit one model for all the subjects at location 1 and another for the subjects at location 2 to get the effects of the other variables conditional on location. $\endgroup$
    – P Schnell
    Commented Mar 4, 2014 at 1:30

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.