3
$\begingroup$

I have some revisions to make on my Master's thesis. One was to add dummy and control variables as I am also looking into demographics.

I have 5 independent variables which were previously shown to relate to customer satisfaction. 4/5 were significant. I now added age, sex and education. However, then one of the 4 significant independent variables becomes insignificant.

Does it make sense -- after checking the dummy variables along with the independent variables -- to redo the regression with just the independent variables? Seems more robust to just have the primary variables, and also more simplistic. What is best practice here?

Improve this question
$\endgroup$
4
  • 8
    $\begingroup$ Do not throw out insignificant variables, report your model results for all the variables. $\endgroup$
    – user2974951
    Commented Aug 6 at 11:47
  • 1
    $\begingroup$ I wonder if part of what's going on has to do with a data sample that is too small to fit the model adequately with the larger number of predictors. The answer from Graham Wright (+1) is certainly correct in general, but you might have information that could point toward what's going on in your specific case. Consider editing the question to provide more details: the nature of the outcome variable, the number of observations (size of minority class for binary outcome), number of independent variables, summaries of the model results with and without the added variables. $\endgroup$
    – EdM
    Commented Aug 6 at 12:34
  • $\begingroup$ It also might help to show results of what Frank Harrell recommends in a comment on that answer: a model in which you predict the values of the independent variable that is no longer "significant" from the other independent variables in the model. $\endgroup$
    – EdM
    Commented Aug 6 at 13:11
  • $\begingroup$ Your two previous threads asked essentially the same question, and got essentially the same answers! $\endgroup$
    – Nick Cox
    Commented Aug 7 at 11:59

1 Answer 1

Reset to default
13
$\begingroup$

Regression is a tool designed to help you answer particular research questions. If your question is "Does variable A impact some dependent variable Y?" then you obviously need to include variable A in your model; otherwise you aren't answering the question you care about.

My guess is that reviewers suggested adding "control" variables to your model (like age, sex and education) to guard against the possibility that the observed correlation between A and Y was merely due to the fact that people who have "more" A also tend to be older or more educated or something, and it is those background differences, rather than the effect of A itself, that is causing the correlation. This is precisely what regression analysis is for - isolating the effect of one key independent variable after holding other "control" variables constant.

If you find that, after controlling for other variables, A is no longer significant then you have found the answer to your question: A does not relate to the dependent variable after controlling for demographic factors. So make that be your conclusion.

An insignificant result is not a failure. The goal of research is not to find significant results, but to answer specific questions. If the question you are asking is "Does A matter?" and the answer the model gives you is "No, it only looks like A matters because it is correlated with background characteristics" then that is what you should report.

Improve this answer
$\endgroup$
6
  • 5
    $\begingroup$ I wish researchers would print this answer and tape it to the wall. It would be also helpful to predict the main covariate from the adjustment covariates to expose the inter-relationships. And this is a good place to emphasize that translating research questions into statistical models is key, and the models should be pre-specified to the extent possible. “Significance” judgments should not play into model selection. $\endgroup$
    – Frank Harrell
    Commented Aug 6 at 12:21
  • $\begingroup$ First of all, thanks for your input. Here is some clarification on specifics: I have 5 independent variables which have been pervious shown to relate to customer satisfaction, 4/5 where significant. $\endgroup$
    – Carl Berglund
    Commented Aug 6 at 12:24
  • $\begingroup$ Now I have to check for control variables to pass, I put in age, gender and sex but they are all insignificant to customer satisfaction, does this not mean that they do not significantly contribute? So could I not then exclude those from the regression, what’s the point of having them if they are insignificant? Seems more robust to just have the primary variables and also more simplistic. P.S. also I have wrote the entire results, analysis and conclusion on certain results, I don't have to change much as long as the I can do it in this way and still pass. $\endgroup$
    – Carl Berglund
    Commented Aug 6 at 12:25
  • $\begingroup$ Otherwise one of the 4/5 variables that were previously significant come up as insignificant. $\endgroup$
    – Carl Berglund
    Commented Aug 6 at 12:25
  • 1
    $\begingroup$ The question of what variables to include in a regression model is super complex, with no simple answers or algorithmic solution. Everything depends on what you are trying to do and the specific problem you are studying. The best quick advice I can give is to use your substantive knowledge about the problem to decide on a set of control variables that might be important, run the model once with those variables, and report the results, whatever they are. Running the model again and again with different variables undermines the logic of significance tests (search for "multiple comparisons"). $\endgroup$
    – Graham Wright
    Commented Aug 7 at 0:20

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.