3
$\begingroup$

I set up a (multinominal) logit model (model 1) and then, holding all else constant, set up a second model (model 2) in which I add a control variable ($X_2$) that did not feature in model 1. The rationale is to test whether the strength of an independent variable of interest, $\hat\beta_1$, changes significantly between model 1 and model 2. I find that adding ($X_2$) in model 2 does decrease the strength of $\hat\beta_1$ compared with model 1, but I am unsure of how to test whether this difference is significant. Until now I am using a crude method: examining whether the confidence intervals for $\hat\beta_1$ overlap between model 1 and model 2. If not, I am interpreting the difference as significant. Yet this test seems harsh: if the confidence intervals marginally overlap, we can surely be more than 95% confident that the true value of $\hat\beta_1$ does not lie at the extremes of both estimates. My question: is there a better way of testing for a significant between $\hat\beta_1$ in model 1 an model 2?

(PS. I estimated the models using the mlogit function in R.)

$\endgroup$
2
  • $\begingroup$ I think, given the setup, you can do no better. However, even if the result of this "test" comes out significant, you should always argue that it has a real world implication. For instance, we might estimate the real wage of women at \$149 and at $150 for men. With enough observations this difference will eventually become significant - but does it actually matter? $\endgroup$
    – Repmat
    Commented Dec 26, 2015 at 15:11
  • $\begingroup$ This question sounds strange to me; if you think that the variable $X_2$ is relevant for your problem, then you should include it. If you don't then - see 'omitted variable bias' - your $\hat{\beta}_1$ estimate might be biased. So the 'increase' or 'decrease' might be due to 'bias' ? $\endgroup$
    – user83346
    Commented Dec 26, 2015 at 15:29

1 Answer 1

2
$\begingroup$

So there is a way (a couple ways actually) to do this, but it is kind of a pain, so ask yourself how important this really is and how much sense it makes in the context of your research before proceeding.

Let $\hat \beta^{(1)}_1$ and $\hat \beta^{(2)}_1$ be the coefficients from the the model without and with $X_2$ respectively. Your hypothesis test is the following: $$ H_0:\beta^{(1)}_1 - \beta^{(2)}_1 = 0 $$ $$H_1:\beta^{(1)}_1 - \beta^{(2)}_1 \neq 0 $$ Less degrees of freedom adjustments, the z-score test statistic would be of the form $$ z = \frac{\hat \beta^{(1)}_1 - \hat \beta^{(2)}_1}{\sqrt{\mathrm{var}(\hat \beta^{(1)}_1 - \hat \beta^{(2)}_1)/n}} $$ To run the test, it is necessary to estimate the variance of $(\hat \beta^{(1)}_1 - \hat \beta^{(2)}_1)$, which can be broken down below as $$ \mathrm{var}(\hat \beta^{(1)}_1 - \hat \beta^{(2)}_1) = \mathrm{var}(\hat \beta^{(1)}_1) + \mathrm{var}(\hat \beta^{(2)}_1) -2\mathrm{cov}(\hat \beta^{(1)}_1,\hat \beta^{(2)}_1) $$

$\mathrm{var}(\hat \beta^{(1)}_1)$ and $\mathrm{var}(\hat \beta^{(2)}_1)$ are easily estimated using the standard error's of the coefficients, the hard part is estimating the covariance term, $\mathrm{cov}(\hat \beta^{(1)}_1,\hat \beta^{(2)}_1)$ . There are two ways I know of:

  1. Estimate both mlogit models simultaneously and use the resulting hessian matrix to estimate covariance: This would most likely require you to program your own bivariate multinominal likelihood function and optimize over it (not as hard as it sounds). I do not have time to write a likelihood for you, explain how to use the optim function in r, get standard errors from the hessian matrix, and so forth, but this is what you will have to do in this approach. Perhaps look here for the multinominal log likelihood and here for an example of estimating a logit model using a negative log-likelihood function and optim in R.
  2. Use a bootstrap to estimate $\mathrm{var}(\hat \beta^{(1)}_1 - \hat \beta^{(2)}_1)$ directly: This will allow you to use the same mlogit you were using before. However, you will need to employ a bootstrap in which you estimate both models and collect $\hat \beta^{(1)}_{1,g} - \hat \beta^{(2)}_{1,g}$ for $g=1,2,...,G$ iterations . This will result in a vector of bootstrap estimates from which you can estimate $\mathrm{var}(\hat \beta^{(1)}_1 - \hat \beta^{(2)}_1)$.

There are caveats to both techniques which I refrain from diving into depth here. The bootstrap is generally considered more robust since it does not rely on asymptotic normally quiet so much, but you need to make sure you are employing the correct bootstrap (i.e. is your data cross-sectional?, auto-correlated?, etc.... these will imply different bootstrap techniques), it is also computationally expensive since it requires you to re-estimate the model thousands of times. In the end, both techniques are asymptotically equivalent, so given enough data the difference would be inconsequential.

$\endgroup$
3
  • $\begingroup$ Some fundamental inconsistency may be underlying this answer. Consider the analogous problem in a multiple regression setting. The two fits concern two different, incompatible models. One asserts $y =\beta_1X_1+\varepsilon$ while the other asserts $y=\beta_1X_1+\beta_2X_2+\delta$ where in each case the random variables $\varepsilon$ and $\delta$ have zero expectations. The $\epsilon$ and $\delta$ are not the same, nor can both sets of assumptions simultaneously hold when $\beta_2X_2\ne 0$. The same inconsistency holds in logistic regression. This seems to invalidate your tests. $\endgroup$
    – whuber
    Commented Dec 26, 2015 at 16:29
  • $\begingroup$ @whuber I didn't really like the idea of doing this test in the first place, and probably should not have answered. I think you are right in the context that we usually think about regression, However, if you are just interested in prediction and think about regression as simply a conditional expectation function, then in the example above you would willing to accept the possibility that $\beta^{(1)}_1 \neq \beta^{(2)}_1$. So there is really no bias per say, just two different conditional expectation functions, with two different partial correlation coefficients...correct me if I am wrong $\endgroup$ Commented Dec 26, 2015 at 16:59
  • $\begingroup$ Of course if you where just interested in prediction, you wouldn't really care about tests for significance so much as AIC, BIC, and so forth...which makes this whole hypothesis test sort of useless...point taken. $\endgroup$ Commented Dec 26, 2015 at 16:59

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.