5
$\begingroup$

I recently read a paper in which an academic compared two competing models of voting theory. One of the advanced diagnostic tests he did was to try and see whether either of the models could "formally encompass one another". The way he was able to interpret the results was by comparing the reduction in residual deviance or AIC when M1 was added to M2, as opposed to M2 being added to M1.

I have joylessly been trying to do this in R. Does anybody have any experience with encompassing tests? Effectively all I am attempting to do is compare two models which are non-nested. The models are completely different apart from the control variables, therefore anova(m1, M2) does not work.

Can anyone advise or suggest ways to proceed?

$\endgroup$
6
$\begingroup$

If you have two competing linear models with different regressors, say:

m1 <- lm(y ~ x1 + x2, data = ...)
m2 <- lm(y ~ z1 + z2, data = ...)

then the encompassing model is

m12 <- lm(y ~ x1 + x2 + z1 + z2, data = ...)

and the encompassing test compares

anova(m1, m12)
anova(m2, m12)

The "hope" would be that one of the models (m1, m2) is significantly worse than the encompassing model m12 while the other is not. The one that is not significantly worse would be preferable then. However, it may happen that both models are significantly worse than m12 in which case neither model alone is entirely convincing.

If you want a convenience function to carry out this test in R, you can use the encomptest() function from the lmtest package:

encomptest(m1, m2)

The package also provides other tests for non-nested model comparisons, e.g., jtest() and coxtest().

$\endgroup$
  • $\begingroup$ That's very very helpful- thank you Achim. The only issue is I get the error that the two datasets are not the same size, even though I have used the same dataset? Thanks $\endgroup$ – HenryBukowski Mar 3 '15 at 19:12
  • $\begingroup$ This can be caused by different patterns of missingness in the sets of regressors x1/x2 vs. z1/z2. If you want to use the anova() route by hand, you have to make sure that all models are fitted to the same subset of data. If you use encomptest() you can also use the formula interface: encomptest(y ~ x1 + x2, y ~ z1 + z2, data = ...) which selects the right subset for you. $\endgroup$ – Achim Zeileis Mar 3 '15 at 19:18
  • $\begingroup$ Am not out of the woods yet it seems :D. I've tried that argumentation and I get 'subscript out of bounds'. This is certainly something I've done wrong.. Wait, no it worked: My results: <code> Res.Df Df F Pr(>F) M1 vs. ME 283 -11 5.1302 2.446e-07 *** M2 vs. ME 283 -9 4.3358 2.630e-05 *** Now to work out what it means!! </code> $\endgroup$ – HenryBukowski Mar 3 '15 at 20:26
  • $\begingroup$ @AchimZeileis Very good. For reference purposes, is there a particular name associated with this test in the (econometric) literature? $\endgroup$ – Graeme Walsh Mar 4 '15 at 1:16
  • 1
    $\begingroup$ In Greene's book (in Chapter 8.3.1 in the 5th edition from 2003) he just calls it a test based on the "encompassing principle". It is often called "the" encompassing test although - as correctly pointed out in the other reply - other tests can be constructed based on the same principle. $\endgroup$ – Achim Zeileis Mar 4 '15 at 9:38
3
$\begingroup$

There are different encompassing tests. I'll give you an example of forecast encompassing test from "A Companion to Economic Forecasting" edited by Michael P. Clements, David F. Hendry, see Eq (14.2) on p.302.

Let's say you have competing forecast $\hat y_{1,t+1}$ and $\hat y_{2,t+1}$, and the actual observations $y_{t+1}$. Run a regression: $$y_{t+1}=\alpha_0+\alpha_1 \hat y_{1,t+1}+\alpha_2 \hat y_{2,t+1}+u_{t+1}$$

Under $H_0$: model 1 encompasses model 2, you have $\alpha_1=1$ and $\alpha_2=0$. This is a simple linear restriction test, that can easily be done in R or any other stat package. Note, that you don't need to know anything about the models, their independent variables etc. All you need is their forecasts and actual observations.

$\endgroup$
  • $\begingroup$ Sorry to open this up again @AchimZeileis ...would you know how to computer AIC/BIC resulting from the encompassing tests? $\endgroup$ – HenryBukowski Mar 19 '15 at 14:14

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.