4
$\begingroup$

I have 2 non-nested models which I would like to compare. Both models are based on the same dataset but use different predictors.

Model1 predictor A+B Model2 predictor B+C

I know there are multiple tests available to select the "best" method: 1) jtest (Davidson-MacKinnon J test) 2) coxtest (Cox test) 3) encomptest (Davidson & MacKinnon)

All of the test are described in r for the comparison of non-nested models. However, which test is prefered?

If I understand the test correctly, all test say that Model1 is the best.

> coxtest(Model1,Model2)
Cox test

Model 1: group ~ A + B
Model 2: group ~ C + B
                Estimate Std. Error  z value Pr(>|z|)    
fitted(M1) ~ M2  -3.0809     3.1646  -0.9735   0.3303    
fitted(M2) ~ M1 -31.1339     2.0889 -14.9043   <2e-16 ***
---
Signif. codes:  0 ‘***’ 0.001 ‘**’ 0.01 ‘*’ 0.05 ‘.’ 0.1 ‘ ’ 1


> jtest(Model1,Model2)
J test

Model 1: group ~ A + B
Model 2: group ~ C + B
                Estimate Std. Error t value  Pr(>|t|)    
M1 + fitted(M2)  0.18681    0.21166  0.8826    0.3786    
M2 + fitted(M1)  0.93740    0.13155  7.1257 2.149e-11 ***
---
Signif. codes:  0 ‘***’ 0.001 ‘**’ 0.01 ‘*’ 0.05 ‘.’ 0.1 ‘ ’ 1


> encomptest(Model1,Model2, data=data)
Encompassing test
Model 1: group ~ A + B
Model 2: group ~ C + B
Model E: group ~ A + B + C
          Res.Df Df       F   Pr(>F)    
M1 vs. ME    188 -1  1.2402   0.2669    
M2 vs. ME    188 -1 24.3536 1.76e-06 ***
---
Signif. codes:  0 ‘***’ 0.001 ‘**’ 0.01 ‘*’ 0.05 ‘.’ 0.1 ‘ ’ 1
$\endgroup$
4
  • 1
    $\begingroup$ What is the objective? Interpretation or prediction? $\endgroup$ Commented Jan 23, 2019 at 14:33
  • $\begingroup$ Also, what is your definition of "best" for this decision? $\endgroup$
    – Roland
    Commented Jan 23, 2019 at 14:39
  • $\begingroup$ The aim of the model comparison is to choose the method that predicts best in which group the data fits (group 0, or 1). $\endgroup$ Commented Jan 23, 2019 at 14:52
  • $\begingroup$ You can consider using r2redux cran.r-project.org/web/packages/r2redux/index.html or github.com/mommy003/r2redux $\endgroup$
    – Biyong
    Commented May 22, 2023 at 4:25

2 Answers 2

9
$\begingroup$

For your particular problem, the answer would be: "None of these tests is appropriate."

The help pages for these 3 functions indicate that they all are expecting standard linear regression models (from lm() in R), implicitly with a continuous outcome variable.

In your case, however, your outcome variable is a choice between 2 groups, not a continuous outcome variable. Thus you should be using logistic regression instead of linear regression. Finding the "best" linear regression, as you seem to be attempting, could be far from finding the best model for predicting group membership.

There is considerable discussion on this site about the best ways to compare non-nested generalized linear models (like logistic regression). This page provides arguments for using the Akaike Information Criterion (AIC) to choose among non-nested models, while further providing a link to an informed difference of opinion.

The standard stats package in R has 2 functions to determine the AIC from models (AIC() and extractAIC()); this page shows a way to use them for comparing 2 models. Just be careful to stick with one or the other as the outputs from the 2 functions differ by additive constants.

$\endgroup$
3
  • $\begingroup$ I do perform logistic regression, however I was struggling a little to find the correct statistical methods to verify the models. $\endgroup$ Commented Jan 24, 2019 at 8:06
  • $\begingroup$ @EdM isn't the AUC (plus confidence interval) on the same test set held out from the data set the best method of comparison? $\endgroup$
    – Omry Atia
    Commented May 8, 2022 at 6:30
  • $\begingroup$ @OmryAtia only if you have tens of thousands of cases. See Frank Harrell's post on the perils of split-sample model validation. $\endgroup$
    – EdM
    Commented May 8, 2022 at 11:48
0
$\begingroup$

If your only goal is simply to choose a model with the best fit, then either one of these models is valid since they all work on the same principle. If you are asking this question in order to know which test is preferred for your data type, then it is best to refer to literature in the field.

Also, based on the p-values alone, the fit of Model 2 is statistically significant in this case.

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.