0
$\begingroup$

I have two models, one restricted, the other full. Is there a test-statistic decision criteria I could use to make a decision as to which model might be better?

$\endgroup$
3
  • 1
    $\begingroup$ Are the two models nested? $\endgroup$ Dec 6, 2014 at 17:45
  • $\begingroup$ One could also reject the notion that one model is "better" than another, and argue that both reflect insights into the modeled phenomena based on differing sets of assumptions. The degree to which results differ indicate the sensitivity of modeled behavior to those assumptions. $\endgroup$
    – Alexis
    Dec 8, 2014 at 2:19
  • $\begingroup$ Define "better". What are you trying to achieve with your model? Lower mean square prediction error? $\endgroup$
    – Glen_b
    Dec 8, 2014 at 6:07

2 Answers 2

1
$\begingroup$

The easiest way is likelihood ratio test if the models are nested, i.e. if 6 variables contain 4 variables. You get likelihood ratio for the 6-variable model as unrestricted. Then get likelihood ratio for the 4-variable model as restricted. Then plug into the test and get the statistic.

$\endgroup$
0
$\begingroup$

To expand a little on Aksakal's answer, you could use AIC (Akaike information criterion) or BIC (Bayesian Information Criterion) to make this distinction.

$$ AIC = 2k - 2ln(L) $$

Given $k$ parameters and $L$ maximum likelihood. Though depending on the sample size you are looking at, you may have to penalize more strongly for additional parameters such that

$$ AICc = AIC + \frac{2k(k+1)}{n-k-1} $$

These formulas directly from wikipedia. You will be looking to minimize the AIC.

This is how I would approach model selection, though it does not give a test statistic and a P value such that you have asked for. The methods for model selection are many and varied, and you can read about several here. To get a test statistic, you may want to look into $\chi^2$ model selection. Another possibility, if you have a binary response, would be to look into ROC curves. If you build two ROC curves for your two models and test for a difference using, perhaps, the Delong et al. approach for correlated (read: built from the same data set) detailed in R. If there is no significant difference, then the model with more parameters is not a significant improvement over the original model.

Hope that gives you something to think about. Good luck!

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.