5
$\begingroup$

I was asked a question recently which I could not find an answer for and was hoping someone could enlighten me.

The question was regarding the significance of a single variable in a linear model.

What is the difference between a traditional p-value which you would obtain from the t-statistic formula

$(\beta_1 - \beta_0) / \sigma(\beta_1)$

and log-likelihood ratio between a model with the variable in question, and a model without.

In the log-likelihood case, would a significant LLR suggest that the variable had a significant impact on the model? Assuming the LL for the full model was better than the subset model.

Is this result comparable to a traditional t-distribution p-value one would normally get from a linear regression? If not, why not?

Thanks in advance, and please ask for any additional details if I was not clear.

Improve this question
$\endgroup$
0

1 Answer 1

Reset to default
4
$\begingroup$

What is the difference between a traditional p-value {(b1 - b0) / se(b1)} and log-likelihood ratio between a model with the variable in question, and a model without.

Depends on what you do with the likelihood ratio. If - as is sometimes done - you work out the LR test statistic and then manipulate it in order to get a statistic whose small sample distribution can be worked out, you're rejecting the same cases as the LR test should, without using asymptotic results. But usually when people apply an LR test they usually base their likelihood ratio test on applying Wilk's theorem, yielding an asymptotic chi-square approximation.

If you take the t-statistic you mentioned, it's asymptotically normal... if you square that, you have what is (that same) asymptotic chi-square test statistic.

As a result, in small samples from normal distributions, the two tests aren't identical, but as n increases they become more similar (reject exactly the same cases, as the significance level of the approximation becomes exact).

In the log-likelihood case, would a significant LLR suggest that the variable had a significant impact on the model?

Yes, up to the fact that your test is not exactly at the desired significance level.

Assuming the LL for the full model was better than the subset model

It's always at least as good.

Here's an example in R. I am going to use the glm function because I can get the t-ratio and the asymptotic chi-square test statistic from the output. I will use the cars data available in R (a few less interesting lines have been cut out).

summary(glm(dist~speed,cars,family=gaussian))

Coefficients:
            Estimate Std. Error t value Pr(>|t|)    
(Intercept) -17.5791     6.7584  -2.601   0.0123 *  
speed         3.9324     0.4155   9.464 1.49e-12 ***

(Dispersion parameter for gaussian family taken to be 236.5317)

    Null deviance: 32539  on 49  degrees of freedom
Residual deviance: 11354  on 48  degrees of freedom
AIC: 419.16

Number of Fisher Scoring iterations: 2

Note that the $236.5317$ in the output is simply $11354/48$ (up to rounding error, since the actual residual deviation is closer to 11353.52)

LR test statistic = $\frac{(32539-11353.54)}{236.5317} = 89.565$

t-test statistic = $9.464$

$9.464^2 = 89.567$ (difference in last figure is simply rounding error in both calculations)

So the difference is basically the same as the difference between looking up 9.464 in $t_{48}$ tables and in normal tables.

Improve this answer
$\endgroup$
2
  • $\begingroup$ Thanks Glen. An very thorough answer. +1. I have one follow up question. I am actually using this in regard to spatial auto regressive parameters, but feared no one would respond if i was more specific. Do you think that we could make the same conclusions? That the test statistic for an auto correlation parameter will be approximately the same as a LLR test statistic between a model with and without the parameter? Thanks again $\endgroup$
    – SamPassmore
    Commented Jan 30, 2014 at 9:37
  • 1
    $\begingroup$ I guess it depends on which test statistic you mean. For example, the z-test for an autocorrelation will likely be almost identical to the LR test (though I haven't worked out the LR test, I think from the form of the likelihood for an AR(p) (which apart from a term involving the first p observations, looks the same as a regression) I think it should work out. $\endgroup$
    – Glen_b
    Commented Jan 30, 2014 at 9:46

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.