Comparison between Log-likelihood ratios and beta coefficients

Question

I was asked a question recently which I could not find an answer for and was hoping someone could enlighten me.

The question was regarding the significance of a single variable in a linear model.

What is the difference between a traditional p-value which you would obtain from the t-statistic formula

$(\beta_1 - \beta_0) / \sigma(\beta_1)$

and log-likelihood ratio between a model with the variable in question, and a model without.

In the log-likelihood case, would a significant LLR suggest that the variable had a significant impact on the model? Assuming the LL for the full model was better than the subset model.

Is this result comparable to a traditional t-distribution p-value one would normally get from a linear regression? If not, why not?

Thanks in advance, and please ask for any additional details if I was not clear.

Answer 1

What is the difference between a traditional p-value {(b1 - b0) / se(b1)} and log-likelihood ratio between a model with the variable in question, and a model without.

Depends on what you do with the likelihood ratio. If - as is sometimes done - you work out the LR test statistic and then manipulate it in order to get a statistic whose small sample distribution can be worked out, you're rejecting the same cases as the LR test should, without using asymptotic results. But usually when people apply an LR test they usually base their likelihood ratio test on applying Wilk's theorem, yielding an asymptotic chi-square approximation.

If you take the t-statistic you mentioned, it's asymptotically normal... if you square that, you have what is (that same) asymptotic chi-square test statistic.

As a result, in small samples from normal distributions, the two tests aren't identical, but as n increases they become more similar (reject exactly the same cases, as the significance level of the approximation becomes exact).

In the log-likelihood case, would a significant LLR suggest that the variable had a significant impact on the model?

Yes, up to the fact that your test is not exactly at the desired significance level.

Assuming the LL for the full model was better than the subset model

It's always at least as good.

Here's an example in R. I am going to use the glm function because I can get the t-ratio and the asymptotic chi-square test statistic from the output. I will use the cars data available in R (a few less interesting lines have been cut out).

summary(glm(dist~speed,cars,family=gaussian))

Coefficients:
            Estimate Std. Error t value Pr(>|t|)    
(Intercept) -17.5791     6.7584  -2.601   0.0123 *  
speed         3.9324     0.4155   9.464 1.49e-12 ***

(Dispersion parameter for gaussian family taken to be 236.5317)

    Null deviance: 32539  on 49  degrees of freedom
Residual deviance: 11354  on 48  degrees of freedom
AIC: 419.16

Number of Fisher Scoring iterations: 2

Note that the $236.5317$ in the output is simply $11354/48$ (up to rounding error, since the actual residual deviation is closer to 11353.52)

LR test statistic = $\frac{(32539-11353.54)}{236.5317} = 89.565$

t-test statistic = $9.464$

$9.464^2 = 89.567$ (difference in last figure is simply rounding error in both calculations)

So the difference is basically the same as the difference between looking up 9.464 in $t_{48}$ tables and in normal tables.