How to test the statistical significance for categorical variable in linear regression?

Question

If in a linear regression I have categorical variable ... how do I know the statistical significance of the categorical variable?

Let's say the factor $X_1$ has 10 levels ... there will be 10 different resultant t-values, under the umbrella of one factor variable $X_1$ ...

It seems to me that the statistical significance is tested for each level of the factor variable? No?

@Macro: Following your suggestion, I've built the following example:

It seems that x3 is useful and must be included in the model, from the below model comparison.

But actually that's wrong ...

    n=100    
    x1=1:n
    x2=(1:n)^2 
    x3=rnorm(n)
    ee=rnorm(n)
    y=3*x1-2*x2+x3+3+ee
    lm1=lm(y~x1+x2+x3)
    summary(lm1)
   
    lm2=lm(y~x1+x2) 
    summary(lm2)
   
    anova(lm1, lm2)

    > anova(lm1, lm2)
    Analysis of Variance Table
    
    Model 1: y ~ x1 + x2 + x3
    Model 2: y ~ x1 + x2
      Res.Df     RSS Df Sum of Sq      F    Pr(>F)    
    1     96  82.782                                  
    2     97 146.773 -1    -63.99 74.207 1.401e-13 ***
    ---
    Signif. codes:  0 '***' 0.001 '**' 0.01 '*' 0.05 '.' 0.1 ' ' 1 

Answer 1

You are correct that those $p$-values only tell you whether each level's mean is significantly different from the reference level's mean. Therefore, they only tell you about the pairwise differences between the levels. To test whether the categorical predictor, as a whole, is significant is equivalent to testing whether there is any heterogeneity in the means of the levels of the predictor. When there are no other predictors in the model, this is a classical ANOVA problem.

When there are other predictors in the model. you have two options to test for the significance of a categorical predictor:

(1) The likelihood ratio test: Suppose you have an outcome $Y_i$, quantitative predictors $X_{i1}, ..., X_{ip}$ and the categorical predictor $C_i$ with $k$ levels. The model without the categorical predictor is

$$ Y_i = \beta_0 + \beta_1 X_{i1} + ... + \beta_p X_{ip} + \varepsilon_i $$

In R you can fit this model with the lm() command and extract the log likelihood with the logLik command. Call this log-likelihood $L_0$. Next, you can fit the model with the categorical predictor:

$$ Y_i = \beta_0 + \beta_1 X_{i1} + ... + \beta_p X_{ip} + \sum_{j=1}^{k-1} \alpha_j B_j + \varepsilon_i $$

where $B_j$ is a dummy variable which is $1$ if $D_i = j$ and $0$ otherwise. The $k$'th level is the reference level, which is why there are only $k-1$ terms in the sum. R will automatically do this dummy coding for you if you pass the categorical variable to lm(). You can fit this model similarly and extract the log likelihood as above. Call this log-likelihood $L_1$. Then, under the null hypothesis that $D_i$ has no effect,

$$ \lambda = 2 \left( L_1 - L_0 \right ) $$

has a $\chi^2$ distribution with $k-1$ degrees of freedom. So, you can calculate the $p$-value using 1-pchisq(2*(L1-L0),df=k-1) in R to test for significance.

(2) $F$-test: Without going into the details (which are similar to the LRT except sums of squares are used rather than log-likelihoods), I'll explain how to do this in R. If you fit the "full" model (i.e. the model with all of the predictors, including the categorical predictor) in R using the lm() command (call this g1) and the model without the categorical predictor (call this g0), then the anova(g1,g0) will test this hypothesis for you as well.

Note: both of the approaches I've mentioned here require normality of the errors. Also, the likelihood ratio test is a very general tool used for nested comparisons, which is why I mention it here (and why it occurs to me first), although the $F$-test is more familiar in comparing linear regression models.