If in a linear regression I have categorical variable... how do I know the stastical signifance of the categorical variable?

Let's say the factor $X_1$ has 10 levels... there will be 10 different resultant t-values, under the umbrella of one factor variable $X_1$...

It seems to me that the statistical signifiance is tested for each level of the factor variable? No?

@Macro: Following your suggestion, I've built the following example:

It seems that x3 is useful and must be included in the model, from the below model comparison.

But actually that's wrong...



anova(lm1, lm2)

> anova(lm1, lm2)
Analysis of Variance Table

Model 1: y ~ x1 + x2 + x3
Model 2: y ~ x1 + x2
  Res.Df     RSS Df Sum of Sq      F    Pr(>F)    
1     96  82.782                                  
2     97 146.773 -1    -63.99 74.207 1.401e-13 ***
Signif. codes:  0 '***' 0.001 '**' 0.01 '*' 0.05 '.' 0.1 ' ' 1 
  • 1
    $\begingroup$ @Luna, why is that wrong? It appears you used x3 to generate the ys, so it should be included in the model and the $p$-value agrees with that conclusion. $\endgroup$ – Macro Jul 5 '12 at 18:46
  • $\begingroup$ @Seth - you are right. I was just giving a toy example of using anova generally in model comparison. So it's not linked to my original question. $\endgroup$ – Luna Jul 5 '12 at 19:01
  • $\begingroup$ @Macro - you are right. Now I see the point. Thank you! $\endgroup$ – Luna Jul 5 '12 at 19:02
  • $\begingroup$ The 'Anova' function from the R 'car' package (pdf) lets you test the overall significance of a categorical variable. It works with a lot of different packages and types of regression. $\endgroup$ – SK4ndal May 31 '18 at 13:21

You are correct that those $p$-values only tell you whether each level's mean is significantly different from the reference level's mean. Therefore, they only tell you about the pairwise differences between the levels. To test whether the categorical predictor, as a whole, is significant is equivalent to testing whether there is any heterogeneity in the means of the levels of the predictor. When there are no other predictors in the model, this is a classical ANOVA problem.

When there are other predictors in the model. you have two options to test for the significance of a categorical predictor:

(1) The likelihood ratio test: Suppose you have an outcome $Y_i$, quantitative predictors $X_{i1}, ..., X_{ip}$ and the categorical predictor $C_i$ with $k$ levels. The model without the categorical predictor is

$$ Y_i = \beta_0 + \beta_1 X_{i1} + ... + \beta_p X_{ip} + \varepsilon_i $$

In R you can fit this model with the lm() command and extract the log likelihood with the logLik command. Call this log-likelihood $L_0$. Next, you can fit the model with the categorical predictor:

$$ Y_i = \beta_0 + \beta_1 X_{i1} + ... + \beta_p X_{ip} + \sum_{j=1}^{k-1} \alpha_j B_j + \varepsilon_i $$

where $B_j$ is a dummy variable which is $1$ if $D_i = j$ and $0$ otherwise. The $k$'th level is the reference level, which is why there are only $k-1$ terms in the sum. R will automatically do this dummy coding for you if you pass the categorical variable to lm(). You can fit this model similarly and extract the log likelihood as above. Call this log-likelihood $L_1$. Then, under the null hypothesis that $D_i$ has no effect,

$$ \lambda = 2 \left( L_1 - L_0 \right ) $$

has a $\chi^2$ distribution with $k-1$ degrees of freedom. So, you can calculate the $p$-value using 1-pchisq(2*(L1-L0),df=k-1) in R to test for significance.

(2) $F$-test: Without going into the details (which are similar to the LRT except sums of squares are used rather than log-likelihoods), I'll explain how to do this in R. If you fit the "full" model (i.e. the model with all of the predictors, including the categorical predictor) in R using the lm() command (call this g1) and the model without the categorical predictor (call this g0), then the anova(g1,g0) will test this hypothesis for you as well.

Note: both of the approaches I've mentioned here require normality of the errors. Also, the likelihood ratio test is a very general tool used for nested comparisons, which is why I mention it here (and why it occurs to me first), although the $F$-test is more familiar in comparing linear regression models.

  • $\begingroup$ Thanks a lot Macro. I found that my data is highly non-normal. The QQ plot is as follows: the curve is all below the straight 45 degree line. The curve is tangent to that straight line. And the curve looks like the curve of f(x)=-x^2 ( shape-wise). What kind of problem am I facing? And how shall I fix this? Thank you! $\endgroup$ – Luna Jul 5 '12 at 18:21
  • 1
    $\begingroup$ @Luna, Your data is highly non-normal or the residuals are highly non-normal? Also, I don't think it's possible for the entire set of points to lie under the 45 degree line. $\endgroup$ – Macro Jul 5 '12 at 18:23
  • $\begingroup$ oh actually you are right... I just took one more look at the QQ plot. It's not the entire set of points that's under the 45 degree line. It's the curve with the shape of f(x)=-x^2 is "tangent" to the 45 degree line. By "tangent" I should have meant that those points around the "tangent" point are actually above the 45 degree line, very slightly though. Therefore, visually speaking, most of the data (~98%) are below the 45 degree line... what shall I do first to fix this problem before doing model comparison? Thank you! $\endgroup$ – Luna Jul 5 '12 at 18:32
  • 2
    $\begingroup$ If your sample size is pretty large, the $p$-values should still be reasonable (by the central limit theorem) as long as your error distribution isn't long-tailed. If you just want to test the categorical variable in isolation, you can use a non-parametric ANOVA: en.wikipedia.org/wiki/… but, as I said, this is really becoming an entirely different question from the one posed and may be more appropriately posed as a new question or answered by searching the site for a related question. $\endgroup$ – Macro Jul 5 '12 at 19:11
  • 1
    $\begingroup$ @Druss2k, yes that's correct. $\endgroup$ – Macro Dec 19 '12 at 21:09

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.