Can I ignore coefficients for non-significant levels of factors in a linear model?

Question

After seeking clarification about linear model coefficients over here I have a follow up question concerning non-signficant (high p value) for coefficients of factor levels.

Example: If my linear model includes a factor with 10 levels, and only 3 of those levels have significant p values associated with them, when using the model to predict Y can I choose to not include the coefficient term if the subject falls in one of the non-signficant level?

More drastically, would it be wrong to lump the 7 non-significant levels into one level and re-analyze?

Answer 1

If you are putting in a predictor variable with multiple levels, you either put in the variable or you don't, you can't pick and choose levels. You might want to restructure the levels of your predictor variable to decrease the number of levels (if that makes sense in the context of your analysis.) However, I'm not sure if this would cause some type of statistical invalidation if you're collapsing levels because you see they are not significant.

Also, just a note, you say small $p$-values are insignificant. I assume that you meant small $p$-value are significant, ie: a $p$-value of .0001 is significant and therefore you reject the null (assuming an $\alpha$ level of $> .0001$?).

Answer 2

@Ellie's response is a good one.

If you are putting in a variable with a number of levels, you need to retain all those levels in your analysis. Picking and choosing based on significance level will both bias your results and do very weird things to your inference, even if by some miracle your estimates manage to stay the same, as you'll have gaping holes in your estimated effects over different levels of the variable.

I would consider looking at your estimates for each level of the predictor graphically. Are you seeing a trend as you go up levels, or is it erratic?

Generally speaking, I'm also opposed to recoding variables based on statistical tests - or based purely on statistical moments. The divisions in your variable should be based on something more firm - logically meaningful cut-points, field interest in a particular transition point, etc.

Answer 3

Expanding on the two good answers you've already gotten, let's look at this substantively. Suppose your dependent variable is (say) income and your independent variable is (say) ethnicity, with levels, per census definitions (White, Black/Afr.Am., Am. Indian/Alaska Native, Asian, Native Hawaii/Pac Islander, other and multiracial). Let's say you dummy code it with White being the reference category and you get

$ Income = b_0 + b_1BAA + b_2AIAN + b_3AS + b_4NHPI + b_5O + b_6MR $

If you are doing this study in New York City, you will probably get very few Native Hawaiians/Pacific Islanders. You might decide to include them (if there are any) with the others. However, you can't use the full equation and just not include that coefficient. Then the intercept will be wrong, and so will any predicted values for income.

But how should you combine categories?

As the others said, it has to make sense.

Answer 4

To give a different opinion: why not include it as a random effect? That should penalize those levels with weak support and make sure their effect size is minimal. That way you can keep them all in without worrying about getting silly predictions.

And yes, this is more motivated from a Bayesian view of random effects than the whole "sample of all possible levels" view of random effects.

Answer 5

I was also wondering whether I could combine non-significant categories with the reference category. The following statements in the book "Data Mining for Business Intelligence: Concepts, Techniques, and Applications in Microsoft Office Excel® with XLMiner®, 2nd Edition by Galit Shmueli, Nitin R. Patel, Peter C. Bruce", p87-89 (Dimension Reduction section) (Google Search Result) seem to support the second sentence of @Ellie's response:

• "Fitted regression models can also be used to further combine similar categories: categories that have coefficients that are not statistically significant (i.e. have a high p-value) can be combined with the reference category because their distinction from the reference category appears to have no significant effect on the output variable"
• "Categories that have similar coefficient values (and the same sign) can often be combined because their effect on the output variable is similar"

However, I plan to check with subject matter experts whether combining the categories makes logical sense (as implied in previous answers / comments, e.g. @Fomite, @gung).