# Can I ignore coefficients for non-significant levels of factors in a linear model?

After seeking clarification about linear model coefficients over here I have a follow up question concerning non-signficant (high p value) for coefficients of factor levels.

Example: If my linear model includes a factor with 10 levels, and only 3 of those levels have significant p values associated with them, when using the model to predict Y can I choose to not include the coefficient term if the subject falls in one of the non-signficant level?

More drastically, would it be wrong to lump the 7 non-significant levels into one level and re-analyze?

-
Well, you could get biased inference by doing that - for example, if you're forming prediction intervals, the coverage probabilities would probably be wrong for individuals in any of the 7 insignificant levels. – Macro Mar 8 '12 at 3:43
You've gotten some good answers here, but you might also be interested in why it is inappropriate to drop factors with high p-values. It's worth pointing out that this is logically equivalent to an automatic model selection procedure, even though you're doing it yourself, instead of the computer doing it for you. Reading through this question & the answers offered can help w/ understanding why these things are true. – gung Mar 8 '12 at 16:37
This Q has an exact duplicate from November 2012: stats.stackexchange.com/questions/18745/… . There's a bit of thought-provoking info there too. – rolando2 Mar 8 '12 at 20:15

If you are putting in a predictor variable with multiple levels, you either put in the variable or you don't, you can't pick and choose levels. You might want to restructure the levels of your predictor variable to decrease the number of levels (if that makes sense in the context of your analysis.) However, I'm not sure if this would cause some type of statistical invalidation if you're collapsing levels because you see they are not significant.

Also, just a note, you say small $p$-values are insignificant. I assume that you meant small $p$-value are significant, ie: a $p$-value of .0001 is significant and therefore you reject the null (assuming an $\alpha$ level of $> .0001$?).

-
 (Corrected my p-value typo.) Good points here. So collapsing levels, provided it is based on some real-world and logical reason justifiable in the context of the study (that might also happen to parse them out along the significance break) is reasonable, but not just lumping them arbitrarily based on their significance. Got it. – Trees4theForest Mar 8 '12 at 12:05

@Ellie's response is a good one.

If you are putting in a variable with a number of levels, you need to retain all those levels in your analysis. Picking and choosing based on significance level will both bias your results and do very weird things to your inference, even if by some miracle your estimates manage to stay the same, as you'll have gaping holes in your estimated effects over different levels of the variable.

I would consider looking at your estimates for each level of the predictor graphically. Are you seeing a trend as you go up levels, or is it erratic?

Generally speaking, I'm also opposed to recoding variables based on statistical tests - or based purely on statistical moments. The divisions in your variable should be based on something more firm - logically meaningful cut-points, field interest in a particular transition point, etc.

-

Expanding on the two good answers you've already gotten, let's look at this substantively. Suppose your dependent variable is (say) income and your independent variable is (say) ethnicity, with levels, per census definitions (White, Black/Afr.Am., Am. Indian/Alaska Native, Asian, Native Hawaii/Pac Islander, other and multiracial). Let's say you dummy code it with White being the reference category and you get

$Income = b_0 + b_1BAA + b_2AIAN + b_3AS + b_4NHPI + b_5O + b_6MR$

If you are doing this study in New York City, you will probably get very few Native Hawaiians/Pacific Islanders. You might decide to include them (if there are any) with the others. However, you can't use the full equation and just not include that coefficient. Then the intercept will be wrong, and so will any predicted values for income.

But how should you combine categories?

As the others said, it has to make sense.

-

To give a different opinion: why not include it as a random effect? That should penalize those levels with weak support and make sure their effect size is minimal. That way you can keep them all in without worrying about getting silly predictions.

And yes, this is more motivated from a Bayesian view of random effects than the whole "sample of all possible levels" view of random effects.

-