20
$\begingroup$

After seeking clarification about linear model coefficients over here I have a follow up question concerning non-signficant (high p value) for coefficients of factor levels.

Example: If my linear model includes a factor with 10 levels, and only 3 of those levels have significant p values associated with them, when using the model to predict Y can I choose to not include the coefficient term if the subject falls in one of the non-signficant level?

More drastically, would it be wrong to lump the 7 non-significant levels into one level and re-analyze?

Improve this question
$\endgroup$
4
  • 2
    $\begingroup$ Well, you could get biased inference by doing that - for example, if you're forming prediction intervals, the coverage probabilities would probably be wrong for individuals in any of the 7 insignificant levels. $\endgroup$
    – Macro
    Mar 8, 2012 at 3:43
  • 2
    $\begingroup$ You've gotten some good answers here, but you might also be interested in why it is inappropriate to drop factors with high p-values. It's worth pointing out that this is logically equivalent to an automatic model selection procedure, even though you're doing it yourself, instead of the computer doing it for you. Reading through this question & the answers offered can help w/ understanding why these things are true. $\endgroup$
    – gung - Reinstate Monica
    Mar 8, 2012 at 16:37
  • 1
    $\begingroup$ This Q has an exact duplicate from November 2012: stats.stackexchange.com/questions/18745/… . There's a bit of thought-provoking info there too. $\endgroup$
    – rolando2
    Mar 8, 2012 at 20:15
  • 3
    $\begingroup$ This is such an important question, and yet there is no answer backing the argument with theory. As it stands, they are just opinions. Not even the book linked in one of the answers (which conclusion differs from the other answers) provide references. As this stands, I do not trust any of them, and thus would rather do nothing (i.e. keep all categories/factors in). $\endgroup$
    – luchonacho
    Sep 22, 2018 at 9:43

5 Answers 5

Reset to default
15
$\begingroup$

If you are putting in a predictor variable with multiple levels, you either put in the variable or you don't, you can't pick and choose levels. You might want to restructure the levels of your predictor variable to decrease the number of levels (if that makes sense in the context of your analysis.) However, I'm not sure if this would cause some type of statistical invalidation if you're collapsing levels because you see they are not significant.

Also, just a note, you say small $p$-values are insignificant. I assume that you meant small $p$-value are significant, ie: a $p$-value of .0001 is significant and therefore you reject the null (assuming an $\alpha$ level of $> .0001$?).

Improve this answer
$\endgroup$
2
  • $\begingroup$ (Corrected my p-value typo.) Good points here. So collapsing levels, provided it is based on some real-world and logical reason justifiable in the context of the study (that might also happen to parse them out along the significance break) is reasonable, but not just lumping them arbitrarily based on their significance. Got it. $\endgroup$
    – Trees4theForest
    Mar 8, 2012 at 12:05
  • $\begingroup$ "you can't pick and choose levels", why? $\endgroup$
    – PeJota
    Nov 18, 2022 at 14:24
16
$\begingroup$

@Ellie's response is a good one.

If you are putting in a variable with a number of levels, you need to retain all those levels in your analysis. Picking and choosing based on significance level will both bias your results and do very weird things to your inference, even if by some miracle your estimates manage to stay the same, as you'll have gaping holes in your estimated effects over different levels of the variable.

I would consider looking at your estimates for each level of the predictor graphically. Are you seeing a trend as you go up levels, or is it erratic?

Generally speaking, I'm also opposed to recoding variables based on statistical tests - or based purely on statistical moments. The divisions in your variable should be based on something more firm - logically meaningful cut-points, field interest in a particular transition point, etc.

Improve this answer
$\endgroup$
9
$\begingroup$

Expanding on the two good answers you've already gotten, let's look at this substantively. Suppose your dependent variable is (say) income and your independent variable is (say) ethnicity, with levels, per census definitions (White, Black/Afr.Am., Am. Indian/Alaska Native, Asian, Native Hawaii/Pac Islander, other and multiracial). Let's say you dummy code it with White being the reference category and you get

$ Income = b_0 + b_1BAA + b_2AIAN + b_3AS + b_4NHPI + b_5O + b_6MR $

If you are doing this study in New York City, you will probably get very few Native Hawaiians/Pacific Islanders. You might decide to include them (if there are any) with the others. However, you can't use the full equation and just not include that coefficient. Then the intercept will be wrong, and so will any predicted values for income.

But how should you combine categories?

As the others said, it has to make sense.

Improve this answer
$\endgroup$
4
$\begingroup$

To give a different opinion: why not include it as a random effect? That should penalize those levels with weak support and make sure their effect size is minimal. That way you can keep them all in without worrying about getting silly predictions.

And yes, this is more motivated from a Bayesian view of random effects than the whole "sample of all possible levels" view of random effects.

Improve this answer
$\endgroup$
0
$\begingroup$

I was also wondering whether I could combine non-significant categories with the reference category. The following statements in the book "Data Mining for Business Intelligence: Concepts, Techniques, and Applications in Microsoft Office Excel® with XLMiner®, 2nd Edition by Galit Shmueli, Nitin R. Patel, Peter C. Bruce", p87-89 (Dimension Reduction section) (Google Search Result) seem to support the second sentence of @Ellie's response:

  • "Fitted regression models can also be used to further combine similar categories: categories that have coefficients that are not statistically significant (i.e. have a high p-value) can be combined with the reference category because their distinction from the reference category appears to have no significant effect on the output variable"
  • "Categories that have similar coefficient values (and the same sign) can often be combined because their effect on the output variable is similar"

However, I plan to check with subject matter experts whether combining the categories makes logical sense (as implied in previous answers / comments, e.g. @Fomite, @gung).

Improve this answer
$\endgroup$
4
  • $\begingroup$ This answer is contradicted by the other answers here. $\endgroup$
    – kjetil b halvorsen ♦
    Aug 12, 2019 at 7:27
  • $\begingroup$ @kjetilbhalvorsen all the answers are just opinions $\endgroup$
    – PeJota
    Nov 18, 2022 at 14:25
  • $\begingroup$ @PeJota: If you want to combine categories that invalidates inference. If you only want prediction, that is different ... but beware that if you follow the advice in the first bullet point in this answer, results might depend on choice of reference level. At least try regularization, as in stats.stackexchange.com/questions/146907/… $\endgroup$
    – kjetil b halvorsen ♦
    Nov 18, 2022 at 16:12
  • $\begingroup$ I only want prediction, would it be fair to ignore non-predictive levels on a dummy variable then? $\endgroup$
    – PeJota
    Nov 22, 2022 at 17:54

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy