1
$\begingroup$

Suppose you have a very simple linear model to predict Salary. Your ridiculously simple model is based on a persons age and their education level which can have the values: None, High School, College, Post Grad.

gm0 <- lm( Salary ~ Age+EducationLevel, data=dfWork )
summary(gm0)

Now you run the summary. The coefficient for Age turns out to be so significant that p is almost zero. As for education level the results are mixed. The only value with a significant p is EducationLevel=College.

So ... umm ... what do you do with this information? Do you include EducationLevel in the model even though only one of the values is significant? Do you scrap it because only one value is significant? Or do you make a new column with a value=1 when EducationLevel=College, 0 otherwise and include THAT categorical variable into the model (and not the original EducationLevel).

Improve this question
$\endgroup$
3
  • $\begingroup$ en.wikipedia.org/wiki/Principle_of_marginality $\endgroup$
    – Bensstats
    Apr 8, 2019 at 23:30
  • $\begingroup$ If the new column is binary for "less than college" or "more than high school", that would seem a reasonable choice to me. $\endgroup$
    – James Phillips
    Apr 9, 2019 at 0:20
  • $\begingroup$ You need to go back to the primary data to get full continuous values for age and education level. These categorized versions can cause problems for the analysis. And abandon "statistical significance". Look at compatibility intervals for effects of interest. $\endgroup$
    – Frank Harrell
    Apr 9, 2019 at 12:19

3 Answers 3

Reset to default
3
$\begingroup$

I would not do anything on the basis of "statistical significance" alone.

First of all, the choice of the significance level is completely arbitrary and often dictated by ritual.

Second, "statistical significance" is highly dependent on sample size.

Third, "statistical significance" says absolutely nothing about practical significance. You might have a highly statistically significant estimate of an extremely small and uninteresting effect.

Fourth, the non-statistically significant estimate may be interesting in itself.

Fifth, re-categorizing a variable in order to chase p-values could completely mislead your audience, if the removed levels do have an association with the outcome but you simply didn't have enough statistical power.

Sixth. Form a research hypothesis (perhaps consider that there might also be an interaction and/or non-linear association), conduct a power analysis, collect the data, fit the model corresponding to your research hypothesis, check model assumptions, and report the unadulterated results.

Improve this answer
$\endgroup$
2
  • $\begingroup$ Thanks for the very thorough answer. As you can see I am kind of a noob when it comes to data analysis. I get the feeling my whole approach is wrong here in using significance to pick out interesting variables. What do you mean by power analysis? Are there any links on what you would consider the proper way of performing a linear regression on a data set with possibly dozens of variables? $\endgroup$
    – Dano13
    Apr 10, 2019 at 13:03
  • $\begingroup$ @Dano13 other things being equal, the larger your sample, the smaller an "effect" you are able to detect. Therefore, at the study design stage you should decide what effect you are hoping to detect, and this will imply a minimum sample size to detect such an effect - in other words you will have sufficient "power" to detect the effect. This is power analysis and you can search this site and many others for more information about it. As for variable selection, this is an enormous topic - again you can search this site and others. $\endgroup$
    – Robert Long
    Apr 10, 2019 at 13:20
2
$\begingroup$

You should ignore what you've already done and run a subset (joint) F-test of significance for the entire educational variable (all dummy variables for edu). If the p-value for this subset F-test is significant, then go ahead and look at the individual tests for the different levels of education in reference to the base group. Irrespective of p-value for the individual levels of education, you should leave them all in (as a general rule) because the subset F-test told you the variable of "Education" was important and it's nonsensical to remove portions of this variable and can turn into p-hacking. If the subset F-test is not significant, then do not make the individual comparisons of educational level. There are some very good points made by @Robert Long.

Improve this answer
$\endgroup$
0
0
$\begingroup$

If only one of the factors in EducationLevel is significant consider omitting EducationLevel as a whole. If you want to keep EducationLevel but with only one or two factors, consider collapsing those factors to only include rows in your data that contain the desired factor levels. A way to do this:

library(tidyverse)

gm0 <- gm0 %>%
  filter(EducationLevel==College)

However, modeling is an art form, there is no correct answer. You may keep all EducationLevel factors even if certain factors aren't significant if the entire model is significant. That is up to you.

Improve this answer
$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge that you have read and understand our privacy policy and code of conduct.

Not the answer you're looking for? Browse other questions tagged or ask your own question.