Model selection for linear regression with categorical variables

Question

I regressed the dependent variable Rating (numeric) on Judge which is categorical. The output of the first model is given at the end of the question. Only Judge John Foy and Linda Murphy came out to be significant. In the next step of modeling, should I consider only 3 factors which are John, Linda and rest judges, and run the regression again?

Please guide me to take the next step to create the final model. Ultimately I want to know that should we combine all the factors which are not significant and run the regression again with new set of factors of a categorical variable.

> levels(wine_data$Judge)
[1] "Danièle Meulders"   "Francis Schott"      "Jamal Rayyis"       
[4] "Jean-Marie Cardebat" "John Foy"            "Linda Murphy"       
[7] "Olivier Gergaud"     "Robert Hodgson"      "Tyler Colman"       

>wine_lm <- lm(Rating ~ Judge, data = wine_data)

>summary(wine_lm)

# Call:
# lm(formula = Rating ~ Judge)
# 
# Residuals:
#    Min     1Q Median     3Q    Max 
# -7.850 -1.625  0.325  1.400  4.825 
# 
# Coefficients:
#                          Estimate Std. Error t value Pr(>|t|)    
# (Intercept)               13.6000     0.5164  26.337  < 2e-16 ***
# JudgeFrancis Schott        1.2500     0.7303   1.712  0.08877 .  
# JudgeJamal Rayyis          1.0750     0.7303   1.472  0.14285    
# JudgeJean-Marie Cardebat  -1.0500     0.7303  -1.438  0.15232    
# JudgeJohn Foy              3.0250     0.7303   4.142  5.4e-05 ***
# JudgeLinda Murphy          2.0500     0.7303   2.807  0.00558 ** 
# JudgeOlivier Gergaud       1.0000     0.7303   1.369  0.17269    
# JudgeRobert Hodgson       -1.4000     0.7303  -1.917  0.05690 .  
# JudgeTyler Colman         -0.4500     0.7303  -0.616  0.53858   
# ---
# Signif. codes:  0 ‘***’ 0.001 ‘**’ 0.01 ‘*’ 0.05 ‘.’ 0.1 ‘ ’ 1
# 
# Residual standard error: 2.309 on 171 degrees of freedom
# Multiple R-squared:  0.2713,  Adjusted R-squared:  0.2372 
# F-statistic: 7.957 on 8 and 171 DF,  p-value: 4.345e-09

Answer 1

In general, you should not just include some levels of a categorical dependent variable.

First, that changes the question you are asking from "Are judges different?" to "Are these three judges different from the others?"

Second, it is "fishing". If you really wanted to do this (and I don't think you do) then you should separate the data into training and test sets, find the three judges in the training set and then look for those three in the test set.