0
$\begingroup$

Suppose we are assessing the impact three factors, each with two levels, have on some response $Y$. Let's call the factors $A$ with levels $\{a_1, a_2\}$, $B$ with levels $\{b_1, b_2\}$and $C$ with levels $\{c_1, c_2\}$.

Suppose that first of all we want to determine the 'best' model in R using step-wise regression techniques. 

full_model = lm(Y ~ A*B*C)
null_model = lm(Y ~ 1)

forward_model = step(null_model, scope=list(lower=null_model, upper=full_model), direction="forward", k = 2)  
backward_model = step(full_model, direction="backward", k = 2)                                                 
bidirec_model = step(full_model, direction="both", k = 2)

After running the code suppose that we find that each of the main effects, two-way interactions and three-way interactions are significant. That is, full_model = lm(Y ~ A*B*C) is our 'best' model.

When add the summary wrapper around full_model, i.e. summary(full_model) and run it we get a result like this, 

Call:
lm(formula = Y ~ A * B * V)

Residuals:
   Min     1Q Median     3Q    Max 
 -5.50  -1.25   0.00   1.50   4.75 

Coefficients:
                                   Estimate Std. Error t value Pr(>|t|)
(Intercept)                          30.150      1.095  29.801  < 2e-16
a2                                   11.250      1.548   7.267 2.05e-10
b2                                   -0.500      1.548  -0.323  0.74754
c2                                   38.250      1.548  24.708  < 2e-16
.      
.
.
a2:b1                                 5.250      2.189   2.398  0.01879
.
.
.
a2:b1:c1                              4.500      3.096   1.453  0.14998
a2:b1:c2                              3.250      3.096   1.050  0.29699
.
.
.

('...' represents omitted rows) Note: I just made these values up

My questions are,

  1. Why does $R$ use the factor as a WHOLE (i.e. $A$ instead of $a_1$ and $a_2$, $B$ instead pf $b_1$ etc) in step-wise regression? R instead will exclude the factor as a WHOLE, that is, it will remove $A$ completely from the model rather than just a particular level. Why is this?

  2. When (if at all) should we remove insiginifcant terms from our model obtained with step-wise regression?

Improve this question
$\endgroup$

1 Answer 1

Reset to default
3
$\begingroup$

First, you should never use stepwise regression. All of the output is wrong. Parameter estimates are biased away from 0, p values are too small, standard errors are too small and so on.

Second, if you are going to use stepwise (or any other selection) it really does make sense to treat factors as a whole. Otherwise, results don't make sense. Suppose you have a variable "race" and you have levels "White", "Black", "Other" (not an uncommon classification). Then if you drop just (say) "Black" you wind up with nonsense. Where do the Black people go? Who is different from whom?

Third, using statistical significance as your sole means of model fitting is a mistake. You should never remove terms just because they are not significant. There are lots of reasons to include non-significant variables e.g.

  • Adding the variable affects other parameters
  • The effect is important, even if not significant
  • You hypothesized about that variable
  • Deleting that variable will get you laughed at
  • Theory says that variable is important, so finding a small effect is important
  • The variable is involved in an interaction
Improve this answer
$\endgroup$
4
  • $\begingroup$ Hello Peter, dropping a level from a categorical variable would be equivalent to lumping the data with whichever category was used as reference. It's not quite nonsense, just requires a more careful interpretation. Your second bullet point is somewhat unclear to me, as generally importance is measured in terms of effect size, which one would consider as indistinguishable from zero under the significance test paradigm. Would you mind to clarify this? $\endgroup$
    – overdisperse
    Commented May 9, 2018 at 0:59
  • 1
    $\begingroup$ That's only true of levels if you do it in a certain way. And effects can be very important and not significant. "not significant" does not mean "indistinguishable from 0" it means that, if the null hypothesis is true in the population, you'd be unlikely to get a test statistic at least as extreme as the one you got in a sample the size of the one you have. A variable can be significant and unimportant, important and not sig., or any other combination. $\endgroup$
    – Peter Flom
    Commented May 9, 2018 at 1:43
  • $\begingroup$ How would you define importance? You already address theoretical importance lower in the list. Surely trying to measure it through any estimation-derived quantity would suffer from problems related to those that arise when using stepwise selection? $\endgroup$
    – overdisperse
    Commented May 9, 2018 at 2:15
  • 1
    $\begingroup$ You define importance based on substanitve knowledge of the field. And stepwise introduces a whole bunch of additional problems. $\endgroup$
    – Peter Flom
    Commented May 9, 2018 at 10:42

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.