2
$\begingroup$

Usually in backward elimination, we start with the full model of all covariates, check the $p$-value of $t$-statistic for each covariate (which is compared between the full model and the model minus the given covariate), and remove the covairate with the highest $p$-value until the maximum of $p$-value is all below some threshold $\alpha$.

What about when the data has both categorical (of multiple levels) and numeric data? I know the most natural thing is to do a similar procedure, but now with the $t$-statistic replaced with the $F$-statistic, and compare the entire group of covariates against the full model to eliminate (we remove the categorical variable of all levels all at once). But how can I easily implement this in $R$ and if there is already some package that integrates the scenario. In the usual case of all numeric covariates, I can perform something like

run_backward_elimination = function(alpha){
    S = 1:p
    while(TRUE){
    pvalues = summary(lm(highway_mpg~[,S]))$coefficients[-1,4]
    if(max(pvalues) <= alpha){
      break
    }
        remove_ind = S[which.max(pvalues)]
        S = setdiff(S,remove_ind)
    }
    XS = X[,S,drop=FALSE]; colnames(XS) = S
    print(summary(lm(Y~XS))$coefficients)
}

Then it should be fine, as I can directly check the $t$-static at the summary table of linear regression. But with the Anova table, the $F$-statistic is compared among the inner models, not against the full model like $t$-static does. So I have to manually run run $p$ many regressions to remove one covariate, which has a rather high complexity and is also tedious to implement.

$\endgroup$

1 Answer 1

3
$\begingroup$

I would argue that the variable under consideration is the entire factor with all levels; therefore the appropriate test is a "chunk" test of the entire factor variable, not individual levels. In a linear regression, the standard chunk test is the F-test of nested models...

...but stepwise procedures like this are known to be problematic, which just come up on Meta. If fact, the most downvoted post I have seen is a self-answer that argues in favor of stepwise procedures after many members argued against stepwise procedures.

While this is on a Stata website, the discussion is purely regression theory, and I think Harrell makes a compelling case against stepwise procedures.

$\endgroup$
2
  • $\begingroup$ Uh... CV.SE (not meta) is loaded with answers arguing against stepwise model build methods. $\endgroup$
    – Alexis
    Dec 9, 2021 at 21:27
  • $\begingroup$ I find the case less than compelling, because the linked arguments implicitly suppose that certain things are and are not done and assumed, including (a) alleging there is no "thinking" about the data that occurs among those using stepwise procedures and (b) assuming the sole purpose is explanation rather than prediction. This is not to defend stepwise procedures, only to point out that blanket condemnation of them might be counterproductive insofar as there plausibly exist circumstances in which such procedures might be an informative or helpful part of an analysis. $\endgroup$
    – whuber
    Feb 7 at 18:11

Not the answer you're looking for? Browse other questions tagged or ask your own question.