4
$\begingroup$

No regular here will be unaware of the perils of using stepwise and similar automatic methods for variable selection in regression analysis. But preferred alternatives, such as the lasso or elasticnet, have there own difficulties.

I can't find anywhere in the archive here a discussion of the methods provided by the subselect package in R, which I've just come across - the package has, so far as I can see, existed for a decade and more, and presumably has proved useful.

In addition to a variation on the leaps procedure, subselect offers three algorithms (which it calls anneal, genetic and improve) for variable selection for different kinds of analysis.

Have these procedures (or any of them) proved to be of value in variable selection?

$\endgroup$
5
$\begingroup$

Given the plethora of things in the package, I think this is too broad to answer, in the sense of 'could take a book to answer'.

However, let me give you some broad principles:

1) Where it involves finding the best (by some criterion of fit say) subset of variables, it will generally suffer from pretty much exactly the same set of problems as stepwise or all subsets regression. [Leaps and Bounds comes in here, for example, even if it's based on AIC or BIC - some of the issues may be mitigated somewhat by using such a criterion, but the main underlying problems are unaltered]

Let's look at the vignette under 'Measuring quality'

1.2 Measuring the quality of a subset
Selecting variable subsets requires the definition of a numerical criterion which measures the quality of any given variable subset. In a univariate multiple linear regression, for example, possible measures of the quality of a subset of predictors are the coefficient of determination $R^2$, the $F$ statistic in a goodness-of-fit test, its corresponding $p$-value or Akaike's Information Criterion (AIC), to give a few examples

Yeah, like that. If this is what is happening, it's essentially going to leave you with the same shopping list of problems as stepwise (less, perhaps, the problem that stepwise often misses the 'optimal' model).

Broadly speaking, it doesn't matter if you use this or that algorithm to find the optimum, or this or that statistic in your criterion, it's the use of optimization itself (without properly accounting for the effects of doing that) that screws everything so badly. Use any fancy optimizer you like, it's still optimizing, so you're still screwed.

2) If there's some kind of regularization (such as shrinkage, as you can get with the lasso and a number or other approaches) then many of those problems may either substantially reduced or avoided.

3) where there's proper out of sample assessment of the performance of competing models in the class (such as via cross validation), the inferences tend to be more 'honest' - to have closer to the required properties, like approximate coverage of confidence intervals and so on. With variable selection, this would tend to involve having a subset for identification, a subset for estimation and a subset for testing. (Cross validation would then work by looking at what happens with repeated subsetting like that.)

More detailed specifics depend on exactly what is being done with each function, but I think that gives a framework which should usually give you a good idea.

| cite | improve this answer | |
$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.