4
$\begingroup$

I have a mixed model with a continuous outcome variable and a certain number of predictors. Some need to be included in the model no matter what (sex, age, and a "main factor"), and others must be selected from a list of potential confounders.

I know some software packages have very well developed procedures to do proper variable selection, but I am looking for a simple and "reasonable" method to select the variables manually.

The stategy used until now was to first conduct simple linear regressions with every predictor separately, and proceed to the multiple regression that includes every potential confounder whose p value in the simple regression was ≤ .250. I'm not sure if this threshold is commonly used, and I don't know what threshold to use to "bump out" the variables which don't contribute to the model. I might add that I have a good sample size (500), but that some of the variables have missing values which may not be MCAR (missing completely at random) - hence the need to be parsimonious.

My efforts to find clear and simple guidelines were not successful. I thank you in advance for sharing your advice.

$\endgroup$

4 Answers 4

4
$\begingroup$

Slightly better than your current method is stepwise forward regression. Please read the criticism on that page, though: it holds some of the many reasons why I don't like it (note that most of those reasons also apply to your current method, and there is even more criticism for that).

Bottom line there is to add one variable at the time (obviously the one for which there is most evidence that it must be added, i.e. smallest p-value in a likelihood ratio test or similar) up to a certain point. When the threshold is reached it is customary to perform clean-up, that is: remove some variables that are below some p-value threshold from the model again. The advantage of this method is that you can easily ensure that some variables are indeed guaranteed to be in your model (you simply start with these variables in the model, and exclude them from the cleanup. In a similar fashion you can also add interaction terms, once you're finished with the main effects.

If you're willing to go one step further, you can use any of the modern penalized regression techniques (LASSO, ridge, ...) though these cannot be applied manually. Software like R makes them easy to use, though (package glmnet).

With regards to your missing data: especially since you are asking for a 'manual' technique: I doubt you'll find any that properly accounts for missing data. One of the easiest solutions (that is statistically correct) is multiple imputation, but that will require a lot of work to do manually.

$\endgroup$
3
  • $\begingroup$ +1 for glmnet. I believe you can also force inclusion of certain variables, and it will do variable selection/shrinkage on the rest. It's also very fast. $\endgroup$
    – Zach
    Jun 23, 2011 at 13:17
  • $\begingroup$ Hm. Yes indeed, I just checked that. Parameter penalty.factor can be used for that. Thanks! $\endgroup$
    – Nick Sabbe
    Jun 23, 2011 at 13:20
  • $\begingroup$ I stumbled upon the stepwise function from the relimp package (integrated in R commander), which proposes various options of directions and criterions (bic/aic). That's much more accessible to me than glmnet, but I'm still unsure about which options to use. I know the results won't be affected much; I just need to pick a goodenough method and report it, really. Any advice? $\endgroup$ Jun 24, 2011 at 6:13
2
$\begingroup$

I disagree that there is any program that can do just what you want. Of the automated methods, LASSO is probably best, but it does not allow certain variables to be forced into the equation. It also does not use your substantive knowledge.

There is also no algorithm to replace an automated method with a manualized version.

The right method depends on number of variables and their interrelationship, but, if there are not a huge number, then I suggest coming up with several sets of variables, based on your knowledge, and comparing them based on AIC, BIC or some other similar measure (I don't have a strong preference among these).

This is a case where our human brains are better than computers.

$\endgroup$
1
$\begingroup$

Variable selection in your context is not recommended, in the absence of heavy penalization (e.g., using lasso or elastic net). It will result in invalid standard errors, P-values, regression coefficients, etc.

$\endgroup$
2
  • $\begingroup$ So you would recommend deciding beforehand of a set of variables and stick with those whatever their "contribution" might be? $\endgroup$ Jun 24, 2011 at 18:43
  • $\begingroup$ yes that's how you avoid bias. In ordinary regression that's how you get the right $\sigma^2$. If you do variable selection you must do simultaneous penalization unless perhaps the alpha for keeping variables is 0.5 or greater. $\endgroup$ Jun 24, 2011 at 22:07
1
$\begingroup$

Unless you want to know what features are relevant, or there is a cost associated in collecting all of the attributes, rather than just some of them, then don't perform feature selection at all, and just use regularisation (e.g. ridge-regression) to prevent over-fitting instead. This is essentially the advice given by Miller in his monograph on subset selection in regression. Feature selection is tricky and often makes predictive performance worse, rather than better. The reason is that it is easy to over-fit the feature selection criterion, as there is esentially one degree of freedom for each attribute, so you get a subset of attributes that works really well on one particular sample of data, but not necessarily on any other. Regularisation is easier, as you generally only have one degree of freedom, hence you tend to get less over-fitting (although the problem doesn't go away completely).

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.