Cross Validated is a question and answer site for people interested in statistics, machine learning, data analysis, data mining, and data visualization. Join them; it only takes a minute:

Sign up
Here's how it works:
  1. Anybody can ask a question
  2. Anybody can answer
  3. The best answers are voted up and rise to the top

I have 18 predictors in a binary generalized linear mixed model (repeated measurements, over a 1000 subjects). I would like to trim the model a bit and remove some noise and useless predictors. Unfortunately, PROC GLIMMIX does not have any facility to do this. I could not find an R package that would do this (step() function style). If I were to try this manually, say begin with a full model (all predictors in) and do a 'backward selection', what criterion could I use to do this quickly? Could I use, say, p-values? But at what significance level?

share|improve this question
In general predictor selection is a bad thing to do. To help understand why, you may want to read my answer here: algorithms-for-automatic-model-selection. – gung Oct 25 '12 at 22:55

Rather than using a stepwise procedure, I would fit an L1-regularized model, and discard predictors whose coefficients are effectively forced to be zero. See [Ng 2004].

share|improve this answer
That would be great, if I could only find an implementation of it specific for a GLMM. I could just ignore the within-individual clustering (yikes!) and attempt something like what you propose, or something related like the elastic-net procedure and see how well it does in cross-validation. If the within-individual nesting is ignored, then a whole plethora of predictive techniques can be used... my main problem was that I did not want to ignore the within-subject repeated measures. But I guess it wouldn't hurt to see what happens if the analysis is conducted that way. – user16263 Nov 25 '12 at 18:46
I could think of some options here, but maybe it's better to open another question on what software exists to fit regularized GLMM models (or even better, specifically the kind of GLMM you want, which I think is a logistic regression). – Jack Tanner Nov 25 '12 at 18:49

The most robust way to do backward selection using a given model, would be to remove one predictor, fit the model to the the remaining predictors and evaluate (in cross-validation of course). Repeat for all predictors and remove the one which has the smallest contribution. Then iterate the process until you achieve a combination of predictor number/performance that satisfies you. If you have 18 predictors, it would take 18+17+16+15+..+(k-1) model fittings/evaluations to find a model with k predictors.

If this your model fitting/evaluation is too expensive in term of resources, you can try any kind of predictor evaluation instead, you can use one of several possible measures (e.g. statistical tests/correlation/information gain) and use the same process of fitting and evaluating. You don't care about a threshold because at each iteration you remove the one with the lowest contribution regardless of the actual value. The true evaluation will be on what you are trying to predict (CV etc.).

share|improve this answer
Thanks. It's probably going to come down to this. The size of the dataset allows splitting into training, calibration, and testing. – user16263 Nov 25 '12 at 18:34
That would require at least 20,000 subjects for the results to be replicable and not subject to "lucky splits". – Frank Harrell Dec 26 '12 at 13:27
@FrankHarrell typically this is done in cross-validation and does not use any additional data. Robustness of the procedure depends on the data itself (for example the structure of intercorrelations), but in many cases this approach works well for many Machine Learning problems. Obviously, this is a heuristic and will not be as comprehensive as evaluating all combinations of features - but that would be NP-hard in general. My point is that even without theoretical guarantees this often works well in practice. – Bitwise Dec 26 '12 at 16:58
Correct; I was referring to the non-cross-validation ideas others mentioned above. Cross-validation can work quite well if you repeat it 50-100 times and average over the (say 10-fold) cross-validations. Or just use the optimism bootstrap which requires fewer iterations. – Frank Harrell Dec 27 '12 at 14:10

Your Answer


By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.