5
$\begingroup$

I wanted to create a predictive model of mortality after patients had undergone a surgical procedure. But I also wanted to avoid doing what most researchers do by first performing univariate analysis then using the variables that are found to be significant to perform multivariate analysis using some sort of step-wise feature selection. So I used glmnet to perform feature selection, and found about 20 of the initial 80 variables to be significant. I then used some of these variables (as supported by literature) to create a statistical model to predict mortality using the glm function in R. I think the model does fairly well as it has a ROC of 0.8. However when I use the summary function I notice that of the 15 variables that I am using, 5 of them do not have significant p-values. But if I remove these variables, in my mind the model would not make sense (since the literature supports their use), and in addition the ROC decreases to about 0.75.

Given this situation, how does one go about when analyzing these variables? It seems that they are useful and necessary (as they aid in the discrimination of patients who will die or live when having a procedure performed on them) but do not have a significant p-value.

Forgive me for being light on code, as I wanted more of a 10,000 feet overview of this rather than to get into the nitty-gritty from the get-go. As always, I appreciate the help!

Improve this question
$\endgroup$

1 Answer 1

Reset to default
7
$\begingroup$

First of all, it seems like glmnet is a reasonable tool for your problem - good choice!

If all you want is a predictive model, you don't need to worry about p-values. A simple way to assess the predictive accuracy of your model is to use cross validation. Glmnet will cross validate automatically for you (try cv.glmnet), so implementation should not be a problem. The model cv.glmnet produces can then be used as is.

A thing to keep in mind is that glmnet (or lasso) simultaneously shrinks and selects features. The fact that glmnet shrinks features allows it to use more features without overfitting. The upshot is that if you just take the features selected by glmnet and plug them into glm, you'll probably start overfitting (since glm won't do any of the shrinking).

Anyways, once you start using glmnet, you need to stay within the world of penalized regression. You should just take the output of glmnet as your model, and not try to use glm to refit it.

Improve this answer
$\endgroup$
7
  • $\begingroup$ To add to Stefan's comment. Stepwise is used with regular glm to arrive at a parsimonious model. With glmnet, variables that aren't adding much to the analysis get zero coefficients. So I wouldn't be confident that stepwise on top of glmnet would add much value. $\endgroup$
    – conjectures
    Commented Jun 3, 2013 at 7:16
  • $\begingroup$ @conjectures: I'm not sure if what I was doing could be called stepwise since I just took the variables that glmnet had chosen as being significant and plugged them into glm without further modification.Then to rephrase my question - for glm, why do we even bother with p-values in the first place? $\endgroup$
    – oort
    Commented Jun 4, 2013 at 0:25
  • $\begingroup$ @Stefan Wager: Why must I stay in the world of penalized regression? Why can't I use glmnet as an intermediate step as I did here to simply find features that are significant? Both GLM and GLMNET are performing classification against the same variable (mortality) - shouldn't the "true" features that are significant be the same for both? (Hope that makes sense) $\endgroup$
    – oort
    Commented Jun 4, 2013 at 2:43
  • 2
    $\begingroup$ The key concept here is `degrees of freedom'. If your model has more degrees of freedom, then it can fit the data more accurately, but it is also susceptible to fitting noise (i.e. overfitting). Glm needs to pay one degree of freedom for every feature you look at. Glmnet, by using penalization, is able to look at more features while spending less degrees of freedom. One way of understanding this is that glmnet earns the right to scan through all the features and select the ones that fit the data best by then shrinking the predicted coefficients. $\endgroup$
    – Stefan Wager
    Commented Jun 4, 2013 at 4:24
  • 1
    $\begingroup$ If you remove the shrinkage and pass the features selected by glmnet into glm, you're adding in more degrees of freedom and will probably end up overfitting. Anyways - the take home message is that glmnet provides a holistic way of building a model that integrates shrinkage and variable selection. You're better off using the output of glmnet as is rather than trying to do some other regression after it. (By the way... Regarding your other comment: If you want to build a predictive model, you really don't need to look at p-values. Cross validation is more useful.) $\endgroup$
    – Stefan Wager
    Commented Jun 4, 2013 at 4:28

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.