I'm trying to determine the influence (direction and relative strength) of certain attributes of incoming students to an academic program on their successful completion of the program. My sample size is in the range of several hundred students. There are about 20 variables all together, all of which we could reasonably expect to have some influence on successful completion of the program. (I'm not throwing in everything I could conceive of). My variables are as follows:

  1. Variables which are necessary to control for as determined by previous investigations of a similar nature. These are things like race, gender, age, incoming GPA, number of credits, etc.
  2. Variables which are more specific to the particular program which haven't been investigated yet, such as pre-admission tests of subject matter specific to the program or grades in courses which are pre-requisites for the program.

I've examined the data from a bunch of angles by now, but here are the three approaches which, when compared, are causing me grief:

  • Performing a backward step-wise feature selection on a logistic regression model which at the beginning includes all the features.
  • Fitting a LASSO model to the data, cross validating to find the optimal lambda.
  • Including all the variables in a logistic regression model. No feature selection.

All the numeric variables going in to the analyses have been scaled, so that to answer my original question, I would just have to look at the absolute values and signs of the coefficients. The problem is that I don't know which model to go by.

Including all the variables leaves me with 16/20 variables having p-values above .1. The four variables which do have small p-values are those which were left in the step-wise routine. The LASSO model, when lambda=lambda.min, leaves me with approximately half the variables having non-zero coefficients.

My not particularly well informed feeling is that, if I were just trying to get an idea of the influence of these variables for my own use, I would use the relative sizes and directions of the lasso coefficients, but attribute more certainty to those variables which came up as significant in the full logistic regression model, ignoring the step-wise model altogether.

Is this intuition justified?

There's also the issue of presenting these findings to others. Standard regression techniques with p-values are pretty well known to anyone who has taken a course covering regression, while LASSO is not. Are LASSO coefficients ever presented analysis models such as the one I've described?

EDIT based on comment:

If my main concern is coming up with an ordered list of variables in terms of their effect size, along with the direction, then which of the following options should I expect to get me better answers: 1) using the coefficients of the full model or 2) using the coefficients from the Lasso model, or perhaps the relaxed Lasso, and accepting that I'm not going to have any p-values or confidence intervals, etc.?

  • 3
    $\begingroup$ It is not appropriate to use variable selection methods that do not incorporate shrinkage. Lasso or quadratic penalty (or mixed: elastic net) should work OK. P-values from ordinary stepwise methods are not valid, and using measures of association to guide model selection is frought with difficulties. Note that variable selection guided by associations with Y does not help in any way with the "too many variables too few events" problem. $\endgroup$ – Frank Harrell Dec 2 '12 at 21:35

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Browse other questions tagged or ask your own question.