1
$\begingroup$

I have a dataset where I am trying to identify what group of people (i.e the predictors) are most likely to do X. I have ~25 predictors, ~5k cases and a binary outcome Y.

The predictors are categorical with many levels. I am mainly interested on what drives the outcome the most - it would be helpful to have coefficients that are interpretable as well.

I have quite a lot of multicollinearity in my dataset so my plan was to do ridge/lasso + logistic regression.

However, I have read some posts here where it is stated that the coefficients should be interpreted with caution (?due to bias), also the lack of p-values and CI is making me think this isn't the best solution for the question I am trying to answer. Should I go straight for a logistic model? Or is the smoothing provided by ridge useful in this case as it can highlight the predictors that are the most important?

$\endgroup$
5
  • $\begingroup$ Could you explain how such a model would answer your original question? $\endgroup$
    – whuber
    Jul 22 at 19:06
  • 1
    $\begingroup$ Welcome to Cross Validated! Why apply the ridge or lasso penalty? You seem to have enough data to handle $25$ features. $\endgroup$
    – Dave
    Jul 22 at 19:08
  • $\begingroup$ The usual rule of thumb to avoid overfitting for logistic regression is 10-20 cases in the minority class per predictor you are evaluating (including levels beyond the first of categorical predictors, interaction terms, and any non-linear coefficients you estimate for continuous predictors). So I agree with @Dave, provided that you have on the order of 10% or more of cases (~500 cases) in the minority class. Multicollinearity by itself isn't necessarily a problem, unless it's perfect so that your design matrix isn't of full column rank (and your software doesn't automatically handle it). $\endgroup$
    – EdM
    Jul 22 at 19:29
  • 1
    $\begingroup$ If the predictors are categorical with "many" levels, the OP's concerns may well be justified. Can you please explain what your predictors are? If you are mainly interested in the main drivers of the outcome, it sounds like p-values are not of major concern, no? $\endgroup$ Jul 22 at 19:31
  • $\begingroup$ Yes just to add that the predictors are categorical with many levels and my outcome is binary Y. I am mainly interested on what drives the outcome the most - it would be helpful to have coefficients that are interpretable as well. Would the coeff for ridge not be helpful to answer my question? Or am I overcomplicating it and I should just stick to logistic model? If anyone has any other ideas I'd be open to hearing! thanks $\endgroup$
    – test1234
    Jul 22 at 21:31

2 Answers 2

2
$\begingroup$

First, the question of "what drives the outcome the most" is very tricky and can be poorly posed. Consult this page and its links before you proceed.

Second, binary regression has inherent omitted-variable bias. If you omit any outcome-associated predictor, you risk bias in the coefficient estimates for the included predictors. It's usually a good strategy to include as many predictors as reasonable without overfitting.

Third, your data might be adequate to fit without penalization. As discussed in comments, that depends on the number of coefficients you need to estimate and the number of cases in the minority class. For a multi-level categorical predictor you need to estimate coefficients for one less than the number of levels. A useful rule of thumb is that about 15 minority-class cases per coefficient is adequate to avoid serious overfitting. See Chapter 4 of Frank Harrell's course notes or book.

Fourth, if you are in danger of overfitting, first apply data-reduction techniques combined with your understanding of the subject matter to reduce the dimension of your predictor space without looking at the outcomes. The Harrell chapters cited above contain many suggestions for things like combining related predictors, removing predictors with narrow distributions, combining related levels of categorical predictors, principal-components analysis (or related techniques), etc.

Fifth, you need to avoid focusing on the individual predictor coefficients reported for multi-level categorical predictors. Each typically represents the significance of a difference from whatever level you chose as reference, so "significance" of individual coefficients depends on that choice. You want an overall estimate that combines all levels, for example a Wald test on all coefficients together. See Section 5.4 of either of the Harrell references for how to do this, in expositions of why such attempts to evaluate predictor importance can be difficult or misleading .

Related to that problem, with standard lasso your final model might keep some levels of a categorical predictor while omitting others. You need the group lasso to keep sets of related coefficients together.

Sixth, if you still need to use a penalized approach like ridge or lasso, then you won't be able to use the simplest out-of-the-box measures with your multi-level categorical predictors. The default is typically to normalize all predictors to zero mean and unit standard deviation so that penalization of coefficients for continuous predictors doesn't depend on whether distances, say, are measured in millimeters or miles. That approach has problems even with binary predictors; with multi-category predictors, the choice of reference level can change the penalization! See this page and its links.

$\endgroup$
0
$\begingroup$

If you have a lot of multicollinearity in your independent variables and want to get rid of it, then you could start by excluding variables based on an analysis of VIF (Variance Inflation Factor), and based on your knowledge about your dataset. As explained by the comments under your questions, it might not be necessary depending on the characteristics of your dataset.

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.