Let's suppose I have a dataset with classes A and B, with class A occurring in 1% of cases and class B occurring in 99% of cases. Perhaps class A is a loan default.

Suppose I want to "understand" what factors make one class A vs B by fitting a logistic regression to dependent variables X and then looking at model coefficients. Does it make sense to put a higher class_weight on class A, such as putting a weight of 99 on class A and 1 on class B? Or does the intercept already take care of this?

What if the logistic regression has a regularization parameter, would class imbalance matter more in this scenario? (because the model would be more inclined towards a constant "Predict B" model to reduce the penalty on coefficient size).

I've seen many economics papers in which a unregularized logistic regression is run on data and then the authors interpret coefficient sizes and significance, just wondering how valid this is.


I assume by "factors" you mean which features or variables are "most" important for distinguishing between the 2 classes? I think that you first need to establish a performance metric to check if the results make sense. E.g., on such a dataset, just always predicting the majority class will already give you a classification accuracy of 99% (or error of 1%). Depending on the size of the dataset, you may want to look at ROC auc or precision-recall aucs (maybe F1) to get an idea how well your model discriminates between the classes.

In addition (also depending on the size of your dataset) you may want to cross-validate, e.g., 0.632+ bootstrapping, 10-fold CV, etc to get an idea about the stability of your model. If you have a sufficiently large dataset, regularization (L1 or L2) may give you a good idea of how important certain features are, answering your question "Suppose I want to "understand" what factors make one class A vs B " -- but this only makes sense if a linear model is appropriate given the data, and if there's sufficient stability of your model in CV I'd say.

| cite | improve this answer | |
  • $\begingroup$ Thanks Sebastian. I've read many papers in which the implicit assumption is that the underlying data was generated from a linear model, y = Xb + e, and then inference is done by looking at the coefficient sizes and standard errors. But this doesn't quite make sense to me since why should we expect for the true model to be linear? Doesn't it make more sense to try to find a model with a high AUC and look at the coefficients of that model? $\endgroup$ – convolutedstatistic Sep 6 '16 at 13:03
  • $\begingroup$ "why should we expect for the true model to be linear?" That's the assumption of logistic regression since it is a generalized linear model. Let's say your data is not linear, then your model will perform poorly but you still have the linear relationship between inputs and weights z = w_0 + x_1*w_1 + ... + x_n*w_n. That's not going to change, but if your model doesn't capture the structure of the data, the weight coefficient's magnitudes may be meaningless for an importance interpretation -- they still tell you about which features are important for the classifier to make the decision though. $\endgroup$ – user39663 Sep 6 '16 at 14:24
  • $\begingroup$ " Doesn't it make more sense to try to find a model with a high AUC and look at the coefficients of that model?" A (standard) logistic regression model is always linear, so if you have a non-linear problem, it maximizing accuracy or AUC via model selection is not going to help. You could try algorithms for non-linear hypothesis spaces, but then you can't simply read the weight coefficients (if its parametric) as importances. $\endgroup$ – user39663 Sep 6 '16 at 14:27
  • $\begingroup$ but very few models in reality would be perfectly linear, so I was thinking of interpreting the model more in the sense of being a good linear approximation to the true conditional expectation E[Y|X] $\endgroup$ – convolutedstatistic Sep 6 '16 at 16:10
  • $\begingroup$ I absolutely agree with you. What I was trying to explain was that it is not about the "goodness" of the linear approximation but is something like "the feature coefficients tell you how important each feature is for the logistic regression output" rather than "the coefficients tell you if a feature is generally a good predictor or not, whether you assume a linear model or not". Or in other words the coefficients tell you how important the features are under a linear assumption. $\endgroup$ – user39663 Sep 6 '16 at 18:31

Not the answer you're looking for? Browse other questions tagged or ask your own question.