3
$\begingroup$

I have a dataset with the following characteristics:

  • moderate sample size (~300 samples)
  • moderate class imbalance (~20% positives)
  • high-dimensional (the number of independent variables, again ~300, is comparable to the number of samples)

I fitted a logistic regression model with Lasso regularization and two-level 3-fold cross-validation (the outer split is used exclusively for performance evaluation, while the inner CV is used for tuning the regularization hyperparameter). I am using scikit-learn's LogisticRegressionCV with class_weigh='balanced' to try and make up for at least some of the imbalance.

The performance as evaluated on the outer CV split is quite good: the model achieves a ROC AUC of around 0.97 ± 0.02 and a Matthews' correlation coefficient of 0.8 ± 0.05 when using a decision threshold of 0.5. This all sounds fun until I look at the model parameters. The regularized model contains around 60 non-zero coefficients, and the optimized regularization hyperparameter is tiny ($\frac{1}{C} = \frac{1}{50}$). Furthermore, when I check the predictions on the training set, I can see that the model achieves complete separation of the classes, which is a known problem with optimizing logistic regression using the standard cross-entropy loss (which is what scikit-learn does).

Now my problem with this is that the number of non-zero coefficients is comparable to the number of positive samples, i.e., the cardinality of the smaller class. I know of the common advice that in an imbalanced-class logistic regression modell, one should adjust the number of predictors based on the cardinality of the smaller class, and this seems reasonable enough (even obvious): if there's one predictor for each positive sample, then it would be trivial to overfit the model and basically have it memorize every single training data point.

So, my question is: do you think my model is overfit? The small regularization parameter and the relatively high remaining dimensionality of the final model would suggest so, but the cross-validated performance suggests that it really is as good as it sounds like. What do you think?

P.s.: I was careful enough to apply at least some basic measures for preventing information leakage. The dataset I am trying to model is from a biomedical study, and sometimes, multiple samples are present by certain patients. I am using the StratifiedGroupKFold splitter to take this into account, by ensuring that the same patient never shows up in multiple splits during cross-validation.

Improve this question
$\endgroup$
0

1 Answer 1

Reset to default
2
$\begingroup$

The sample size is not large enough to be able to reliably choose the shrinkage coefficient (penalty; regularization). The sample is barely large enough to estimate the overall prevalence of the outcome ignoring predictors, i.e., to estimate just the intercept in a logistic model. For a margin of error of 0.05 on the probability scale the sample size needed (with zero predictors) is 384; for a margin of error of 0.1 it is 96 observations. Your effective sample size is 3p(1-p)n where p = 0.2, which helps you determine the dimensionality of the features that you are allowed to analyze.

The only real hope is to reduce the feature space to something the effective sample size will support, which is perhaps 2-5 collapsed features. Features can be collapsed using unsupervised learning (data reduction) blinded to Y.

These issues are discussed here and here.

Improve this answer
$\endgroup$
4
  • $\begingroup$ I'm afraid I don't really follow how this answers my question. How does this explain the apparently good generalization power, then? Another question would be why I experience the same phenomenon when I do not try estimating the shrinkage coefficient using cross-validation, if that is an issue. For example, I since tried separately selecting the 10 top features most correlated with the target, based on your claim of the effective sample size ($\approx 144$), then applying LDA, both with Ledoit-Wolf shrinkage and without. The results were approximately the same in all three cases. $\endgroup$
    – ladislaw94
    Commented Nov 24, 2023 at 8:34
  • $\begingroup$ I also couldn't find the sample size formula (let alone a derivation of it) that you mentioned. Based on the binomial distribution, $np(1 - p)$ seems logical enough, but where exactly does the factor of 3 come from? $\endgroup$
    – ladislaw94
    Commented Nov 24, 2023 at 8:40
  • $\begingroup$ I found a common definition of "effective sample size" as used in the context of weighting samples, which is $N_{eff} = \frac{(\sum{w_i})^2}{\sum(w_i^2)}$. If I plug in the inversely-weighted case (i.e., the negative class is weighted $\frac{p}{1-p}$ times compared to the positive class), I get $N_{eff} = 4Np(1-p)$. That's closer, but still 4, not 3. What gives? $\endgroup$
    – ladislaw94
    Commented Nov 24, 2023 at 9:40
  • $\begingroup$ That’s a slightly different context. I’m refer to the effective sample size in terms of power/information content using a Wilcoxon test to compute power, which is identical to considering the variance of a log odds ratio in a proportional odds semiparametric ordinal model. Said more simply we are computing the $N_\text{eff}$ for a continuous $Y$ with no ties that gives the same power of a two-group comparison (even though we may not be doing this comparison) with the original $N$ observations that may contain ties. Details at hbiostat.org/rmsc/multivar#sec-multivar-overfit $\endgroup$
    – Frank Harrell
    Commented Nov 24, 2023 at 13:51

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.