2
$\begingroup$

I'm using sklearn LogisticRegression with a training data set of 279 inputs. Each input point belongs to $[0,1]^2$ and to a class. There are two classes: $\{0, 1\}$.

I evaluate AUC score with cross_val_score with below code snippet.

aucs = []
num_runs, num_splits = 40, 5
for seed in range(num_runs):
    # Use logistic regression with L2 penalty
    estimator = LogisticRegression(penalty="l2", C=0.05, solver="liblinear")

    cv = StratifiedKFold(n_splits=num_splits, shuffle=True,
                         random_state=seed)

    # Cross validation on the training set
    auc = cross_val_score(estimator, X=features_train, y=labels_train,
                          cv=cv, scoring="roc_auc", verbose=0)

    aucs.append(auc)

aucs = np.array(aucs)
print(f"AUC: max {aucs.max()}, min {aucs.min()}, mean {aucs.mean()}, std {aucs.std()}")

The issue I have is that AUC score has large variations depending on the training / validation split:

AUC: max 0.9696969696969696, min 0.659846547314578, mean 0.8303383462619687, std 0.05924693385612475

The main questions I'm asking myself:

  1. Is the range of those variations "normal" considering the size of the training data set?
  2. Would regularization be a good topic for increasing AUC stability?

Overall, is there a way to obtain a more stable model?

$\endgroup$
4
$\begingroup$
  1. We can't say. The stability of AUC (or any other quality KPI) depends on the size of the training set (which is modest at 279 samples), but also on the number of predictors (which you didn't tell us).

    If you have a single predictor, your AUC will likely be more stable than if you have twenty, simply because you can overfit twenty predictors more easily.

  2. Yes, regularization should make your AUC more stable. Which is just another way of looking at point 1 above. The more you regularize, the less "wiggle room" your model will have. If you regularize enough, you will be left with a very simple model that only uses the overall proportion of successes in your training sample, and the variability of that will not depend on the predictors any more, so you have removed a big source of variation.

$\endgroup$
  • $\begingroup$ Thanks for your answer Stephan. I'll try to answer your predictor question based on my current ML knowledge! If I refer to the code snippet I provided, I have only one predictor which is the probability of a sample to belong to class 1. $\endgroup$ – mathcounterexamples.net Aug 28 '19 at 8:06

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.