1
$\begingroup$

Quick Intro

Sorry for the long read. I added a lot in here because I wanted to describe what I've worked on so far, but I wanted to quickly summarize the issue I've been having, just so you have it in mind when you continue reading.

I've seen a lot of posts that go over the appropriate way of oversampling an imbalanced dataset and performing cross validation for model selection purposes. Usually these issues are described to ensure that the modeler has an accurate scoring metric on the cross validated model.

I've tried testing two approaches:

1) Wrong Approach: Oversampling before GridSearch CV

2) Right Approach: Oversampling within GridSearch CV

and added the extra step of validating test sets which I haven't seen many of these posts go over. To my surprise, the scoring of the models on the test sets aren't that different and to me, this brings up the issue of whether this means there is no real difference in model selection as the overfitted model can still perform just as well.

I don't want to leave you hanging at this point, and included a lot of the steps I took in the following sections. Maybe I missed something, or took an inappropriate step, but please feel free to let me know. It somewhat bothers me to think that calibrating an overfitted model can lead to a similar performance as one that has been fitted more reasonably. I'd like to better understand and discuss why this happens.

Description of Approaches

I was trying to compare two approaches to optimal model selection/ hyperparameter tuning based on the following:

1) Wrong Approach: Oversampling before GridSearch CV

The oversampling is done on the data before it's split into folds. This can lead to bleeding of data (that the model has already seen) to the test fold during cross validation

2) Right Approach: Oversampling within GridSearch CV

This ensures that oversampling is performed on selected training folds and observations that the model has seen don't bleed-over to the test fold during cross validation

The Overall workflow:

1. Generate dataset: 2 different datasets were created to do some testing and are described in more detail:

  • Randomized dataset - X is composed of 10 IID normally distributed variables and Y is composed of random uniform sampling.

        n = 10000
        np.random.seed(48)
        mean = np.zeros((10))
        cov = np.identity(10)
        X = np.random.multivariate_normal(mean, cov, n)
        y = np.random.uniform(0, 1, n)
    

    Y is also binarized to have class imbalance ~ 95%: 5%. Expected AUC is to be 0.5

        y[y <= 0.05] = 0
        y[y > 0.05] = 1
    
  • sklearn's make_classification - I made an imbalanced binary class dataset ~ 95%:5% and made the classification difficult by having class separation = 0. 10 Features were added like previous dataset, but 1 was informative. Wanted to see how well both approaches could utilize the 1 informative feature while having no class separation.

        X, y = make_classification(n_samples=10000, n_features=10, n_informative=1,
                        n_redundant=0, n_repeated=0, n_classes=2,
                        n_clusters_per_class=1,
                        weights=[0.95, 0.05],
                        flip_y = 0,
                        class_sep=0, random_state = 48)
    

2. Split into a train and test dataset:

    # 60/40 split for train/test sets
    X_train, X_test, y_train, y_test = train_test_split(X, y, test_size= 0.4, 
    random_state  = 48, stratify = y)

3. Run GridSearchCV (scoring = AUC) with both oversampling approaches

I ran these tests on

Gradient Boosted Machine (n_estimators = 1000, learning_rate = 0.01)

Random Forest (n_estimators = 1000)

Logistic Regression.

I kept the parameter grids simple for now, but open to more suggestions:

param_grid_gbm = [
    {'classifier__max_depth': [5, 10, 15],
     }
    ]

param_grid_rnd = [
    {'classifier__max_depth': [5, 10, 15],}
    ]

param_grid_log = [
    {'penalty': ['l1', 'l2']}
    ]

4. Compare .best_params_, training and test scores of both approaches

I definitely expect there to be a good amount of overfitting based on approach 1, but am interested to see how the model compares with approach 2 based on test scores and best_params_.

Results

Note:

Post OS - Wrong Approach

Pre OS - Right Approach

  • Randomized dataset

    Model Type:  GBM
    Best Params from Post-Oversampled Grid CV:  {'max_depth': 10}
    Best Params from Pre-Oversampled Grid CV:  {'classifier__max_depth': 10}
    AUC of Post-Oversampled Grid CV - training set:  1.0
    AUC of Post-Oversampled Grid CV - test set:  0.5014220616374434
    AUC of Pre-Oversampled Grid CV - training set:  0.5064537046306599
    AUC of Pre-Oversampled Grid CV - test set:  0.5020803028170115
    
    Model Type:  RF
    Best Params from Post-Oversampled Grid CV:  {'max_depth': 15}
    Best Params from Pre-Oversampled Grid CV:  {'classifier__max_depth': 10}
    AUC of Post-Oversampled Grid CV - training set:  1.0
    AUC of Post-Oversampled Grid CV - test set:  0.49986835176408634
    AUC of Pre-Oversampled Grid CV - training set:  0.49992743583114635
    AUC of Pre-Oversampled Grid CV - test set:  0.5011079306982831
    
    Model Type:  LOG
    Best Params from Post-Oversampled Grid CV:  {'penalty': 'l2'}
    Best Params from Pre-Oversampled Grid CV:  {'classifier__penalty': 'l2'}
    AUC of Post-Oversampled Grid CV - training set:  0.5367667764608921
    AUC of Post-Oversampled Grid CV - test set:  0.5171403396263796
    AUC of Pre-Oversampled Grid CV - training set:  0.4959255247658574
    AUC of Pre-Oversampled Grid CV - test set:  0.5171403396263796
    

Randomized Dataset

Note: The red bars indicate results from the test set

As expected, Post-OS models have very high AUC on training data. Pre-OS models end up being close to 0.5.

As far as the test scores go, both models give a similar score of around 0.5 as well. The discrepancy between both approaches is much lower for a Logistic Regression.

  • sklearn's make_classification

    Model Type:  GBM
    Best Params from Post-Oversampled Grid CV:  {'max_depth': 10}
    Best Params from Pre-Oversampled Grid CV:  {'classifier__max_depth': 15}
    AUC of Post-Oversampled Grid CV - training set:  0.9999858347032252
    AUC of Post-Oversampled Grid CV - test set:  0.5144736842105263
    AUC of Pre-Oversampled Grid CV - training set:  0.563613940767598
    AUC of Pre-Oversampled Grid CV - test set:  0.5098684210526315
    
    Model Type:  RF
    Best Params from Post-Oversampled Grid CV:  {'max_depth': 15}
    Best Params from Pre-Oversampled Grid CV:  {'classifier__max_depth': 5}
    AUC of Post-Oversampled Grid CV - training set:  0.9999406990733853
    AUC of Post-Oversampled Grid CV - test set:  0.5211842105263158
    AUC of Pre-Oversampled Grid CV - training set:  0.5701708880319654
    AUC of Pre-Oversampled Grid CV - test set:  0.5522368421052631
    
    Model Type:  LOG
    Best Params from Post-Oversampled Grid CV:  {'penalty': 'l1'}
    Best Params from Pre-Oversampled Grid CV:  {'classifier__penalty': 'l1'}
    AUC of Post-Oversampled Grid CV - training set:  0.5532500750193888
    AUC of Post-Oversampled Grid CV - test set:  0.5246052631578947
    AUC of Pre-Oversampled Grid CV - training set:  0.4913954088676672
    AUC of Pre-Oversampled Grid CV - test set:  0.5246052631578947
    

make_classification dataset

Note: The red bars indicate results from the test set

As mentioned earlier, this dataset was made in mind of making the classification task very difficult, by having virtually no separation between the classes, but having one variable among 10 that could be a predictor. This was to see if both cross validation approaches could construct the same model. In this case, it seems like the correct approach wins out by a small margin for the Random Forest model based on the test score.

Note that as far as best_params_ go, both approaches result in differences when looking at GBM & RF models.

I also tried increasing the number of informative variables in the make_classification dataset, and doing so does bring the scores closer.

Question

In this case, it seems to bring up the question of which model to proceed with? The one that is overfitted but performs similarly on the test set as a model with a more realistic score?

Both approaches could converge when choosing the best parameters when there's absolutely no information to pickup. However, in the second dataset, where there was an informative variable, there was some deviance, which leads to a different model depending on the approach you take.

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Browse other questions tagged or ask your own question.