2
$\begingroup$

I have a highly imbalanced dataset of about 8000 observations, with 11 features and one binary target variable. I want to predict the target labels, considering that the "1" target label occurs for 1.5% of the observations in my data.

Given that this classification problem is very unbalanced and that my features are all categorical, I use the balanced random forest method provided by h2o (which directly support categorical variables), with a 6-fold cross-validation. Moreover, I perform a cartesian grid-search to find a couple of hyperparameters.

Before training the classifier, I split the dataset into train, validation (for grid-search) and test set using a 70%-15%-15% stratified split.

I am very confused by the results of my analysis. I can reproduce the same results by running my code several times with the same pseudo-random seed. However, my results vary a lot when I change the seed. While I can expect some variation depending on the seed, I'm puzzled by how much they vary with it. Below I report some examples of confusion matrices I find (computed using the test set) using exactly the same code but seed. The seed comes into play for the random forest classifier and the train-validation-test split:

seed = 7

|---------------------|------------------|
|      TP = 17        |     FN = 118     |
|---------------------|------------------|
|      FP = 588       |     TN = 7813    |
|---------------------|------------------|

seed = 692

|---------------------|------------------|
|      TP = 23        |     FN = 112     |
|---------------------|------------------|
|      FP = 1042      |     TN = 7359    |
|---------------------|------------------|

seed = 1864

|---------------------|------------------|
|      TP = 1         |     FN = 134     |
|---------------------|------------------|
|      FP = 42        |     TN = 8359    |
|---------------------|------------------|

As you can see in all cases the performance is poor (this is a very complex problem, but that's not the point) and the True Positives, False Negatives, False Positives and True Negatives vary a lot depending on the seed.

Honestly I have only one explanation for that: given different random seeds, when splitting the data into train, test and validation sets I'm selecting different subpopulations/subsamples. These subsamples have different relationships between features and target variable, hence when selecting one subsample over the other the classifier tries to adjust itself to that particular relationship features-target that holds on that specific subsample.

However, I'm not fully convinced that this is what is happening here and I'd appreciate any feedback/idea on this problem.

Improve this question
$\endgroup$
3
  • 1
    $\begingroup$ Your proposed explanation looks very plausible to me especially with such an unbalanced data-set. $\endgroup$
    – mdewey
    Sep 18, 2018 at 15:55
  • $\begingroup$ You are possibly overfitting then? $\endgroup$
    – Tom
    Sep 18, 2018 at 16:04
  • $\begingroup$ @Tom indeed I'm clearly overfitting, the point is why the random seed changes so much the "way/strength" we overfit the data. $\endgroup$
    – black_cat
    Sep 19, 2018 at 16:58

1 Answer 1

Reset to default
1
$\begingroup$

The seed will change which data gets placed in the train, test, and validation groups. If you only have 11 cases in one class and 7989 in the other, then you can imagine that each seed differs in the amount the small class is distributed to the three train, validation, and testing groups. Decision trees, like random forest, use the traits in a class to split branches. If you only have a class size of 2 vs 8, the split will vary very widely.

Improve this answer
$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge that you have read and understand our privacy policy and code of conduct.

Not the answer you're looking for? Browse other questions tagged or ask your own question.