# classification on imbalanced dataset via random forest: results vary with random seed

I have a highly imbalanced dataset of about 8000 observations, with 11 features and one binary target variable. I want to predict the target labels, considering that the "1" target label occurs for 1.5% of the observations in my data.

Given that this classification problem is very unbalanced and that my features are all categorical, I use the balanced random forest method provided by h2o (which directly support categorical variables), with a 6-fold cross-validation. Moreover, I perform a cartesian grid-search to find a couple of hyperparameters.

Before training the classifier, I split the dataset into train, validation (for grid-search) and test set using a 70%-15%-15% stratified split.

I am very confused by the results of my analysis. I can reproduce the same results by running my code several times with the same pseudo-random seed. However, my results vary a lot when I change the seed. While I can expect some variation depending on the seed, I'm puzzled by how much they vary with it. Below I report some examples of confusion matrices I find (computed using the test set) using exactly the same code but seed. The seed comes into play for the random forest classifier and the train-validation-test split:

seed = 7

|---------------------|------------------|
|      TP = 17        |     FN = 118     |
|---------------------|------------------|
|      FP = 588       |     TN = 7813    |
|---------------------|------------------|


seed = 692

|---------------------|------------------|
|      TP = 23        |     FN = 112     |
|---------------------|------------------|
|      FP = 1042      |     TN = 7359    |
|---------------------|------------------|


seed = 1864

|---------------------|------------------|
|      TP = 1         |     FN = 134     |
|---------------------|------------------|
|      FP = 42        |     TN = 8359    |
|---------------------|------------------|


As you can see in all cases the performance is poor (this is a very complex problem, but that's not the point) and the True Positives, False Negatives, False Positives and True Negatives vary a lot depending on the seed.

Honestly I have only one explanation for that: given different random seeds, when splitting the data into train, test and validation sets I'm selecting different subpopulations/subsamples. These subsamples have different relationships between features and target variable, hence when selecting one subsample over the other the classifier tries to adjust itself to that particular relationship features-target that holds on that specific subsample.

However, I'm not fully convinced that this is what is happening here and I'd appreciate any feedback/idea on this problem.

• Your proposed explanation looks very plausible to me especially with such an unbalanced data-set. Sep 18, 2018 at 15:55
• You are possibly overfitting then?
– Tom
Sep 18, 2018 at 16:04
• @Tom indeed I'm clearly overfitting, the point is why the random seed changes so much the "way/strength" we overfit the data. Sep 19, 2018 at 16:58