1
$\begingroup$

Together with two friends I participate in a university course about data mining in R and we chose the topic of bankruptcy prediction. We started with some "clean" data found on an "In class" kaggle competition and comparing to the leaderboard our classification seemed to perform very good.

Now we've gone on to use the public dataset about Polish companies from UCI, that has some missing values; apart from that, it's basically the same data. We get AUC scores of 0.96-0.98 depending on the chosen seed, so the classification seems to perform pretty good. The score is better than in the paper by Zieba et al. (2016) that gathered and first worked with the data.

We apply k-fold-cross validation and use the Extreme Gradient Boosting method "xgbLinear". Now dealing with this data we discovered a few strange facts about the performance:

For the kaggle data without missing values, SMOTE Oversampling definitely improved the classification performance. Now for the UCI data, running it without Oversampling performs slightly better. Any ideas how that can be?

Next, we experimented with imputation for the missing values. The best scoring method is to put all of them equal to "-99999" (as Zieba et al. have done as far as I can tell). This makes sense as the distribution of missing values is different for the two classes in our data and so has some predictive power.

We thought we might be able to slightly improve on that method by using KNN imputation for the missing values and then add a dummy variable for every predictor that is 1 for a missing value (prior to imputation) and 0 if there was no missing value. But it turns out this method performs worse than the simpler (and more arbitrary?) method of making all of them equal to "-99999". My intuition is that the dummies should capture the information stored within the distribution of missing values, and additionally to that the KNN imputation should definitely provide a more realistic picture of missing values than the -99999 does. But apparently my intuition is wrong here;)

So in short:

Why does Oversampling lead to worse performance here? Is it connected to the occurrence of missing values in our data?

Why does the "-99999" method perform better than KNN imputation + dummies? I'd be incredibly thankful for any ideas and input on this! If helpful I can supply our code.

$\endgroup$
1
$\begingroup$

The fact that you are having to consider balance in the Y distribution means that you don't understand the need to predict tendencies rather than make arbitrary classifications (premature decisions). This is discussed in detail here. If you have to sample from your data there is a fundamental logic flaw in the approach.

$\endgroup$
  • 1
    $\begingroup$ Thank you very much for your feedback, we will look into this in detail the next days. From my current understanding I can just say this: I believe I understand the point about predicting tendencies instead of a binary class, because predicting whether an active company will default within the next year is essentially a probabilistic question. So classifying based on some probability threshold obviously involves a utility trade-off, which needs to be made by the respective decision maker. I will add a comment as soon as I have figured out whether/how this changes our approach. $\endgroup$ – Ferdinand Berr Aug 8 at 7:54
  • $\begingroup$ The "xgbLinear" algorithm we used so far predicts probabilities for new data points to show Y=1 (companies that will go bankrupt). In a next step we performed the classification based on some threshold. I've read your blog posts and completely get the point about classification being context-/preference-dependent, thank you for that, we will emphasize that. My question now is: Are the probability outputs of "xgbLinear" ready to interpret as such, or do they need any transformation/calibration? And what scoring rule is proper for our task, is AUC correct? $\endgroup$ – Ferdinand Berr Aug 13 at 8:38
  • $\begingroup$ Others will have to answer the xgbLinear output question. As for scoring rules, take a look at the Brier score, pseudo $R^2$ (a version of the logarithmic proper scoring rule). Use the c-index (concordance probability; AUROC) as a descriptive additional measure. $\endgroup$ – Frank Harrell Aug 13 at 11:11
  • $\begingroup$ My plan is now to compare the results for different models using (1) AUROC and (2) Brier Score as performance metrics; being especially interested in whether/how the ranking of models changes depending on the metric. But as far as I know, Brier Score (or log loss) will give misleading results when working with imbalanced data. That means I'd need to implement Brier Skill Score for example, correct? As I'm having trouble implementing this metric in "caret", I'm probably starting to use simple Brier Score (or log loss) after balancing the data with SMOTE, which is not my favorite solution.. $\endgroup$ – Ferdinand Berr Aug 20 at 12:32
  • $\begingroup$ No, the Brier score works fine with imbalanced data. Also look at the measures in fharrell.com/post/addvalue $\endgroup$ – Frank Harrell Aug 20 at 20:15
0
$\begingroup$

SMOTE stands for "Synthetic Minority Over-sampling Technique". As you can deduct from the word synthetic - it creates additional minority instances. (which are similar to the original minority instances). That way it reaches class balance.

This creating of new minority instances follows some algorithm (just google for SMOTE). If this new instances do not resemble the actual ground truth your performance will be worse. Meaning you tried to create new instances for "bankruptcy" - but these synthetically created instances are not representative for "bankruptcy". These new - but wrong "bankruptcy" instances mislead your algorithm.

Try another over- or undersampling technique. (maybe one that does not create synthetic instances). This does not mean synthetic instances are bad - it just did not work in this case.

Might also be affected by the missing data - but I would rather think the two datasets are just different. Just because both are about bankruptcy does not mean they behave the same or the same set of algorithms will be best.

About the KNN imputation: This can also mislead your algorithm. Especially KNN imputation is prone to this. KNN - as the name says just looks for the nearest neighbors - the most similar instances. So what might happen e.g for knn (k=1) is the following: variable x is missing for instance 3, knn finds instance 5 which is exactly the same but with variable x. Then the x value of instance 3 gets imputed with the x value of variable 5. The imputation just repeats what is already there in the dataset. Instance 5 is now basically overrepresented in the dataset. Which might lead to a worse performance.

You could try other imputation methods. Generally the binary missingness indicator dummy variable is a good idea. The problem probably rather lies with the imputation method you've chosen. Sometimes also no imputation is the best solution.

$\endgroup$
  • $\begingroup$ Both datasets are actually based on identical data, but they are different "subsamples" from this Polish Companies Bankruptcy Data; one is very neat and clean, the other contains observations with missing values. But I can see how there might be systematic differences, if the clean data has been constructed by only sampling observations with no missing values for example. We will look into the imputation problem in more detail, thank you! But what do you mean by no imputation? Just omitting those observations would lead to significant bias I suppose. $\endgroup$ – Ferdinand Berr Aug 8 at 8:00
-1
$\begingroup$

There are multiple things to consider.

  1. SMOTE operates in the Euclidean space implied by the data, thus, the range of the attributes matters when SMOTE determines which are the closest neighbors of a sample. Extreme values for missing values will definitely ruin this. I would recommend the standardization/normalization of all attributes, and the use of means/medians/modes for data imputation.
  2. Classification after SMOTE is highly sensitive to the number of samples being generated. Balancing the dataset might work, but in many cases not. Consider two classes sampled from two manifolds, the manifold of the minority class having half the size of the manifold of the majority. In this case, balancing the dataset to have the same number of samples will result a situation where the density of minority samples is two times the density of the majority ones near the decision boundary, which clearly deteriorates the results, but in the reverse way. Thus, it is highly important to carry out model selection for the number of samples to generate using SMOTE.
  3. You could test various oversampling techniques to find the best one for the data. Implementations can be found in packages like imblearn or smote-variants (the latter is my own).
$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.