Binary Classification in Imbalanced Data; Oversampling and Imputation Together with two friends I participate in a university course about data mining in R and we chose the topic of bankruptcy prediction. We started with some "clean" data found on an "In class" kaggle competition and comparing to the leaderboard our classification seemed to perform very good.
Now we've gone on to use the public dataset about Polish companies from UCI, that has some missing values; apart from that, it's basically the same data. We get AUC scores of 0.96-0.98 depending on the chosen seed, so the classification seems to perform pretty good. The score is better than in the paper by Zieba et al. (2016) that gathered and first worked with the data.
We apply k-fold-cross validation and use the Extreme Gradient Boosting method "xgbLinear". Now dealing with this data we discovered a few strange facts about the performance:
For the kaggle data without missing values, SMOTE Oversampling definitely improved the classification performance. Now for the UCI data, running it without Oversampling performs slightly better. Any ideas how that can be?
Next, we experimented with imputation for the missing values. The best scoring method is to put all of them equal to "-99999" (as Zieba et al. have done as far as I can tell). This makes sense as the distribution of missing values is different for the two classes in our data and so has some predictive power.
We thought we might be able to slightly improve on that method by using KNN imputation for the missing values and then add a dummy variable for every predictor that is 1 for a missing value (prior to imputation) and 0 if there was no missing value. But it turns out this method performs worse than the simpler (and more arbitrary?) method of making all of them equal to "-99999". My intuition is that the dummies should capture the information stored within the distribution of missing values, and additionally to that the KNN imputation should definitely provide a more realistic picture of missing values than the -99999 does. But apparently my intuition is wrong here;)
So in short:
Why does Oversampling lead to worse performance here? Is it connected to the occurrence of missing values in our data?
Why does the "-99999" method perform better than KNN imputation + dummies?
I'd be incredibly thankful for any ideas and input on this! If helpful I can supply our code.
 A: The fact that you are having to consider balance in the Y distribution means that you don't understand the need to predict tendencies rather than make arbitrary classifications (premature decisions).  This is discussed in detail here.  If you have to sample from your data there is a fundamental logic flaw in the approach.
A: SMOTE stands for "Synthetic Minority Over-sampling Technique". As you can deduct from the word synthetic - it creates additional minority instances. (which are similar to the original minority instances). That way it reaches class balance.
This creating of new minority instances follows some algorithm (just google for SMOTE). If this new instances do not resemble the actual ground truth your performance will be worse. Meaning you tried to create new instances for "bankruptcy" - but these synthetically created instances are not representative for "bankruptcy". These new - but wrong "bankruptcy" instances mislead your algorithm.
Try another over- or undersampling technique. (maybe one that does not create synthetic instances). This does not mean synthetic instances are bad - it just did not work in this case. 
Might also be affected by the missing data - but I would rather think the two datasets are just different. Just because both are about bankruptcy does not mean they behave the same or the same set of algorithms will be best.
About the KNN imputation:
This can also mislead your algorithm. Especially KNN imputation is prone to this. KNN - as the name says just looks for the nearest neighbors  - the most similar instances. So what might happen e.g for knn (k=1) is the following: variable x is missing for instance 3, knn finds instance 5 which is exactly the same but with variable x. Then the x value of instance 3 gets imputed with the x value of variable 5. The imputation just repeats what is already there in the dataset.
Instance 5 is now basically overrepresented in the dataset. Which might lead to a worse performance.  
You could try other imputation methods. Generally the binary missingness indicator dummy variable is a good idea. The problem probably rather lies with the imputation method you've chosen. Sometimes also no imputation is the best solution.
A: There are multiple things to consider.


*

*SMOTE operates in the Euclidean space implied by the data, thus, the range of the attributes matters when SMOTE determines which are the closest neighbors of a sample. Extreme values for missing values will definitely ruin this. I would recommend the standardization/normalization of all attributes, and the use of means/medians/modes for data imputation.

*Classification after SMOTE is highly sensitive to the number of samples being generated. Balancing the dataset might work, but in many cases not. Consider two classes sampled from two manifolds, the manifold of the minority class having half the size of the manifold of the majority. In this case, balancing the dataset to have the same number of samples will result a situation where the density of minority samples is two times the density of the majority ones near the decision boundary, which clearly deteriorates the results, but in the reverse way. Thus, it is highly important to carry out model selection for the number of samples to generate using SMOTE.

*You could test various oversampling techniques to find the best one for the data. Implementations can be found in packages like imblearn or smote-variants (the latter is my own).

