I have a dataset consisting of 23 features for a number of clients :

  • Client ID
  • yearly financial ratios
  • couple of qualitative features
  • and a binary default variable

I'm trying to create a model that predicts whether the client will default or not , i tried , for starters , with svm and xgboost but the results were pretty bad (high accuracy and very bad recall - F1) due to the imbalance of the data (~minority about 6%).
A problem that i tried to fix by trying SMOTE,undersampling and trying to adjust the built-in tools for balancing data like class_weight parameter for SVC and scale_pos_weight for xgboost respectively but the results are still bad .( whatever i do if the recall of one of the two classes rises the other one decreases ).

Also the Correlations between the predictors and the target are very low ( 0.18 is the highest ) ,
i want to also know, at which point can i tell for sure that the data at hand is not explanatory enough to predict a certain variable?

  • $\begingroup$ Just for clarification purposes, is your question related to these particular modelling choices (XGBoost, SVC) and their performance on this dataset? Or are you just trying to build a good predictive model on the data you have? In other words, does the solution to your problem have to include XGBoost/SVC or will any method with good enough results suffice? $\endgroup$ – Emil Jul 11 '19 at 9:09
  • $\begingroup$ No , i'm not restricted to these two methods. I'm planning to try other methods and compare between all of them and eventually choose the best one so any suggestions are welcome. $\endgroup$ – Whatever Jul 11 '19 at 9:29

If you are looking to just build a "good" (aka good enough) model and are not restricted to one type of model or another, I would recommend the following sequence of actions.

  1. Perform a single factor analysis first, where you calculate the Accuracy Ratio between your default indicator and each of your independent variables (since the outcome variable is binary -- default or no default -- the AR is equivalent in this case to Somers' D which can be easily calculated with any statistical software).
  2. Rank the results from best value of AR to worst, while also calculating the correlations between your predictors. This should address multicollinearity, as it will give you an idea of which factors are good candidates for the final model, while also telling you which ones to exclude due to correlation with other, well-performing factors, or because their correlation sign is counter-intuitive (this is particularly important in credit risk because it makes no sense to include a variable that has a negative correlation sign with default, like e.g. debt, where you would expect the opposite).
  3. Hopefully now you have decreased your predictors from 23 to e.g. 5 or 6, and you can then perform a simple logistic regression to estimate probability of default for future customers. To measure how well this method performs, you could split your data to development/validation samples, where you take for example 80% of the data and use it to train the model, then test the results on the remaining 20%. In general, a good model is one with discriminatory power $\geq 70\%$.
  4. It should be noted here that the data characteristics themselves could also be at play if you still are not getting good results. As a rule of thumb, the method described above requires at least 50 defaults present in the portfolio, otherwise you are dealing with a so-called "low-default portfolio", and those are generally more difficult to model in this way. Finally, it could also be that you simply have bad predictors as features, in which case not much can be done from a modelling point of view.

You might be disappointed at my suggestion for a logistic regression instead of something fancier, but when it comes to this type of classification, it's a surprisingly well-performing method, especially if you look at the banking industry: fancier methods like neural networks etc. rely on both quantity and quality of data, two areas in which banks traditionally suffer when it comes to model building. Hope this helps.


Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.