0
$\begingroup$

I am trying to build a credit scoring model and have discovered and interesting approach for feature selection. I am looping through all features and removing them one by one (using variable importance criterion) from model until test set has approximately the same AUC as the train set.

What supprised me was when I selected a model that satisfied AUC criteria, most of the variables had bad significance:

Coefficients:
                Estimate   Std. Error   z value               Pr(>|z|)    
(Intercept) -0.993380858  0.051309749 -19.36047 < 0.000000000000000222 ***
X17gr_woe   -0.007743923  0.002244585  -3.45005             0.00056049 ***
X43gr_woe   -0.009534972  0.001512110  -6.30574       0.00000000028682 ***
X45gr_woe   -0.008539861  0.005766576  -1.48092             0.13862683    
X47gr_woe   -0.006138579  0.004134777  -1.48462             0.13764412    
X48gr_woe    0.012399043  0.006592004   1.88092             0.05998259 .  
X53gr_woe   -0.011698312  0.017367607  -0.67357             0.50058427    
X54gr_woe    0.010207240  0.018934665   0.53908             0.58983381    
X55gr_woe   -0.006290728  0.003805058  -1.65325             0.09827915 . 

The ROC curve on train and test sets are approximately the same: enter image description here

I am inclined to use this model because it does not overfit and performs the same on test set. My questions is - what are your thoughts, would you use this model if the goal is to use it in real-time?

EDIT: Here is the variable removing process. I choose the model for which train AUC and test AUC were one the same level enter image description here

$\endgroup$
3
  • 3
    $\begingroup$ If I understand what you did correctly, it seems like you took multiple looks at the test set, removing variables each time to get better AUC. This procedure defeats the purpose of a hold-out test set which you aren't supposed to use to train your model. If you use this model on a completely fresh test set, it won't perform well. $\endgroup$
    – Heisenberg
    Jun 7, 2016 at 14:18
  • 2
    $\begingroup$ It seems like you used your test set to train your model, which isn't kosher. $\endgroup$
    – Heisenberg
    Jun 7, 2016 at 14:22
  • $\begingroup$ I edited original post and added variable removal process $\endgroup$
    – MiksL
    Jun 7, 2016 at 14:24

1 Answer 1

2
$\begingroup$

As @Heisenberg pointed out in the comments, before doing any model training/evaluation/selection, you should create a hold-out test set, which you in the end use once to test your single, final, ready-trained model.

You could do this e.g. as follows:

  1. Create a training and hold-out test partition.
  2. Using the training partition, do repeated cross validation for training and evaluating different model types, model parametrizations, and combinations of features$^1$:
    1. This gives you multiple performance measures for each model type, model parametrization, and combination of features, from which you can derive how well exactly this configuration is suited for your task, and what its performance spread might be.
    2. Chose a model type, parametrization, and combination of features from those results, then train this model using the complete training partition.
  3. Test the obtained model on the test partition once to assure its performance is correct.

$^1$ The deletion of features you explained seems to be close to backwards feature selection, e.g. recursive feature elimination: this starts with all features and removes features until a peak is reached in the average repeated cross validation performance. This means it re-evaluates the model each time. The big difference to what you mentioned is that this does not involve the test data at all: if you overfit, you do it just on the training data and will at last notice on the test data.

As you are using R: the caret package has all this ready-made in their ?rfe implementation, so its easy to use, just takes a while to evaluate all the options.

PS: in your example, it could theoretically be that you initially overfitted using all features, then by removing some of them decreased variance at the cost of (some little) bias. But much more likely you just overfitted your test data as well, as you used it like training data in an optimization process.

UPDATE

If you did not use any test data in training and only used test data to ensure that your final model actually has a reasonable performance, using such features, which are labeled "insignificant" by your feature importance metrics, is legit. This could e.g. be the case with some "weak" features, which only in their combination will solve a problem well - for example, a binary classification over two features $A$ and $B$, that are organized as as $AND$ or $OR$:

    B 0 1      B 0 1
  A +----    A +----
  0 | 0 0    0 | 0 1
  1 | 0 1    1 | 1 1

Each of those features might end up being labeled insignificant, but their combination enables the problem to be solved - so using them in your model is reasonable.

$\endgroup$
2
  • $\begingroup$ Thanks @geekoverdose. Let's assume I got the same results but with the approach you described (not fitting to hold out sample). Would you still select such model if some of the parameters for logit would have been insignificant? $\endgroup$
    – MiksL
    Jun 10, 2016 at 9:17
  • $\begingroup$ @MiksL yes, if those boost your performance, using them is reasonable (and you did not use test data during training, your CV performance is OK, and your test performance confirms that performance ;)). I've updated my answer accordingly. $\endgroup$ Jun 10, 2016 at 11:57

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.