0
$\begingroup$

I'm working on a credit scoring model (logistic regression), and I have divided my dataset (5082 obs with 580 negatives) in two samples: 75% training set and 25% test set. The result of the estimation is disappointing, the quality evaluated by the roc curve is 0.63 (and gini coefficient=0.26). I thought, the reason maybe that I have not enough data, and so not enough negatives after I split the data. Is there a method to use the whole data set for estimation? And do the validation on the same dataset? How might this work? I'm new to this methods so my questions is not detailed. Thank you for an suggestion/explication

$\endgroup$
  • $\begingroup$ Could you tell more about your covariates? Their quality could be easily the most important factor. You have in my opinion enought observations to split data into test/dev branches. $\endgroup$ – Analyst Jun 3 '15 at 9:26
  • $\begingroup$ I think you are right, the predictive power of my variables is indeed very poor (information value around 0.03). I had to bin the variables, there was no other way. Is there a way to improve them? $\endgroup$ – Charlotte Jun 3 '15 at 9:35
  • $\begingroup$ It might be that you must gather new variables which might have more predictive power. $\endgroup$ – Analyst Jun 3 '15 at 12:38
1
$\begingroup$

I dont think that its the problem from the data size, do you think that your model is perfectly build? there are many ways to increase the quality of fitting. For more detail, you should read the book of

Hosmer, Lemeshow - Applied Logistic Regression

$\endgroup$
1
$\begingroup$

It's almost certainly not the sample size. There are various things it might be

  1. The variables. In a comment, you asked about ways to improve them. The biggest thing is probably to get more variables that are more related to risk. But, if you are stuck with the ones you have, you can explore non-linear relationships e.g. with spline models.

    1. The model. You say you are using "logistic regression" so I am guessing it's binary logistic with values something like "paid" and "defaulted" but there are surely more levels than that. You might need ordinal or multinomial logistic with levels such as "paid on time", "paid late", and so on (I don't know the credit industry but you can surely have more levels)

    2. Another possibility is that the model should include the amount of credit in the DV. This might call for a hurdle model or something similar

$\endgroup$
1
$\begingroup$

As has been written about extensively on this site, your dataset is far too small to trust split-sample validation. I recommend using the bootstrap to estimate the likely future performance of a whole-sample fit. The bootstrap allows you to use the whole sample for both model development and model evaluation, if you are honest about things like variable selection so that the bootstrap knows to repeat all modeling steps for each of, say, 400 re-samples. Details are at http://biostat.mc.vanderbilt.edu/rms.

$\endgroup$
  • 1
    $\begingroup$ Do have a source or more details how this might be implemented practically (with R or other software)? When using the bootstrap, which parameters/statistics do I need to calculate for each step? The AUC? $\endgroup$ – Charlotte Jun 3 '15 at 15:01
  • 1
    $\begingroup$ The above web site has my handouts that go into great detail about this. This goes along with the 2nd edition of my book Regression Modeling Strategies to be published in July 2015. $\endgroup$ – Frank Harrell Jun 3 '15 at 17:34

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.