0
$\begingroup$

I have a dataset that contains a credit card flag for members(1/0) and certain set of characteristics. I have a final set of 60 variables which I applied in logistic regression and obtained an roc of 0.99. I thought this might be happening as any of the variable might be strongly related to dependent variable, so I started by introducing one by one variable and observed the ROC and it was increasing like 0.51,0.55,0.58,0.61......0.99. About in 8 variables I reached 0.92. I select some random 10 variables and same thing is happening for them as well. This gives the impression that its not the variable information, but just addition that is giving such ROC.

Pop Size: 450,000(98% negative cases and 2% positive cases) Hosmer and Lemeshow goodness of fit test is significant.

Any idea, why I am observing such a result? Please let me know, if you need any additional information.

Improve this question
$\endgroup$
1
  • $\begingroup$ Its the credit card use vs non-use data. Also I think its worth highlighting that I am talking about these statistics on the training dataset, on which model is built. Is it giving so much high ROC, because its own values? $\endgroup$
    – user83685
    Aug 5, 2015 at 11:08

2 Answers 2

Reset to default
4
$\begingroup$

I agree with Cagdas - it seems like you may be looking at the training error, in which case you would just be overfitting to your data as you add more features.

An Introduction to Statistical Learning provides a good overview of different validation techniques. Chapter 2 explains the difference between training and test error, and chapter 5 provides an overview of different validation approaches.

I have found that repeated k-fold cross validation works well.

Given that you are working with a largely imbalanced dataset, it would be surprising (but nice!) if you could achieve AUCs that high. There's been a lot of discussion on working with imbalanced datasets, but in practice I have found Kaggle competitions to be a good source of ideas. You may want to check out the Liberty Mutual Fire Loss discussion boards, as that competition dealt with a highly imbalanced dataset. My read was that an ensemble of undersampled models worked well.

Improve this answer
$\endgroup$
4
  • $\begingroup$ Thanks for your input. I read somewhere on the same website, this much data should be sufficient for modeling: 10000(events)/450000(total)=~2%. Or should I still treat it as unbalanced data? My objective here is not exactly the prediction, but identifying the prospective members $\endgroup$
    – user83685
    Aug 5, 2015 at 12:48
  • $\begingroup$ It will be sufficient to get predictions, but often in practice it can be difficult to get useful results with highly imbalanced data, since the model could perform really well (in terms of accuracy) by just predicting 0 all of the time. You may be fine just building a model with the data as-is, but I wouldn't be surprised if you could get better results by using an approach that's tailored towards imbalanced data (e.g., cost-sensitive training, oversampling positives, undersampling negatives) $\endgroup$
    – Tchotchke
    Aug 5, 2015 at 12:56
  • $\begingroup$ I would be surprised if those methods helped him get better test accuracy because methods like over sampling in my experience only help sometimes and it is hard to know when. $\endgroup$
    – Brash Equilibrium
    Aug 5, 2015 at 14:23
  • $\begingroup$ True, just like any other modeling technique there's "no free lunch." That being said, it is easy to find many papers that have found those approaches to be useful. In my work, depending on the dataset, I've found ~1-5% improvement from an ensemble of models built on undersampled negatives. $\endgroup$
    – Tchotchke
    Aug 5, 2015 at 16:33
2
$\begingroup$

The concordance probability ($c$-index; ROC area) is a good measure of pure discrimination useful for describing a single model. It is not sensitive enough for comparing models. The log-likelihood is the gold standard.

Note that the Hosmer-Lemeshow test is virtually obsolete.

Stepwise variable selection (as opposed to full pre-specification of models) has a host of problems as discussed at length on this site.

Improve this answer
$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.