I know that there are various posts regarding variable selection but I am asking something particular. With respect to the question that I posted today in the following link:

Low accuracy in out of time validation

If you had a look at the above link, you have seen that my problem is low detection rate in out of time data (ie, low true positives) though I had a very good accuracy in out of sample (80.5%). Please comment on the thoughts below that I have for this problem. Since I need to have a model which has reasonably good accuracy with the past as well as future data would the following things be of any use to me?

  1. Trying and selecting those variables which are shock resistant to time variation in data (not really sure whether there are such variables but trying to think intuitively that model is as good as its data and variables)-- what would this variable look like?

  2. I had done profiling of both sample and out of time validation data; should I consider dropping the variables which have high variation or difference in distribution or statistics (in case of continuous variables). Agree, it might decrease my model accuracy from 80 to 70 (may be) but, I guess, it would help me in keeping only those variables which are more shock proof to the seismic waves of time -- please suggest.

All in all, I want suggestions on which variables to keep so as to maintain my initial accuracy.

I dont mind initial accuracy of 65% detection and out of time accuracy of 50% finally but a drop from 80 to 35 is a worry.

  • $\begingroup$ Is it safe to assume that the data are being generated by the same process over time? $\endgroup$ – Zach Jun 1 '11 at 16:09
  • $\begingroup$ @zach, yes thats safe to assume. $\endgroup$ – ayush biyani Jun 1 '11 at 16:22

Whatever variable selection technique you use, be sure to cross-validate it to keep from overfitting. It seems that you may have overfit your initial model, so this step is very important.

One idea would be to use the elastic net or lasso for regularization and variable selection. You can use the package glmnet in R to run a logistic regression using the elastic net for regularization and variable selection. It's pretty straightforward and could improve your results.

| cite | improve this answer | |
  • $\begingroup$ Thanks. I have been getting this verdict that this model could be overfitted. If that matters I have just 25 variables in my dataset which has 4.5 lakh rows. Forgive my ignorance but is it necessary that this model is overfitted ? My idea towards variable reduction was not for reducing overfitness but to only contain variables which are time shock proof though I agree that there are no such variables called time shock proof. $\endgroup$ – ayush biyani Jun 1 '11 at 16:24
  • 2
    $\begingroup$ @ayush biyani: The behavior you are experiencing: good accuracy on one dataset, with poor accuracy on another, is usually taken as an indication of overfitting. However, it could also be the case that some sort of "structural shift" has happened between the 2 time periods, and you need a different model for the old data. Your concept of "shock proof" is probably analogous to "overfitting:" in both cases removing certain variables allows you to generalize further from your model. $\endgroup$ – Zach Jun 1 '11 at 17:28
  • $\begingroup$ Overfitting can occur any time the model is selected in a way that did not mask the analyst to Y. But the data splitting approach, when n < 15000, is prone to imprecision problems. You can even find better prediction in the 2nd half than you did in the model development sample, which is statistically. Bootstrapping or repeated cross-validation is warranted for non-huge datasets. Use of improper scoring rules (e.g., % classified correct) will result in the selection of "wrong" models, not to mention huge loss of power. $\endgroup$ – Frank Harrell Jun 2 '11 at 12:50
  • $\begingroup$ Sorry - left out a word. That should be statistically nonsensical. If doing cross-validation, one recommended approach is 100 repeats of 10-fold cross-validation (the bootstrap would be faster however; you might need 300 bootstrap reps). When the sample size is huge, the number of repetitions of cross-validation or the number of bootstrap resamples can be much lower. $\endgroup$ – Frank Harrell Jun 2 '11 at 13:10

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.