I know that there are various posts regarding variable selection but I am asking something particular. With respect to the question that I posted today in the following link:
If you had a look at the above link, you have seen that my problem is low detection rate in out of time data (ie, low true positives) though I had a very good accuracy in out of sample (80.5%). Please comment on the thoughts below that I have for this problem. Since I need to have a model which has reasonably good accuracy with the past as well as future data would the following things be of any use to me?
Trying and selecting those variables which are shock resistant to time variation in data (not really sure whether there are such variables but trying to think intuitively that model is as good as its data and variables)-- what would this variable look like?
I had done profiling of both sample and out of time validation data; should I consider dropping the variables which have high variation or difference in distribution or statistics (in case of continuous variables). Agree, it might decrease my model accuracy from 80 to 70 (may be) but, I guess, it would help me in keeping only those variables which are more shock proof to the seismic waves of time -- please suggest.
All in all, I want suggestions on which variables to keep so as to maintain my initial accuracy.
I dont mind initial accuracy of 65% detection and out of time accuracy of 50% finally but a drop from 80 to 35 is a worry.