Recently, Hannes Leeb from Yale University and Benedikt Pötscher from the University of Vienna have published a series of papers dealing with what they call Post Model Selection Inference problems.* Let me be clear: I'm not a professional researcher myself, and my understanding of this might be not as good as I hope. But if I understand what I read correctly, their discovery is that selecting variables based on model selection criteria (e.g., Akaike (AIC), Bayesian/Schwartz (BIC), Final prediction error (FPL), ... ) and then treating the selected model with standard inference methods usually yield spurious results. The reason is that the probability distributions of the estimators are changed dramatically if the model selection step is being used. This is particularly noticeable if a lot of variables/features are selected. But even if one only has two variables, extreme distortions can be encountered. I actually ran a Monte Carlo myself to check this for the case of a regression with two regressors. The confidence intervals are badly undersized (80% coverage probability when they should have 95%), and the estimates severely biased.
I think this is a major issue - particularly because model selection is indispensable for big data/machine learning applications. I was wondering which solutions there are to circumvent the problem for practicioners, but it has not been easy to find any myself, so I thought I would give it a shot and ask here. Any kind of resources/links elaborating on the subject further are also appreciated!
*e.g., "Model Selection and Inference: Facts and Fiction", Econometric Theory, 21, 2005, p.21-59