5
$\begingroup$

Recently, Hannes Leeb from Yale University and Benedikt Pötscher from the University of Vienna have published a series of papers dealing with what they call Post Model Selection Inference problems.* Let me be clear: I'm not a professional researcher myself, and my understanding of this might be not as good as I hope. But if I understand what I read correctly, their discovery is that selecting variables based on model selection criteria (e.g., Akaike (AIC), Bayesian/Schwartz (BIC), Final prediction error (FPL), ... ) and then treating the selected model with standard inference methods usually yield spurious results. The reason is that the probability distributions of the estimators are changed dramatically if the model selection step is being used. This is particularly noticeable if a lot of variables/features are selected. But even if one only has two variables, extreme distortions can be encountered. I actually ran a Monte Carlo myself to check this for the case of a regression with two regressors. The confidence intervals are badly undersized (80% coverage probability when they should have 95%), and the estimates severely biased.

I think this is a major issue - particularly because model selection is indispensable for big data/machine learning applications. I was wondering which solutions there are to circumvent the problem for practicioners, but it has not been easy to find any myself, so I thought I would give it a shot and ask here. Any kind of resources/links elaborating on the subject further are also appreciated!

*e.g., "Model Selection and Inference: Facts and Fiction", Econometric Theory, 21, 2005, p.21-59

Improve this question
$\endgroup$
4
  • 4
    $\begingroup$ This is most certainly true and known for a long time (see e.g. Breimann, who called it "a quiet scandal" in 1992 and various papers in the 80s and likely even before). Things like model averaging and random forests are some of the better ways to deal with this. $\endgroup$
    – Björn
    Commented Jan 20, 2016 at 19:51
  • $\begingroup$ How would you use random forests for inference? For prediction, surely - but what if you are interested in specific parameters? This is a recurrent issue in econometrics, where parameters have theoretical meaning, and making a good prediction is sometimes of secondary interest. Is there a way to do that with random forests? $\endgroup$
    – Jeremias K
    Commented Jan 20, 2016 at 19:54
  • 1
    $\begingroup$ As far as I know there is no magic bullet that would solve the problem. On the other hand, Leeb and Pötscher's results are sometimes argued to relate to a worst case scenario (given their focus on uniformity), so just how serious the issue is in any given application is not easy to say. See econpapers.repec.org/article/sprempeco/… for some simulation results. $\endgroup$
    – Christoph Hanck
    Commented Jan 21, 2016 at 7:05
  • $\begingroup$ @JeremiasK, just discovered your question, a while after having posted a related one here. Interesting. $\endgroup$
    – Richard Hardy
    Commented Mar 15, 2017 at 18:38

1 Answer 1

Reset to default
1
$\begingroup$

Just wanted to point you to the slides of the talk given by Robert Tibshirani at the NIPS Conference last year, which shows recent research done by his team in this subject.

Hope this helps.

Improve this answer
$\endgroup$
2
  • $\begingroup$ Thanks, coming from a statistics background and drifting into machine learning, this is really useful! $\endgroup$
    – Jeremias K
    Commented Jan 21, 2016 at 18:34
  • $\begingroup$ No problem! While I'm more into prediction, I do work for people (economists mostly) who are concerned about parameter inference. So this issue makes it hard to convince them to adopt all these new methods that truly deliver in terms of predictive performance, but where we don't have yet the proper tools for inference. In fact, I believe that this situation is happening in many research groups, so its likely that we'll see more interesting results in the near future. $\endgroup$
    – mbiron
    Commented Jan 22, 2016 at 12:02

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.