21
$\begingroup$

Suppose I'm trying to estimate a large number of parameters from some high-dimensional data, using some kind of regularized estimates. The regularizer introduces some bias into the estimates, but it can still be a good trade-off because the reduction in variance should more than make up for it.

The problem comes when I want to estimate confidence intervals (e.g. using Laplace approximation or bootstrapping). Specifically, the bias in my estimates leads to bad coverage in my confidence intervals, which makes it hard to determine the frequentist properties of my estimator.

I've found some papers discussing this problem (e.g. "Asymptotic confidence intervals in ridge regression based on the Edgeworth expansion"), but the math is mostly above my head. In the linked paper, Equations 92-93 seem to provide a correction factor for estimates that were regularized by ridge regression, but I was wondering if there were good procedures that would work with a range of different regularizers.

Even a first-order correction would be extremely helpful.

| cite | improve this question | |
$\endgroup$
  • 4
    $\begingroup$ +1 timely and important question - though I'm not sure anyone can at present answer this in the affirmative (I guess we simply don't know how to do that properly and if I knew, I'd have a couple of Annals of Statistics papers lined up). Related question: stats.stackexchange.com/questions/91462/… We do know that bootstrapping performs purely in such situations but that won't help. $\endgroup$ – Momo Jun 22 '15 at 16:40
  • $\begingroup$ Thanks for the link. Could you clarify what you meant regarding bootstrapping? $\endgroup$ – David J. Harris Jun 22 '15 at 16:55
  • $\begingroup$ Also, I'm still holding out hope that someone could have methods that work well for non-sparse regularizers. I'd imagine that the L1 penalty makes things especially difficult because of all the estimates piled up at zero. Thanks again. $\endgroup$ – David J. Harris Jun 22 '15 at 17:02
  • 1
    $\begingroup$ Dave, would the so-called selection intervals of Tibshirani and coauthors be suitable? They've worked them out for at least the Lasso, LARS, & step-wise regression via something called a polyhedral form. From that you can essentially form confidence intervals in the usual way but using a truncated normal with limits $c$ & $d$ informed from the data. Superficial details plus links to actual papers (most of which are on ArXiv) are in Taylor & Tibshirani (2015, PNAS). $\endgroup$ – Gavin Simpson Jun 23 '15 at 20:51
  • 1
    $\begingroup$ The paper by Ruben Dezeure, Peter Bühlmann, Lukas Meier and Nicolai Meinshausen is to the best of my knowledge the most recent and comprehensive account on inference in a high-dimensional setting. $\endgroup$ – NRH Jun 25 '15 at 9:58
4
+200
$\begingroup$

There is a recent paper which address precisely your question (if you want to perform regression on your data, as I understand) and, luckily, provides expressions which are easy to calculate (Confidence Intervals and Hypothesis Testing for High-Dimensional Regression).

Also, you may be interested in the recent work by Peter Bühlmann on that very topic. But I believe that the first paper provides you with what you are looking for, and the contents are easier to digest (I am not an statistician either).

| cite | improve this answer | |
$\endgroup$
  • $\begingroup$ +1 Interesting paper. So, it appears there are at least three competing ideas of how to approach these problems and from what I can see they are not closely related. Then there also is the impossibility theorem from journals.cambridge.org/action/… Will be interesting to see how this plays out and what emerges as canonical. $\endgroup$ – Momo Jun 28 '15 at 22:21
  • $\begingroup$ Thanks. This may not be something I'm actually able to implement, but it does seem like the math works for a variety of regularized estimates. $\endgroup$ – David J. Harris Jun 29 '15 at 23:21
1
$\begingroup$

http://cran.r-project.org/web/packages/hdi/index.html

Is this what you're looking for?

Description
Computes confidence intervals for the l1-norm of groups of regression parameters in a hierarchical
clustering tree.
| cite | improve this answer | |
$\endgroup$
  • $\begingroup$ I was hoping for something that would work for a variety of (mostly non-sparse) regularizers. Thanks though. $\endgroup$ – David J. Harris Jun 29 '15 at 23:21

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.