22
$\begingroup$
  1. What is the oracle property of an estimator?
  2. What modelling goals is the oracle property relevant for (predictive, explanatory, ...)?

Both theoretically rigorous and (especially) intuitive explanations are welcome.

$\endgroup$
9
$\begingroup$

An oracle knows the truth: it knows the true subset and is willing to act on it. The oracle property is that the asymptotic distribution of the estimator is the same as the asymptotic distribution of the MLE on only the true support. That is, the estimator adapts to knowing the true support without paying a price (in terms of the asymptotic distribution.)

By the asymptotic optimality properties of the MLE discussed in, for instance, Keener's theoretical statistics in theorem 9.14, we know, under some technical conditions which hold when, for instance, the error is Gaussian, that $$\sqrt{n} \left( \hat\beta_S - \beta^*_S \right) \to \mathcal{N} (0, I^{-1}(\beta^*_S)),$$ where we assume that $\beta^*_S$ is the true coefficient on the true support $S$. Notice that the variance of the asymptotic distribution is the inverse of the Fisher information, showing that $\hat\beta_S$ is asymptotically efficient. Since the MLE knowing the true support achieves this, it is also required as part of the oracle property.

However, we do pay a steep nonasymptotic price: see, for instance,

Hannes Leeb, Benedikt M. Pötscher, Sparse estimators and the oracle property, or the return of Hodges’ estimator, Journal of Econometrics, Volume 142, Issue 1, 2008, Pages 201-211,

which shows that the risk of any "oracle estimator" (in the sense of Fan and Li, 2001) has a supremum which diverges to infinity.

| cite | improve this answer | |
$\endgroup$
  • $\begingroup$ -so the oracle property for the lasso states the followng:the oracle property is that the asymptotic distribution of the estimator is the same as the asymptotic distribution of the LASSO logistic regression on only the true support $\endgroup$ – Annalise Azzopardi Sep 21 '19 at 15:15
7
$\begingroup$

The definition of Oracle property is related highly to the context. The very short but precise answer in linear regression (precisely high dimensional one) is this:

an oracle estimator must be consistent in parameter estimation and variable selection.

Notice that an estimator that is consistent in variable selection is not necessarily consistent in parameter estimation. See adaptive lasso paper for mathematical definitions or simply see this slides.

| cite | improve this answer | |
$\endgroup$
  • $\begingroup$ In the adaLASSO paper (linked in my comment) they say the convergence rate has to be optimal, too (extra to consistent estimation). That is an important and a bit difficult concept. Could you elaborate on that? $\endgroup$ – Richard Hardy Aug 10 '16 at 11:30
  • $\begingroup$ Convergence rate is a context-related assumption. In lasso it is $\sqrt{n}$ for $n$ the number of observations. However, consistency is an asymptotic result in lasso. $\endgroup$ – TPArrow Aug 10 '16 at 13:09
  • $\begingroup$ So would you suggest to remove the requirement of the rate being optimal in the definition of oracle property? $\endgroup$ – Richard Hardy Aug 10 '16 at 13:14
  • $\begingroup$ In the general definitions, I see no obligation to mention the speed. But in theory, we need to know/determine the optimal speed, obviously. $\endgroup$ – TPArrow Aug 10 '16 at 13:17
  • $\begingroup$ Thanks. I am picking on this because we talk about a definition here, so I am trying to be precise. $\endgroup$ – Richard Hardy Aug 10 '16 at 13:31

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.