- What is the oracle property of an estimator?
- What modelling goals is the oracle property relevant for (predictive, explanatory, ...)?
Both theoretically rigorous and (especially) intuitive explanations are welcome.
$\begingroup$
$\endgroup$
1
asked Aug 10, 2016 at 8:56
-
1$\begingroup$ It would be nice to have a solid one-stop-shop answer for the question. Some related material: Zou "The Adaptive LASSO and its oracle properties", p. 1 (pp. 1418). $\endgroup$– Richard HardyAug 10, 2016 at 9:11
Add a comment
|
2 Answers
$\begingroup$
$\endgroup$
2
An oracle knows the truth: it knows the true subset and is willing to act on it. The oracle property is that the asymptotic distribution of the estimator is the same as the asymptotic distribution of the MLE on only the true support. That is, the estimator adapts to knowing the true support without paying a price (in terms of the asymptotic distribution.)
By the asymptotic optimality properties of the MLE discussed in, for instance, Keener's theoretical statistics in theorem 9.14, we know, under some technical conditions which hold when, for instance, the error is Gaussian, that $$\sqrt{n} \left( \hat\beta_S - \beta^*_S \right) \to \mathcal{N} (0, I^{-1}(\beta^*_S)),$$ where we assume that $\beta^*_S$ is the true coefficient on the true support $S$. Notice that the variance of the asymptotic distribution is the inverse of the Fisher information, showing that $\hat\beta_S$ is asymptotically efficient. Since the MLE knowing the true support achieves this, it is also required as part of the oracle property.
However, we do pay a steep nonasymptotic price: see, for instance,
Hannes Leeb, Benedikt M. Pötscher, Sparse estimators and the oracle property, or the return of Hodges’ estimator, Journal of Econometrics, Volume 142, Issue 1, 2008, Pages 201-211,
which shows that the risk of any "oracle estimator" (in the sense of Fan and Li, 2001) has a supremum which diverges to infinity.
-
$\begingroup$ -so the oracle property for the lasso states the followng:the oracle property is that the asymptotic distribution of the estimator is the same as the asymptotic distribution of the LASSO logistic regression on only the true support $\endgroup$ Sep 21, 2019 at 15:15
-
$\begingroup$ @AnnaliseAzzopardi The oracle estimator is not penalized (i.e. is just logistic regression on the true support). $\endgroup$ Oct 22, 2022 at 18:58
$\begingroup$
$\endgroup$
9
The definition of Oracle property is related highly to the context. The very short but precise answer in linear regression (precisely high dimensional one) is this:
an oracle estimator must be consistent in parameter estimation and variable selection.
Notice that an estimator that is consistent in variable selection is not necessarily consistent in parameter estimation. See adaptive lasso paper for mathematical definitions or simply see this slides.
-
1$\begingroup$ In the adaLASSO paper (linked in my comment) they say the convergence rate has to be optimal, too (extra to consistent estimation). That is an important and a bit difficult concept. Could you elaborate on that? $\endgroup$ Aug 10, 2016 at 11:30
-
$\begingroup$ Convergence rate is a context-related assumption. In lasso it is $\sqrt{n}$ for $n$ the number of observations. However, consistency is an asymptotic result in lasso. $\endgroup$– TPArrowAug 10, 2016 at 13:09
-
$\begingroup$ So would you suggest to remove the requirement of the rate being optimal in the definition of oracle property? $\endgroup$ Aug 10, 2016 at 13:14
-
$\begingroup$ In the general definitions, I see no obligation to mention the speed. But in theory, we need to know/determine the optimal speed, obviously. $\endgroup$– TPArrowAug 10, 2016 at 13:17
-
$\begingroup$ Thanks. I am picking on this because we talk about a definition here, so I am trying to be precise. $\endgroup$ Aug 10, 2016 at 13:31